Resolve "Could not hpack encode POST" error when sending requests to Solr over HTTP/2 – Lucidworks

Issue

When sending update requests to Solr over HTTP/2 using ContentStreamUpdateRequest, the request fails with an exception similar to the following:

org.eclipse.jetty.http2.hpack.HpackException$SessionException: Could not hpack encode POST
    at org.eclipse.jetty.http2.hpack.HpackEncoder.encode(HpackEncoder.java:300)
    ...
Caused by: java.nio.BufferOverflowException
    at java.base/java.nio.Buffer.nextPutIndex(Buffer.java:722)

This issue may occur when passing large XML or parameter values in the request, particularly when the total size of HTTP headers exceeds the default compression or buffer limits.

Diagnosis

The error originates from HPACK, the header compression algorithm used by HTTP/2. In this case, the header list generated by the request exceeds the size that the server can process.
If the same request works over HTTP/1.x or when submitted through Solr Admin UI, but fails over HTTP/2, the problem is likely related to HTTP/2 header compression limits rather than Solr indexing logic.

Environment

Solr 9.x
Jetty-based Solr deployment
Requests sent via HTTP/2

Cause

The default request header size in Jetty and the HTTP/2 HPACK encoder may be too small for large header payloads. HTTP/1.x requests are unaffected because they do not use HPACK compression.

Resolution

1. Increase Jetty request header size

Adjust the requestHeaderSize property in jetty.xml to accommodate larger headers.

<Set name="requestHeaderSize">
  <Property name="solr.jetty.request.header.size" default="29000" />
</Set>

Test different values as appropriate for your environment, and restart Solr after making the change. Always back up configuration files before modification.

2. Reduce request header size

Where possible:

Move large values from query parameters to the request body.
Shorten parameter names or values.
Remove unnecessary headers.

3. Test with HTTP/1.x

HTTP/1.x does not use HPACK compression and may process the request successfully.
Note: HTTP/1.x APIs are deprecated in Solr 9.x and may be removed in future versions.

4. Update HPACK compression implementation

If feasible in your environment:

Upgrade Jetty to a version with improved HTTP/2 header handling.
Review server HTTP/2 configuration to increase allowed header size or compression buffer.