Hybrid search query fails due to interaction between KNN and highlight components – Lucidworks

Issue

Hybrid search queries using KNN vector clauses fail with ArrayIndexOutOfBoundsException in the highlighting component.

"msg" : "Index 364 out of bounds for length 364", 
"trace" : "java.lang.ArrayIndexOutOfBoundsException: Index 364 out of bounds for length 364\n\t
at org.apache.lucene.util.FixedBits.get(FixedBits.java:37)\n\t
at org.apache.lucene.search.AbstractKnnVectorQuery$1.match(AbstractKnnVectorQuery.java:159)\n\t
at org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc(FilteredDocIdSetIterator.java:64)\n\t
at org.apache.lucene.util.BitSet.or(BitSet.java:110)\n\t
at org.apache.lucene.util.FixedBitSet.or(FixedBitSet.java:326)\n\t
at org.apache.lucene.util.BitSet.of(BitSet.java:42)\n\t
at org.apache.lucene.search.AbstractKnnVectorQuery.createBitSet(AbstractKnnVectorQuery.java:162)\n\t
at org.apache.lucene.search.AbstractKnnVectorQuery.getLeafResults(AbstractKnnVectorQuery.java:128)\n\t
at org.apache.lucene.search.AbstractKnnVectorQuery.searchLeaf(AbstractKnnVectorQuery.java:104)\n\t
at org.apache.lucene.search.AbstractKnnVectorQuery.lambda$rewrite$0(AbstractKnnVectorQuery.java:89)\n\t
at org.apache.lucene.search.TaskExecutor$TaskGroup.lambda$createTask$0(TaskExecutor.java:117)\n\t
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n\t
at org.apache.lucene.search.TaskExecutor$TaskGroup.invokeAll(TaskExecutor.java:152)\n\t
at org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:76)\n\t
at org.apache.lucene.search.AbstractKnnVectorQuery.rewrite(AbstractKnnVectorQuery.java:91)\n\t
at org.apache.lucene.search.KnnFloatVectorQuery.rewrite(KnnFloatVectorQuery.java:45)\n\t
at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:326)\n\t
at org.apache.lucene.search.BoostQuery.rewrite(BoostQuery.java:76)\n\t
at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:769)\n\t
at org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:145)\n\t
at org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74)\n\t
at org.apache.lucene.search.uhighlight.MemoryIndexOffsetStrategy.getOffsetsEnum(MemoryIndexOffsetStrategy.java:119)\n\t
at org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:80)\n\t
at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFieldsAsObjects(UnifiedHighlighter.java:944)\n\t
at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFields(UnifiedHighlighter.java:814)\n\t
at org.apache.solr.highlight.UnifiedSolrHighlighter.doHighlighting(UnifiedSolrHighlighter.java:165)\n\t
at org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:161)\n\t
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:472)\n\t
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:226)\n\t

Diagnosis

This issue can be diagnosed by:

Observing stack trace errors such as Index X out of bounds for length Y from the Solr highlight component.
Identifying that the query includes a KNN clause and that highlighting is enabled.
Confirming that the query succeeds when highlighting is disabled.

Environment

Fusion 5.9.5 and above
Hybrid search using KNN query clauses in combination with Solr highlighting
Managed Fusion deployments (Lucidworks-hosted or self-managed on Kubernetes)

Cause

The issue seems to be with the Highlight component conflicting with the Knn parser, causing query failures. Disabling highlighting stages resolves the problem temporarily.

"lex_q" : "{! type=edismax pf=body_txt_en^3 pf2=title_txt_en^4 pf2=description_txt_en^3 bq='' bf='' boost='' v=$raw_lex_q}", 
"q" : "(({!query v=$lex_q}^=0 AND {!func v=$vecSim}) OR {!knn f=long_pwnygpt_summary_custom_512_v topK=1000 preFilter=$lex_q_exclude v=$vec_q})^0.8",

Resolution

To resolve this issue, apply the following adjustments in the query pipeline:

Prevent conflict between hybrid query and highlighting

Update the “Highlighting Params” stage in the pipeline to explicitly set the highlight query parameter (hl.q) to the main search term. This ensures Solr uses the correct query context for highlighting.

Additional notes

This issue is related to the interaction of unified highlighter and hybrid queries leveraging the Lucene KNN vector search capability. If vector-based hybrid search is required and highlighting is also necessary, configuring hl.q explicitly is the preferred workaround until deeper compatibility is improved in future releases.