Resolve single-document indexing from nested JSON in REST V2 connector – Lucidworks

Goal

Ensure that a REST V2 connector indexes all documents from a JSON response containing an array of records, rather than treating the full response as a single document.

Environment

Fusion 5.9.12 and above
REST V2 connector (in JSON editor mode or UI mode)

Guide

When configuring a REST V2 connector to ingest data from a remote source such as Salesforce, ensure that the connector knows how to access the array of records in the JSON response. If the response structure is nested and the record path is not specified, the connector will treat the entire JSON object as a single document.

Understand the issue

If your source API returns a response like the following:

{
  "records": [
    { "id": "1", "name": "Document A" },
    { "id": "2", "name": "Document B" }
  ]
}

And the connector is not configured with the correct path to the records array, it will index the entire JSON as a single document instead of indexing each item in the array.

Step 1: Examine the response structure

Use a tool like Postman or curl to inspect the actual JSON response returned by the API endpoint. Identify the key that contains the array of documents to index. In this case, it is records.

Step 2: Set the record path in the connector configuration

Configure Root response mapping, and use the right key containing the array

dataPath The name of a specific data object from a datasource that is returned within a response. For example, in order to extract a list of elements named objects in the datasource, the dataPath would be objects, with each element indexed as a separate Solr document. If not provided or left blank as "", the entire response body will be indexed as a single Solr document. This property also accepts JSONPath expressions,for example, objects[] or $.objects[].

Step 3: Validate indexing results

After updating the configuration:

Clear the datasource.
Re-run the connector.
Query the collection to verify that multiple documents were indexed.

For example, run a query in the application’s default pipeline to confirm:

_lw_data_source_s: <connector_datasource_id>

You should see multiple documents corresponding to each item in the array.

Step 4: (Optional) Enable diagnostic logging

To help debug indexing behavior:

Enable diagnostic logging in the connector run settings.
After execution, check logs for parsing or indexing issues.

Once verified, disable diagnostic logging to reduce verbosity.

Additional notes

Note: If no record_path is configured, the connector assumes the entire response body is a single document.