Troubleshoot OAuth and pagination errors when configuring REST connectors

Issue

When configuring a REST connector to access OAuth-protected APIs, the connector fails to validate the datasource and does not index results. Common errors include:

• OAuth validation failures

• PKIX certificate path errors

• 404 errors with unresolved pagination variables

• The connector indexing the full JSON response as a single document

Diagnosis

Check the connector logs for errors such as:

• PKIX path building failed: unable to find valid certification path to requested target

• Access token request failed

• 404 Not Found errors referencing unresolved variables like ${LW_INDEX_START}

• REST connector logs showing documents not split correctly from JSON response arrays

If these symptoms are present, this article provides configuration guidance.

Environment

• Fusion 5.9.7 (Kubernetes-based deployment on Amazon EKS)

• REST V2 connector (lucidworks.rest)

• OAuth 2.0 client credentials or password grant APIs (e.g., Microsoft AAD, Gainsight)

• TLS-enabled environment with possible firewall and certificate constraints

Cause

• The OAuth configuration lacked a full token endpoint URL, causing authentication failures.

• Pagination was misconfigured due to incorrect use of unsupported placeholders ({{page}}, {{pageSize}}) instead of Fusion-recognized variables.

• The JSON response from the API was not parsed correctly due to a missing dataPath in the response handling configuration.

Resolution

Configure the REST connector with OAuth authentication

Ensure your datasource JSON contains a valid OAuth configuration. Use the complete token endpoint URL under oAuthEndpoint. For example:

"oAuthEndpoint": "https://login.microsoftonline.com/<TENANT_ID>/oauth2/v2.0/token"

Avoid splitting the domain and token path into oAuthServiceUrl and oAuthEndpoint. Instead, specify the full URL under oAuthEndpoint and leave oAuthServiceUrl blank if not used.

Set OAuth credentials and grant parameters

Use plainVariables to supply client credentials:

"plainVariables": [
  { "keyName": "client_id", "keyValue": "<client_id>" },
  { "keyName": "client_secret", "keyValue": "<client_secret>" },
  { "keyName": "grant_type", "keyValue": "client_credentials" },
  { "keyName": "scope", "keyValue": "https://<resource>/.default" }
]

In some environments, certificate issues may block the OAuth request. If running in a pod where truststore import is unavailable, test with:

JAVA_TOOL_OPTIONS="-Dcom.lucidworks.connectors.disableCertificateValidation=true"

Configure pagination using index offset values

Do not use {{page}} or {{pageSize}} in request URLs. Instead, configure using Fusion’s supported variables:

"queries": [
  { "queryKey": "page", "queryValue": "${LW_INDEX_START}" },
  { "queryKey": "pageSize", "queryValue": "50" }
],
"pagination": {
  "paginationByBatchSize": {
    "paginationStopConditionValue": "[]",
    "paginationStopConditionKey": "result",
    "batchSize": 1,
    "indexStart": 1
  }
}

Note: Using batchSize: 1 forces LW_INDEX_START to increment by one and simulate a page-based pattern like page=1, page=2, etc., even though it was originally intended as a record offset.

Parse JSON array properly with response handling

In the responseHandling section of the datasource configuration, specify the JSON path to the array of documents:

"responseHandling": {
  "dataPath": "$.results"
}

Adjust the dataPath to match the structure of the actual API response (e.g., $.items, $.data, etc.).

Optional: Validate connector behavior

Use connector logs to confirm that:

• OAuth tokens are fetched successfully (status 200)

• Requests are being made to expected paginated URLs like ...?page=1&pageSize=50

• Documents are being emitted individually (not as a full raw JSON blob)

If necessary, test API behavior using curl from a pod to confirm firewall or certificate issues:

curl -v -k "https://<host>/oauth2/token"

If a 400 Bad Request - Invalid Header is returned, inspect custom headers or payload encoding.