Upgrading from forked Tika parser to asynchronous Toka parser leads to asynchronous Tika pod not runinng – Lucidworks

Issue

When upgrading from the forked Tika parser to the asynchronous Tika parser, users may find that the asynchronous Tika parsing pod is not deployed or running in the Kubernetes cluster. This prevents asynchronous parsing from functioning, potentially resulting in slower indexing performance for large document volumes.

Diagnosis

To verify whether the asynchronous Tika parser pod is running:

Run the following command to check for the asynchronous parsing pod:

kubectl get pods | grep async

If the pod is running, the output should resemble the following:

<YOUR_NAMESPACE>-async-parsing-0 2/2 Running

If the command returns no results, the asynchronous parsing pod is not deployed. This indicates that adjustments to the Kubernetes deployment configuration may be required.

Review the cluster events to check for errors related to pod scheduling or resource constraints, particularly those associated with the asynchronous parsing pods.

Environment

Fusion 5.10+

Cause

The asynchronous Tika parsing pod might not be deployed due to one or more of the following:

Helm charts used for deployment are outdated or missing the async-parsing microservice definition.
Persistent storage requests exceed available quotas, preventing the pod from being scheduled.
Updates to the StatefulSet configuration are disallowed after initial deployment.

Resolution

Confirm Helm chart version and deployment source

Determine whether the Fusion Helm charts are sourced directly from the Lucidworks Helm repository or maintained internally.
If using internally hosted charts, ensure that you pull and deploy Fusion Helm charts for version 5.10 or later to include the asynchronous parsing microservice.

Failure to deploy the updated charts may result in the async-parsing pod being omitted from the deployment.

Address storage quota issues

If asynchronous parsing pods fail due to excessive storage requirements, Kubernetes may display errors similar to the following:

25s Warning FailedCreate statefulset/fusion-sandbox-async-parsing
create Claim async-parsing-data-claim-fusion-sandbox-async-parsing-0
for Pod fusion-sandbox-async-parsing-0 in StatefulSet fusion-sandbox-async-parsing
failed error: persistentvolumeclaims "async-parsing-data-claim-fusion-sandbox-async-parsing-0" is forbidden:
exceeded quota: limit-storage, requested: requests.storage=100Gi,
used: requests.storage=843Gi, limited: requests.storage=918049259520

To reduce storage requirements, add or adjust the following configuration in the fusion_values.yaml file:

async-parsing:
volume:
storage: 10Gi

Apply the updated Helm values to redeploy the services.

Resolve StatefulSet patch errors

Applying new Helm configurations can result in the following error:

Error: UPGRADE FAILED: cannot patch "fusion-sandbox-async-parsing" with kind StatefulSet:
StatefulSet.apps "fusion-sandbox-async-parsing" is invalid: spec: Forbidden:
updates to statefulset spec for fields other than 'replicas', 'template',
'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

If this error occurs:

Delete the StatefulSet:

kubectl delete statefulset <statefulset-name>

Delete the associated PersistentVolumeClaim (PVC):

kubectl delete pvc -l <label-key>=<label-value>

Recreate the StatefulSet using the updated YAML file:

kubectl apply -f <statefulset-config-file.yaml>

After redeployment, re-run the command:

kubectl get pods | grep async

Verify that the asynchronous parsing pod is running as expected.