Fusion jobs fail after upgrade due to missing Argo CRDs – Lucidworks

Issue

After upgrading to Fusion 5.9.11 or 5.9.12, aggregation jobs complete successfully, but any jobs that rely on Argo Workflows (such as ML pipelines or workflow-based jobs) fail to execute. The Argo controller pod continually restarts and its logs contain errors such as:

the server could not find the requested resource (get workflowtaskresults.argoproj.io)
the server could not find the requested resource (get workflowtasksets.argoproj.io)
the server could not find the requested resource (get workflowartifactgctasks.argoproj.io)

Additionally, users may notice that workflows do not appear in the Argo UI, or encounter authentication prompts when attempting to port-forward to the Argo server.

Diagnosis

To confirm this issue:

Inspect Argo controller pod logs for repeated messages referencing missing CRDs (Custom Resource Definitions).
Run the following command to list CRDs in the cluster and verify the absence of Argo workflow CRDs:

kubectl get crds | grep argo

If the following CRDs are missing, the issue described in this article is present:

workflowtaskresults.argoproj.io
workflowtasksets.argoproj.io
workflowartifactgctasks.argoproj.io

Environment

Fusion 5.9.11 or 5.9.12 (self-hosted, Kubernetes deployment)
Kubernetes version 1.30 (AKS or other platforms)
Helm-based installation or upgrade from Fusion 5.9.9 or earlier

Note: Fusion 5.9.11 and 5.9.12 Helm charts do not automatically install all required Argo CRDs during upgrade. This behavior may occur on any supported Kubernetes environment.

Cause

The method for installing Argo CRDs changed in recent Fusion Helm charts (beginning with Fusion 5.9.11). The upgrade process does not install new CRDs if they are not already present in the cluster, which results in missing Argo resources and failed jobs. Helm will not update or reapply existing CRDs, nor install new ones on upgrade, unless a fresh deployment is performed.

Resolution

To restore the missing Argo CRDs, use one of the following approaches:

Option 1: Temporary deployment with Helm (recommended for most environments)

Create a temporary namespace:

kubectl create namespace tmp-crd-install

Install Fusion 5.9.10 in the temporary namespace.
Note: Fusion 5.9.10 is the last release that automatically includes all required Argo CRDs with the Helm chart.

helm install --namespace tmp-crd-install tmp-crd lucidworks/fusion --version 5.9.10

Delete the temporary release and namespace:

helm delete --namespace tmp-crd-install tmp-crd
kubectl delete namespace tmp-crd-install

Verify the CRDs are present:

kubectl get crds | grep -E "workflowtaskresults|workflowtasksets|workflowartifactgctasks"

Important:
If you encounter Helm errors such as "rendered manifests contain a resource that already exists" related to CRDs (for example, seldondeployments.machinelearning.seldon.io), this is due to Helm's release annotations. Only one Helm release can own a given CRD at a time. If this occurs, proceed with Option 2 below.

Option 2: Manually apply CRDs from the Fusion chart

Download and extract the Fusion 5.9.12 Helm chart:

helm pull lucidworks/fusion --version 5.9.12 --untar

Apply the Argo CRD YAML files directly:

kubectl apply -f fusion/charts/argo/crds/

Confirm the CRDs are now present:

kubectl get crds | grep -E "workflowtaskresults|workflowtasksets|workflowartifactgctasks"

Once these CRDs are installed, Argo workflow jobs should function as expected in Fusion 5.9.11/12.

Additional information

Adjusting memory limits for recommender jobs

If jobs fail with OOMKilled (out-of-memory) errors during the training phase of Fusion's content recommendations, increase the memory limit for the affected pod. Since memory settings for the fusion-recommender container are not configurable via values.yaml, you must modify the deployment manifest for that pod to specify higher memory resource limits.

Argo UI authentication

If unable to access the Argo UI when port-forwarding to the Argo server, authentication is required. For information on configuring service account access or SSO, refer to the official Argo Workflows service account documentation and Argo server SSO documentation.