Spark job fails with AccessDenied Exception – Lucidworks

Issue

A Spark job running in Kubernetes fails during initialization with the following error:

Exception in thread "main" java.nio.file.AccessDeniedException: ./job-impl-5.7.0-SNAPSHOT-all.jar

This prevents the job from completing and marks the Kubernetes pod as failed.

Diagnosis

This issue can occur when a Spark jobs using fusion-spark-3.2.2 version, it prevents token refresh under OIDC authentication.

This may happen in scenarios such as:

Upgrading Fusion to a newer version (for example, from 5.9.4 to 5.9.14) while retaining old job configurations or container image references.
when spark jobs uses fusion-spark-3.2.2 version.

You can confirm the root cause by checking the container logs from the Spark driver pod:

kubectl -n <namespace> logs <spark-driver-pod> | grep 'AccessDenied'

Also verify the JAR file being referenced in the pod spec or log output:

kubectl -n namespace logs driver-pod_name | grep -i 'spark.jars\|repl.local.jars'

Environment

Fusion 5.9.14
Spark-based jobs (aggregation or enrichment jobs using job-launcher)

Cause

This is related to spark image where it fails to resolve a Fabric8 token. The Fabric8 Kubernetes client in the spark image prevented token refresh under OIDC authentication, spark throws a java.nio.file.AccessDeniedException.

This often happens when spark jobs uses fusion-spark-3.2.2 version.

Resolution

To fix this, you need to update the fusion-spark-3.2.2 image to resolve a Fabric8 token refresh bug.
The Fabric8 Kubernetes client in this Spark image has been patched to fix a bug that prevented token refresh under OIDC authentication. This ensures that Spark jobs using fusion-spark-3.2.2 run reliably in Kubernetes environments that require token-based authentication. Please follow the below steps to resolve this:

Update the configmap of job-launcher

Update the job-launcher ConfigMap with the correct image values of driver and executor container as below:

kubectl edit configmap <job-launcher-configmap> -n <namespace>

driver:
  container:
    image: lucidworks/fusion-spark-3.2.2:5.9.14
executor:
  container:
    image: lucidworks/fusion-spark-3.2.2:5.9.14

Re-start the job-launcher deployment

After updating the job-launcher ConfigMap with the correct image values. Then apply a rollout restart of the job-launcher deployment:

kubectl rollout restart deployment job-launcher -n <namespace>

Re-run the spark job

After re-starting the job launcher deployment, re-run the spark job to confirm the issue is resolved.

If successful, the job will run successfully and the spark driver should initialize without throwing an AccessDeniedException.