Spark jobs fails with “Invalid JWT Signature” when accessing GCS bucket

Issue

A SparkSQLLoader job fails with the following error when attempting to access a Google Cloud Storage (GCS) bucket:

400 Bad Request: Invalid JWT Signature

In logs, the following error may also be observed:

java.io.IOException: Error accessing gs://<bucket-path>
Caused by: com.google.api.client.auth.oauth2.TokenResponseException: 400 Bad Request

The job may have been functioning correctly prior to the failure.

Diagnosis

This issue typically occurs when the JWT token or service account credentials used to access the GCS bucket are invalid or expired. To confirm:

• Check the logs of the SparkSQLLoader or job launcher pods for GCS access errors, particularly:

  • Invalid JWT Signature

  • TokenResponseException

• Verify the exact bucket path used in the job.

• Use the Google Cloud Console to navigate to Storage > Buckets and confirm the path is accessible and exists.

• Confirm that the credentials file (e.g., *.json) is properly mounted to the pod running the Spark job.

Environment

Fusion 5.x 
Applies to Spark jobs reading from GCS buckets

Cause

The JWT token or service account credentials used by the job have expired, were rotated, or are otherwise invalid. As a result, the pod attempting to read from the GCS bucket fails authentication.

Resolution

Validate bucket access

• Ensure that the specified GCS bucket path exists using the GCP Console under Storage > Buckets.

Review logs

• Check the Spark pod logs to confirm the failure is related to an invalid JWT token.

kubectl logs <pod-name> -n <namespace>

Validate and regenerate credentials

• Navigate to IAM & Admin > Service Accounts in the GCP Console.

• Locate the service account used by the Spark job.

• If the current key is expired or invalid, generate a new key.

Note: Ensure proper permissions exist to manage service account keys.

Mount updated credentials

• Update the credentials in the Fusion deployment by modifying the secret or mount configuration associated with the Spark job.

• Ensure the secret is correctly referenced in the job parameters using --conf flags or Kubernetes volume mounts (depending on deployment method).

   

Restart the job

• After the credentials have been updated and verified, rerun the job from the Fusion UI or API.

If the issue persists, validate the new token is correctly mounted and that the pod reflects the updated configuration.