Skip to main content

Resolve Spark job timeout during driver pod creation

Updated

Issue

Spark SQL jobs fail consistently during the driver pod creation phase. The driver pod is never created, and the submission exits with a generic Kubernetes client exception and socket timeout.

The error logs display the following output during the submission phase:

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
...
Caused by: java.io.IOException: timeout
...
Caused by: java.net.SocketTimeoutException: timeout

 

Diagnosis

The socket timeout exception during the driver pod creation indicates that the Spark Kubernetes client opened a connection and sent a request to the Kubernetes API server but stalled without receiving response headers. This timeout behavior masks the underlying job configuration error.

To expose the actual underlying failure, the submission request timeout must be increased. When the timeout is extended, the root cause surfaces in the logs, typically displaying an analysis exception regarding duplicate columns:

org.apache.spark.sql.AnalysisException: [COLUMN_ALREADY_EXISTS] The column params_source_ss already exists. Consider to choose another name or rename the existing column.

 

Environment

  • Fusion 5.9.x

  • Spark 3.4.1

  • Kubernetes 1.32 or higher

  • OpenShift 4.19 or higher

 

Cause

The initial timeout is caused by a short submission timeout configuration that prevents the Spark client from properly logging the failure response from the API server.

The root cause of the job failure is a case sensitivity conflict in the input fields, where field names differ only by case, leading to a duplicate column exception during Spark SQL execution.

 

Resolution

To resolve the timeout masking and the underlying column collision, apply the following properties to the Spark job configuration.

  • Navigate to the configuration for the failing Spark job.

  • Add the following parameter to increase the request timeout and allow subsequent errors to be properly logged:

spark.kubernetes.submission.requestTimeout=120000

  • Add the following parameter to enforce case sensitivity in Spark SQL, which prevents column collision errors:

spark.sql.caseSensitive=true

  • Save the job configuration and rerun the job.

Powered by Zendesk