Introduction
The Lucidworks Fusion platform leverages Apache Spark for distributed data processing, integrating seamlessly with Kubernetes (K8s) for efficient job orchestration. In Kubernetes environments, the use of service accounts becomes pivotal in ensuring proper access control and authorization within the cluster.
Environment
Fusion 5
Background
Kubernetes service accounts play a crucial role in controlling access and authorization within a Kubernetes cluster. When Spark jobs operate in a Kubernetes setting, they might require specific permissions to work with the cluster's resources, such as creating and managing pods. It's common to encounter permission issues if the service account, for example, "fusion-job-launcher-spark" in the "fusion" namespace, doesn't have the necessary permissions, leading to job failures. Additionally, failing the upgrade script due to account duplication can pose challenges.
This article will guide you in addressing such permission challenges to ensure the smooth execution of Spark jobs in Lucidworks Fusion on Kubernetes.
Guide
1. Insufficient Permission
The following error message indicates that the service account "f5-job-launcher-spark" within the "f5" namespace lacks the necessary permissions to list resources of type "persistentvolumeclaims" in the Kubernetes cluster. This issue commonly occurs due to insufficient RBAC (Role-Based Access Control) permissions.
ERROR [shutdown-hook-0:spark.internal.Logging@94] - {} - Uncaught exception in thread shutdown-hook-0
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/f5/persistentvolumeclaims?labelSelector=spark-app-selector%3Dspark-ab7f6898d98f431289431e9576ad1223.
Message: Forbidden! Configured service account doesn't have access.
Service account may have been revoked. persistentvolumeclaims is forbidden:
User "system:serviceaccount:f5:f5-job-launcher-spark" cannot list resource "persistentvolumeclaims" in API group "" in the namespace "f5".To resolve this, follow these troubleshooting steps:
Check permissions by running the below command and observe the forbidden error.
kubectl get persistentvolumeclaims -n f5- Ensure sufficient privileges to modify RBAC configurations in your K8s cluster.
- Access the RBAC configuration, usually a YAML file, defining roles and role bindings in your cluster.
- Locate the role or cluster role associated with the "f5-job-launcher-spark" service account, ensuring it has permission to list "persistentvolumeclaims."
- If the role does not exist, create one with the necessary API group and resource permissions.
- Verify that the role binding or cluster role binding for the service account references the correct role or cluster role.
- Example RBAC YAML:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: f5-job-launcher-spark-role
namespace: f5
rules:
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["list"]- Apply the modified RBAC configuration to your cluster:
kubectl apply -f name-of-file.yaml- After applying the changes, confirm that the "f5-job-launcher-spark" service account has the necessary permissions:
kubectl get persistentvolumeclaims -n f5