Issue
This error is usually observed while jobs/connectors are failing, but there is a possibility that we can receive this error some other times, as well.
Caused by: java.io.FileNotFoundException: /home/fusion/.kube/config (No such file or directory)
Environment
Fusion 5
Cause
There are multiple things that can cause this error.
-
Non matching Fusion version with Kubernetes version
-
Service account tokens are expired
-
Using gp2 SSD volumes in production environment for larger workloads
-
High disk IO latency
Resolution
Short-term solution
A rollout restart of deployments and stateful sets should fix the issue.
In the case of EKS
Check the output of kubectl get pods.
This should return the age of the pods. This issue often happens after exactly 90 days. The cause of this issue is related to service accounts using volume projection. It's likely the service account tokens expired and were not able to auto-renew. The default expiration for the tokens is 90 days which corresponds to when the issue occurred.
This was a mandatory change since Kubernetes 1.22 for EKS.
To fix this long term, upgrade to Fusion 5.5.1-3 or the most recent version of Fusion, which have support for volume projection.
In other cases
- Check if your Fusion version is compatible with Kubernetes. Search for your fusion version here you will find the compatibility details https://doc.lucidworks.com/
e.g. https://doc.lucidworks.com/fusion/5.5/301ya6/fusion-5-5-0?q=fusion%205.5 - Look out for what volume type you are using.
We suggest using at gp3 rather than gp2. gp2 is the default SSD-backed EBS volume for EC2 instances. AWS recommends using this type of volume for smaller workloads like boot volumes, non-production environments, or small-scale applications.
Find more details here,
https://zesty.co/blog/guide-to-ebs-pricing/