Issue
Solr pods may stop processing data or queries when the allocated disk storage in their Persistent Volume Claim (PVC) reaches capacity. Attempts to increase PVC size may not reflect in the running pods.
Diagnosis
To confirm the issue:
Check Solr pod logs for errors related to insufficient disk space.
Verify PVC size and usage:
kubectl get pvc -n <fusion-namespace>
kubectl describe pvc <pvc-name> -n <fusion-namespace>Check pod filesystem usage:
kubectl exec -n <fusion-namespace> <solr-pod-name> -- df -hIf the PVC has been expanded but pods are not recognizing the new size, this confirms the issue.
Environment
Fusion 5.5.0 and above, deployed on Kubernetes.
Cause
Kubernetes StatefulSets running Solr may not automatically mount the resized PVC storage. In these cases, pods continue to use the original allocation until they are recycled or the StatefulSet is recreated.
Resolution
Step 1: Free temporary space
If immediate relief is needed, you can delete unnecessary logs or temporary files within the Solr pod:
kubectl exec -n <fusion-namespace> <solr-pod-name> -- rm -rf /var/solr/logs/*This provides only short-term relief and should not be considered a full resolution.
Step 2: Resize the PVC
Increase the PVC size to a larger value:
kubectl edit pvc <pvc-name> -n <fusion-namespace>Update the spec.resources.requests.storage field with the new desired size.
Step 3: Restart Solr pods
Recycle the Solr pods to pick up the new PVC size:
kubectl delete pod <solr-pod-name> -n <fusion-namespace>Step 4: Recreate the StatefulSet if needed
If the pod does not claim the resized volume, delete and recreate the StatefulSet without cascading deletion of PVCs:
kubectl delete statefulset <solr-statefulset> -n <fusion-namespace> --cascade=falseThen reapply the StatefulSet manifest to restore the pods.
Step 5: Validate Solr health
After pods restart, verify that:
PVC size reflects the new allocation:
kubectl exec -n <fusion-namespace> <solr-pod-name> -- df -hAll Solr collections have healthy replicas:
kubectl exec -n <fusion-namespace> <solr-pod-name> -- solr healthcheck