Skip to main content

Stop Kafka consumers from trying to access deleted topics

Updated

Issue

Kafka consumers are repeatedly logging warnings indicating attempts to fetch metadata for topics that no longer exist. These warnings take the form:

WARN [pool-*-thread-*] - [Consumer clientId=..., groupId=...] Error while fetching metadata ... UNKNOWN_TOPIC_OR_PARTITION

Despite the topics being deleted from Kafka, the logs persist and indicate ongoing fetch attempts by consumers. (particularly in kafka under /bitnami/kafka/data path)

Diagnosis

To confirm this issue is caused by stale Zookeeper metadata:

Connect to one of your Zookeeper pods:

kubectl exec -it <zookeeper-pod-name> -- bash

Launch the Zookeeper CLI:

bin/zkCli.sh

List the subscriptions path:

ls /lwfusion/5.0/core/subscriptions

If the output includes subscription entries related to deleted topics, Zookeeper still holds metadata that is prompting the Kafka clients to retry fetching.

 

Environment

Fusion 5.9.2 and later, running on Kubernetes (EKS or compatible)

 

Cause

Zookeeper acts as the centralized coordination service for Fusion. When Kafka topics are deleted, their corresponding subscription entries may persist in Zookeeper unless explicitly removed. These entries can continue to trigger Kafka consumer activity and logging, even if the topics no longer exist on disk or in the Kafka broker metadata.

 

Resolution

To resolve the issue, delete the obsolete subscriptions from Zookeeper and restart the Fusion indexing pods:

Step 1: Delete subscription entries from Zookeeper

Connect to a Zookeeper pod:

kubectl exec -it <zookeeper-pod-name> -- bash

Open the Zookeeper CLI:

bin/zkCli.sh

List all current subscriptions:

ls /lwfusion/5.0/core/subscriptions

For each obsolete subscription listed, delete it:

delete /lwfusion/5.0/core/subscriptions/<subscription-name>

  1. Repeat for each stale entry.

 

Step 2: Restart the Fusion indexing deployment

Once cleanup is complete, restart the Fusion indexing pods to stop retry attempts:

kubectl rollout restart deployment fusion-indexing -n <fusion-namespace>

Replace <fusion-namespace> with the appropriate Kubernetes namespace for your Fusion deployment.

Powered by Zendesk