Set persistent replica count for individual connectors in Fusion 5.9+ – Lucidworks

Goal

Ensure that the replica count for a specific Fusion connector, such as the ldap-acls plugin, remains persistent and does not reset to the default value (typically 3) when the connectors-backend pod is restarted.

Environment

Fusion 5.9.0 and later
Kubernetes (self-hosted, including AKS, GKE, EKS, etc.)
Helm-based deployment of Fusion
Applicable to any connector plugin requiring custom replica configuration

Guide

Background on default replica behavior

In Fusion 5.9 and above, if replica counts are not explicitly configured per plugin, the number of replicas defaults to the value defined in the connectors-backend configuration. When connectors-backend is restarted, it uses the values from its ConfigMap to redeploy connector pods, which can override manual replica scaling.

To make connector replica counts persistent, use the pluginValues field in your connectors-backend Helm values configuration.

Step 1: Locate your connectors-backend configuration

Retrieve the current ConfigMap for the connectors-backend:

kubectl get configmap <release-name>-connectors-backend -n <namespace> -o yaml

Replace <release-name> and <namespace> with the appropriate values from your deployment.

Step 2: Update the Helm chart with pluginValues

In your Helm values YAML file (used during deployment or upgrade), configure the pluginValues section to specify the desired replica count for the target connector.

Example:

connector-plugin:
  pluginValues:
    - id: ldap-acls
      replicaCount: 1
      resources:
        limits:
          cpu: 7000m
          memory: 20000Mi
        requests:
          cpu: 400m
          memory: 2000Mi

This configuration ensures that:

The ldap-acls connector is always deployed with 1 replica
Other connectors will default to values defined in the base configuration or their own pluginValues entries

Step 3: Apply the changes

Upgrade the release using Helm to apply the new configuration:

helm upgrade <release-name> lucidworks/fusion-values -f values.yaml -n <namespace>

This will redeploy the necessary components with the updated settings, ensuring that the ldap-acls connector runs with the specified replica count, even after restarting connectors-backend.

Step 4: Confirm deployment state

Verify that only one pod for ldap-acls is running:

kubectl get pods -n <namespace> | grep ldap-acls

Also confirm the replica configuration has been applied:

kubectl get deployment <release-name>-connector-plugin-ldap-acls -n <namespace> -o yaml | grep replicas

You should see replicas: 1 in the deployment specification.

Notes

Note: If you do not configure a specific plugin under pluginValues, it will inherit the default replica count from the connectors-backend settings. To maintain fine-grained control, configure each connector requiring a custom replica count explicitly.

Note: This method is applicable to all connector plugins in Fusion, not just ldap-acls. You can define multiple entries in the pluginValues list to control behavior across various connectors.