Issue
During high-volume performance testing or peak traffic periods, external applications routing search profiles (such as product details, search, or carousel requests) experience persistent 500, 503, and 504 errors.
Simultaneously, internal Fusion internal service metrics and logs display corresponding clusters of 401 Unauthorized errors for query profile endpoints, despite configurations using valid credentials and active authentication tokens.
Diagnosis
When downstream clients report unmapped 5xx gateway drops under severe concurrency, inspect internal application routing components to distinguish network drop-offs from structural token degradation.
To locate token errors, examine cloud logging outputs filtered for 4xx transaction status lines mapped to internal API gateway paths. You can isolate the exact response signatures by executing queries across application logs:
resource.type="k8s_container"
labels.k8s-pod/app.kubernetes.io/name="api-gateway"
jsonPayload.status="401"If the logs display recurring 401 signatures isolated strictly to peak traffic lines, the system is actively invalidating or dropping validation queries. Check the JVM argument profiles on active API Gateway service pods to observe the current active limits assigned to credential validation pools.
kubectl exec -it deployment/proxy -- jcmd 1 VM.system_properties | grep -E "cache-size"Default parameters restrict these arrays to low values (typically 40 entries). Under high concurrency, this threshold leads directly to high thrashing, cache eviction, and authentication drops.
Environment
Managed Fusion
Cloud Native Deployment (Kubernetes)
Fusion API Gateway (Proxy Microservice)
Cause
The issue is driven by authentication token cache exhaustion within the Fusion API Gateway layer.
By default, the microservice limits password validation arrays and JSON Web Token (JWT) tracking objects via tight constraints:
-Dsecurity.auth.password-cache-size=40-Djwt.token.user-cache-size=40
When parallel transaction loads scale aggressively (e.g., exceeding several hundred requests per second across unique authorization parameters), cache spaces fill up instantly. Continuous entry eviction forces the gateway to repeatedly re-authenticate incoming sessions against internal datastores.
Under excessive thread contention, validation processes fail or timeout, manifesting internally as 401 Unauthorized errors. Downstream load balancers or edge reverse-proxies map these internal stalls or authenticating drops as generic 500, 503, or 504 status alerts back to the client interface.
Resolution
To remediate authentication drops, increase the maximum security token cache capacities allocated to the API Gateway runtime engines via your centralized Kubernetes environment configuration.
Modify Gateway Deployment Arguments
Adjust your environment's service configuration files or active deployment maps to increase your cache values to 1000 entries. Append or update the following parameters within your global Java options arrays:
-Dsecurity.auth.password-cache-size=1000
-Djwt.token.user-cache-size=1000Apply Configuration Changes
Execute a rolling restart across all API Gateway pods to ensure that the newly provisioned cache sizes are successfully initialized:
kubectl rollout restart deployment/proxyValidate Pod Initialization
Verify that all running replicas return to a healthy, balanced state and confirm the parameters have taken effect inside the container runtimes:
kubectl get pods -l app.kubernetes.io/name=api-gatewayEstablish Pre-Scale Operations During Load Adjustments
When testing aggressive scaling profiles, run a 10-minute baseline dummy transaction pass prior to kicking off your true performance workload.
This initial run forces Horizontal Pod Autoscalers (HPA) to systematically ramp up replica metrics (such as scaling API Gateway and Search Solr pods to their designated maximum ceilings) without injecting sudden latency spikes into critical production verification windows.