API Gateway Timeouts

Issue

Occasionally you may observe api calls that error out with the "error": "service-timeout" and "msg": "Did not observe any item or terminal signal within 15000ms in 'circuitBreaker' (and no fallback has been configured)". This means the api-gateway pod timed out before it could complete the request. It could be normal for some requests to take over the default 15 second timeout, and the timeout just needs to be increased.

Environment

Fusion 5

Resolution

The API Gateway timeout can be increased by adding an additional JVM parameter to the api-gateway pod startup. One way this can be done is by editing the ..fusion_values.yaml file and overriding the default. Example below is changing the timeout to 30 seconds by adding the text in bold.

api-gateway:
logstashEnabled: false
service:
externalTrafficPolicy: "Local"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
javaToolOptions: "-Dresilience4j.timelimiter.instances.query.timeoutDuration=30s -Xms2g -Xmx2g -Djwt.token.user-cache-size=100 -Dhttp.maxConnections=1000 -XX:+ExitOnOutOfMemoryError"
pod:
annotations:
prometheus.io/port: "6764"
prometheus.io/scrape: "true"
prometheus.io/path: "/actuator/prometheus"