Goal
Enable logging of HTTP response headers returned to Fusion when using Web v2 connectors, to support debugging and validation of upstream responses during crawling and indexing.
Environment
Fusion 5.9.x and later deployed on Kubernetes
Web v2 connector plugin deployments
Kubernetes-based Fusion deployments, including managed and self-managed environments
Guide
Understand default behavior
HTTP response headers returned by upstream web servers are not logged by default in Fusion Web v2 connector deployments. There is no Helm chart value or Fusion-level setting that enables this logging automatically.
To capture HTTP response headers, Java logging must be configured at the connector plugin runtime using Apache HttpClient logger categories.
Identify the correct deployment
HTTP response header logging must be enabled on the connector plugin deployment responsible for executing the crawl. For Web v2 connectors, this is the connector plugin deployment, not the API gateway or other Fusion services.
In Kubernetes environments, this deployment typically runs as a connector-plugin pod.
Configure Java logging options
Set Java system properties using the JAVA_TOOL_OPTIONS environment variable on the connector plugin deployment. These options enable HTTP header logging while avoiding verbose wire-level body output.
Add the following Java options to the deployment:
-Dorg.apache.commons.logging.simplelog.log.org.apache.http=DEBUG
-Dorg.apache.commons.logging.simplelog.log.org.apache.http.wire=ERROR
-Dorg.springframework.http=DEBUG
-Dorg.springframework.web=DEBUGAfter applying these changes, restart the connector plugin deployment to ensure the new settings take effect.
Run a controlled crawl
Trigger a limited crawl using the Web v2 connector to generate outbound HTTP requests. For sitemap-based crawls, limiting crawl depth helps reduce noise and ensures the crawl completes quickly.
Allow the connector to complete its run before reviewing logs.
Review connector plugin logs
Inspect the logs for the connector plugin pods using the Kubernetes logging mechanism appropriate for the environment.
When logging is enabled correctly, HTTP response headers appear in the logs with entries similar to the following pattern:
http-outgoing << Header-Name: Header-ValueThese entries represent the actual HTTP response headers returned by the upstream server to the connector.
Disable logging when finished
HTTP header logging increases log volume and should only be enabled temporarily for diagnostics.
Once analysis is complete, remove the Java logging options from the connector plugin deployment and restart the pods to return to default logging behavior.