Managing large log and data files in Fusion 4.x environments

Issue

Fusion environments running on Windows, Linux may experience large log or data files accumulating in specific directories, which can lead to storage concerns and may impact overall system stability. Log cleanup jobs may not cover all log directories, and certain Solr data directories can grow unexpectedly.

Diagnosis

To determine if this article is relevant to your environment, check for the following symptoms:

• Log folders such as admin-ui, spark-worker, or connectors contain files exceeding several gigabytes, sometimes dating back months or years.

• Solr data directories (for example, under data\solr\<core>\data) are consuming a significant portion of disk space.

• Existing log cleanup jobs (such as "delete-old-system-logs") do not remove files from all directories.

To identify affected folders:

• Review the disk usage in the Fusion installation’s var\log and data\solr directories.

• Verify the run history and configuration of any log cleanup jobs scheduled in Fusion.

Environment

• Fusion 4.2.x (Self-Hosted Windows, Linux deployments)

• Solr 8.4.0 and compatible versions

• Applies to all Fusion 4.x deployments with manual or script-based log management

Cause

Fusion’s built-in log cleanup scheduler ("delete-old-system-logs") typically targets a limited set of log directories. Some log-producing components, such as admin-ui, connectors, and spark-worker, may generate logs that are not automatically included in this cleanup process. Additionally, Solr data directories grow as more documents are indexed or as signals accumulate, which may require periodic maintenance.

Resolution

Clearing large log files safely

To manage disk usage and prevent downtime, follow these steps:

Immediate manual cleanup:

• Stop the relevant Fusion service or component if possible, to avoid deleting log files in use. However, for many log files, deletion of old logs (not the active file) can be performed while the service is running.

• Delete or archive old log files from affected directories, such as:

  • <Fusion_install_dir>\var\log\admin-ui

  • <Fusion_install_dir>\var\log\connectors

  • <Fusion_install_dir>\var\log\spark-worker

• Retain only recent logs as required by your retention policy (for example, the last 7 days).

• Confirm free disk space after cleanup.

Note: Deleting the currently active log file may cause the application to create a new file. Avoid deleting *.log files currently being written to; instead, clear out historical files with date stamps or rotated file names.

Updating log rotation and retention settings:

• Edit the log configuration for each service (typically in log4j or log4php XML or properties files) to enable file size-based or time-based rolling and retention.

• Reference the official log4php or log4j documentation to set rolling policies:

  • Log4php Rolling File Appender documentation

• After updating log configurations, a service restart is required for changes to take effect.

Automating future log management:

• Adjust the Fusion log cleanup job schedules to target all required directories, if possible.

• Supplement with OS-level scripts or scheduled tasks to regularly delete or archive files in custom log directories not covered by Fusion’s scheduler.

Managing large Solr data directories

If Solr core data directories are growing rapidly:

• Evaluate retention policies for signals and indexed data.

• Use Solr’s built-in APIs or Fusion’s data management tools to delete or purge obsolete collections, signals, or documents.

• If appropriate, reindex collections or optimize Solr segments to reduce storage overhead.

Note: Large Solr data directories are expected as document count grows. Deleting or truncating files directly in the data directory is not supported; always use Fusion or Solr APIs for data management.

Minimizing downtime

• Most log file deletions do not require downtime, unless the active log file itself needs to be removed or log configuration changes are being made.

• When updating log configuration files, restart the relevant Fusion services during scheduled maintenance windows to avoid business impact.

• For production environments, perform cleanup activities during non-business hours and test changes in a staging environment if possible.