Fusion logs guide to configuring retention and rollover periods – Lucidworks

Issue

How to configure the retention or rollover period of my Fusion logs?

Environment

Fusion (Any version)

Resolution

Fusion provides logging for each of the respective services in the path Fusion_Home/var/log/<service-name>/<service-name>.log files.

In Fusion, these logs tend to accumulate and grow large and taking up a lot of disk space (in the order of MB’s to GB’s), but we can configure the log retention period or rollover period as per our requirement.

The logs here have been divided into 3 categories

Fusion Service Log
Jetty Request log and Jetty Stderrout
GC Logging

Fusion Services Log

All the below services log files by default are stored for 30 days after which they are rolled over.

agent
api
Connectors-classic
connectors-rpc
log-shipper
proxy
solr
spark-master
spark-worker
sql
Zookeeper

We can alter the default settings based on our requirements. To alter the particular service log, navigate to the path Fusion_Home/conf/<service-name>log4j2.xml

Here we have a log4j file for each of the services. We can work around the below 3 properties for log retention/rollover period as per our requirement.

This causes a rollover once the file has reached the specified size. That is a new file is created once it reaches the specified size. The size can be specified in bytes, with the suffix KB, MB or GB, for example, 20MB.

Specifies a duration. The no of days of old logs we want to retain. The condition accepts files that are as old or older than the specified duration.

This is to limit the disk space used by logs. The size can be specified in bytes, with the suffix KB, MB or GB, for example 20MB

Example of api.logs:

Jetty Request log and Jetty Stderrout

We have Jetty logging which can come in very handy during performance and general troubleshooting.

Jetty Request Log

jetty-logging.xml (for configuring jetty*.stderrout.log).

Here, changing this single file will affect the jetty*.stderout.log files for all the fusion services

Path: Fusion/apps/jetty/home/etc/jetty-logging.xml

Modify the parameter <Arg type=“int”>90</Arg> from 90 to the required “x” number of retaining days.

So if we change the parameters of retaining days from 90 days to the required “x” number of days then the log file will start to roll over after that x number of days.

Example:

Jetty stderrout Log

start.ini (for configuring jetty*.request.log). Here individual services has specific file at the below path

Path: Fusion_Home/<service_name>/jetty/*/start.ini

Modify the following parameter from 30 to “x” number of retaining days.

jetty.requestlog.retainDays = 30

Example for api jetty stderrout log:

GC Logging

In Fusion by default, the GC logs are getting created as fusion.properties have a parameter configured as shown below:

So as per this if default.gcLog is configured as true then as GC logs will be created into each service's directory, and rollover after 10 MB, with the 20 most recent files retained.

To configure GC logs as per our requirement, you can follow the below steps:

Make the value of default.gcLog as False.
For each of the services for which you have to enable GC logs add the below code in their respective jvm options

-verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=1M -Xloggc:/Users/madhu/Downloads/NEW_Fusion/fusion-4.1.1/4.1.1/var/log/api/gc-%t.log

Here you can modify the number of GC log file and GC file size as per your convenience. To get more logging than just the last 24 hours then increase the file size.

To store the file at specified location we can modify -Xloggc:/Users/madhu/Downloads/NEW_Fusion/fusion-4.1.1/4.1.1/var/log/api/gc-%t.log

Example for api service: