Log4j vulnerabilities: CVE-2021-44228, CVE-2021-45046, CVE-2021-45105, and CVE-2021-44832

A major security vulnerability (CVE-2021-44228) has been discovered in Apache Log4j, which is used by all versions of Fusion and Solr for logging.  This vulnerability is considered a “zero-day” because it was publicized before the Log4j community could determine mitigation and a fix.  This CVE, as well as the follow up CVEs CVE-2021-45046, CVE-2021-45105, and CVE-2021-44832, are colloquially referred to as the "Log4Shell" CVEs.

If Fusion and Solr are properly secured behind a firewall and queries are sanitized by middle-ware, the risk of this vulnerability being exploited is low.  However, as with all vulnerabilities, if an attacker gains access to the server somehow, this vulnerability could be exploited.

Response Matrix

Log4j Information

Log4j is a commonly used library for application logging.

Impacted Log4j Versions

See the Apache Log4j Security Vulnerabilities page for a complete list of impacted Log4j versions based on each CVE.

Updated Versions

Log4j has released a new version 2.17.1 to solve the CVEs and has published several options for mitigation steps.  Upgrading Log4j is not required when we are able to mitigate the problem in the ways described below.

Mitigation for Solr-only (Only if upgrade to 8.11.1 is not possible)

• Note: This section is intended for customers using standalone Solr, or Solr installations external to Fusion. Fusion customers should follow the guidance referenced in the Response Matrix above for their specific version of Fusion.
• Note: Per the Solr community security bulletin, Solr releases are not vulnerable to the followup CVE-2021-45046 and CVE-2021-45105, because the MDC patterns used by Solr are for the collection, shard, replica, core and node names, and a potential trace id, which are all sanitized and injected into log files with "%X". Passing system property log4j2.formatMsgNoLookups=true (as described below) is suitable to mitigate.

Lucidworks recommends upgrading to the new Solr version 8.11.1 which addresses the Log4j vulnerabilities. In scenarios where an urgent upgrade to 8.11.1 is not possible, the Solr community has published a list of simple mitigation options on its website:

https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228

If Solr has been deployed via Docker containers, new images have been pushed to Solr's Docker Hub for all supported versions which mitigate the Log4j vulnerability.  You may need to re-pull the image.

If Solr has been deployed with the .tar.gz or .zip packages, the simplest mitigation is either of the following:

1. Append Solr’s ./bin/solr.in.sh (Unix) or ./bin/solr.in.cmd (Windows) on all nodes anywhere in the file and restart Solr (on all nodes):

   1. Unix: SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"

   2. Windows: SOLR_OPTS="%SOLR_OPTS% -Dlog4j2.formatMsgNoLookups=true"

2. Start (or restart) Solr with the system property -Dlog4j2.formatMsgNoLookups=true added to the start parameters (i.e., bin/solr start -c ...). Option 1 is preferred since it keeps all system properties in a single place but this is an alternate approach.

Fusion 5.4.5 Information

• Note: Fusion 5 relies on multiple open source software components which may utilize older Log4j 1.x versions.  As these open source components are not Lucidworks-developed products, Lucidworks is unable to update these Log4j 1.x dependencies to a later version.  As detailed on the Apache Log4j Security Vulnerabilities page, Log4j 1.x is not vulnerable to any of the Log4Shell CVEs.  Lucidworks' Engineering has reviewed the known CVEs impacting Log4j 1.x (including CVE-2021-4104 and CVE-2022-23302), and Fusion 5 does not ship with any configuration which would leave it vulnerable to exploitation.  As such, although a security scanner may flag the Log4j 1.x files as a security risk, there is no concern unless these files are intentionally configured in such a way as to make them exploitable.  Fusion 5 does not provide access to these configuration files through the running service containers.

Fusion 5.4.5 (released on December 17th, 2021) upgrades Log4j to version 2.16.0 in all Fusion 5 services.

However, Lucidworks was not able to upgrade Solr to 8.11.1 before releasing Fusion 5.4.5, and so it ships with Solr 8.8.2.  The containers produced by the Solr community for all affected versions (including Solr 8.8.2) were updated to include the initial mitigation by default.

Analysis by the Solr community, in consultation with the Log4j community, determined that there was no possibility that Solr installations running with the mitigation were vulnerable to the original CVE (CVE-2021-44228) nor the follow-up CVEs (CVE-2021-45046, CVE-2021-45105, and CVE-2021-44832).  Solr has published details of this here. 

This means that the Solr containers running with Fusion 5.4.5 are not vulnerable, even though they use an earlier version of Log4j.

Fusion 5.5.0 (released on March 18th, 2022) upgrades Solr to 8.11.1, and also upgrades Log4j in the Lucidworks-developed Fusion containers to 2.17.1.

Mitigation for Fusion 5.x (Only if upgrade to 5.4.5 is not possible)

Lucidworks recommends upgrading to the new Fusion version 5.4.5 (or later) which addresses the Log4j vulnerabilities.  In scenarios where an urgent upgrade to 5.4.5 is not possible, the following steps can be taken to mitigate on existing versions of Fusion 5.x:

1. If utilizing an nginx ingress controller as documented here, first mitigate nginx in order to “close the front door” on malicious requests:

   1. Download the attached lua-scripts.yaml and helm-values.yaml.

   2. Apply the configmap:

      kubectl -n <NGINX-NAMESPACE> apply -f lua-scripts.yaml

   3. Upgrade the nginx controller with the attached values file:

      helm upgrade --install nginx-ingress ingress-nginx/ingress-nginx --namespace <NGINX-NAMESPACE> --values helm-values.yaml

   4. Test with the following (they should all return a 403 Forbidden):

      curl -X GET https://<FUSION-HOST>  -H "x-header: {jndi:ldap://testing:000/RCE/Command}"
      curl -X GET https://<FUSION-HOST>  -A "{jndi:ldap://testing:000/RCE/Command}"
      curl -X GET https://<FUSION-HOST>  -d "{jndi:ldap://testing:000/RCE/Command}"  
      

2. Update configurations for the Java-based Fusion services that utilize log4j in order to turn on the formatMsgNoLookups setting via your fusion-values.yaml Helm config file:

   [...]
   
        api-gateway:
             javaToolOptions: "-Dlog4j2.formatMsgNoLookups=true"
   [...]
        job-launcher:
             javaToolOptions: "-Dlog4j2.formatMsgNoLookups=true"
   
   [...]
   
        job-rest-server:
             javaToolOptions: "-Dlog4j2.formatMsgNoLookups=true"
   
   [...]
   
   pulsar:
        [...]
        proxy:
             javaToolOptions: "-Dlog4j2.formatMsgNoLookups=true"
   
   [...]
   

3. Mitigate the Solr server component of Fusion by updating your fusion-solr-values.yaml Helm configuration file's solrOpts settings as follows:

   fusion:
        solr:
             [...]
             solrOpts: "-Dsolr.disableConfigSetsCreateAuthChecks=true -Dlog4j2.formatMsgNoLookups=true"
             [...]
   

4. Apply these configuration changes via a helm upgrade command to the Fusion Kubernetes namespace.

Mitigation for Prometheus solr-exporter

The Solr community has determined that the the mitigation is not necessary because the solr-exporter does not log user/external input or data. 

The solr-exporter Helm chart also does not currently support setting JVM properties, so Kubernetes deployments do not have an avenue to implement a mitigation at the current time. This is slated to be changed so that jvm properties will be availability to be changed

Fusion 4.2.6 SP1, SP2, and SP3 Information

• Note: Fusion 4 relies on multiple open source software components which may utilize older Log4j 1.x versions.  As these open source components are not Lucidworks-developed products, Lucidworks is unable to update these Log4j 1.x dependencies to a later version.  As detailed on the Apache Log4j Security Vulnerabilities page, Log4j 1.x is not vulnerable to any of the Log4Shell CVEs.  Lucidworks' Engineering has reviewed the known CVEs impacting Log4j 1.x (including CVE-2021-4104 and CVE-2022-23302), and Fusion 4 does not ship with any configuration which would leave it vulnerable to exploitation.  As such, although a security scanner may flag the Log4j 1.x files as a security risk, there is no concern unless these files are intentionally configured in such a way as to make them exploitable.
• Note: To upgrade to the latest SP release, please see Upgrade Fusion Server 4.2.6 to Latest SP.
• Note: All of the Service Pack (SP) releases, starting with SP1, address the Log4Shell CVEs, and it is therefore unnecessary to apply the mitigation patch if a SP has already been applied.  Applying the mitigation patch on top of an SP installation may result in unforeseen issues.

Fusion 4.2.6 SP1 (released on February 8th, 2022) upgrades Log4j to version 2.17.1 in all Fusion 4 services except proxy and log-shipper. 

Fusion 4.2.6 SP2 (released on April 20th, 2022) upgrades Log4j to version 2.17.1 or higher in all Fusion 4 services, Solr, as well as App Insights and App Studio. 

Fusion 4.2.6 SP3 (released on June 27th, 2022) modifies or removes some Log4j 1.x files and also upgrades Log4j to version 2.17.2 for App Insights and App Studio. 

Fusion 2.x - 4.2.6 Mitigation Patch

• Note: This patch program is only intended for Fusion version 2.x - 4.2.6, and is not intended for Fusion installations which already have a SP (Service Pack) patch applied.  Applying this patch to an SP installation is unnecessary as the Log4Shell vulnerabilities have already been removed, and doing so may result in unforeseen issues.
• Note: Fusion must be run on JDK 1.8. We no longer support JDK 1.7. See here for details.
• Note: Those who have patched Fusion with the previously available JAR versions should apply the patch again with the latest available JAR version fusion-log4shell-vulnerability-patch-6.0.13.jar.  You can apply the latest patch to a previously patched Fusion installation.
• Note: For patched Fusion 2.x and 3.x versions, warnings will be shown during startup due to the missing JNDI classes. These warnings messages are benign and will not affect normal startup and operation of Fusion. 
• Note: Starting from patch version 6.0.11, the patch program can also be used on any server-based deployments of TikaServer to fix Log4j vulnerabilities.
• Note: This patch will not prevent a security scanner from flagging multiple Log4j 2.8.2 files as possible vulnerabilities.  However, these files are not exploitable to the Log4Shell CVEs as they do not contain JNDI class.  If the Log4j 2.8.2 files are of concern, Fusion must be upgraded to 4.2.6 SP3 (see the section above).

This patch program is for Fusion 2.x, 3.x, and 4.x to fix CVE-2021-44228, CVE-2021-45046, CVE-2021-45105, and CVE-2021-44832.  The program will apply the remediation change: {nolookups} will be added to the logging patterns.

For Fusion versions 4.1.1, 4.1.2, 4.1.3, 4.2.0, 4.2.1, 4.2.2, 4.2.3, 4.2.4, 4.2.5, and 4.2.6, it will patch the Log4j library to 2.17.1 where compatible, and utilize a modified version of the 2.8.2 library elsewhere.  The modified Log4j 2.8.2 library was rebuilt to remove the vulnerability with JNDI lookups.

For older versions (4.1.0 and earlier), the latest Log4j library 2.17.1 is not compatible, so the modified version of the Log4j 2.8.2 library is used instead.

Steps to apply patch:

1. Download the attached JAR fusion-log4shell-vulnerability-patch-6.0.13.jar to the server(s) hosting Fusion.
2. Stop all Fusion services on each Fusion node.
3. Make a backup of the $fusion/apps directory on each Fusion node.
4. Run the program using the non aliased (canonical) path to the Fusion install on each Fusion node:
   

   java -jar fusion-log4shell-vulnerability-patch-6.0.13.jar <fusion_directory>

   Example:

   java -jar fusion-log4shell-vulnerability-patch-6.0.13.jar /home/fusion/4.2.6

   It will print the changes as it runs.
5. Start Fusion services on each Fusion node.

If an issue is encountered after applying the patch, open a Support ticket (or append the information to an existing Support ticket) based on the type of issue:

1. If Fusion is not operating properly after applying the patch, provide the following:
   • The patch version that was applied (6.0.8, 6.0.9, etc.)
   • The output of the patch application (the output that is printed when the patch is run)
   • Logs of affected Fusion service(s) including stderr logs
2. If a security scan is detecting a vulnerability after applying the patch, provide the following:
   • The patch version that was applied (6.0.8, 6.0.9, etc.)
   • The output of the patch application (the output that is printed when the patch is run)
   • File name(s) and path of the JAR(s)/file(s) in question
   • Type of security scanner used

Fusion 2.x - 4.2.6 Mitigation Patch Release Notes