Matteo Merli, StreamNative CTO, Apache Pulsar PMC Chair; Addison Higham, Lead Engineer, StreamNative, Apache Pulsar Committer.
This post was updated Tuesday with detailed information on how the latest Log4j vulnerability affects other Pulsar ecosystem tools, the StreamNative Cloud, and the StreamNative Platform.
Log4j 2.16.0 has been released as an emergency release due to a new vulnerability exposed over the weekend. This article provides an update to the latest version of Apache Pulsar regarding the critical vulnerability of Log4Shell, namely log4j. The following lists the status of vulnerabilities in the open source versions of Apache Pulsar and StreamNative products and the steps that need to be taken to address security vulnerabilities. If you have any questions or questions, please fill out the form at support.streamnative. IO or send an email to [email protected] for StreamNative support.
Vulnerability impact and product status updates
Apache Pulsar Open Source project status update
A total of three Cves have been found that affect Log4J. Of these three vulnerabilities, only two affect Apache Pulsar by default. The Pulsar community has been working to patch the impact of the vulnerability caused by all three Cves. As of now, Pulsar versions 2.9.1 and 2.8.2 have updated log4J versions to address the effects of all known vulnerabilities. New releases of Pulsar 2.7 and 2.6 are in progress. The following table summarizes the impact of the Log4J vulnerability and actions to address the impact.
| CVE | describe | Pulsar is affected | The solution |
|---|---|---|---|
| CVE-2021-44228[1] | This vulnerability allows external remote code execution (RCE) attacks on vulnerable versions of the system. This vulnerability can also lead to data /secret leaks. Log4j version 2.15.0 fixes this problem. | Pulsar versions prior to 2.9.1 or 2.8.2 are vulnerable by default. However, currently no known RCE threat has been found (in the case of Docker images or the latest JRE), but data or Secret may leak through environment variables. | • Update to Pulsar 2.9.1 or 2.8.2 • And add the formatMsgNoLookups attribute or LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable |
| CVE-2021-45046[2] | If configured using some log4J format, this vulnerability can bypass the workaround implemented with formatMsgNoLookups. This is the result of an incomplete fix in Log4j 2.15.0 that causes the same risk as the first CVE when some additional configuration is added. Log4j version 2.16.0 fixes this problem. | Pulsar is not attacked by default; Unless the user has configured Pulsar Functions using the Process runtime or has configured a non-standard LOG4J format. See non-standard Log4J configurations for more details. | • Update to Pulsar 2.9.1 or 2.8.2 • Or disable Pulsar Functions process runtime or remove non-standard log formats |
| CVE-2021-45105[3] | This vulnerability exposes the system to the risk of DoS attacks when configured using certain Log4J formats. This vulnerability still exists in log4j 2.16.0. Log4j version 2.17.0 fixes this problem. | Pulsar is not attacked by default unless the user has configured Pulsar Functions using the Process runtime or has configured a non-standard LOG4J format. See additional details on non-standard Log4J configurations. | • Update to Pulsar 2.9.1 or 2.8.2 • Or disable Pulsar Functions process runtime or remove non-standard log formats |
StreamNative product status updates
StreamNative is committed to quickly resolving and fixing these issues for all customers. The following table provides detailed information on current StreamNative product status updates and recommended user actions.
| product | state | Recommended Operations | The future trend |
|---|---|---|---|
| StreamNative Cloud Hosted | Fully resolved as of 10 December | No need to do anything | As part of regular maintenance work, StreamNative will upgrade all clusters to Pulsar with Log4j 2.17.0 in the coming weeks. There will be no impact on customers. While the formatMsgNoLookups solution is sufficient to fix the vulnerability, StreamNative ensures that all clusters are running the latest version of Pulsar. |
| StreamNative Cloud Managed | As of December 15, all clusters have fully resolved the impact of the vulnerability. For special cases, the StreamNative technology team has communicated directly separately. | Some customers may receive information about additional actions that need to be taken when using Java Pulsar Functions. Please refer to the details below. | As part of regular maintenance work, StreamNative will upgrade all clusters to Pulsar with Log4j 2.17.0 in the coming weeks. There will be no impact on customers. While the formatMsgNoLookups solution is sufficient to fix the vulnerability, StreamNative ensures that all clusters are running the latest version of Pulsar. |
| StreamNative Platform | We contacted and communicated on December 13 how to solve the impact of the bug (by formatMsgNoLookups). The StreamNative Platform released a new release for the 2.8 branch of Pulsar on December 13. | The vulnerability can be resolved by directly updating the deployment configuration file (values file) using the formatMsgNoLookups configuration (see details below) or by upgrading to a version later than Sn-Platform 2.8.1.29. Please refer to the details below. | A version will be deployed to Helm Charts and these changes will be applied by default to all clusters. This release is expected in the next 1-2 weeks. |
| Support | We contacted and communicated on December 13 how to solve the impact of the bug (by formatMsgNoLookups). | This can be resolved by setting the formatMsgNoLookups property or updating the open source version of Pulsar to 2.9.1 or 2.8.2 (in the final RC version validation). Please refer to the details below. | The release of the open source Pulsar 2.7 and 2.6 branches is being prepared and will take another 1-2 weeks. Until then, resolve the effects of the vulnerability by using the formatMsgNoLookups attribute. |
The solution
Non-standard Log4J/Pulsar Functions and process runtime user solutions
As shown above, if the user has configured the Customer Log4J template string or Pulsar Functions through Process Runtime, even if no message Lookup is configured, Log4j can also be used externally for vulnerability attacks. In summary, if the user has configured the Log4J template string to contain references to context objects ($${CTX :}, %x, % MDC, etc.) or uses Pulsar Functions at process runtime, your system may be at risk. We recommend that users remove custom Settings and upgrade to Pulsar version 2.8.2 or 2.9.1 if they must use Pulsar Functions at process runtime.
Pulsar Functions user solutions
Pulsar Functions written in Java need to be redeployed to get the updated values. We will contact StreamNative Cloud Managed customers directly in the coming weeks to resolve any outstanding functions issues. This operation is similar to using a pulsar – admin functions provides the update command, sample see pulsar.apache.org/docs/en/fun…
StreamNative Platform user solutions
If you are using the StreamNative Platform, you can edit the Values file accordingly to deploy the formatMsgNoLookups operation. We do not recommend updating the chart version at this time as the latest version contains some changes that are not related to this vulnerability. Below are the new values that need to be added or changed in the value file.
Note: here are the solutions to the previous plan — to add – Dlog4j2 PULSAR_EXTRA_OPTS. FormatMsgNoLookups = true operation are equivalent, can choose any one of them.
Bookkeeper: configData: LOG4J_FORMAT_MSG_NO_LOOKUPS: "true" Broker: configData: LOG4J_FORMAT_MSG_NO_LOOKUPS True proxy: configData: LOG4J_FORMAT_MSG_NO_LOOKUPS: true Streamnative_console: configData: LOG4J_FORMAT_MSG_NO_LOOKUPS: true Zookeeper: configData: LOG4J_FORMAT_MSG_NO_LOOKUPS: trueCopy the codeYou can upgrade the version in the following ways:
Images: Autorecovery: repository: Streamnative/Sn-platform Tag: 2.8.1.30 Bookie: repository: Streamnative/SN-platform tag: 2.8.1.30 Broker: repository: Streamnative/SN-platform tag: 2.8.1.30 functions: Repository: streamnative/sn-platform tag: 2.8.1.30 Presto: Repository: Streamnative /sn-platform tag: 2.8.1.30 Proxy: Repository: streamnative/sn-platform tag: 2.8.1.30 Zookeeper: repository: streamnative/sn-platform tag: 2.8.1.30 streamnative_console: repository: streamnative/sn-platform-console tag: "1.10-rc2"Copy the codeThe preceding two operations can be deployed at the same time.
Apache Pulsar Open Source version solution
If you’re using the open source version of Pulsar, see the blog post. In addition, we encourage you to upgrade to version 2.8.2 or 2.9.1 whenever possible. In addition, users of open source or StreamNative Helm Charts do not need to wait for an image update in Helm Chart and can specify a new version of Platform Chart similar to the one shown above.
Related updates of surrounding ecology
Below we list some of the tools in the Pulsar ecosystem and ask if they are affected.
- Pulsar Manager – Pulsar Manager is not directly affected by the Log4j vulnerability. Pulsar Manager is built using the Spring framework and uses LogBack for logging. Logback is also affected by the vulnerability, but unlike the Log4j vulnerability, it requires permission to edit the Logback configuration file directly, which greatly reduces the severity of the problem. We will release a new version of Pulsar Manager after spring releases a fix.
- Pulsar Spark/Flink Connector – Neither Apache Flink nor Apache Spark Connector of Pulsar will be directly affected. Neither Connector contains log4J directly, but is based on the log4j bundled with the Flink or Spark distribution you are using. Upgrading Flink or Spark will solve this problem.
- Pulsar IO Connector – The Connector in Pulsar IO (by default) does not contain log4j itself, but relies on log4j provided by the Pulsar IO framework. Following the instructions above and redeploying the Connector will resolve the vulnerability. This recommendation applies to all connectors included in Pulsar and those supported by StreamNative open source. Custom developed connectors may require independent validation.
Other FAQ about log4j security vulnerabilities
- The severity of this security breach. The potential problem can be so severe that a remote code execution (RCE) vulnerability exposes arbitrary code to more damage done by an attacker, potentially gaining full system access. There are no known EXPLOITS of RCE vulnerability against Apache Pulsar. However, we recommend paying attention to this issue because the mechanism for executing code is complex and has many potential possibilities. Exposing environment variables or other environment data is a potential attack. The Apache Pulsar community and StreamNative strongly recommend that users take immediate action.
- Why does it take so long to release an open source version? Releasing a new version of open source Pulsar involves following a process clearly defined in the community. As additional vulnerabilities were discovered in the past week, the community/Pulsar PMC (Project Management Committee) decided to restart the release process several times to include the latest version of Log4J to resolve all outstanding issues.
- Is there any known impact of this Log4j security breach on StreamNative customers and their data? At present, we have not found any loss of Secret or customer data due to the use of this vulnerability. Some cloud vendors automatically report security issues when someone tries to exploit these vulnerabilities, but so far no successful attacks have been detected. We are in the process of replacing credentials for customers who have received warning reports.
reading
- StreamNative Cloud and StreamNative Platform provide Log4j vulnerability solutions
- Apache Pulsar is a solution to the Log4j2 vulnerability (CVE-2021-44228)
Reference links
[1] CVE – 2021-44228: NVD. Nist. Gov/vuln/detail… [2] CVE – 2021-45046: NVD. Nist. Gov/vuln/detail… [3] CVE – 2021-45105: NVD. Nist. Gov/vuln/detail…