December 30th, 2021 Update: Apache released another new version (2.17) after the previous patch was discovered to be vulnerable as well.
This article has been updated to reflect the latest recommendations. It remains relevant for those looking to understand what measures beyond patching could be at their disposal.
With notifications about the log4j vulnerabilities (CVE-2021-44228 & CVE-2021-45046 - aka, log4shell) all over the internet, you have likely already heard the steps you need to take to remediate. That said, keep in mind that this is a recent and evolving threat. It will be around, in different iterations, for the foreseeable future. In case you haven’t, here are the remediation steps that Apache recommended, which were updated after additional holes were discovered in the 2.15 patch (...and the 2.16 patch). Long story short, patch to version 2.17 asap!
In this post, we’ll take you through the steps we took to go beyond patching. You can use these steps to help harden your defenses against the log4shell exploits. We also discuss measures you can take to protect your software supply chain more generally, and evaluate them in the context of this threat.
Sound the alarm!
First off, we notified our customers and prospects of the developing threat, and the remediation steps above, in step with the broader community. We kept them updated as the situation developed.
The nature of our solution demonstrated that we were confident in our ability to block, detect and respond to exploits of log4shell. We began by confirming that we protect against attacks against these vulnerabilities. To this, we added vulnerability scanning, and threat hunting to monitor and identify potential log4shell events through multiple sources of threat Intelligence, and auditing of raw data from customer endpoints, combined with machine-learning models for prioritization and escalation. The attack is blocked immediately/automatically, upon detection.
However, given the insidious nature of log4j, we really wanted to do more. Similarly, you may be wondering what else you can do? Do you know which software uses this logging package? Do you know which endpoints they are installed on? Are you aware of other steps can you take once you have patched the software (or, at least, the instances that you know of)?
A step further: proactive scanning, threat hunts, and automated/immediate response
Here are the steps we took to get ahead:
1. Created a Custom Vulnerability Scanner
The idea here was to help our clients identify the endpoints that are vulnerable. Because log4j is “buried in so many other applications” it is very hard to identify everything that is vulnerable. This is true both for end-users, who would likely be unaware which of the apps they have added include it, but also for administrators. While admins or devs might have a better idea of certain software that used log4j (especially within their own SDLC), it’s still difficult for them to enumerate everything vulnerable within the environment.
Further, certain network structures can make it difficult to scan for this. Not wanting to delve “too deeply” here, there needs to be a callback from the vulnerable endpoint to the scanner… so, a “work-from-home topography” can get in the way here. Not to suggest that this wouldn’t still be a difficult problem to scan for, even if everybody was operating in the office.
Remediation (patching) was ultimately the responsibility of the customer, but adding the visibility was certainly a boon.
2. Created Custom Threat Hunts
We knew that with news breaking about this vulnerability, that new attackers would be actively looking for vulnerable organizations.
Though every indication is that our pre-existing protection was solid against these attacks, we wanted a contingency. So, we opted to create a way to enumerate whether anything was being targeted. This resulted in custom threat hunts that we proactively use to search for Indicators of Compromise (IoCs) and Indicators of Attack (IoAs) - in case something slipped through. From there, anything we discovered would be addressed through our usual detection and response processes.
So, we searched for any indication of an attacker reconnoitering their environment. Unsurprisingly, we did find evidence of probing from known bad IPs. We also saw indications of exploit attempts, of course.
3. Offer to Inspect Logs
For those organizations who were uncertain as to whether or not they may have been at risk and need assistance — even if they're not an ActZero customer — we welcomed them (and you) to submit your logs before end of day, December 16, 2021 for a complimentary scan for attack activity.
So, in sum, beyond general patching recommendations, we proactively scanned endpoints to identify vulnerable software, we searched for any indication of an attacker reconnoitering their environments, and we searched for evidence of exploitation above and beyond our base prevention and detections. Then, we communicated our findings.
Ultimately, in most cases this resulted in peace-of-mind and some prescriptive advice. While in others, it resulted in responsive action being taken. Win-win.
Other proactive steps, in the context of Software Supply-chain Compromises
Many products use log4j without end-users knowing it. Even software used by security researchers - Ghidra for example. Or, more widely-adopted consumer software - Minecraft, for example (don’t ask us how we found out about this one ;) ). In the wake of the Solarwinds Solarigate attacks earlier this year, we produced a white paper discussing 6 Steps to Secure Your IT Supply Chain that we think are relevant here.
While a purist might argue that this isn’t a supply-chain attack per se (usually defined as one targeting a specific partner/provider in order to get at an individual organization), we’ve seen this term confounded with the “IT” or “Software” Supply Chain, in reference to wide-scale attacks such as Solarwinds Solarigate, Microsoft Exchange “HAFNIUM”, or the more recent Kaseya attacks we saw this year. We think that it’s appropriate to classify it as a “software or IT supply chain attack” as the source of the vulnerability is part of your IT supply chain. The Log4j2 library is a component which isn't part of Apache, but is very widely used. The impact is just as significant, affecting vast numbers of organizations.
As such, we’ve listed the high level steps to mitigating the risk of an IT Supply Chain attack here, along with how they might have helped related to log4shell.
- Inventory and Manage IT Assets
While this was unlikely to have happened at this level of detail, except in the most rigorously secured environments, it is applicable to this situation. Doing this now helps administrators know what software/systems might potentially be running Log4J libraries and require patching/disabling its use - especially for folks without access to our custom scanner. - Monitor 3rd Party Risk
While this is generally a good practice, in this case it is safe to assume that 3rd parties are impacted by this, because it is so wide-spread. - Address Software Supply Chain Hygiene
See above… In the context of log4j, the software is open-source, so while you COULD technically have evaluated the software yourself, this is likely too resource intensive for all but the most diligent of security practices. - Segment The Network
While this would potentially have mitigated the impact of log4shell exploits, it is difficult to engineer one’s network for every possibility. Read the white paper for more info. - Reduce Your Attack Surface
Another generally good practice… had you done so, you would likely have seen fewer vulnerable software (do you really need to have Minecraft on your corporate machines?). - Monitor the Environment
Catching any potential attacks in progress could greatly reduce the impact of log4shell (really, any attack). If such capabilities are in reach for your organization, keep them up. For everybody else, consider augmenting your capabilities with a service like ours.
Call to action
We applaud the community’s effort to highlight the risk of this attack, and quickly put forward measures to mitigate that risk. There is, of course, always more that can be done. We hope that you can leverage some of the proactive steps listed in this post to further mitigate the risk to your company, customers, partners, and employees.
In case you missed it, be sure to patch, immediately. Once you have done so, we have some resources that may help. Check out our white paper 6 Steps to Secure Your IT Supply Chain for more detail on the steps above, in the context of Solarwinds Solarigate.
Consider our offer to review your logs for indications of this attack. Submit your request here.
We also have an “emergent threats” threat insight piece that can help you understand and deal with developing/emergent threats like these.
