In December, a critical zero-day vulnerability was reported in the widely used Log4j framework. Log4j is so ubiquitous, in fact, that Cybersecurity and Infrastructure Security Agency (CISA) Director Jen Easterly called this “the most serious vulnerability I have seen in my decades-long career.”
It’s not an exaggeration. An estimated 3 billion devices run Log4j. Public cloud providers, major software vendors, and private and public sector organizations have all been affected on a massive scale. So what can we learn from this event? Let’s go deeper into how you can be ready for an event of this scale with excellent systems hygiene and a ready-to-launch patch program.
What Is Log4j?
It may sound obscure to the non-developer, but Log4j is actually one of the most widely used pieces of open source code on the internet. The Java-based code, developed under the Apache Software Foundation, does what it sounds like it does: It logs an application’s activity as entries in a record. Logging is important in maintaining the health and integrity of software and applications, and as I’ve written about before, log data can also be an important tool in security.
But once Log4j adds data to that record, that code—clean or malicious—is inside the system or application. And that’s the key to the vulnerability.
What Is the Log4j Vulnerability?
Turns out, Log4j can be asked to log malicious code, which can then be executed. Since Log4j runs as a privileged system process, the malicious code that it can execute can also run as a privileged system process without requiring authentication. It’s a backdoor that could lead to major exploitations.
The Race to Update and Patch (and the Risks of Not Updating)
The convenience and availability of this open source piece of code (which developers can use in place of creating their own new logging module each time) means the vulnerability is also widespread. Experts say the amount of services and sites it is linked to makes it the biggest vulnerability of all time.¹
Cybersecurity professionals at companies large and small had to work over the holiday break to remedy the code in all of its internet-facing instances. Developers worked day and night to locate breaches, backdoors, or malicious code, and will likely be doing follow-up work to resolve issues in the coming weeks.
An update with a patch has been released, but the lesson is that too many organizations were caught off guard without a plan. Those unable to patch quickly face serious consequences—the private sector faces consequences from the FTC for not updating, and CISA issued a deadline for civilian federal agencies to make updates. It’s an unprecedented degree of oversight, and we all need to be ready in the event this becomes the norm.
Maintain a Better Security Posture and Data Hygiene with a Patching Program
In the event something similar happens again, there are a few things you can do to be ready:
- Practice security posture hardening. Regardless of whether there’s a specific issue that you’re trying to protect your environment against, employ security posture best practices.
- Restrict management interfaces to a trusted set of networks.
- Additional security posture hardening may be achieved by restricting all control plane access through a jump box (bastion host). Restrict outbound internet access to trusted destinations.* Pure Storage strongly encourages the widely endorsed best practice of highly restricting—if not blocking altogether—internet access to administrative login interfaces, including connections via SSH, TLS, remote consoles, and remote desktop mechanisms.
- Closely monitor arrays for abnormal or unexpected workload/IO spikes or utilization as a leading indicator.
- Enable edge detection/protection mechanisms in the firewall/IDS/IPS systems to detect anomalous access or traffic patterns.
- Practice excellent system hygiene. This is the most important aspect of any security program. Data and system hygiene are all part of the collective processes that help ensure data is clean, deduplicated, organized, and accurate.
- Have a communications plan ready. Read my article “A 6-Point Plan for the ‘During’ of a Data Breach” where I cover in detail how to prepare external messaging to customers, the media, and regulatory authorities.
- Make sure all of your systems can be patched in a timely manner. Some general guidance for timelines could look like:
- 24 hours for critical vulnerability
- One to three days for high vulnerability
- One to two weeks for medium risk
- Up to two to four weeks for low-risk vulnerabilities
- Have a patch team ready to be deployed at a moment’s notice. It’s imperative that you have a team at the ready to quickly understand the vulnerability, build, and test your patch before deploying it to your customers. Establish an operations team that is specifically focused on deployment of critical patches. If your organization creates a lot of code, enlist developers who can build software patches.
- Stay on top of any open source code you’re using. If you’re leveraging other people’s code, have a program to stay on top of open source code so you know what versions you’re using, and where, to the broadest extent possible.
- Get visibility into what, if any, open source code and systems your suppliers use, as well as where and how they’re used by your products. Open source code is widely used, but disruptions you cannot control or remediate can lead to disruptions in the code supply chain. In the same vein, have a backup and recovery plan ready for critical systems reliant on third-party systems. For example, if your public cloud instance fails, you might be ready to get critical systems back online via a co-location or private cloud environment if you can.
- Have all users update systems to the latest versions. Updating systems and conducting ongoing education and training of your employees, customers, and partners will always be key. Stay extra vigilant for phishing attacks during a breach—it’s one of the most common ways hackers gain entry into your system when there’s a known vulnerability.
Read more of my cybersecurity resources and learn more about Pure Storage enterprise data protection solutions.
Like this article and want to read more? Sign up for our monthly Perspectives email today. And we promise not to spam you, just inform and inspire you!
[footnote]
https://www.washingtonpost.com/technology/2021/12/20/log4j-hack-vulnerability-java/
*Phone Home and Remote Assist (RA) require port 443 (https) to be open to CloudAssist subnet 52.40.255.224/27 only for outbound traffic and your firewall will need to accept inbound traffic for the established connection.
[/footnote]
