The Cyber Safety Review Board (CSRB), formed in 2021 to review major cyber events, released a report last summer summarizing the discovery of the Log4j vulnerability in 2021. Its disclosure sparked a global race between malicious hackers trying to exploit a flaw in Java code and the security researchers trying to stop them. Among other things, the report found that answering Log4j by a federal cabinet department required 33,000 hours of staff time — the equivalent of hiring more than 16 full-time employees for a year. It’s a big figure, but hardly shocking. Many federal agencies do not have the tools to easily detect and remediate IT vulnerabilities, including Log4j exploits. Because so many applications rely on the logging framework, this vulnerability had the ability to disrupt entire software supply chains – and it did.
Using customer data, a Veracode report found that nearly 58 percent of organizations were using a vulnerable version of Log4j after the zero-day vulnerability was discovered on December 9, 2021. (The remaining “lucky” 42 percent were not saved by proper sanitation practices, necessarily. Many were simply using an older version of Log4j that had not yet introduced the vulnerability into the codebase.)
With a library this ubiquitous, it’s difficult to find all instances of direct use; Finding linked dependencies is an even more arduous challenge. The CSRB report was a wake-up call – a reminder for authorities to improve their cyber hygiene practices before the next Log4j-like vulnerability emerges. How? The path to fix starts with patching.
The reported time and cost of patching vulnerable areas affected by Log4j (after countless scans to figure out exactly what needed patching) didn’t surprise cybersecurity professionals. We’ve known for some time that widespread vulnerabilities like Log4j can be difficult to contain.
As software, Log4j is incredibly widespread. It’s been around for a while, and everyone was using it, either directly or indirectly (through the use of something dependent on Log4j), hence the monumental impact on the software supply chain after discovery.
In addition, the Log4j vulnerability was particularly easy to exploit. Its short and simple code was ideal for exploitation in a tweet, allowing malicious actors to automate exploitation and scale quickly. The stars were well aligned for hopeful hackers in terms of the proliferation of Log4j and the ease with which it could be exploited.
After discovering the Log4j vulnerability, authorities scrambled to uncover its use in their systems. For some organizations, this was a multi-phased process: determining the best tool for the task, obtaining licenses, and assembling a team to accomplish the mission. Agencies that hadn’t invested in software testing prior to the discovery of Log4j faced a daunting task.
On the other hand, agencies that had carefully built software inventory capacities could consult a list to look for affected areas.
The Software Supply Chain: A Not-So-Hidden Culprit
While it was certainly a labor-intensive and costly fix, fixing the Log4j vulnerability itself was unlikely to be the cause of the massive workforce. The real problem arose from the complexity of the software supply chain. Stopping the bug was fairly easy for those who knew exactly where patches were needed. Not surprisingly, most didn’t.
As the subject of various pieces of legislation, agencies have long been asked to improve the security and integrity of their software supply chains. When the Executive Order on Improving the Nation’s Cybersecurity was released in May 2021, organizations should have strived to stay ahead of the threat curve and avoid vulnerability issues before they happen.
To this end, the authorities should be aware of the components of the software that their organizations use. This begins with the use of software bills of materials (SBOMs) – the IT equivalent of labels that list ingredients in packaged foods. SBOMs give companies the ability to ensure they are using software that has been developed following common cybersecurity practices.
A key component of the “left shift”: Organizations using tools to generate SBOMs in December 2021 would have found and patched Log4j vulnerabilities in much less time than those who had to scan their supply chains for affected areas before beginning the patching effort started. The tool of choice? Analysis of software composition.
Software Composition Analysis: The key to staying one step ahead of vulnerabilities
Using an SCA tool allows agencies to take care of the first part of the equation: inventory. A robust tool provided by SCA keeps users informed about constantly evolving open source libraries by automating the detection and remediation of vulnerabilities. Ensure SCA tools are applied across your agency’s portfolio.
Once you understand the scope of your engineering work, you can start the to-do list. In the case of Log4j, users who had applied appropriate SCA tools across their portfolios could see exactly where they were using Log4j, so no time was wasted tracing back and determining where to start their mitigation efforts. A curated list to prioritize and act on is key to keeping up with and staying ahead of security debt.
While open source software (OSS) saves developers from having to reinvent the wheel in many cases, we need to recognize the inherent risk of OSS and protect against it. Knowing that vulnerabilities are inevitable allows us to sharpen and build up cyber hygiene before the next Log4j happens. Keeping code inventory and updating software gives you the upper hand against future attacks.
Going forward: preparing for the next Log4j
We cannot solve the 2021 challenge, but there are steps authorities can take now to avoid the impact of another Log4j-like disaster.
Updating the software is of paramount importance. According to Veracode’s State of Software Security v11: Open Source Edition, an alarming 79 percent of developers never update third-party libraries after including them in the codebase. The more outdated your software is, the more difficult it becomes to patch.
It is important to note inventory and update the list as necessary, but simply knowing inventory is not enough. With new vulnerabilities being discovered every day, agencies must take action now to minimize the accumulation of security debt in code. This saves effort, time and money across the board.
Robust software development practices should be encouraged. Engineering teams will appreciate the rigor when the next major vulnerability comes to light, so it’s important to make the investment as soon as possible.
Now is the time to invest in tools that can analyze how a particular vulnerability is affecting your agency. As recently as December 2021, a tool that provided SCA scans could have helped authorities fix the effects of the Log4j vulnerability more quickly. Instead, many struggled to locate the use of the faulty library in their systems, wasting valuable time and resources.
Agencies should secure their software supply chains with regular scanning and quick patching when needed before it’s too late. While the task may seem intimidating, creating an inventory will save your agency time, money, and stress in the future—a worthwhile effort given today’s ever-evolving cyber threat landscape.
The views expressed here are those of the author and are not necessarily endorsed by Homeland Security Today, which welcomes a wide range of viewpoints in support of securing our homeland. To submit an article for consideration, email Editor@Hstoday.us.
