The Cyber Safety Review Board (CSRB) just released their report on the Log4j vulnerability, CVE-2021-44228, and it’s a fascinating read. The report (which it should be noted - is incredibly well written) illustrates a painful fact - the operational overhead of dealing with a major vulnerability is as dangerous as the vulnerability itself, if not more so.

The CSRB worked with over 80 organizations to understand exactly what happened with Log4j, step by step. They recount how the vulnerability was introduced in 2013 through a community-submitted feature intended to ease data storage and retrieval called “JNDI Lookup plugin support”. Then how the vulnerability was discovered by the Alibaba Cloud security team in November 2021, and how the Apache Software Foundation acted fast to remidient. And then how all hell broke loose. 

The report makes an interesting point about how much of the global response to the vulnerability “went right”. The cybersecurity and open source communities banded together as the race between attackers and defenders began, and everybody’s weekends were ruined. But even with the response generally being swift, many organizations suffered, while the CSRB notes that it was difficult to “identify an authoritative source from which to understand exploitation trends across geographies, industries, or ecosystems”.

So what happened? Security and development teams found themselves in a situation where they quickly had to answer the question “who’s using Log4j?” and the more difficult question of “who’s using something that uses Log4j?”. For most modern applications, 70% of code is code you didn’t write, but depend on through open source packages. A handful of these are direct dependencies selected by developers, but the majority are indirect dependencies automatically pulled into your projects.

This means that even smaller organizations needed to navigate a complex dependency graph to figure out what depends on Log4j. Beyond that, they didn’t have time to consider that “who uses Log4j” is an important question, but equally important is “which of those Log4j instances are actually reachable and exploitable?”

Our founders here at Endor Labs faced exactly these challenges when they were running a 400-person engineering team. A common occurrence they ran into was developers going on slack to ask “who’s using this dependency? I plan on updating it and you might be impacted.” Getting visibility into the dependency graph and analyzing the potential impact of something as common as an update was next to impossible. The problem was so prevalent that it eventually drove them to team up and start Endor Labs.

This is where we start seeing the true impact of Log4j, the amount of hours lost dealing with mitigating it. According to the report, “one federal cabinet department reported dedicating 33,000 hours to Log4j vulnerability response to protect the department’s own networks. These costs, often sustained over many weeks and months, delayed other mission-critical work, including the response to other vulnerabilities.” 

This problem is captured perfectly by this line in the report: "Industry experts have called for increased automation to help organizations identify vulnerable software and execute Enterprise Risk Management at scale, but they also recognize that cataloging software components at this depth can be prohibitively labor intensive. Solutions to enable these necessary capabilities do not exist." We hope to change that with Endor Labs.

When it comes to open source security, solutions tend to fall into two major categories: vulnerability scanners (SCA), and continuous update solutions. By their nature, they address half the problem at the most. As we’ve seen, the true impact of Log4j was made apparent by a company’s ability to react to it quickly, and the overhead of that response was more dangerous than the vulnerability itself. The report event goes as far as to say that:

“Perhaps most significantly, the force exerted on the urgent response and the challenges in managing risk also contributed to professional “burnout” among defenders that may, compounded with the generally intense pace of many cybersecurity jobs, have a long-term impact on the availability of cybersecurity talent.”

Now that we’ve spent some time exploring the problem, let’s think about some of the solutions, and a bit of insight into what we’re building at Endor Labs. 

Contextually Aware Vulnerability Management Practices

Contextually aware vulnerability management practices can help improve vulnerability response times and reduce tangible and intangible response costs. Log4J has cost the national economy enormously in terms of both remediation costs as well as opportunity cost.  The Cyber Safety Review Board highlights that “Log4j is an ‘endemic vulnerability’ and that vulnerable instances of Log4j will remain in systems for many years to come, perhaps a decade or longer.”

But how can this possibly be the case with the most widely publicized vulnerability in recent memory?

For many organizations the answer is one or more of these:

• It is prohibitively expensive to address the issue holistically
• A holistic application component inventory does not exist for the organization
• They are dependent on a third party, either open source or vendor supported and can’t accept the cost of migrating off or mitigating the risk in that dependency

For the first two problems the issue is often one of visibility and scale.

Scalable solutions don’t exist today that enable organizations to understand if a vulnerability applies in the context of their application and business. The lack of scalable assessment solutions has led to an industry paradigm where security practitioners are updating everything and making risk management decisions with only partial context. In many of these circumstances, if the organization had full context, patching for critical vulnerabilities would be made not a priority for an application because it didn’t apply in the context of the app. This includes scenarios where the package version is not used or where the package version is used but the vulnerable method is not. 

Today, what happens is:

• A vulnerability is discovered
• Vulnerabilities are published to vulnerability feeds that are consumed by vulnerability management solutions
• Vulnerability management solutions compare the package version with those listed in the vulnerability feed.
• If the package version exists in an application or server, then a vulnerability is reported.

This lacks three important pieces of context that are absolutely necessary to make informed risk management decisions:

1. How important the application is to the business
2. The relative likelihood of the application being attacked 
3. The context of which the application uses the code.
4. When issues like log4j are exposed through indirect usage of the dependency, upgrades may not even be made available, resulting in reliance on self patching or waiting for another dependency to be updated. 

Even if a security practitioner had all of this context, they would still need to understand if an update to the dependency may break their application, either through direct API breaking changes or dependency conflicts. While this may not be the case in a small patch release, it may be the case if the vulnerable dependency is indirectly used and not directly used in the code. 

How CISA and Industry Security Leaders Can Help

In order to help important vulnerabilities scale more effectively, CISA should partner with NIST to ensure that vulnerable methods are standard searchable fields that are applied to every applicable vulnerability in the National Vulnerability Database (NVD). This should be prioritized for vulnerabilities in the known exploited vulnerabilities catalog.  By including this metadata, static analysis solutions will be able to detect vulnerabilities in a scalable manner down to the level of if the vulnerable method is actually being used. 

Beyond the NVD, the foundation of vulnerability remediation is a software bill of materials. The Cyber Safety Review Board says: “As designed today, SBOMs are limited, for example by variances in field descriptions and a lack of version information about cataloged components, and lack of automation on the consumption end due to these variances. Addressing SBOM standardization gaps would support a faster software supply chain vulnerability response.”

Further, as the industry pushes to adopt and improve SBOM tooling the industry should focus on data quality and rates of change. Assuming that an SBOM did have accurate information about the versions of software packages, and had high fidelity accurate data at all times, an SBOM is limited in that it is only a snapshot in time and doesn’t evolve over time. Even if the same version of an application is deployed the dependencies of the application are frequently continuously changing with no action required by development teams or customers.

While dependency pinning has been designed to reduce the likelihood of operational issues from dependency changes over time, it's unlikely that all of your direct dependencies have pinned every one of their dependencies. Dependency pinning is a practice that has clustered usage throughout the industry and as a result, it's very frequent that as your dependencies release new versions of your package, the SBOM will change without any action on the part of the software producer or software consumer.

Lockfiles do solve this, but their support is package manager dependent and not supported everywhere. 

This moving target problem make SBOMs a snapshot proxy to improve vulnerability management but not an accurate inventory that can be fully trusted. Enabling SBOM tooling to track changes over time, and encouraging the use of dependency pinning and lockfiles where applicable should also be key recommendations for the cybersecurity industry.

The Risks of Requiring Financial Commitments in OSS Projects

Software is the foundation on which society is built. Traffic lights, cars, and water supply are controlled using software. The foundation on which software is built is open source software, such as log4j. Our society is built on the backs of the giants who are the maintainers of the open source community. This community is volunteer, sometimes sponsored but frequently free. Often, maintainers of open source packages do so out of good will as a side project from the comfort of their homes. They have families and other priorities, just like all of us. Sponsoring of these volunteers is a noble pursuit, but requiring financial commitments from organizations using open source software may be an impractical solution. The Cyber Safety Review Board indicates that “Private sector companies that build commercial software that includes open source libraries or dependencies should commit financial resources toward the open source projects that they deploy” such as “paying developers, security engineers, and other essential roles in supporting secure software development, vulnerability disclosure and handling processes, and vulnerability response for open source projects; and  contributing funding and knowledge to centralized open source security projects.”

The taxing of projects used by organizations should be designed so as not to disincentivize the ethos that the open source community was built on, which was one of collaboration, experimentation, and group innovation. Taxing the open source community to secure it if done, should be done in a way that doesn’t disincentivize usage and collaboration with open source software and regulation may do just that. 

How Endor Labs Supports Safe OSS Adoption

Endor Labs helps teams scale by giving them the context they need to make more informed risk management decisions. Endor Labs performs static analysis of code to help our users understand which package versions are actually used and which are not by your application. When an application uses a software dependency, typically that software dependency relies on others transitively. Transitive dependencies are often never used and even those dependencies directly imported by your application may be unused over time due to frequent refactoring of code.  This problem varies by software language and ecosystem. Organizations should put a focus on removing unnecessary dependencies from their application code. By narrowing the graph of dependencies, promoting dependency reuse and removing unnecessary dependencies, organizations can further scale vulnerability management as well as improve their application quality. Endor Labs further uses this static analysis to identify areas where changes between versions of packages may impact applications by detecting conflicting dependencies. This helps teams prioritize remediation efforts by the effort to fix an application. Context also helps in prioritizing remediation efforts. By ensuring that remediation efforts can be focused on areas where they are actually using vulnerable code, organizations improve the impact and reduce the cost of their remediation efforts over time. Industry can reduce the economic cost of security as well as decrease defender burnout rates and help to reduce the skill gap in our industry by focusing on improving context to optimize for impact over output. 

‍

At Endor Labs we help organizations take an approach we believe to be more practical and aligned with the ethos of open source. Organizations need a strategy for how they manage their software supply chain to help them manage critical incidents like Log4j. If a software dependency is unmaintained and transitively relies on log4j, your organization needs a plan for how to update that package version. Too often that answer is, “wait for someone else to fix it”. Organizations should take conscious action with open source packages and with conscious action there can be three choices:

• Accept the risk
• Switch to a different package as your software dependency
• Maintain that dependency yourself and contribute back to the community by accepting the cost of maintenance as a cost of doing business. 

We help organizations to identify unmaintained software so they can make more informed risk management decisions on how to approach managing open source. By helping organizations select higher quality software dependencies, giving them visibility into unmaintained software packages and helping organizations to narrow their dependency graph Endor Labs helps your teams make better choices around open source package selection and maintenance.

Our Final Thoughts

As the industry deals with the shellshock (no pun intended) of the log4j incident we’re reminded how much further the cybersecurity industry has to go to meet the dual goals of managing national security and risk along with doing so in a cost effective manner. We’re reminded painfully that our industry still has significant room to mature and that as we mature, the problems we face will change. As proprietors of developer productivity and cybersecurity, we’re ready to partner with the community to solve these problems. Only through partnership with communities can we truly make critical incident response, scalable, cost effective and timely.