fgj.codes

Practical Considerations for AI Incident Reviews

Sun, 25 Jan 2026 12:48:30 -0800

I’ve seen a lot of interest recently in using large language models to generate incident review documents, but very few actual attempts that I’ve observed at this have been successful. Why is that? Many people are approaching this problem without an understanding of the limitations of their tools, or what the point even is of doing incident reviews. In this post I’d like to expand on some things to consider if you want to take a stab at using LLMs for incident reviews without wasting anyone’s time.

The broadest category of problems that LLM incident reviews encounter is with input data quality. Sometimes this problem is obvious: if you can’t get structured data into your LLM tooling at all about your incident (maybe you’re copying and pasting Slack messages into a chatbot text input, or even trying to one-shot an incident review document with just a description of the incident) then whatever output you get is likely to be much lower quality than even a cursory human-drafted incident review document.

Even as you start getting access to some sources of structured input data, like a model context protocol (MCP) server for Slack data, it’s easy to underestimate how much additional data you need. When I’m doing a routine, one or two hour quick incident review draft I’m at minimum going to review Slack transcripts, telemetry data, and version control artifacts related to the incident. If you have access to all of these via MCP, integrated with one LLM, you are probably in the 99th percentile of AI adoption. If you don’t, you are going to find your generated incident summaries lacking in key context that would make them useful (more on this later).

There are some smaller, funnier, data quality problems you can run into as you’re trying to string together multiple data sources. My favorite is user identifiers: even for people it’s hard to tell that <@U123ABC> in a Slack transcript is the same as @mygithubusername in a pull request and Jane Doe in a Zoom meeting. When we built AI-generated incident narratives in Jeli Narrative Builder we could normalize user identifiers by querying a database that linked user accounts across all the data sources we ingested. Can you do the same?

I’d be surprised if these problems exist in exactly the same form in a couple years – I like to say that MCP today reminds me of the early years of OpenTelemetry, and OTel adoption today has become vastly easier than it was when I first dipped my toes into it six years ago. These aren’t insurmountable issues, but it’s important to understand that the value you get out of your AI incident reviews will be directly proportional to the effort you put into building infrastructure to provide your AI tooling with high quality input data.

With that said, there are some other issues I’ve seen that have persisted across multiple generations of foundation models, that seem relatively sticky even as model capabilities improve across the board.

One issue I’ve been surprised to see little improvement on: one- or no-shot incident review documents tend to have bad information architecture and often summarize input data without grouping it in a meaningful way. Bad human writing often does this as well of course, and I think this also mirrors the sorry state of incident review “best practices” that you see across the industry.

A broader problem that I think contributes to poor information architecture in these documents is that a lot of the most useful data you can present in an incident review document is interstitial. A shallow version of this issue you’ll run into almost immediately: you almost certainly have structured data about your incident that starts when your incident was “opened”, but can you trace that back to when the incident was actually reported and the investigation actually started?

It only gets harder from there. It’s easy to record that responders made a deductive leap to connect a particular symptom to a particular cause, but one of the most important things an incident review author can do is go ask those responders how they made that leap, and then record it for other people to learn from. Similarly, incidents often expose implicit tradeoffs that are left undocumented because they expose realities that your organization would prefer not to acknowledge: how often do you document that you are choosing not to make an investment in training or reliability because you’re prioritizing delivering features that actually drive revenue?

Just as smarter LLMs are getting better at expressing uncertainty rather than hallucinating, it seems like it should be possible to prompt a state of the art model to identify these interstitial gaps in the official incident record so that a human can work with other humans to fill them in. Many people driving AI adoption for incident review don’t do this though, because they don’t actually understand why incident reviews are valuable.

A simple model I use for how organizations benefit from incident reviews is:

One person learns a lot about the incident from writing an incident document, and maybe facilitating an incident review meeting
That person is able to take their learnings and present them to the responders of the incident, who come away with a stronger understanding of the incident they were involved in
The artifacts from the previous two steps become part of a long term record that a broader group of people in your organization can use to understand the systems they work on

As you go down that list, more people are impacted by the process but their engagement also becomes more shallow. Critically, the whole process is anchored by a small group of investigators and responders who are most invested in producing learnings from the incident review process.

If you’ve never thought about this before – and I am frequently shocked by the number of people who advocate for incident review processes but have never really considered the mechanisms of how their organizations actually benefit from incident reviews – it’s possible to completely collapse the entire process by introducing an AI-generated incident review document in the wrong place. When you remove the person acting as a quality gate in the first step, there’s no one to drive engagement with the process in the second step. And then your outputs from the process sit neglected in a Google Drive folder somewhere.

Bringing this back to a more neutral stance: incident reviews are fundamentally a socio-technical process, and they do not provide benefit if people don’t engage with them. When you’re thinking about introducing AI tooling to your incident reviews, it matters a lot if people see it as an opportunity to extend your capacity for learning from incidents. If not, it’s unlikely that teams who are disengaged with your process will change their mind because you’ve created tooling that allows them to engage even less with it.

To that end, I’m not particularly optimistic about “agentic” workflows for drafting incident reviews (if you disagree, I’d be happy to be proven wrong). I’d be more interested to see something like NotebookLM for incidents, where the goal is to help an analyst better keep track of all the sources of data they have for an incident. In particular I think LLMs could be particularly useful for analyzing recordings of incident calls, which are often long enough that it’s not worth the effort for a human analyst to track through them.

I was pretty skeptical that AI tools would ever be valuable for incident reviews a couple years ago, and I’ve since softened my position a lot. But, like I’ve laid out in this post, I think the value from this tooling will go to organizations that already value learning from incidents, that are willing to put the necessary effort into building the right data infrastructure to provide AI incident tools with the context they need to function, and that understand incident reviews well enough to be able to tell when their tooling is producing low quality output. Conversely, if you aren’t learning from your incidents today you probably don’t have the right organizational muscle memory to get any benefit from producing a higher quantity of lower quality incident documentation. The work of understanding your incidents is hard but it isn’t too late to get started.

Why do we do blameless incident reviews?

Sat, 29 Mar 2025 10:00:00 -0700

Blamelessness is a widely accepted and poorly understood practice in the software industry. It was introduced by Etsy engineers with a deep background in resilience engineering and a desire to improve safety in their organization, but the roots of blamelessness in safety research have been lost as the concept has made its way around the world. Today, software engineers routinely misunderstand the value of blamelessness and practice it in ways that are sometimes actively counterproductive. Let’s talk about why.

There are two primary benefits to practicing blameless incident reviews:

Creating a “just culture” in your organization
Actually understanding your incidents

The first benefit, from what I’ve seen, is what most engineers are familiar with. You may not know the term “just culture” (see the article linked in the first paragraph) but it encapsulates the idea that people should be treated fairly when they’re involved in incidents, and that doing so has a lot of organizational benefits. Some of these include: avoiding undue risk-aversion in engineers, making it easier to speak freely about the events of an incident, and encouraging teams to freely participate in the post-incident process. This is generally how the issue is framed in the articles you find if you google “why do blameless postmortems”. This isn’t wrong, but it’s incomplete.

In organizations that over-index on this concept of blameless, you see a couple of common anti-patterns:

Senior engineers are comfortable blaming themselves for incidents they participated in, understanding that they won’t be punished for their actions
Blame shifts from individuals to teams, since diffused blame implies less individual responsibility
People assume mistakes just happen, and don’t discuss the details of participants’ actions during an incident. If you use the passive voice a lot in your incident writeups you are probably doing this
Incidents are attributed to “human error”, while the human who committed the “error” is not personally blamed

To use some safety jargon, this is what happens when you apply blamelessness in a Safety-I culture. Safety-I assumes that normal operations are safe, accidents (or incidents, in software terms) happen because of abnormal operations, and abnormal operations happen because of human error.

If it feels oxymoronic to prevent yourself from talking about “human error” in a safety culture where “human error” is the main cause of incidents, that’s because it is. It’s not surprising that a lot of people chafe at this particular iteration of blamelessness; honestly, it’s surprising more people don’t question it. Paraphrasing an engineer I spoke with recently at SREcon:

When we do blameless incident reviews, it’s like we can’t actually talk about what happened in the incident. On teams that don’t do blamelessness, you might get yelled at but at least you can actually talk about what you did.

If you want to correctly practice blamelessness, we need to understand the second reason it exists: because it’s the only way to understand your incidents.

This involves a key principle of Safety-II thinking: systems constantly run in a degraded mode, humans constantly adapt to changing circumstances, and accidents happen when normal operations collide with environmental factors that make failures possible. Put a little more simply: people generally don’t do things they think will cause accidents, so people’s actions are rarely sufficient to explain why incidents occurred.

Put even more simply: human error doesn’t exist.

An anecdote from Sidney Dekker’s Field Guide to Human Error Investigations lays this out well:

Someone called me on the phone, demanding to know how it was possible that train drivers ran red lights. Britain had just suffered one of its worst rail disasters—this time at Ladbroke Grove near Paddington station in London. A commuter train had run head-on into a high-speed intercity coming from the other direction. Many travelers were killed in the crash and ensuing fire. The investigation returned a verdict of “human error”. The driver of the commuter train had gone right underneath signal 109 just outside the station, and signal 109 had been red, or “unsafe”. How could he have missed it? A photograph published around the same time showed sensationally how another driver was reading a newspaper while driving his train.

The Ladbroke Grove verdict of “driver error” lost credibility very soon after it came to light that signal 109 was actually a cause célèbre among train drivers. Signal 109 and the entire cluttered rack on which it was suspended together with many other signals, were infamous. Many drivers had passed an unsafe signal 109 over the preceding years and the drivers’ union had been complaining about its lack of visibility. In trains like the one that crashed at Ladbroke Grove, automatic train braking systems (ATB) had not been installed because they had been considered too expensive. Train operators had grudgingly agreed to install a “lite” version of ATB, which in some sense relied as much on driver vigilance as the red light itself did.

If you actually speak to incident participants who “caused” incidents about their involvement, a story you will almost always hear is:

I was following a normal procedure for doing something [where the “normal” procedure is rarely the “official” procedure]
I did enough checks to satisfy my understanding of what could potentially go wrong with this action
I did the thing, and stuff started breaking

Based on that information, we want to identify a couple of critical details about the person’s actions:

What is missing from the official procedure such that operators are required to deviate from it?
What was insufficient about the operator’s understanding of the system that they weren’t able to prevent the incident?
What environmental factors made it possible for the incident to occur?

When you decompose the problem into its constituent parts, “human error” tends to disappear, and performance of individual actors in the incident becomes uninteresting. If you were to focus on “mistakes” made by the people involved, you’d miss all the parts of the incident that actually matter to the rest of your organization.

I think it’s important to call out that in a safety-II culture that practices blamelessness this way, there is no conflict between blamelessness and accountability. Rather than blamelessness allowing people to dodge accountability for “mistakes” (a thing that can genuinely happen in safety-I cultures) blamelessness becomes the only way a team can hold themselves accountable for incidents because it allows them to understand how they actually operate and how they can make durable changes to prevent future incidents.

If you want to move your organization towards a blameless safety-II culture, the most important thing you can do in incident reviews is to have people speak directly about their own involvement in the incident. The review facilitator (or the author of a written document) should focus the conversation on the mental model of the person involved in the incident and help everyone understand the actual process that they were following prior to the incident. I highly recommend the Howie Guide and the Etsy Debriefing Facilitation Guide as background for these discussions.

If you’ve made it this far in this post, you probably also owe it to yourself to read through the Field Guide to Human Error Investigations.

You should also be careful about practices that diffuse blame rather than facing it head on. Never use the passive voice to describe someone’s action in an incident¹ (“the change was deployed”), and if you feel like a matter of fact description an action makes someone look bad that’s almost always a sign that you should surround that description with more context about the process someone was following. Smart people disagree about if you should use the names of individual engineers in incident review documents, but even if you are going to avoid people’s names you should be careful to not also strip out human agency from your document.

In general, I’d also encourage people to engage more critically with this type of industry “best practice”. Software engineering culture isn’t very literate; we tend to distribute knowledge through conference talks, blog posts, and folk knowledge rather than papers and books, and that ultimately means that big ideas tend to quickly lose the context in which they were introduced. This post is a couple years in the making, and is the product of me asking questions about blamelessness and having trouble overcoming many of the objections that blameless-skeptics often raise. Understanding the safety research roots of the concept is what finally made me click, and it’s made me a much better incident analyst overall.

Fred’s post here (and the linked paper) is an interesting read if you want a counterpoint to this suggestion: https://ferd.ca/notes/paper-agentive-language-in-accident-investigations.html. Some day I will figure out a good synthesis of the two perspectives ↩︎

AWS App Runner is really cheap?

Sun, 06 Nov 2022 11:51:37 -0800

I was playing around with AWS App runner recently and learned two things that I hadn’t previously known about its pricing. The first is from the pricing page:

When your application is deployed, you pay for the memory provisioned in each container instance. Keeping your container instance’s memory provisioned when your application is idle ensures it can deliver consistently low millisecond latency

When your application is processing requests, you switch from provisioned container instances to active container instances that consume both memory and compute resources

The second, not mentioned on the pricing page, is that you don’t have to pay for an Elastic Load Balancer instance when using App Runner.

As someone who uses AWS Elastic Container Service a lot professionally but avoids it for personal stuff because of cost, those are both a pretty big deal:

Not having to pay for a load balancer saves you a flat $15 or so per month
You can’t scale an ECS service to zero instances without dealing with a cold start, so you’re typically going to be paying for a minimum of one active instance at all times

This felt like a good opportunity to try to quantify the difference, so I made a quick graph that shows how much you pay for each option as the level of traffic your service receives increases:

If you’re using a single provisioned instance with 1 vCPU and 2GB of memory, App Runner is basically always cheaper than ECS. Even if you were to scale your ECS service to zero instances when you’re not using it, app runner still comes out ahead at low levels of traffic because you’re not paying for an ALB.

In conclusion: I’m definitely going to start using App Runner for personal projects, although I’m a little more ambivalent about its advantages for a more professional operation. You lose a lot of control of load balancer configuration and task settings, and if you’re running more instances with more vCPUs and a higher baseline of activity the ECS price premium starts to go away.

With that said, App Runner definitely isn’t the cheapest option out there for cloud compute. Specifically for containers, Google Cloud Run is less money at least for low-traffic services, and it’s hard to compete with zero dollars for a t3.micro in EC2. But as a container person and an AWS person it’s probably the option I’m sticking with for the conceivable future.

10 Things I Learned From My First Incident Review (via jeli.io)

Tue, 16 Aug 2022 21:25:42 -0700

A post by me on the Jeli blog about my first incident review. I covered a pretty good range of subjects, from trust relationships between teammates and the value of mentorship, to classic resilience engineering subjects like above the line/below the line, to my experience with Jeli’s Howie incident analysis guide.

Read the full post here: https://www.jeli.io/blog/10-things-i-learned-from-my-first-incident-review

Building a Toy Container Vulnerability Scanner

Sun, 20 Mar 2022 13:30:42 -0700

After doing some work around container vulnerability scanning recently, I realized that I had a pretty incomplete mental model of how scanners like Snyk or AWS Inspector actually work. A couple questions came up for me, the most pressing being: why can’t I use AWS Inspector to scan the otel/opentelemetry-collector image we use as a base for deploying the OTel collector in prod?

The answer was pretty simple: because the main stage of the collector uses scratch as a base, and Inspector’s ECR integration doesn’t support scratch base images. But I also learned that I don’t know all that much about container scanning and decided it was worth investing some more time to improve that situation. Doing more research, I realized that vulnerability scanning is a surprisingly straightforward process and this was a good opportunity to get some hands on experience working directly with Docker images.

This post explains what I did to build a “toy” vulnerability scanner, and what I learned. I used Go but my hope is that the code is simple enough for non-Go programmers to follow, and I’ve tried to link to as many sources as possible to make it easier to follow the decisions made. The completed project is on GitHub at jemisonf/toy-scanner.

But first:

Some background

Understanding what actually constitutes a container image¹ is really important to understanding container scanning. If you’re already familiar with image manifests and layers, feel free to click here to go to the next section.

Container images, as specified in the Open Container Initiative image-spec document, are made up of “layers” of files, with a “manifest” document that describes what layers exist in the image.

You can think of individual layers as just a bundle of files compressed with tar and gzip. Layers correspond 1:1 with lines in your Dockerfile; the line

ADD package.json package-lock.json .

will create a single layer containing those two files at whichever location they exist at in your container’s filesystem.

When you run a container, Docker applies those layers one at a time to the container filesystem in the order that they were created. So by inspecting each layer in a image, you can see every file that will appear in the container when it’s started.

Manifests are just metadata that tells container clients what layers exist for a given image. You don’t need to know a ton about them for the purposes of this post, just that they can tell us how to find the layers for the image we’re scanning. You can see the full specification in the OCI spec.

It’s also important to mention that manifests are just JSON files, and layers are essentially just tarballs. There are software libraries I’ll talk about in the next section that make it substantially easier to interact with container images, but a running Docker daemon isn’t required and even the libraries are optional – it would be possible to implement this toy scanner in a shell script using curl and tar if you were into that kind of thing.

How do you scan a container for vulnerabilities?

For this section, I wanted to be careful to avoid reinventing the wheel. So I did some research on existing container scanners to see what information is available about their approach to scanning containers. Snyk, a leading container security product, is unfortunately not open source and I didn’t find readily available resources about how they implement their product. But Clair, developed by Red Hat subsidiary Quay, is open source, and has actually published an overview of their architecture and implementation here: https://quay.github.io/claircore/.

Clair’s architecture has two key components:

Indexers continuously scan for new image layers, and can pull in an image to identify which packages are installed in the image, and then generate a “report” that can be consumed by other Clair components
Matchers, which take reports produced by indexers and compare the packages in the report to public vulnerability databases to identify if a package in a report contains any vulnerabilities

A lot of engineering work in Clair and other projects goes into supporting continuous scanning of container registries and checking for a wide range of operating system and language dependencies. So I added some constraints to make my life a little easier:

The toy scanner should just be a CLI tool that scans a single image on-demand
The scanner will only support Alpine packages

I picked Alpine because my impression from the Clair source code was that Alpine is one of the simpler distributions to scan for installed packages. At the same time, learning to scan Alpine will teach you most of what you’d need to know about container scanning in general because most individual scanners tend to follow a similar set of patterns.

With those restrictions in place, our much-simplified scanner architecture looks like:

Scanner architecture

Without much further ado, let’s get to it!

Step 1: setup and image metadata

I used Go for building the components of the scanner so I could use Google’s go-containerregistry module, which makes working with remote container images extremely simple. Reading an image manifest just requires parsing the image tag with name.ParseReference and then downloading the image with remote.Image. The collected code looks like:

func main() {
	var image string

	flag.StringVar(&image, "image", "", "Image to scan")
	flag.Parse()

	ref, err := name.ParseReference(image)

	if err != nil {
		fmt.Printf("Error parsing image name: %s\n", err.Error())
		os.Exit(1)
	}

	docker_image, err := remote.Image(ref)

	if err != nil {
		fmt.Printf("Error fetching image: %s\n", err.Error())
		os.Exit(1)
	}

	manifest, err := docker_image.Manifest()

	if manifest != nil {
		// do something
	}
}

Having manifest and docker_image gives us all the information we need for the next section. manifest has a Layers field we can use to find the digest² for each layer of the image, which we can then use to get a Layer object with the LayerByDigest field on docker_image.

Step 2: identifying packages

Before we start: I would like to say that I am not an expert at Alpine, vulnerability databases, or Linux package management in general, and that I can vouch for the educational value of this section in regards to learning how vulnerability scanners work but I can’t promise there aren’t some errors as I start to get into specifics.

For this step, I had to do some research on actually scanning Alpine for vulnerabilities. I started at the end, with asking what the minimum information you’d need to actually match a package to a vulnerability was. Looking at Alpine SecDB, which is listed as Clair’s vulnerability database for Alpine, I found that the three most relevant pieces of data are: Alpine distro version, package name, and package version. Here’s how I encoded that information:

type AlpineReport struct {
	Packages []AlpinePackage
	Version  string
}

type AlpinePackage struct {
	Name    string
	Version string
}

The term “report” here is borrowed from Clair, where indexers produce reports that are consumed by vulnerability matchers.

The outline of what I called the AlpineScanner method looks like this:

func AlpineScanner(manifest v1.Manifest, image v1.Image) (*AlpineReport, error) {
	layers := []v1.Layer{}
	for _, layer_desc := range manifest.Layers {
		layer, err := image.LayerByDigest(layer_desc.Digest)

		if err != nil {
			return nil, fmt.Errorf("Error fetching layer %s: %w", layer_desc.Digest, err.Error())
		}
		layers = append(layers, layer)
	}

	packages := []AlpinePackage{}
	var version string
	for _, layer := range layers {
		// scan layer
	}

	return &AlpineReport{
		Packages: packages,
		Version:  version,
	}, nil
}

In short: collect the metadata for each layer based on its digest, then iterate over the layers one by one and collect the packages for each layer into a report containing the full list of packages in the image.

That leaves left two things to implement while scanning the layers: find the Alpine distro version, and find each package installed.

First, we have to figure out how to scan individual files in each layer. I had to do some research on this, but it turned out to be pretty straightforward. layer.Uncompressed() will return an io.ReadCloser representing a readable stream of the compressed tarball for the layer, and we can then pass that into tar.NewReader and get a tar.Reader object that we can use to iterate over individual files:

reader, _ := layer.Uncompressed()

tar_reader := tar.NewReader(reader)

header, err := tar_reader.Next()
for header != nil && err == nil {
  // process the file
  // ...
  // now advance to the next file
  header, err = tar_reader.Next()
}

Calling tar_reader.Next() here will return a header object for each subsequent file in the tarball, with we can use to get the name of the file as well as reading its contents.

One reason I picked Alpine is that the next steps are really simple. Reading through Clair’s implementation, there are two files we need to look for:

A file in /lib/apk/db/installed that has a list of every package installed on the system
A file in etc/os-release that contains the distribution version

When we find /lib/apk/db/installed, we need to do a little additional parsing. If you run docker run alpine cat /lib/apk/db/installed you can see a full example, but here’s a single package for reference:

C:Q1tSkotDvdkl639V+pj5uFYx3AGIQ=
P:libssl1.1
V:1.1.1n-r0
A:aarch64
S:207620
I:536576
T:SSL shared libraries
U:https://www.openssl.org/
L:OpenSSL
o:openssl
m:Timo Teras <timo.teras@iki.fi>
t:1647383879
c:455e966899a9358fc94f5bce633afe8a1942095c
D:so:libc.musl-aarch64.so.1 so:libcrypto.so.1.1
p:so:libssl.so.1.1=1.1
r:libressl
F:lib
R:libssl.so.1.1
a:0:0:755
Z:Q1jtZ7ec5Jx9TJ+IUk5mJnXnn+Gd4=
F:usr
F:usr/lib
R:libssl.so.1.1
a:0:0:777
Z:Q18j35pe3yp6HOgMih1wlGP1/mm2c=

Each one of these package blocks is separated by a double newline, and then each line within the block represents a different field. Because I couldn’t find documentation about the details of the format, my implementation borrows really heavily from Clair’s Alpine package scanner:

if header.Name == "lib/apk/db/installed" {
	contents, _ := io.ReadAll(tar_reader)

	entries := bytes.Split(contents, []byte("\n\n"))

	for _, entry := range entries {
		lines := bytes.Split(entry, []byte("\n"))

		var name, version string

		for _, line := range lines {
			if len(line) == 0 {
				continue
			}
			switch line[0] {
			case 'P':
				name = string(line[2:])
			case 'V':
				version = string(line[2:])
			}
		}

		if name != "" && version != "" {
			packages = append(packages, AlpinePackage{Name: name, Version: version})
		}
	}
}

I’m deliberately dropping a bunch of fields here from the package block to keep the implementation simpler. I suspect that means there are edge cases this code might not handle, but I’m not super concerned about that since my intent here is not to write an airtight vulnerability scanner.

Next, we need to find the Alpine version used by the container. This is straightforward enough to do just by checking /etc/os-release, a standard across Linux distributions for where the distribution version is stored. As an example:

$ docker run alpine cat /etc/os-release      
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.15.1
PRETTY_NAME="Alpine Linux v3.15"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

I don’t think that just relying on VERSION_ID here will necessarily cover every edge case, but like with the package versions that should be ok for the purposes of this tool. Here’s the implementation:

if header.Name == "etc/os-release" {
	contents, _ := io.ReadAll(tar_reader)

	lines := bytes.Split(contents, []byte("\n"))

	for _, line := range lines {
		if strings.Contains(string(line), "VERSION_ID") {
			fmt.Sscanf(string(line), "VERSION_ID=%s", &version)
		}
	}
}

With that done, we can print out the return from the scanner and get something like:

{Packages:[{Name:musl Version:1.2.2-r7} {Name:busybox Version:1.34.1-r4} {Name:alpine-baselayout Version:3.2.0-r18} {Name:alpine-keys Version:2.4-r1} {Name:ca-certificates-bundle Version:20211220-r0} {Name:libcrypto1.1 Version:1.1.1n-r0} {Name:libssl1.1 Version:1.1.1n-r0} {Name:libretls Version:3.3.4-r2} {Name:ssl_client Version:1.34.1-r4} {Name:zlib Version:1.2.11-r3} {Name:apk-tools Version:2.12.7-r3} {Name:scanelf Version:1.3.3-r0} {Name:musl-utils Version:1.2.2-r7} {Name:libc-utils Version:0.7.2-r3}] Version:v3.15}

Step 3: matching packages and vulnerabilities

Let’s start with some data structures. I relied pretty heavily on the Alpine SecDB data format to figure out how to structure the data in the program. You can see an example here.

Based on that, the return value we’re looking for would like something like:

type Vulnerability struct {
	CVEs        []string
	PackageName string
	Version     string
}

It’s also handy to have a partial data structure to unwrap the SecDB data into. I used three structs for that:

type SecDBReport struct {
	Packages []SecDBPackage
}

type SecDBPackage struct {
	Pkg SecDBPkg
}

type SecDBPkg struct {
	Name     string
	Secfixes map[string][]string
}

That just leaves some setup to do in the actual method, specifically:

Compute the major.minor version of the distro from the major.minor.patch version we already have
Use that to fetch the vulnerability document from SecDB
Unwrap that value into a SecDBReport object

With some error handling omitted, that looks like:

func AlpineMatcher(report AlpineReport) ([]Vulnerability, error) {
	vulnerable_packages := []Vulnerability{}

	majorMinorVersion := semver.MajorMinor("v" + report.Version)

	client := http.Client{}

	secDBURL, _ := url.Parse(fmt.Sprintf("https://secdb.alpinelinux.org/%s/main.json", majorMinorVersion))

	res, _ := client.Do(&http.Request{
		Method: "GET",
		URL:    secDBURL,
	})

	responseBytes, _ := io.ReadAll(res.Body)

	var contents SecDBReport

	json.Unmarshal(responseBytes, &contents)

	// check vulnerabilities here

	return vulnerable_packages, nil
}

To actually match packages to vulnerabilities, we can compare every installed package to every SecDB entry and see if the installed version matches any CVEs:

for _, installed_package := range report.Packages {
	for _, secdb_package := range contents.Packages {
		if secdb_package.Pkg.Name == installed_package.Name {
			if CVEs, ok := secdb_package.Pkg.Secfixes[installed_package.Version]; ok {
				vulnerable_packages = append(vulnerable_packages, Vulnerability{
					PackageName: installed_package.Name,
					CVEs:        CVEs,
				})
			}
		}
	}
}

Testing it out with go run . -image alpine you get . . . no vulnerabilities found. Which is actually expected – Alpine is a small distribution with good security hygiene, so the latest tag typically won’t have any issues to find.

Can we test it with a known-bad image to make sure we aren’t getting false negatives? I had a surprising (in a good way) amount of difficulty doing this – Alpine doesn’t have any of the vulnerable packages I looked for in its package index, and I’m not sure where to look for an old image that might contain the vulnerable packages. So I’ll, uh, leave further testing as an exercise to the reader.

Conclusion

I had a couple of fairly important takeaways about container vulnerability scanners while working on this that I think are worth sharing, because I suspect they’re related to common misconceptions about container scanning:

Container scanners don’t require any kind of special manifest to identify installed packages. They just read the files in the image; there’s no extra magic happening to make scanning possible.
Container scanners do need packages to be installed in a standardized way. As we saw when building the Alpine scanner, anything not installed through apk would not have been caught by our scanner.

With a normal scanner, you can assume it will also catch anything installed with standard language tooling like pip or npm as well, but you should not assume that something downloaded directly can be scanned.
Container scanners don’t operate on “your” code at all. They won’t catch logic errors or previously-unknown vulnerabilities in your implementation.

Running static analysis of your code would be possible in theory but it isn’t a standard practice for container scanners, and it is not possible to do anything that requires dynamically executing code.
Container scanners are based on reported vulnerabilities so the value of a scanner is going to depend partly on how good a language or distribution is about identifying and publishing vulnerabilities.
Container scanners are for trusted images. Bypassing a container scanner to ensure that an image contains an insecure binary would be trivial, and attackers have various tools to compress layers together to make it even more difficult to detect that a layer contains malicious code.

That means that your scans are useful if you can trust that the images you use are installing packages in a standardized way and otherwise not trying to decieve you about their contents. The scanner protects you from yourself, not from other people.

Considering all those points together, it really helps emphasize that container scanning has an important but limited role to play in making sure that trusted container images are using up-to-date dependencies without known security vulnerabilities, but that scanning doesn’t replace other security best practices. I also learned a lot about interacting directly with container images and I think this is a neat example of how much you can do with an image without running a Docker daemon at all.

Whether or not we’re talking about a “Docker” image is an interesting question. Early in the days when containers were becoming a mainstream technology, Docker containers and “containers” were essentially synonymous. But now, Docker is no longer the only way to run containers or even the most popular; containerd is the container runtime that underlies a lot of managed container services, the Kubernetes project is removing Docker entirely, and you can find a number of container building services like buildah or ko that can build OCI-compliant container images without a Docker daemon. I’m generally going to try to use the term “container” instead of “Docker container” in this post, and you can assume everything mentioned here refers to OCI containers and is not Docker exclusive. ↩︎
The the digest of a layer is a hashed value that can be used to uniquely identify the layer, and that is generated based on the contents of the layer. sha256 is the most common method I have seen for generating digests. See the OCI descriptor specification for more detail. ↩︎

Faster, Smaller, and More Secure Docker Builds

Sun, 26 Sep 2021 00:00:00 +0000

For this post, I’d like to highlight some Docker features and techniques that are useful but often left out of entry level tutorials, or that you may have missed if you learned Docker more than a couple years ago and haven’t closely followed the latest releases. I’m using Docker Desktop 20.10.8 in this post but as long as you’re within a couple major versions of that I wouldn’t expect any compatibility issues.

I created an example application that we’re going to use to test out the steps; you can find it on GitHub at jemisonf/docker_builds. For the final version, check out the master branch of the repo, or start with the start-here tag to follow along with the post.

Note: this post uses a Go app as an example, but should be applicable to non-Go apps as well. If you’re familiar with Docker but not Go, you can just skip over the Go-specific parts.

Our example service is a Go web server. It has a single route, /serialize, that accepts a Kubernetes resource in the body, validates it, and returns it serialized into protobuf. To try it out:

Clone the repository and check out the first commit with

git clone git@github.com:jemisonf/docker_builds.git 
cd docker_builds 
git checkout start-here

Run go run . in one terminal
In a second, run curl localhost:3333/serialize --data-binary @./examples/deployment.yaml --output output.proto

That last command will send the yaml file located in examples/deployment.yaml to the server and then save the output in output.proto. You can check that it worked with cat output.proto; the result should be a more or less meaningless jumble of binary text.

Note: the example service is a little contrived, but it simulates a “real” application by pulling in a number of dependencies via the k8s.io/apimachinery and k8s.io/client-go packages

If we want to package this service into a Docker image, the “naive” approach would look something like this:

FROM golang:1.16

WORKDIR /app

COPY . ./

RUN go build .

CMD ./docker_builds

We can build and run our docker file with

docker build . -t docker_builds
docker run -p 3333:3333 docker_builds

Our build output should look something like this, if you’re using the newest version of Docker:

 => [internal] load build definition from Dockerfile                                                                                                                                                    0.0s
 => => transferring dockerfile: 113B                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                                         0.0s
 => [internal] load metadata for docker.io/library/golang:1.16                                                                                                                                          1.4s
 => [auth] library/golang:pull token for registry-1.docker.io                                                                                                                                           0.0s
 => [internal] load build context                                                                                                                                                                       0.1s
 => => transferring context: 11.82kB                                                                                                                                                                    0.0s
 => [1/4] FROM docker.io/library/golang:1.16@sha256:0056b049979bfcf13ac2ede60b810349396fab1d510cb60701503dccd01f9153                                                                                    0.0s
 => => resolve docker.io/library/golang:1.16@sha256:0056b049979bfcf13ac2ede60b810349396fab1d510cb60701503dccd01f9153                                                                                    0.0s
 => CACHED [2/4] WORKDIR /app                                                                                                                                                                           0.0s
 => [3/4] COPY . ./                                                                                                                                                                                     0.2s
 => [4/4] RUN go build .                                                                                                                                                                               13.2s
 => exporting to image                                                                                                                                                                                  1.6s 
 => => exporting layers                                                                                                                                                                                 1.6s 
 => => writing image sha256:e7e90943fba98c38660bc8361e8c520e07b71fb69d22b6b0a3be2b9189c5d941

If you scroll to the right, notice that the time to complete each step in our build is listed, in seconds. We spend the most time on RUN go build – no surprise there.

Let’s make a small code change – for example, we can make the port configurable in main.go:

	port := ":3333"
   if len(os.Args) > 1 {
   	port = os.Args[1]
   }

   err := server.StartServer(port)

If we run docker build . again, we should get about the same build time. This isn’t a huge problem for our relatively small app, but as you may have noticed a lot of that time is just spent downloading dependencies. This is true for a lot of languages – in general, the build for a service in almost any language is going to look like:

Download depencies.

Requires: package management files, like go.mod and go.sum or package.json or requirements.txt.
In some languages, build or bundle the application code into a binary (like in Go) or a single script bundle (like in Node).

Requires: application code
Copy the application code into the Docker file.

Requires: the artifact from step 2.

When you build a Docker image, each line in your Dockerfile is broken out into a “layer”. Docker understands which files (either inside in the Docker image or in your file system) are required to create the layer, and will use a cached version of the layer if no changes are detected in those files from the previous build. You can read more about layers here.

We can take advantage of this behavior in order to cache our dependencies and save that time in each repeated build. This is not possible in our current approach because we’re copying all of the code in our repo at once, and then running go build, which combines downloading dependencies and building application code into one command.

To make caching our dependencies possible, let’s first copy our dependency files (go.mod and go.sum), then download our dependencies with go mod download, then build our application code with go build:

FROM golang:1.16

WORKDIR /app

COPY go.mod go.sum .

RUN go mod download

COPY . ./

RUN go build .
CMD ./docker_builds

This pattern is not specific to Go. In a node app, it would look like:

FROM node:16

WORKDIR /app

COPY package.json package-lock.json .

RUN npm install

COPY . ./

# build your app ...

This should work for most apps, regardless of language: copy your dependency files, download your dependencies, then copy your application files and build your application.

An even more general way to put this is: place the commands in your Dockerfile in order from least likely to change to most likely to change. For example, if you had to install a system dependency first via a package manager, you’d do that before copying any of your dependency files or application code.

If we try running docker image ls, we’ll see another potential problem:

REPOSITORY   TAG       IMAGE ID       CREATED             SIZE
<none>       <none>    e7e90943fba9   15 minutes ago      1.05GB

Our image is pretty large; Go compiles down to static files, so why do we need a whole gigabyte for our image? This presents a couple problems:

It uses more space on your hard drive
Large docker images are slower to download, which can slow down your deploy time
Larger docker images present a larger “attack surface”. If someone gains access to the running container there are more scripts they can run to potentially gain access to the underlying host running the container

Fortunately, we have a relatively simple solution that can address all of these problems at once: “distroless images”, a project maintained by Google, give you a minimal base image with every unnecessary tool stripped out.

This is not the only option for minimal docker builds; you can use a small Linux distribution like Alpine Linux as your base image, or start from the empty scratch Docker image. I would recommend this talk from Matthew Moore as an introduction to the limitations of those approaches and why distroless is the most flexible approach out of all the available options.

We can introduce a distroless base image for our app using a tool called multi-stage builds, which lets us specify different “stages” for our build that each have their own base image. That way we can use a larger base image to build our application code, and then copy the resulting binary into the distroless base image:

FROM golang:1.16 AS builder

WORKDIR /app

COPY go.mod go.sum .

RUN go mod download

COPY . ./

RUN go build . 

FROM gcr.io/distroless/base

WORKDIR /app

COPY --from=builder /app/docker_builds .

CMD ["/app/docker_builds"]

Now if we run docker image ls, we should see our latest image, with a much smaller size of 46 MB:

REPOSITORY   TAG       IMAGE ID       CREATED        SIZE
<none>       <none>    40ebe2a35240   2 hours ago    46.3MB

For examples of how this works with other languages and runtimes, check out the distroless GitHub repo linked above.

Now we have a secure, minimal image that’s quick to build locally. We can set up a GitHub action to build our image using the example from Docker’s Build and Push Docker Images Action. To get that set up:

Set up a Docker Hub account and create an access token
Create a .github/workflows/build.yaml file containing the sample
Change all references to user/app to ${your Docker Hub username}/docker_builds
Add your Docker Hub username to the DOCKERHUB_USERNAME secret in your GitHub repo and and your Docker Hub access token to the DOCKERHUB_TOKEN secret.

I’d also recommend changing the default tag to push two tags, one for the commit SHA and one latest tag. For my repo that looks like:

-  name: Build and push
   id: docker_build
   uses: docker/build-push-action@v2
   with:
   with:
       push: true
       # if forking, replace `jemisonf` with your docker hub username
       tags: jemisonf/docker_builds:${{ github.sha }},jemisonf/docker_builds:latest

Pushing only latest can be OK for sample projects, but it’s generally a bad idea for production applications. Using the SHA for the commit you’re building means you can precisely associate Docker builds with each release of your application.

If you push this to your GitHub repo, let it build, and then re-run the action, you may notice that the build is no longer taking advantage of the docker cache. For this app, the impact isn’t huge because our build should only take around a minute. For larger apps with more dependencies, this may become a more significant problem.

We can solve this problem using Docker external caches. With external caches, we can pull the cached layers of a previous image from the Docker registry during our Docker build, letting us re-use caches in between builds. This is easy to configure in GitHub actions using the cache-from and cache-to options in our build and push step:

with:
    push: true
    # if forking, replace `jemisonf` with your docker hub username
    tags: jemisonf/docker_builds:${{ github.sha }},jemisonf/docker_builds:latest
    cache-from: type=registry,ref=jemisonf/docker_builds:latest
    cache-to: type=inline

See the cache docs for the Action for more details.

If we push this change, let the build complete, and then trigger a rebuild for the same commit we should see the Docker build complete almost instantly because it’s re-using the previous build’s cache. Future builds should be able to take advantage of the caching behavior we saw locally to minimize the amount of steps in the build.

Note: because this step requires downloading layers from a remote repository, it’s possible it won’t save time in every case if it’s slower to download those layers than to download your application dependencies. I would expect this to be worthwhile for the average project, but it’s worth evaluating on an individual basis if your project actually benefits from it before using it.

Wrap-up

A short summary of the suggestions in this post:

You can make your Docker builds more cache-friendly by copying your dependency files and installing dependencies before copying your application files and building your application.
In general, Docker can cache your build more easily if you run your Dockerfile commands in order of least likely to change to most likely to change
You can make your output builds smaller and more secure using multi-stage builds and a distroless base image. Other alternatives are using Alpine Linux or scratch as your base image
You can speed up CI builds by sharing your Docker build cache between builds using external cache sources, which you can enable using the cache settings in the Docker build and push GitHub Action

That’s it! The steps in this post should be applicable to most projects using Docker for builds, and should help ensure that your Docker images are small, fast to build, and secure. I’ve tried to link documentation and relevent resources wherever possible, and I’d strongly recommend reading those as well; this post is intended as an introduction to each of concepts mentioned and I’ll defer to linked resources to provide the depth that’s missing here.

How New Relic Uses Argo Workflows

Sun, 03 Jan 2021 00:00:00 +0000

Here’s the recording, hosted on Youtube. Our presentation starts at 32:11.

If the embed doesn’t work, here’s a direct link: https://www.youtube.com/watch?v=dxX_Xnp2sX4&t=32m11s

Touches on CI/CD, infrastructure automation, and how to adopt a technology like Argo Workflows at a large organization where it is used across many different teams.