[Problem RFC] Worker Crashes with "No Space Left on Device" Despite a Healthy Node Disk #823

Hello! I am submitting this RFC as part of the UIUC++ evaluation task for Agentic Site Reliability Engineering (Tianyin Xu et al., UIUC). I went through the existing problems and found a gap I think would make a great addition to SREGym.

Motivation

I want to propose a failure that I think is one of the most misleading things an SRE can run into in Kubernetes: a pod crash-loops with "No space left on device" while the node disk is nearly empty with plenty of free space. The real cause has nothing to do with the disk. Every container gets a /dev/shm shared-memory filesystem that the container runtime silently caps at 64 MiB, and when an application writes more than that, the kernel returns ENOSPC ("No space left on device"). The disk error is a red herring.

Here is the story of how an agent (or a human SRE) would experience this failure:

The pod goes into CrashLoopBackOff. The logs say:

dd: error writing '/dev/shm/scratch': No space left on device

The natural next step is to check the disk. But this is what comes back:

Resource       Used              Free            Shown in kubectl?
-----------    ---------------   -------------   -----------------
Node disk      mostly empty      hundreds of GB  Yes (describe node)
/dev/shm       64 MiB  (100%)    0 bytes         No  <-- silent limit

The disk is fine. There is no DiskPressure condition. No eviction. Nothing wrong anywhere. And yet the pod keeps crashing. It starts to feel like something is deeply broken that is not visible anywhere.

I find this interesting because the error message is not wrong. There really is no space left. It is just on a completely different resource that Kubernetes never surfaces.

I found similar type of failure well documented in production across many real systems:

• PostgreSQL inside Kubernetes crashes parallel workers with: could not resize shared memory segment "/PostgreSQL.xxxx": No space left on device, even when the node disk is completely healthy. (https://www.osso.nl/blog/2021/postgresql-inside-kubernetes-no-space-left-on-device/)
• PyTorch / ML DataLoaders share tensor batches between worker processes through /dev/shm. Workers die with bus error or ENOSPC ("No space left on device") when the batch size exceeds 64 MiB. (https://www.sobyte.net/post/2022-04/k8s-pod-shared-memory/)
• Chromium / Puppeteer uses /dev/shm for renderer IPC and crashes so often in containers that the --disable-dev-shm-usage flag exists just to work around this limit. (https://peter.sh/experiments/chromium-command-line-switches/#disable-dev-shm-usage)
• OpenCV / FFmpeg media pipelines use /dev/shm as scratch during decode and resize operations, and hit ENOSPC ("No space left on device") on large frames. (https://docs.docker.com/engine/containers/run/#runtime-constraints-on-resources)

The 64 MiB default traces back to Kubernetes commit cbf1cb81. It is documented, but only if someone already knows to look for it.

Proposal

I am thinking of adding a dev_shm_exhaustion_hotel_reservation problem to the Hotel Reservation benchmark. Hotel images need resizing, and a photo-processing worker (media-processor) writing scratch data to /dev/shm is a completely believable component in this app. I am using a generic name so the fault is not leaked to the agent.

Fault injection: Deploy a worker whose container runs:

dd if=/dev/zero of=/dev/shm/scratch bs=1M count=128 && tail -f /dev/null

The pod template omits a memory-backed emptyDir at /dev/shm, so the container falls back to the 64 MiB default. Writing 128 MiB fails at exactly 64 MiB with ENOSPC ("No space left on device") and the deployment enters CrashLoopBackOff.

The intended fix: Mount an emptyDir with medium: Memory (e.g. sizeLimit: 256Mi) at /dev/shm in the worker's pod template.

Repro

I verified this on my kind cluster (k8s v1.32, 4 nodes) using plain kubectl in a throwaway namespace:

Why Agents Struggle

When I thought about what an agent would do here, the sequence is pretty predictable:

1. Agent sees CrashLoopBackOff and pulls the logs.
2. Logs say No space left on device. Agent concludes: disk is full.
3. Agent checks node conditions and disk. Everything looks healthy. Plenty of free space. No DiskPressure.
4. Agent is stuck. The error clearly says no space, but the disk is fine.

The real answer is that there are two completely separate "out of space" resources: the node filesystem (huge, nearly empty) and the /dev/shm tmpfs (tiny, always full). Kubernetes only surfaces the first one. The second one is invisible until someone looks inside the container with df -h /dev/shm or checks the pod spec for a missing emptyDir volume.

Would love to hear any thoughts before moving forward with the implementation. 😊

Sources

[1] OSSO — "PostgreSQL inside Kubernetes / no space left on device"
https://www.osso.nl/blog/2021/postgresql-inside-kubernetes-no-space-left-on-device/

[2] SoByte — "Setting up the shared memory of a Kubernetes Pod"
https://www.sobyte.net/post/2022-04/k8s-pod-shared-memory/

[3] Chromium --disable-dev-shm-usage flag
https://peter.sh/experiments/chromium-command-line-switches/#disable-dev-shm-usage

[4] Docker documentation — Runtime options: --shm-size
https://docs.docker.com/engine/containers/run/#runtime-constraints-on-resources

[5] TimescaleDB issue #5686 — "could not resize shared memory segment"
timescale/timescaledb#5686