ghostmem

"Monitoring looks fine — so why did we get OOM-killed?"

ghostmem is a single-node diagnostic TUI that hunts ghost memory: /dev/shm tmpfs (and memfd) segments that no process RSS accounts for, yet the cgroup OOM killer counts in full. It traces segment lifecycle with eBPF tracepoints and optional uprobes — no code changes, no restarts — and shows per-segment ownership, orphan status, alloc stacks, CUDA pinned/device axes, and your distance to OOM in one screen.

Live view: a SIGKILLed DataLoader worker (pid 2776018) left two 64M torch_* segments behind — flagged ✗ seconds after the kill — while /dev/shm scan seeding surfaces pre-existing segments (lttng-ust-*, owner unknown) and the event log streams every shm alloc/free in the watched mount namespace, Chrome included.

Workloads like PyTorch DataLoader (num_workers > 0 with the file_system sharing strategy) create named shared-memory segments in /dev/shm (torch_*, pymp-*). When a worker dies abnormally, those segments are never unlinked and pile up as orphans:

• tmpfs pages appear in no process's RSS → RSS dashboards stay flat
• the cgroup OOM killer judges by memory.current (which includes shmem) → the working set climbs invisibly until SIGKILL (exit 137)
• nvidia-smi, nvtop, and RSS dashboards can't see this blind spot

ghostmem answers: how far to OOM, what share of it is ghost memory, and who created it and never freed it.

┌─ eBPF (kernel, C + CO-RE) ───────────────────────────────┐
│ tp:openat  absolute /dev/shm + dirfd-relative via shm_dir │
│ tp:memfd_create            synthetic memfd:<label>        │
│ tp:ftruncate / unlink(at)  success-paired size + free     │
│ tp:close                  memfd lifecycle                 │
│ tp:sched_process_exit      orphan detection               │
│ uprobe:shm_open, cuMem*    stacks + CUDA pin/device       │
│ maps: fd_to_name, shm_dirfds, stack_traces, ringbuf       │
└──────────────────┬───────────────────────────────────────┘
                   │ libbpf-rs skeleton (CO-RE, vmlinux.h)
┌──────────────────┴───────────────────────────────────────┐
│ collector: target mnt-ns/cgroup filter → bounded event queue │
│ poller   : statfs, readdir, cgroup; --target-ns via       │
│            /proc/<pid>/root + that pid's cgroup           │
├───────────────────────────────────────────────────────────┤
│ state: segments · rates · CUDA · footprints · stacks      │
├───────────────────────────────────────────────────────────┤
│ ratatui TUI  │  --json / --record  │  ghostmem replay     │
└───────────────────────────────────────────────────────────┘

The eBPF side and the 1 s poller form a hybrid: tracepoints give real-time ownership (pid/comm) while the directory scan seeds segments created before startup, corrects sizes, and reconciles events eBPF can miss (cwd-relative opens without a tracked dirfd, truncated path reads — failures are counted and shown in the footer). By default both sides are restricted to ghostmem's current mount namespace, so event ownership and the /dev/shm directory view always describe the same scope.

• Linux ≥ 5.8 (BPF ring buffer) with BTF (/sys/kernel/btf/vmlinux)
• installation: the default system-wide install location is /usr/local/bin, so installation requires the superuser (root; normally via sudo)
• runtime: root, or CAP_BPF + CAP_PERFMON and read access to /sys/kernel/tracing — that directory is commonly 700, so programs can load fine with just those two caps and then fail to attach with Permission denied. Either run as root or add CAP_DAC_READ_SEARCH:

  $ sudo setcap cap_bpf,cap_perfmon,cap_dac_read_search+ep target/release/ghostmem

• build: Rust toolchain, clang, bpftool, libbpf-dev, libelf-dev
• prebuilt archive runtime: glibc 2.35 or newer, libelf, zlib, libzstd

Tagged releases include a prebuilt x86_64-unknown-linux-gnu archive and SHA-256 checksum. Other Linux architectures can build directly from source.

Install the latest release system-wide to /usr/local/bin (requires sudo):

$ curl --proto '=https' --tlsv1.2 -fsSL \
    https://raw.githubusercontent.com/ziwon/ghostmem/main/install.sh | sudo sh

The installer verifies the release checksum before installing. ghostmem itself also needs root privileges at runtime unless the required Linux capabilities are granted. To inspect the installer before running it:

$ curl --proto '=https' --tlsv1.2 -fsSLO \
    https://raw.githubusercontent.com/ziwon/ghostmem/main/install.sh
$ less install.sh
$ sudo sh install.sh

An explicit user-writable destination remains available for non-system installs, but you must invoke the binary with its full path under sudo (or configure capabilities):

$ GHOSTMEM_INSTALL_DIR="$HOME/.local/bin" sh install.sh
$ sudo "$HOME/.local/bin/ghostmem" --preset pytorch

Or build directly from source:

$ just build            # cargo build --release (generates vmlinux.h on demand)
$ just run              # sudo target/release/ghostmem
$ just test             # unit tests (no root needed)

Typical hunts:

# watch PyTorch segments from any python process
$ sudo ghostmem --preset pytorch --comm python

# focus on one training job
$ sudo ghostmem --pid 31337

# headless NDJSON for pipelines / sidecars
$ sudo ghostmem --json | jq 'select(.type=="snapshot") | .orphans_bytes'

# alert demo: page when orphans exceed 1 GiB
$ sudo ghostmem --json | ./scripts/orphan-alert.sh

# record for post-mortem, then replay without root/BPF
$ sudo ghostmem --record /tmp/gm.ndjson --json --preset pytorch
$ ghostmem replay /tmp/gm.ndjson --speed 2

# stacks + CUDA axes (libcuda required for --cuda)
$ sudo ghostmem --stacks --cuda --python-stacks

# target a container's /dev/shm by host pid (mount namespace filter)
$ sudo ghostmem --target-ns 12345

# optionally require the exact target leaf cgroup too
$ sudo ghostmem --target-ns 12345 --target-cgroup-filter exact

# de-noise a long-running host: only show orphaned segments
$ sudo ghostmem --orphans-only

• memory — cgroup gauge selects the constrained ancestor with the least OOM headroom and separately shows memory.high throttling distance and PSI when available. memory.events, swap, and shmem fields are included in recordings. The /dev/shm gauge also tracks inode pressure.
• rates — 60 s alloc/s and free/s sparklines, net Δ, and a ⚠ LEAK badge when net growth exceeds --leak-threshold
• unified footprint (u) — per-PID device │ pinned │ shm (orphan) │ rss when CUDA/shm data is present
• live segments — name, owner pid/comm, allocated/logical bytes, age, ALIVE (✓/✗/?), source (bpf/scan/mfd), and namespace/dev/inode identity in NDJSON; orphans are red. o (or --orphans-only) hides everything else.
• alloc stack (S, needs --stacks) — symbolized user stack for the selected row (blazesym; optional /tmp/perf-<pid>.map with --python-stacks)
• events — alloc/free log plus warnings like PID 31337 (python3) exited leaving 14 segment(s) (2.3G)

k considers only named /dev/shm segments whose creator exited and whose age exceeds --cleanup-min-age (60 s default). Before either a dry-run validation or real unlink, ghostmem requires a reconciled dev/inode identity, rejects non-single-component names and symlinks, scans /proc/*/fd and /proc/*/maps for the same inode, and rechecks the inode immediately before unlinkat on a pre-opened /dev/shm dirfd. An incomplete /proc scan, a partial/failed directory scan, a remaining visible reference, or a lost target refuses real cleanup. memfd rows are never unlinked this way.

Real cleanup requires complete visibility into /proc/*/fd and /proc/*/maps. hidepid, restrictive Yama/LSM policy, container isolation, or insufficient privileges can therefore refuse cleanup even when a segment appears unreferenced. The event log includes the exact /proc/<pid>/... path and OS error that caused the fail-closed refusal.

Successful removal is reported as unlinked, not reclaimed: mappings and open descriptors can keep pages alive after the directory entry disappears. The next successful poll reports the observed /dev/shm and cgroup deltas. Linux has no atomic “unlink only if globally unreferenced” primitive, so concurrent opens remain an unavoidable race; audit records deliberately say “no visible references” rather than “provably orphaned.”

• --pid <PID> / --comm <prefix> — enforced inside BPF, so filtered processes cost almost nothing
• --target-ns <PID> — scan that PID's /dev/shm and filter BPF events by its mount namespace; shared-IPC sidecars in other cgroups remain visible
• --target-cgroup-filter exact — additionally require the target's exact cgroup v2 ID; opt-in because shared IPC and nested scopes can legitimately use different leaf cgroups
• --preset pytorch (torch_*, pymp-*), --preset python-sharedmem (psm_*, wnsm_*), --preset ray (plasma*, ray*)
• --name-glob 'torch_*' — custom globs (repeatable, * and ?)

These narrow what's traced. Every --pid must be in the watched mount namespace; ghostmem rejects a cross-namespace PID filter with guidance to use --target-ns instead of silently returning an incomplete view. The effective target_filter is shown in the TUI footer, attach log, and NDJSON. --orphans-only (or the o key) is a separate, cheap display filter applied after the fact: aggregate counts (segments_total, orphans_bytes, ...) always reflect the full table; only the live TUI table and live NDJSON segments array are trimmed. Recordings and exit summaries remain complete post-mortem records.

Without the exact filter, the cgroup memory gauge follows the target if it moves to another leaf cgroup. With the exact filter, that move is reported separately as TARGET CGROUP CHANGED; only PID reuse, exit, or a mount namespace change is reported as TARGET LOST.

Event attribution is scoped by mount namespace. Processes in another mount namespace that share the same underlying /dev/shm mount may therefore appear as scan-only rows with an unknown creator.

Every record carries type (snapshot / event / summary) and schema_version (currently 4). Replay accepts older compatible records and rejects a future schema instead of silently applying defaults. One object per line: snapshot (each poll interval), event (each lifecycle event), and a final summary. Key snapshot fields include shm, hierarchical cgroup data (limit/high/events/swap/PSI), rates, target_filter, logical and allocated segment bytes, estimated attribution gap, and the full segments array with namespace/dev/inode identity. confidence.scan_status is complete, partial, or failed; partial scans retain and reconcile readable entries without authoritatively deleting unseen rows, while shm_usage_error preserves a separate statvfs failure. scan_complete, event_stream_complete, and reconciled separate inventory confidence from event-attribution confidence; BPF read failures make the event stream uncertain. Replay restores snapshots authoritatively and does not query the replay host's /proc for historical rows.

# terminal 1 — note: PyTorch renames DataLoader workers to `pt_data_worker`
# and its cleanup daemon to `torch_shm_manager`; either skip --comm entirely
# or list all involved comms:
$ sudo ghostmem --preset pytorch \
      --comm python --comm pt_data_worker --comm torch_shm_manag

# terminal 2 — kills a DataLoader worker with SIGKILL mid-epoch
$ just repro          # needs torch; or `just churn` for a torch-free repro

Expected: the dead worker's torch_* segments remain in the table with ALIVE=✗, the event log shows the exit warning, and the cgroup gauge keeps counting bytes no RSS dashboard sees.

The parallel alert pipeline (ghostmem --json | ./scripts/orphan-alert.sh) fired within one second of the kill and kept firing while the orphans persisted — then escalated when the main process died too, leaving all remaining segments behind:

2026-07-20T10:16:14+09:00 ALERT orphaned shm 128MiB (2 segment(s)) exceeds 32MiB — top: torch_3698551_817289923_2
2026-07-20T10:16:15+09:00 ALERT orphaned shm 128MiB (2 segment(s)) exceeds 32MiB — top: torch_3698551_817289923_2
...
2026-07-20T10:16:54+09:00 ALERT orphaned shm 512MiB (8 segment(s)) exceeds 32MiB — top: torch_3698601_3166856870_2

• cwd-relative opens of /dev/shm files without a prior directory openat on /dev/shm still lack in-kernel ownership (scan seeds them as ?)
• /proc reference validation is fail-closed but cannot make reference checking and unlink atomic; cleanup is deliberately presented as a guarded operational action, not a proof of immediate page reclamation
• memfd lifecycle follows the creator's directly observed file descriptor; descriptors duplicated, inherited across fork, or transferred with SCM_RIGHTS are not reference-counted yet
• CUDA uprobes depend on finding libcuda.so symbols (cuMemAlloc_v2, cuMemHostAlloc, …); driver-only or heavily stripped installs may attach partially
• --python-stacks only annotates via /tmp/perf-<pid>.map (no full CPython frame walk yet)
• Roadmap (v0.4): Prometheus exporter, k8s DaemonSet recipe, cargo-dist packaging, PromQL alert examples, tutorial write-up

Licensed under either of the following, at your option:

• Apache License, Version 2.0
• MIT License