A Kubernetes sidecar that monitors memory usage and captures Go heap dumps before OOM kills occur.

┌─────────────────────────────────────┐
│ Pod                                 │
│  ┌──────────┐    ┌──────────────┐   │
│  │  Go App  │    │ OOM Watcher  │   │
│  │  :5999   │◄───│  (sidecar)   │   │
│  │  pprof   │    │              │   │
│  └──────────┘    └──────┬───────┘   │
└─────────────────────────┼───────────┘
                          │ HTTP POST
                          ▼
               ┌────────────────────┐
               │    Dump Server     │
               │   (centralized)    │
               └────────────────────┘

1. OOM Watcher runs as a sidecar, monitoring container memory via cgroups
2. When memory exceeds the threshold, it fetches a heap profile from your app's pprof endpoint
3. The heap dump is pushed to the centralized Dump Server via HTTP
4. Access dumps through the Dump Server's web UI

OOM Watcher includes intelligent adaptive dumping for rapid memory growth scenarios (enabled by default):

Normal Mode:

• Check memory every 5 seconds (configurable via POLL_INTERVAL)
• Capture dump when memory ≥ 90% (configurable via MEMORY_THRESHOLD)
• Wait 60 seconds between dumps (configurable via COOLDOWN_PERIOD)

Rapid Growth Mode (automatic):

• Triggered when memory is above threshold AND growing ≥ 5% per second (configurable via RAPID_GROWTH_THRESHOLD)
• Reduces cooldown to 10 seconds (configurable via RAPID_DUMP_INTERVAL)
• Captures more frequent dumps to track rapidly escalating memory issues
• Automatically returns to normal mode when growth subsides

Example scenario:

Time  Memory  Growth    Action
0s    85%     -         Normal monitoring
5s    92%     +1.4%/s   First dump captured (threshold exceeded)
10s   93%     +0.2%/s   Cooldown active (normal: 60s)
15s   98%     +6.0%/s   🚨 Rapid growth detected! Enters rapid mode
20s   99%     +0.2%/s   Second dump captured (rapid: 10s cooldown)
25s   99.5%   +0.1%/s   Growth subsided, back to normal mode
30s   OOM KILL          Multiple dumps captured for analysis

This ensures you capture sufficient data during critical memory spikes while avoiding dump spam during stable periods.

• Automatic Heap Dumps: Captures Go heap profiles when memory usage exceeds threshold
• Rapid Growth Detection: Automatically increases dump frequency when memory spikes rapidly (enabled by default)
• Pod Restart Tracking: Automatically tracks pod restart counts to identify the last dump before an OOM kill
• Centralized Storage: All dumps stored in one place with SQLite metadata and PVC storage
• Web UI: Browse, filter, search, and group dumps by namespace, deployment, or pod
• Interactive pprof: View dumps directly in browser with flamegraphs and call graphs
• Heap Dump Diff View: Select any 2 dumps to compare and identify memory growth patterns (shows allocation deltas, growing functions, and leak sources)
• Automatic Leak Detection: AI-powered analysis identifies leak sources with confidence scoring, categorizes leak types (goroutine leaks, connection leaks, cache growth, etc.), and provides actionable suggestions
• Prometheus Metrics: Full observability integration with /metrics endpoint for monitoring and alerting
• Auto-Pruner: Configurable background pruning with retention policies (count/age/size/hybrid strategies)
• Cleanup Automation: Automatic removal of dumps for deleted pods
• Data Management: Manual prune, auto-pruner config, cleanup config, and flush via Web UI or API
• Sortable Columns: Click any column header to sort dumps by time, memory, pod, etc.
• Visual Restart Boundaries: Clear dividers mark pod restarts in grouped view
• Auto-Refresh: Configurable page refresh (30s, 1m, 2m, 5m) with pause during configuration
• Cgroup v1/v2 Support: Works with both cgroup versions
• Low Overhead: Minimal resource usage (32Mi memory, 10m CPU)

# Using Helm (recommended - includes RBAC for restart tracking)
helm install oomwatcher ./charts/oomwatcher

# Or manually
kubectl apply -f deploy/rbac.yaml        # ServiceAccount and ClusterRole for dumpserver
kubectl apply -f deploy/pvc.yaml
kubectl apply -f deploy/dumpserver.yaml

import _ "net/http/pprof"

func main() {
    // Start pprof server
    go http.ListenAndServe(":5999", nil)

    // Your app code...
}

spec:
  template:
    spec:
      containers:
        - name: myapp
          image: myapp:latest
          # ... your app config

        - name: oomwatcher
          image: ghcr.io/ethanadams/oomwatcher:latest
          env:
            - name: PPROF_URL
              value: "http://localhost:5999"
            - name: DUMP_SERVER_URL
              value: "http://oomwatcher-dumpserver"
            - name: MEMORY_THRESHOLD
              value: "90"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_UID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.uid
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
            - name: host-cgroup
              mountPath: /host-cgroup
              readOnly: true

      volumes:
        - name: host-cgroup
          hostPath:
            path: /sys/fs/cgroup
            type: Directory

See deploy/example-with-sidecar.yaml for a complete example.

OOM Watcher exposes Prometheus metrics at the /metrics endpoint for monitoring and alerting.

Dump Tracking:

• oom_watcher_dumps_total{namespace, deployment} - Total number of heap dumps captured (counter)
• oom_watcher_dumps_by_namespace{namespace} - Current number of dumps per namespace (gauge)
• oom_watcher_memory_percent{namespace, pod} - Memory percentage when dump was captured (gauge)
• oom_watcher_avg_memory_percent - Average memory percentage across all dumps (gauge)
• oom_watcher_time_since_last_dump_seconds{namespace, pod} - Seconds since last dump (gauge)

Storage:

• oom_watcher_storage_bytes - Total storage used by heap dumps (gauge)

Pod Restarts:

• oom_watcher_pod_restarts_total{namespace, pod} - Total pod restarts observed (counter)

Pruning:

• oom_watcher_prune_operations_total - Total prune operations executed (counter)
• oom_watcher_pruned_dumps_total - Total dumps deleted by pruning (counter)
• oom_watcher_pruned_bytes_total - Total bytes freed by pruning (counter)

Cleanup:

• oom_watcher_cleanup_operations_total - Total cleanup operations executed (counter)

Using Helm (Automatic):

The Helm chart automatically adds Prometheus scrape annotations to the dumpserver service. Prometheus will discover and scrape the metrics endpoint automatically.

To enable ServiceMonitor (for Prometheus Operator):

# values.yaml
dumpserver:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      interval: 30s
      scrapeTimeout: 10s

Manual Configuration:

Add OOM Watcher to your Prometheus scrape config:

scrape_configs:
  - job_name: 'oomwatcher'
    static_configs:
      - targets: ['oomwatcher-dumpserver:80']

Or create a ServiceMonitor manually:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: oomwatcher
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: oomwatcher
      app.kubernetes.io/component: dumpserver
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

groups:
  - name: oomwatcher
    rules:
      - alert: HighMemoryUsage
        expr: oom_watcher_memory_percent > 90
        for: 5m
        annotations:
          summary: "Pod {{ $labels.pod }} in {{ $labels.namespace }} is using {{ $value }}% memory"

      - alert: FrequentOOMs
        expr: rate(oom_watcher_pod_restarts_total[1h]) > 3
        annotations:
          summary: "Pod {{ $labels.pod }} has restarted {{ $value }} times in the last hour"

      - alert: StorageGrowth
        expr: rate(oom_watcher_storage_bytes[24h]) > 10737418240  # 10GB/day
        annotations:
          summary: "OOM Watcher storage growing at {{ $value | humanize }}B/day"

Create dashboards to visualize:

• Memory usage trends across pods
• OOM events timeline
• Storage usage and growth rate
• Dumps per namespace/deployment
• Prune operation effectiveness

In the browser: Click "View" on any dump in the web UI to open the interactive pprof visualization with flamegraph, call graph, and top views.

Comparing dumps: Select any 2 dumps using the checkboxes:

• Click "Compare (2)" for interactive diff view:
  • Growing functions highlighted in red
  • Shrinking functions in green
  • Top allocations ranked by delta
  • Flamegraph showing memory growth areas
• Click "🔍 Analyze" for automatic leak detection:
  • Identifies leak candidates with confidence scoring (HIGH/MEDIUM/LOW)
  • Categorizes leak types (goroutine leaks, connection leaks, cache growth, buffer growth, etc.)
  • Provides specific, actionable suggestions for each leak
  • Ranks findings by severity and growth rate
  • Shows evidence trail for each detected issue

Locally: Download a dump and analyze with go tool pprof:

# Interactive CLI
go tool pprof heap-default-mypod-20240115-120000-85pct.pb.gz

# Web UI
go tool pprof -http=:8081 heap-default-mypod-20240115-120000-85pct.pb.gz

# Compare two dumps
go tool pprof -http=:8081 -base=dump1.pb.gz dump2.pb.gz

docker pull ghcr.io/ethanadams/oomwatcher:latest
docker pull ghcr.io/ethanadams/dumpserver:latest

docker build -t oomwatcher:latest -f Dockerfile .
docker build -t dumpserver:latest -f Dockerfile.dumpserver .

Supports both cgroup v1 and v2. The sidecar mounts the host's cgroup filesystem at /host-cgroup and uses the pod UID to locate the pod-level cgroup, which reflects the correct aggregate memory limits for all containers in the pod.

Remove dumps captured before pod restarts, keeping only the most recent N dumps before each restart boundary.

Note: For automatic background pruning, see Auto-Pruner Configuration below.

# Prune all pods, keeping only 3 dumps before each restart
curl -X POST "http://dumpserver:8080/api/prune?confirm=true&keep=3"

# Also prune current restart (keep only 3 most recent dumps overall)
curl -X POST "http://dumpserver:8080/api/prune?confirm=true&keep=3&prune_current=true"

# Prune a specific pod
curl -X POST "http://dumpserver:8080/api/prune?confirm=true&keep=5&pod_name=my-app-abc123"

# Also prune dumps for pods that no longer exist
curl -X POST "http://dumpserver:8080/api/prune?confirm=true&keep=3&prune_nonexistent=true"

# Response
{
  "deleted_dumps": 27,
  "deleted_bytes": 1048576,
  "deleted_by_pod": {
    "my-app-abc123": 15,
    "my-app-def456": 12
  },
  "keep_per_restart": 3,
  "prune_current": false,
  "status": "success"
}

⚠️ Warning: This permanently deletes all heap dumps and database records.

# Delete all dumps and clear database
curl -X POST "http://dumpserver:8080/api/flush?confirm=true"

# Response
{
  "deleted_files": 42,
  "status": "success"
}

Configure automatic background pruning with retention policies. Can be managed through the Web UI (Data Management modal) or via API.

# Get current auto-pruner configuration
curl http://dumpserver:8080/api/prune/config

# Update configuration
curl -X PUT http://dumpserver:8080/api/prune/config \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "interval_minutes": 60,
    "retention_strategy": "hybrid",
    "keep_before_restart": 3,
    "prune_current_restart": true,
    "min_dumps_per_pod": 1,
    "max_age_days": 30,
    "max_total_size_mb": 10240,
    "dry_run": false
  }'

# Trigger manual run (dry run to preview)
curl -X POST http://dumpserver:8080/api/prune/run?dry_run=true

Retention Strategies:

• hybrid: Combines count, age, and size strategies (recommended)
• count: Keep N dumps before each restart boundary
• age: Delete dumps older than max_age_days
• size: Keep total storage under max_total_size_mb

Configure automatic cleanup of dumps for deleted pods. Can be managed through the Web UI (Data Management modal) or via API.

# Get current cleanup configuration
curl http://dumpserver:8080/api/cleanup/config

# Update configuration
curl -X PUT http://dumpserver:8080/api/cleanup/config \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "interval_minutes": 1440,
    "min_age_hours": 24,
    "namespace_filter": "default,production",
    "dry_run": false
  }'

# Trigger manual cleanup (dry run to preview)
curl -X POST http://dumpserver:8080/api/cleanup/run?dry_run=true

Configuration Options:

• min_age_hours (optional): Only delete dumps for nonexistent pods if the most recent dump is older than this many hours. This prevents accidentally removing dumps immediately after a pod is deleted when you might still need them for debugging.

Note: This cleanup runs independently from the auto-pruner and specifically targets dumps for pods that no longer exist in Kubernetes. Useful for cleaning up after deployments are deleted.

Use Prune when:

• You want to keep recent dumps but clean up old ones
• Storage is filling up but you still need some history
• You want to keep the last few dumps before each restart for analysis
• Regular maintenance (e.g., weekly cleanup)

Use Flush when:

• You need a complete clean slate
• Testing/development environments
• After completing analysis and no longer need any dumps

For most production environments:

# Weekly: Keep 3 dumps before each restart + 3 most recent overall
curl -X POST "http://dumpserver:8080/api/prune?confirm=true&keep=3&prune_current=true"

This approach:

• Preserves the 3 most recent dumps before each OOM event
• Also trims long-running pods that haven't restarted
• Prevents unbounded storage growth
• Keeps enough history for debugging

For critical applications that need more history:

# Keep more dumps for forensic analysis
curl -X POST "http://dumpserver:8080/api/prune?confirm=true&keep=10&prune_current=true"

For technical architecture, database schema, and development guide, see CLAUDE.md.