SGPU

SGVR GPU / Simple GPU monitor for the lab's MLXP H200 nodes. Check GPU ownership, utilization, storage, and usage history before launching another Kubernetes pod.

• Every GPU process is attributed to its pod and owner, not just a PID.
• One client can survey both H200 nodes with sgpu --all once, or pick a node with sgpu -n 1 / sgpu -n 2.
• In-pod TUI via kubectl exec -it: smooth refresh, scrolling, sorting, owner filtering, and a stats screen.
• Usage stats 24/7: a cluster pulse (KST utilization/VRAM sparklines, utilization zones, observed compute windows), owner momentum, GPU-hours, awards, heatmaps, and idle-allocation warnings.
• Shared storage (pv-01/pv-02) usage at a glance.
• Monitor pod is read-only, always-on, and requests no GPU.

Version history: CHANGELOG.md.

uv tool install sgpu   # persistent install via uv (recommended)
pipx install sgpu      # or pipx
pip install sgpu       # or plain pip (needs pip; WSL/Ubuntu often lacks it)
uvx sgpu               # or run once without installing

Upgrade with the tool you installed with — uv tool upgrade sgpu, pipx upgrade sgpu, or pip install -U sgpu. sgpu tells you the right one when it detects your client is behind the server.

Needs kubectl with an MLXP kubeconfig (setup). One kubeconfig covers both H200 servers - the two downloads (sgvr-node-01/-02) share the same token and contexts, so either file works for every node.

sgpu               interactive TUI      sgpu stats [days]  usage report + awards
sgpu once          one-shot dashboard   sgpu apps          processes + owners
sgpu watch [sec]   dumb-terminal loop   sgpu nvitop        raw nvitop
sgpu pods|smi|gpustat|json|health|version|--help

MLXP has two H200 servers (p-sgvr-node-01, p-sgvr-node-02):

sgpu -n 1 once        node-01   (shorthand for p-sgvr-node-01)
sgpu -n 2 once        node-02
sgpu --all once       survey both nodes at once (any text command)
sgpu --all stats      ONE lab-wide merged report: combined leaderboard,
                      awards and heatmaps with a NODE column per owner
sgpu once             uses your current kubectl context's namespace

In the dashboard TUI, n switches node 1/2 without relaunching. In the stats screen, n cycles the current node, the other node, and LAB.

TUI keys:

j/k       scroll
Tab       switch pane
s         sort
o         owner filter
p         pause
t         stats screen
h/d/w/m   stats axis: hour/day/week/month
a         cycle stats axis
r         refresh
q         quit

Options: -n namespace, --pod, -r refresh, --no-color, --json for stable agent-readable JSON on text commands. Env: SGPU_NAMESPACE, SGPU_POD, SGPU_NO_UPDATE_CHECK=1 (silence the upgrade nudge).

Agent-friendly examples:

sgpu -n 2 json --json
sgpu -n 2 pods --json
sgpu -n 2 apps --json
sgpu -n 2 stats 14 --json --scope lab
sgpu --all json --json

The monitor server is upgraded centrally (one image redeploy updates the dashboard/stats UI for everyone — no client action needed). The client (this pip package) only changes for client-side features (-n, --all, reconnect). When your client falls behind the server, sgpu shows a yellow ↑ update available banner in the TUI and a one-line hint after text commands — just run pip install -U sgpu (or uv tool upgrade sgpu).

Anyone with kubectl access can use the monitor pod without installing sgpu.

kubectl exec -it -n p-sgvr-node-01 sangmin-gpu-monitor -- python3 /opt/gpu-monitor/tui.py
kubectl exec -n p-sgvr-node-01 sangmin-gpu-monitor -- curl -fsS http://127.0.0.1:8080/table

Endpoints on :8080:

/table /apps /json /stats /pods /smi /topo /gpustat /health /version
/stats/files /stats/raw?date=YYYYMMDD

Text endpoints support ?color=1&cols=N&ascii=1.

SGPU samples every 60 seconds around the clock into raw JSONL, gzips and rolls up daily summaries, and stores the results on the shared volume at pv-01/sangmin/sgpu (~0.1 MB/day/node gzipped). The interval is set by SGPU_SAMPLE_INTERVAL; the aggregator preserves a compatible daily median while recognizing confirmed sustained cadence changes within a day, so even a large 15-second to 900-second switch neither under-credits normal samples nor fragments Flow. Missing telemetry remains bounded and breaks a Flow window.

Retention defaults to 365 days and is capped at 2 GB. sgpu stats 30 starts with a Cluster pulse: compact KST utilization and VRAM sparklines, a quiet/light/work/hot utilization-zone distribution, and observed compute windows. It then shows the leaderboard, awards, owner momentum, daily activity, and KST hour heatmaps. The curses stats screen keeps the pulse above the leaderboard/grid; on a short terminal it drops awards before it drops the pulse or grid.

Device telemetry is derived from the util, mem, and mem_total fields already recorded in raw sample v1. Daily rollup v2 stores only weighted sums, 24 KST buckets, ten utilization buckets, and window counters, so rendering does not re-scan raw history. On first access, an older rollup is rebuilt once from its preserved raw/gz file and atomically replaced. If raw history has already been removed, its owner totals remain valid but the pulse marks partial coverage (by node-day in LAB scope, so a same-date peer cannot mask a missing node) or stays hidden when no device telemetry is available, rather than treating unknown time as 0% utilization. Contiguous Flow windows are stitched across UTC rollup boundaries. A multi-node LAB pulse labels its exact aggregate as node compute windows and longest per-node window; it does not present that as a global cross-node time-window union.

The pulse is cluster-wide telemetry, not a per-owner estimate. Therefore an owner-filtered /stats?owner=... result intentionally omits it; use that view for the owner's GPU-hours, efficiency, streak, and allocations.

The monitor pod must stay running for stats to accumulate. It is designed to do that with tini init, restartPolicy: Always, and no GPU allocation.

sgpu stats [days] (or sgpu --all stats for the whole lab) ranks everyone by GPU-hours and hands out playful badges. Example:

SGPU usage report — last 7 days — all nodes (node-01+node-02)
data: 7 days, coverage 168.0h

Cluster pulse
KST  UTIL ▁▁▂▃▄▆▇█▇▆▄▃▂▂▃▄▆▇▆▄▃▂▁▁  avg 54%  hot 31%
     VRAM ▁▂▂▃▄▅▆▇▇▆▅▄▃▃▄▅▆▇▇▆▄▃▂▁  avg 47%
UTIL mix  quiet 23%  light 18%  work 28%  hot 31%
Flow      12 node compute windows · longest 9h18m per-node (any GPU >= 50%)

Awards
🏆 Best researcher: jiwon    — 92.4 effective GPU-h (81% avg over 114.1 GPU-h)
⚡ Power user:      jiwon    — 114.1 GPU-h
🎯 Sharpshooter:    minseo   — 97% avg SM over 40.2 GPU-h
🧠 Memory heavyweight: haeun — 139.7 GiB peak
🦉 Night owl:       doyun    — 71% of activity in KST 0-5h
💤 Most headroom:   sangho   — 22% avg util over 48 GPU-h (free speedup waiting)
🪑 Seat warmer:     taemin   — 9.8 idle GPU-h allocated

Leaderboard
#   OWNER    NODE     GPU-H  EFF-H  AVG-SM%  AVG-UTIL%  PEAK-MEM  ALLOC-H  IDLE-H  IDLE%
1.  jiwon    node-01  114.1   92.4       81         81      81.1    114.5     0.4      0
2.  sangho   node-02   48.0   10.6       22         30     129.2     50.1     2.1      4
3.  minseo   node-02   40.2   39.0       97         97      62.9     40.2     0.0      0
4.  haeun    node-01   37.9   27.2       72         72     139.7     39.4     1.6      4
5.  taemin   node-01   11.6    0.1        7         25     139.0     21.4     9.8     46

Momentum
jiwon 7d streak · 7/7d · 81% eff  |  minseo 5d streak · 6/7d · 97% eff

Ranking is by GPU-H (GPU-hours). Each owner holds at most 3 badges.

Column meanings (GPU-H, EFF-H, SM%, PEAK-MEM, ALLOC-H, IDLE-H, …) are in What the numbers mean. The NODE column (lab-wide view) shows each person's home node, or both if they split their work across nodes.

Names above are illustrative. Press ? in the TUI for the same reference in-app.

Press ? in the TUI for this same reference in-app.

UTIL vs SM%: UTIL is the whole card being busy at all; SM% is how saturated the compute cores are for a specific process. High UTIL with low SM% usually means the GPU is waiting on data (I/O, small batches), not computing hard — that's where EFF-H and the "Most headroom" award come in.

The monitor runs from a public image (docker.io/alex6095/sgpu-monitor), so no registry login or pull secret is needed. Deploy one pod per node - always pass -n (a bare kubectl apply would hit your current context's namespace):

# For each node namespace (p-sgvr-node-01 and/or p-sgvr-node-02):
kubectl apply -n p-sgvr-node-01 -f k8s/gpu-monitor.yaml
kubectl wait --for=condition=Ready pod/sangmin-gpu-monitor -n p-sgvr-node-01 --timeout=180s

Pods are immutable, so to roll out a new image: kubectl delete pod sangmin-gpu-monitor -n <ns> then apply again.

Optional, for the pod-allocation view and idle stats (kubelet syncs it in within a minute, no restart; use the same -n):

kubectl -n p-sgvr-node-01 create secret generic sgpu-kubeconfig --from-file=config=$HOME/.kube/config

Anyone with exec access to the monitor pod can read that token. This is fine inside a trusting lab namespace; use a least-privileged kubeconfig.

docker build -f docker/Dockerfile.gpu-monitor -t docker.io/alex6095/sgpu-monitor:X.Y.Z .
docker push docker.io/alex6095/sgpu-monitor:X.Y.Z   # keep the repo public

mkdir -p ~/.local/bin ~/.kube
V=$(curl -fsSL https://dl.k8s.io/release/stable.txt)
curl -fsSL -o ~/.local/bin/kubectl "https://dl.k8s.io/release/${V}/bin/linux/amd64/kubectl" && chmod +x ~/.local/bin/kubectl
cp /path/to/sgvr-node-01-kubeconfig.yaml ~/.kube/config && chmod 600 ~/.kube/config
# Either node's kubeconfig works for both - pick the node with `sgpu -n 1|2`.
kubectl get pods -n p-sgvr-node-02   # connectivity test

SGPU_MOCK=1 python3 tools/gpu-monitor/server.py   # full pipeline, no GPU needed
SGPU_MOCK=1 python3 tools/gpu-monitor/tui.py
python3 -m unittest discover -s tests
python3 tools/render_readme_images.py        # synthetic public screenshots
SGPU_README_LIVE=1 python3 tools/render_readme_images.py  # optional live capture

How it works: sgpu is a thin Python client. It uses kubectl exec to reach the monitor pod, where server.py renders the dashboard. Process-to-pod attribution reads /proc/<pid>/environ (HOSTNAME = pod name), and owner is inferred from the pod-name prefix.

Known limits: pods overriding spec.hostname and MPS may show as ?.

Troubleshooting:

TUI reconnects after an EOF       -> sgpu checks the actual pod UID/Ready state
                                     and server health; a transient stream loss
                                     uses bounded backoff instead of assuming a
                                     rollout
disconnected TUI left in the pod  -> current monitor images stop it after 45s
                                     without terminal-output progress, avoiding
                                     orphan polling processes
broken terminal after dropped TUI -> reset (older clients)
frozen TUI                         -> rerun sgpu
garbled bars                       -> Windows Terminal or --no-color