feat: add tiered CUDA acceleration for routed experts - #16

Summary

• add an opt-in CUDA backend focused on routed experts, the measured bottleneck
• keep resident dense/attention tensors on CPU by default; preserve their CUDA path behind CUDA_DENSE=1 for controlled experiments
• persist uploaded int8/int4/int2 expert weights in VRAM
• support COLI_GPU=0 and deterministic multi-device ownership with COLI_GPUS=0,1,...
• give every device an independent context, scratch buffers, capacity query, and tensor accounting
• promote the hottest experts from an existing PIN profile into a bounded VRAM tier with CUDA_EXPERT_GB
• treat CUDA_EXPERT_GB as a total multi-device budget and place each whole expert on the least-loaded device that can hold it
• preload the selected expert tier at startup and report per-device tensor/VRAM accounting
• keep non-promoted experts in the existing RAM hot-store/LRU and NVMe streaming tiers
• invalidate device tensors safely when live REPIN reuses a pinned slot
• fail fast for invalid CUDA configurations and disable a failed tensor after one error
• preserve the original pure-C build and runtime behavior by default

Reproducible fixture benchmark

The PR includes a deterministic 313M-parameter glm_moe_dsa fixture generator and a fixed-token REPLAY=1 harness. Random weights are used only to preserve the real MLA/MoE/streaming code paths; this is not a language model.

Single RTX 5090, 48 CPU threads, int4, fixed replay tokens, 5-run median decode:

The result drives the default: removing expert I/O is the dominant win, and routed-expert CUDA adds about 19% beyond RAM pinning. Dense CUDA adds only about 4% after experts are hot and does nothing meaningful while streaming, so it remains opt-in. The full 744B model has much larger expert matrices and still requires a separate benchmark.

Validation

• make and make portable
• Python fixture/harness syntax and end-to-end reproduction
• CPU-only fail-fast checks for CUDA configuration
• CPU tiny-model run under ASan + UBSan: no runtime errors
• one-device and two-device backend correctness/accounting tests on RTX 5090 / CUDA 13.3
• two-device backend test under Compute Sanitizer memcheck: 0 errors
• complete six-device expert-only path under Compute Sanitizer memcheck: 0 errors
• medium fixture expert-only replay under Compute Sanitizer memcheck: 0 errors
• CPU and CUDA tiny-model inference produce identical tokens
• six-device tiny-model profile: 16/16 pinned experts placed 3/3/3/3/2/2, 84 expert calls served from VRAM, zero disk misses

Current scope

This PR implements a complete three-tier path: selected hot experts stay in VRAM, other pinned experts stay in RAM, and uncached experts retain the original NVMe streaming fallback. Experts are balanced by placed bytes under live per-device capacity limits. Each GPU-resident expert has one explicit owner.

Multi-device execution currently uses synchronous host-staged activation copies. It does not require or assume P2P/NCCL, matching the tested six-card RTX 5090 host where GPU P2P is unavailable. The kernels remain correctness-first custom kernels.

Next steps

• bind device contexts and CPU backing allocations to the matching NUMA node
• batch or overlap host/device activation copies
• benchmark the full model before tuning placement and eviction policies