Unify continuous batching and heterogeneous runtime acceleration - #68
Conversation
JustVugg
left a comment
There was a problem hiding this comment.
Thank you for consolidating the stack exactly as asked — one reviewable line instead of six overlapping branches is a big improvement, and it builds clean here with the oracle passing (32/32 TF, 20/20 greedy). The continuous batching, physical-core planning, session-local placement, and disjoint VRAM/RAM staging all read well. I'm ready to land this — one line stands between us and merge.
--topp 0.7 still hard-errors by default. Verified on this branch: TOPP=0.7 ./glm exits with "usa COLI_POLICY=experimental-fast esplicitamente", and coli run --topp 0.7 inherits the default quality policy and aborts. I see you documented the requirement rather than changing it — but the documentation now contradicts itself: the knobs list still says --topp 0.7 adaptive expert top-p (30–40% less disk) two paragraphs above the line that says it needs --policy experimental-fast.
I fully agree with your reasoning — top-p is lossy, it drops low-weight experts and changes the router's output, so a 'quality' policy flagging that is philosophically correct. The disagreement is only about the mechanism: a fatal exit vs a one-line warning. Please make it a warning:
fprintf(stderr, "[policy] --topp/--topk drop low-weight experts (~1.6x fewer reads, small quality cost)\n");
and proceed. Three concrete reasons this can't be a hard error:
--topp 0.7is the single most-recommended flag in the project — it's in the README benchmark instructions and every warm-machine reply.- Four contributors are running it right now (#12, #31, #39, #65). Their next
git pull+ rerun breaks with no obvious fix. - The user passing
--toppis the acknowledgment — they typed the lossy flag on purpose.
Flip that exit to a warning (keep the policy concept, keep experimental-fast for the router-editing TOPK path if you like) and I'll merge immediately. Everything else here is genuinely excellent work.
|
The topp fix is exactly right — thank you. Confirmed in the diff: It just needs a rebase now — it went CONFLICTING because a batch of PRs merged into main since you opened it (the CPU perf line: OMP hot-team #77, PIPE #79, and others). Per the new workflow (thanks for the dev-branch idea): please rebase onto |
…device grouping, policy and release-host on top dev absorbed an independent async expert I/O pool (JustVugg#79: gen-tagged cursors, per-miss drain before the nr==0 skip). That loader supersedes this branch's pthread job loader, which is removed; the merge keeps dev's loader and carries what this branch adds on top of it: per-device grouped CUDA expert transfers, VRAM-prefix/release-host pin loading, resource-policy plumbing, dev's tools/function-calling composed with this branch's request-cancel callback, and the service/wait profiling split with dev's English wording.
…ith the CUDA absorb path and service/wait profiling
…tion scores composed with the resident pipeline, device KV shadow and batched router
|
@JustVugg Ping as requested — it's green: base retargeted to Same is true for the rest of the stack, in landing order: #68 → #80 → #111, all conflict-free on |
JustVugg
left a comment
There was a problem hiding this comment.
The reconciliation is exactly right: retiring your pthread loader in favor of the #79 PIPE pool (no competing I/O pools — the thing I worried about with kryptt's #78) and composing in the #110 heap attention-scores fix, then re-smoke-testing on the 6×5090. That's the diligent path. Validated the CPU side here: build 0 warnings, oracle 32/32 TF + 20/20 greedy (default single-request path unchanged), and SERVE still works (clean READY…END — continuous batching doesn't break the stdio protocol). The topp policy fix (warning, not fatal) is confirmed too.
The CUDA/continuous-batching runtime I take on your 6×5090 + sanitizer validation — this is a GPU-less box, same trust model as the Metal backend and the #16 CUDA tier. It lands on dev to soak (not main) with the rest of the GPU work, gated behind COLI_CUDA. Merging #68 as the base of the stack; #80 and #111 next in order. Thank you for the careful reconciliation.
Four Windows-portability fixes that make colibri run natively on Windows 11 (no WSL). All validated on real hardware (JustVugg#128). 1. Serve-mode pipe fix (glm.c + coli): the engine speaks a byte protocol with the 'coli' wrapper. On Windows the MinGW CRT opens stdout in TEXT mode, translating \n to \r\n, so the READY sentinel never matched and chat hung at ~10 GB resident. Fixed by switching stdin+stdout to BINARY mode (_setmode _O_BINARY) in run_serve(). Also moved child stderr from a temp-file redirect (which stalled the pipe) to a PIPE with a bounded non-blocking drain. 2. RAM detection (resource_plan.py): memory_available() only read /proc/meminfo. On native Windows Python that path doesn't exist, so it returned 0, the budget defaulted to 8 GB, and the expert cache was sized to 0 slots/layer. Added a Windows branch using GlobalMemoryStatusEx (ullAvailPhys), matching the C engine's compat_meminfo, with a GetPhysicallyInstalledSystemMemory fallback. Regression test included. 3. POSIX guards (glm.c): guarded select(), mmap/fstat, madvise, and run_serve_mux behind __APPLE__/__linux__ checks so dev's recent continuous-batching (JustVugg#68) and COLI_MMAP code compiles on native MinGW. SERVE_BATCH reports unavailable on Windows and falls back to the single-sequence serve path. 4. warmup.ps1: overnight expert-cache priming script.
…ustVugg#163) expert_gate_up's matmul_i4_pair fuses gate and up into one OMP parallel-for (2*O iterations), changing the static-schedule thread assignment vs the pre-JustVugg#68 path (two separate matmul_qt calls, O each). The different float accumulation order shifts near-tie argmax decisions through 78 layers, collapsing MTP draft acceptance from 33% to 4%. COLI_NO_FUSED_PAIR=1 disables the fusion and restores the pre-JustVugg#68 accumulation order. This is a diagnostic/workaround — the proper fix is making the verify path (step_all at S>1) and the draft path (mtp_draft at S=1) use numerically identical kernels, which requires a deeper rework of the matmul dispatch.
…ock pin wiring + select_ctx device cache (#162) * win: direct I/O via FILE_FLAG_NO_BUFFERING + compat_fsize + VirtualLock primitives compat_open_direct() gives Windows the O_DIRECT twin fd st.h already uses on Linux/macOS: FILE_FLAG_NO_BUFFERING, same 4K-alignment contract as O_DIRECT (the engine's DIRECT=1 path already aligns offset/len and slabs are posix_memalign'd). Measured on GLM-5.2 744B int4, Ryzen 9 9950X3D / 126 GB / PCIe4 NVMe (5.8 GB/s at the engine's 19MBx8T pattern), Windows 11, MinGW GCC 16.1, 32-token greedy runs at --topp 0.7, 40 GB pin, current dev HEAD: buffered: 0.38 tok/s (expert-disk dominates) DIRECT=1: 0.56 tok/s (1.47x) — byte-identical greedy output vs buffered compat_fsize() (GetFileSizeEx): CRT lseek(SEEK_END) returns -1 on NO_BUFFERING fds (measured on UCRT); iobench uses it and gains a NO_BUFFERING branch so disk numbers are comparable across platforms. compat_mlock/compat_munlock: VirtualLock with working-set growth (bare VirtualLock caps at the default working-set minimum, a few hundred KB). Wired into the engine in the next commit. tests/test_compat_direct.c covers the alignment contract, data integrity, fsize on both fd kinds; skips cleanly off Windows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * win: wire VirtualLock into mem_wire, munlock pairing in expert_host_release MLOCK=1 was a silent no-op on Windows: pinned experts could be paged out by working-set trimming under memory pressure. mem_wire now uses compat_mlock (VirtualLock + working-set growth); expert_host_release unlocks before freeing, mirroring the POSIX branch. Validated: 39.6 GB pin wired in 17s on a 126 GB machine, zero failures; TF oracle 32/32 with MLOCK=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * cuda: thread-local current-device cache in select_ctx cudaSetDevice on every call is expensive when the serial expert loop alternates devices. Measured on RTX 5090 + RTX 4090 (Windows, DLL backend, pre-#68 dispatch): expert-matmul 14.3s -> 25.4s per 32 tokens going from 1 to 2 devices, entirely per-call context switching. The current device is per-thread in the CUDA runtime, so a thread_local cache skips redundant switches; multi-GPU expert serving becomes positive-scaling instead of negative. Kernel suite passes on sm_120 + sm_89; TF oracle 32/32 dual-GPU. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: olorin <io@zyphyr.co> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…gg#140) The repo's own dependency-free gate, on the three claimed platforms, for every PR and push to main/dev. The windows job is the one that would have caught JustVugg#68/JustVugg#137 pre-merge. No CUDA, no Metal, no model downloads in v1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hysical-core planning, disjoint VRAM/RAM placement, topp-policy warning (CPU-validated, CUDA on 6x5090) (JustVugg#68) * Fuse CUDA expert MLP execution * Group CUDA expert transfers by device * Instrument grouped CUDA expert execution * Bound grouped CUDA decode scratch * Execute expert groups across GPUs in parallel * Release host backing for multi-GPU experts * Define quality-preserving memory policies * Overlap cold expert loading with resident compute * Adapt expert placement with session LFRU * Fuse q4 expert gate and up dispatch * Plan CPU work on physical cores * Batch grouped expert CUDA kernels * Separate VRAM and RAM expert placement * Add ragged multi-sequence decode forward * feat(runtime): add continuous decode scheduler * Route concurrent API requests through batch scheduler * Harden multiplex request lifecycle and framing * Cancel disconnected multiplex requests * Bind API port before starting the engine * fix automatic KV slot allocation * add native int4 Tensor Core grouped GEMM * add Tensor Core throughput benchmark * optimize packed int4 low-row kernels * add asynchronous CUDA staging streams * document validated six-GPU dense acceleration * tune six-GPU expert hot set * raise validated expert hot-set target * add CUDA MLA absorption core * fuse grouped expert gate and up projections * Warn for explicit lossy routing flags
…ock pin wiring + select_ctx device cache (JustVugg#162) * win: direct I/O via FILE_FLAG_NO_BUFFERING + compat_fsize + VirtualLock primitives compat_open_direct() gives Windows the O_DIRECT twin fd st.h already uses on Linux/macOS: FILE_FLAG_NO_BUFFERING, same 4K-alignment contract as O_DIRECT (the engine's DIRECT=1 path already aligns offset/len and slabs are posix_memalign'd). Measured on GLM-5.2 744B int4, Ryzen 9 9950X3D / 126 GB / PCIe4 NVMe (5.8 GB/s at the engine's 19MBx8T pattern), Windows 11, MinGW GCC 16.1, 32-token greedy runs at --topp 0.7, 40 GB pin, current dev HEAD: buffered: 0.38 tok/s (expert-disk dominates) DIRECT=1: 0.56 tok/s (1.47x) — byte-identical greedy output vs buffered compat_fsize() (GetFileSizeEx): CRT lseek(SEEK_END) returns -1 on NO_BUFFERING fds (measured on UCRT); iobench uses it and gains a NO_BUFFERING branch so disk numbers are comparable across platforms. compat_mlock/compat_munlock: VirtualLock with working-set growth (bare VirtualLock caps at the default working-set minimum, a few hundred KB). Wired into the engine in the next commit. tests/test_compat_direct.c covers the alignment contract, data integrity, fsize on both fd kinds; skips cleanly off Windows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * win: wire VirtualLock into mem_wire, munlock pairing in expert_host_release MLOCK=1 was a silent no-op on Windows: pinned experts could be paged out by working-set trimming under memory pressure. mem_wire now uses compat_mlock (VirtualLock + working-set growth); expert_host_release unlocks before freeing, mirroring the POSIX branch. Validated: 39.6 GB pin wired in 17s on a 126 GB machine, zero failures; TF oracle 32/32 with MLOCK=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * cuda: thread-local current-device cache in select_ctx cudaSetDevice on every call is expensive when the serial expert loop alternates devices. Measured on RTX 5090 + RTX 4090 (Windows, DLL backend, pre-JustVugg#68 dispatch): expert-matmul 14.3s -> 25.4s per 32 tokens going from 1 to 2 devices, entirely per-call context switching. The current device is per-thread in the CUDA runtime, so a thread_local cache skips redundant switches; multi-GPU expert serving becomes positive-scaling instead of negative. Kernel suite passes on sm_120 + sm_89; TF oracle 32/32 dual-GPU. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: olorin <io@zyphyr.co> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…gg#140) The repo's own dependency-free gate, on the three claimed platforms, for every PR and push to main/dev. The windows job is the one that would have caught JustVugg#68/JustVugg#137 pre-merge. No CUDA, no Metal, no model downloads in v1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Summary
This consolidates the previously stacked runtime PRs into one reviewable performance line. It keeps the quality-preserving VRAM/RAM/disk policy while adding continuous decode batching and the validated CPU/CUDA execution improvements.
Runtime and scheduling
CUDA execution
Measured results on six RTX 5090 GPUs
Negative experiments are intentionally excluded: forced low-row Tensor Core dispatch, router-lookahead prefetch, CUDA Graph capture, NCCL tensor parallel on non-P2P RTX 5090 topology, and hard NUMA binding.
Validation
make checkmake cuda-test CUDA_ARCH=sm_120on RTX 5090Supersedes
Supersedes #30, #45, #50, #51, #52, #53, #54, and #61. Those PRs are closed to keep the review queue small.