OpenFugu-rs

A staged Rust reimplementation of OpenFugu — Sakana AI's Fugu ("one model commanding a pool of models") rebuilt in Rust: candle Qwen3-0.6B router + SVF, an OpenAI-compatible serving layer, real heterogeneous worker pools, and gradient-free training. All phases run.

Independent reimplementation. Not affiliated with Sakana AI. No third-party code/weights are redistributed — model weights are fetched at setup. See NOTICE. Licensed under Apache-2.0 (LICENSE).

89 tests across all crates; cargo clippy --all-targets --all-features clean.

Phase 2a hardening (post-review): Router/Backbone are fallible (a backbone forward error propagates → 5xx, never panics — symmetric with Worker); svf_reconstruct rejects non-finite inputs, returns a zero matrix for zero energy, and uses a scale-relative degeneracy guard; apply_agent_mask requires a full-length, not-all-false mask; the SVF_KEYS constant pins the exact 9-matrix state_dict order/shapes the Phase 2b loader must follow; the golden test honestly covers head+argmax only (tokenizer/hidden captured for 2b) and hard-fails under FUGU_REQUIRE_GOLDEN=1 so real-weight parity can't be deferred silently.

Artifact sources (verified reachable 2026-06, sha256-checked): the trained router ships not as model_iter_60.npy but as a structured safetensors set on the HF dataset nshkrdotcom/trinity-coordinator-adapted-qwen3-0.6b:

• router_head.safetensors — F32 [10,1024], key trinity_router_head, sha 7ff2db0e… (loaded + tested).
• 9 checkpoints/000N_*.kernel.safetensors — each ONE BF16 tensor = the SVF-already-applied (adapted) weight matrix (NOT offsets), in Nx/Bumblebee kernel layout [in, out] (transposed vs torch [out, in] for the 7 layer-26 linears; embed/lm_head [vocab,hidden] are identical). source_names match SVF_KEYS exactly.

The 37-case fixture is on GitHub raw; Qwen3-0.6B is on both HF and ModelScope. Tested against real data: load_router_head (41 KB head) and load_adapted_matrix (BF16→f32 + Nx→torch transpose, verified element-wise on the real query matrix).

Phase 2b result. The candle Qwen3-0.6B backbone + the published SVF-adapted weights + the trained head reproduces (and beats) the reference: 100% agent / 100% role on the 37-case fixture (cargo run -p fugu-backbone-candle --bin verify37) — correctly-applied SVF even resolves the 2 documented "agent misses" that base-only (95%/100%) leaves. FUGU_NO_SVF=1 runs the base-only path.

Debugging note (kept honest): an earlier version reported 49% with SVF and wrongly concluded an "artifact/head mismatch". The real cause was a transpose bug — load_adapted_matrix inferred transpose from shape, so the square k/v_proj matrices ([1024,1024]) were loaded as Wᵀ instead of W. A transformers reference shared the same shape-inference bug, so both agreed at 49% (consistent, but both wrong). Fixing the transpose to be explicit (transpose=true for all 7 layer-26 linears, pinned by a square-matrix regression test) gives 100%/100%. Lesson: shape-inferred transpose is unsafe for square matrices, and a reference that shares your bug validates nothing.

Numeric strategy (revised by the artifact format): SVF is pre-applied in the published checkpoints, so serving loads patched matrices directly — no SVD on the hot path (and no offline svf-apply step needed for these weights). For routing only embed_tokens + layer-26 (8 matrices) matter; lm_head's patch is irrelevant (the LM head is never run). svf_reconstruct (nalgebra, cross-checked vs NumPy <1e-4) remains for the from-scratch / training path. The headline 95%/100% is re-validated via scripts/gen_router_golden.py → tests/golden.rs (gated; skips offline, hard-fails under FUGU_REQUIRE_GOLDEN=1).

Phase 1 hardening (post-review): Worker is fallible — a provider error aborts the run and returns 502 rather than a fake 200; the service has per-run concurrency limits (429 when saturated), a wall-clock timeout (504), HTTP connect/read timeouts on the pool, uuid request ids, standard usage token fields, trailing-slash tolerance, JSON 404, and graceful shutdown.

Heterogeneous worker pool + security (post-review): each slot is its own provider via model[@base][@ENV_KEY] (keys referenced by env var name, never in argv); a present-but-empty @ segment or an unset env var is a hard error (no silent wrong-key fallback). Secrets are kept out of logs/errors: Slot has a redacted Debug, the pool summary prints model@base only, and worker errors returned to the client are category-redacted (upstream returned HTTP 4xx), never the URL/headers. Per-request worker timeout (--worker-timeout-secs) is capped to the whole-run --timeout-secs so one upstream call can't outlive the run the client waits on; raise --timeout-secs for slow reasoning models.

Fugu-Ultra (Conductor) serving accepts both conductor output formats: the native 3-list DAG (general models) and the official 3B's ToolScale <answer> tool-call plan (mapped to a linear workflow). --conductor-format threelist|toolscale picks the prompt; parsing honors it (with the other format as fallback). A ToolScale name resolves to a worker only via an explicit in-range worker_<id> — a domain tool like 404/multiply is never misrouted.

Phase 0 deliberately carries zero ML or network dependencies (only regex), so the orchestration logic is fully testable offline — which is exactly where the load-bearing behavioural quirks live.

• fugu-core — constants (VEC_LEN=19456, …), Message/Role/RouteDecision/RunResult, the Router/Worker traits, verbatim prompts.
• fugu-workers — Worker impls: offline MockWorker/UltraMockWorker, the OpenAI-compatible ApiWorker (per-slot heterogeneous pool), and UserAgentConfig — a thin provider over a local UserAgent aggregator gateway (base normalization, /v1/models discovery, SSE delta.content accumulation, useragent:-alias mapping, friendly 401/connect/404 hints) that reuses ApiWorker wholesale.
• fugu-conductor — Fugu-Ultra (Conductor) line: conductor_prompt (asks an LLM to emit the workflow), the 3-list parser (model_id/subtasks/access_list) with the four-stage fallback (smart-quote → literal → CSV), visible_indices (topological, forward-ref-banned, "all"), and the DAG executor. End-to-end via openfugu ultra-demo.
• fugu-coordinator — the step_trinity loop: cold-verifier suppression, obs-updates-only-on-Worker, <suggested_role> override, ACCEPT/max-turns termination.
• fugu-cli — offline entry points mirroring the Python --self-test / --demo.
• fugu-serve — axum OpenAI-compatible /v1/chat/completions (+ /v1/models, /health). --model <qwen3 dir> loads the real candle Qwen3 router (resident, shared via SharedRouter behind a per-route lock — concurrent across requests); without it, the offline MockRouter. --slot-models switches the pool from mock to a real ApiWorker. Verified live: real routing returns one answer in ~2s/query (release, CPU), pool hidden.
• fugu-router — TRINITY router math (split / SVF / head / transcript) + the Backbone trait + FuguRouter. Weight-independent and fully tested.
• fugu-backbone-candle — real Qwen3-0.6B Backbone via candle 0.10 (loads base + optional SVF-adapted weights, returns the penultimate hidden state). verify37 bin = the Rust verify_37 acceptance (100%/100% with SVF).
• fugu-train — gradient-free self-training of the coordinator: a (μ/μ_w,λ)-ES with CSA (the isotropic limit sep-CMA-ES reduces to per docs §5.1) learns feature→specialist routing on a mock world. train-trinity bin: chance→optimal, 4/4 specialists. eval-orchestration bin: routing vs best-single vs random (synthetic, honestly labelled). Zero deps, runs anywhere.
• fugu-backbone-candle also ships bench-real — a REAL small-scale ground-truth benchmark (12 Qs across math/reason/logic/facts EN+ZH/code) over a real heterogeneous API pool via the candle router. Measured (DeepSeek + Qwen + GLM-4-Flash): single 100%/100%/17%, best-single 100%, Fugu routed 100% (tie — no headroom), but +28 pts over random (the router sent 0 calls to the 17% worker — dist [2,10,0]). Honest takeaways printed: no lift over best-single needs complementary models on harder Qs; routing did beat random by avoiding the weak worker, despite not being calibrated for this pool.

cargo test                                   # all crates
cargo run --bin openfugu -- conductor-selftest   # parity with `python openfugu/ultra.py --self-test`
cargo run --bin openfugu -- trinity-demo         # Fugu (TRINITY): offline step_trinity loop
cargo run --bin openfugu -- ultra-demo           # Fugu-Ultra (Conductor): prompt -> DAG -> execute

# serve Fugu as one OpenAI-compatible model (offline mock router + pool)
cargo run --bin fugu-serve -- --port 8088
curl localhost:8088/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"flatten a nested list in one line"}]}'

# REAL end-to-end: the trained candle Qwen3-0.6B router decides each turn
# (build --release; ~2s/query on CPU). Pool is mock here; add --slot-models for live workers.
cargo run --release --bin fugu-serve -- --model artifacts/qwen3-0.6b --port 8090

# HETEROGENEOUS pool — each slot its OWN provider/base/key, no aggregator.
# Slot syntax: model[@base][@ENV_KEY]; the key is an ENV VAR NAME (never in argv),
# a named-but-unset var is a hard error, and local bases need no key.
export OPENAI_API_KEY=sk-...   DEEPSEEK_API_KEY=sk-...
cargo run --release --bin fugu-serve -- --model artifacts/qwen3-0.6b --slot-models \
  "gpt-4o-mini@https://api.openai.com/v1@OPENAI_API_KEY,deepseek-chat@https://api.deepseek.com/v1@DEEPSEEK_API_KEY,qwen2.5:0.5b@http://localhost:11434/v1"
# omit @base/@ENV_KEY on an entry to fall back to --api-base + FUGU_API_KEY (single-provider pool)

# LOCAL aggregator: route over the UserAgent Node (one OpenAI-compatible gateway,
# 100+ models, one key — no cloud OpenAI, no per-provider keys in OpenFugu).
#   API_KEYS="sk-useragent-local:local" AXUM_BIND_ADDRESS="127.0.0.1:58700" axum_server
export USERAGENT_API_KEY=sk-useragent-local   # USERAGENT_BASE_URL defaults to http://127.0.0.1:58700/v1
cargo run --release --bin fugu-serve -- --model artifacts/qwen3-0.6b \
  --useragent-models "deepseek-v4-pro,qwen-3-7-max,zhipu-glm-5-2"
#   --useragent (no list) uses USERAGENT_MODEL (default `auto`, gateway picks the model)

# Fugu-Ultra serving: a Conductor LLM emits a workflow DAG, executed over the pool.
# (--conductor-model = the generating model; without it, offline uses a canned DAG.)
cargo run --release --bin fugu-serve -- --ultra \
  --conductor-model "deepseek-chat@https://api.deepseek.com/v1@DEEPSEEK_API_KEY" \
  --slot-models "gpt-4o-mini@https://api.openai.com/v1@OPENAI_API_KEY,..."

conductor-selftest reproduces the Python self-test output exactly (model_id [2,0,1], access_list [[],[0],[0,1]], dispatch gemini→gpt-5→claude).

• Conductor: forward references rejected (topological order, not arbitrary DAG); "all" ⇒ all earlier steps; >5 steps truncated, not errored; model_id % len(pool) wrap; smart-quote normalization + CSV fallback.
• Coordinator: only the SOLVER updates the router observation (via <reference_thought_N>); a Thinker's <suggested_role> overrides the next turn's role; a cold verifier is re-routed to Worker by default (suppress_cold_verifier), togglable to raw step_trinity behaviour; ACCEPT matched only as a leading prefix.

The parser models the real workflow domain — model_id ints, subtasks strings, access_list of ints/lists/"all" — via PyVal::{Int,Str,List}. It is not a full ast.literal_eval. Exotic Python literals a Conductor never emits fall back to the CSV path or are rejected rather than matched bit-for-bit: True/False/None, floats / scientific notation (1e2), and tuples. Integers larger than i64 degrade to a string (and an un-int-able model_id raises InvalidModelId) instead of Python's arbitrary precision. Common string escapes (\n \t \r) are decoded, matching ast.literal_eval.

Anything needing model weights or the network: the real FuguRouter (SVF/SVD — to be done offline at build time, never in the serving path), live/local worker pools, the HTTP server, and training. See the phase table.