Mubit on CL-Bench

Mubit is the only memory system that consistently beats in-context learning on the Continual Learning Bench — #1 and #2 on the board, measured across seven models.

Benchmarking Mubit as an assistive memory system on the Continual Learning Bench (CL-Bench) — the expert-validated benchmark that measures whether AI agents genuinely improve through sequential experience.

The CL-Bench paper's headline finding: dedicated memory systems (Mem0, ACE, ICL Notepad) all underperform naive in-context learning. Mubit breaks this pattern.

Seven full suites measured — gemini-3.7-flash, gemini-3.5-flash, gemini-3.1-pro-preview, gpt-5.4, gpt-5.4-mini, gemini-3.1-flash-lite, and gemini-3-flash-preview. Sequential evaluation, a fresh Mubit instance per task, 5 runs × 6 tasks per entry.

• The entire podium — Mubit · Gemini 3.7 Flash (+0.255), Mubit · Gemini 3.5 Flash (+0.238), and Mubit · Gemini 3.1 Pro (+0.230) hold the top three spots, 17% clear of the best non-Mubit entry (ICL · Claude Sonnet 4.6, +0.196). All numbers computed by the benchmark's own pipeline (scripts/generate_leaderboard.py) on the submitted artifacts; the formulas were validated by reproducing every official leaderboard entry to its exact published number.
• Mubit · GPT-5.4 beats ICL · GPT-5.4 on its own model (+0.191 vs +0.189) at lower cost ($14.49 vs $18.39) — making it the cheapest GPT-5.4 system on the board.
• The cheapest models on the board land in the top 10. Mubit · GPT-5.4 Mini (+0.135, $4.94) and Mubit · Gemini 3.1 Flash-Lite (+0.134, $4.08) both outrank every ICL Notepad entry — including ICL Notepad · Claude Sonnet 4.6 at $31.53 — and are the two cheapest full suites on the board (cheapest official entry: $7.60).
• No other system is positive on all six tasks — Mubit is, on five of seven models (gemini-3.7-flash, gemini-3.5-flash, gemini-3.1-pro-preview, gpt-5.4, gpt-5.4-mini); flash-lite is positive on five of six. The one clearly negative task across all suites is gemini-3-flash-preview poker — where the stateless model posts the strongest natural poker baseline measured (+240 raw) and the session-stateful memory configuration reduced it. Memory lifts every model that starts weak; on the board's one naturally-strong poker player it got in the way. Every official entry has at least one negative task.
• Costs are token-derived from run telemetry (provider list prices incl. cache-read rates) under the board's accounting — sum across tasks of mean per-run cost; this method reproduces ICL·GPT-5.4's published $18.39 exactly.

The pattern: Mubit leads outright on the three tasks with the strongest continual-learning signal (forecast transfer, spectrum registry, epidemiological accumulation) and is positive-but-not-leading on the other three — where every competitor that scores higher does so from a much lower baseline on a stronger model. No other system on the board is positive on all six.

The BSM task requires an agent to build an increasingly accurate model of a radio frequency band across 90 sequential scans. Transmitters persist across scans — memory is essential, and the learning curve is directly observable:

Mubit · gemini-3.7 climbs from the stateless baseline (0.22) through quartile means of 0.40 → 0.50 → 0.62 → 0.68, ending at ~0.70 per-scan. Suite mean 0.554 (best run 0.582) vs the official field's best stateful ~0.51. Normalized gain +30.1 — the largest single-task gain on the board.

The task introduces a schema migration halfway through (tables renamed, columns reformatted). This is the critical test: when the world changes, does memory help or hurt?

The baseline loses ~75% of its reward when the schema shifts; Mubit's arm holds — and on GPT-5.4 actually improves post-migration. Distilled semantic lessons ("product attribute data lives in the attrs table family, keyed by ref_id") survive renames that break surface-level extracted facts. Suite gains: +1.08 (gemini-3.7), +0.80 (gpt-5.4).

Every official system scores negative bits in the stateful arm (ICL −0.028, Mem0 −0.034, ACE −0.004, Codex −0.038). Mubit · gemini-3.7 posts +0.006 bits/study — small, but the only absolutely-positive stateful score on the board, and +0.120 normalized.

56 of 60 stateful runs beat their own baseline: 30/30 on Gemini 3.7 Flash, 26/30 on GPT-5.4. The four exceptions are one database run (−0.003, noise), one codebase run (−0.03), and two poker runs on GPT-5.4 (cards run cold). Sales: 5/5 above baseline on both models. The Gemini 3.5 Flash suite repeats the pattern — positive on all six tasks.

Gemini 3.1 Pro Preview sits dead last on the official board with ICL (−0.076 reward, +0.036 gain) — the weakest continual learner measured. With Mubit it completes all six tasks positive and lands #3 overall (+0.230) — 6.4× its same-model ICL entry, and above every non-Mubit system on the board.

The pattern this adds to the other suites: the weaker a model's own continual learning, the more of the work the memory layer does. The officially-worst CL model produces a top-3 sales result, the best poker result of any memory system measured, and a top-band BSM score — entirely from the memory layer.

Measurement note: the database and codebase tasks required a hardened transport (streaming responses plus a capped-output/thinking-budget retry ladder) after repeated provider-side 504 DEADLINE_EXCEEDED failures on the standard non-streaming path — this model's thinking-first architecture (21–31s time-to-first-token) combined with long verbose episodes exceeds the server's per-request deadline when responses are delivered in one shot. Model, prompts, memory configuration, and scoring are identical to the other suites; the change is transport-only. The other four tasks ran on the standard path.

Mubit uses 83.5% fewer tokens than context replay — while out-scoring every system on the leaderboard.

An agent without memory must either start every task from scratch or re-read its entire history first (ICL). Re-reading works until the history grows — then you're paying to re-send millions of tokens the model has already seen. Mubit instead distills each task into short lessons and retrieves only the relevant few before the next one. Same model (GPT-5.4), same tasks, same 5-run protocol, measured from per-run token telemetry:

The memory usually pays for itself: on 3 of 6 tasks Mubit's runs used fewer tokens than even a memoryless baseline, because one distilled lesson saves the agent a dozen exploration steps. On Gemini 3.7 Flash the same system runs 22.5M tokens per suite ($18.16) — it spends extra tokens only on poker, where a persistent session tripled the reward for the weaker base model, and saves them elsewhere. Full data: token_savings_measured.json.

Extrapolated over longer task sequences (calibrated so the curves reproduce the measured 6-task values exactly), the gap widens fast — re-reading history grows quadratically, retrieval grows linearly:

Both suites are input-dominated — these are action-selection workloads, so the cost difference between systems comes down to how much history gets re-sent as input, not generation volume.

Each Mubit system is a subclass of CL-Bench's ContinualLearningSystem. The core loop:

1. Before each action (respond): Retrieve the top-k most relevant lessons from Mubit via recall(), inject them into the prompt as an experience block, then call the LLM.

2. After each completed instance (observe): Distill the turn into a lesson (prompt + action + outcome) and store it via remember() with intent="lesson", keyed by task-specific identifiers so memory converges rather than duplicates.

3. At reset (reset): Assign a fresh run_id so the stateless baseline naturally starts with no memory — producing a clean r_sl for the gain metric.

mubit-cl-bench/
├── systems/
│   ├── mubit/              # Core Mubit memory system
│   ├── mubit_genai/        # + native Google genai structured outputs
│   ├── mubit_bsm/          # + structured transmitter registry (BSM)
│   ├── mubit_poker/        # + session-stateful poker variant
│   ├── mubit_tuned/        # + minimal-injection variant for strong models
│   └── mubit_full/         # + reinforcement + reflect()
├── results/                # Viewer artifact JSON.gz files
├── charts/                 # Generated charts + data (all regenerated from the submitted runs)
├── chart_data.json         # Per-task datapoints for both submissions
├── scripts/
│   └── analyze_results.py
├── LICENSE
└── README.md

1. Python 3.13+ and uv
2. CL-Bench cloned and installed:

   git clone https://github.com/pgasawa/continual-learning-bench
   cd continual-learning-bench
   uv sync --all-extras

3. A Mubit API key (see below)
4. An LLM API key (Gemini or OpenAI)

CLBENCH=/path/to/continual-learning-bench
cp -r systems/* $CLBENCH/src/systems/
cd $CLBENCH && uv pip install mubit-sdk

export GEMINI_API_KEY=your-gemini-key      # or OPENAI_API_KEY for gpt-5.4
export MUBIT_API_KEY=your-mubit-api-key
export MUBIT_ENDPOINT=https://api.mubit.ai
clbench setup --all

# gemini-3.7 sales — 5 runs, fresh memory per run group
clbench run sales_prediction --schedule default \
  --system mubit_genai --system.model gemini-3.7-flash --runs 5

# gpt-5.4 poker — minimal-injection variant
clbench run exploitable_poker --schedule default \
  --system mubit_tuned --system.model gpt-5.4 --runs 5

# baselines for the same task/model
clbench run sales_prediction --schedule default \
  --system icl --system.model gemini-3.7-flash --runs 5

Submitted protocol: one task at a time, sequential, a fresh Mubit instance per task (wipe the data dir between tasks so retrieval never crosses task boundaries).

g_b = (r̄_sf − r̄_sl) / (r_max − r̄_sl)

Where r̄_sf is mean stateful reward, r̄_sl mean stateless reward, and r_max the maximum achievable. Gain isolates learning from experience from base-model capability. The leaderboard's Agg. Reward anchors each task to a fixed ICL·GPT-5.4 baseline instead.

The benchmark runs against a hosted Mubit instance — no local setup required.

1. Sign up at console.mubit.ai and create an instance.
2. Copy your API key from the console (format: mbt_...).
3. Export it before running:

export MUBIT_API_KEY="mbt_your_key_here"
export MUBIT_ENDPOINT="https://api.mubit.ai"   # default; override only if you run your own gateway

The SDK picks up both variables automatically. That's the only Mubit configuration the benchmark needs.

• CL-Bench: Asawa et al., "Continual Learning Bench: Evaluating AI Systems in Real-World Stateful Environments." arXiv:2606.05661, 2026.
• Mubit: The Mubit cognitive memory system for agentic AI.

MIT