refactor(inference): an instance does not do inference, and metering lives where someone else's key is spent - #2493

Draft design for hosted AI billing: credit pegged at $0.01, per-model cost
from a models.dev snapshot times a layered markup, chat pre-charges a
guaranteed ceiling and transcription settles under a cap, and self-host
inherits only the model catalog. Includes the profit invariant, worked
examples, a six-phase plan, and the four design-review resolutions
(tokenizer, per-request output cap, 1.5x-2x markup, assertion-gated sync).

Both AGENTS.md and CLAUDE.md declared Codex the sole executor and Claude an
advisory-only lane. That conflated the agent identity with the consult-claude
mechanism, which is read-only by construction, not because Claude cannot
execute. Remove the ownership sections entirely: a user-driven agent executing
its own work is the default and needs no statement, and the consult lane's
usage rules already live in the consult-claude skill description.

Keep the one genuinely Claude-specific note: the /codex:* plugin commands stay
user-invoked.

Phase 1 of the inference credit billing spec. The shared catalog
(@epicenter/constants/ai-providers) now carries only the product vocabulary
(id, provider, label) and no pricing, so the billing-agnostic library route
and self-host import it without touching cost.

Per-model cost moves to a new Cloud-only module
(apps/api/worker/billing/model-pricing.ts): a token-based cost table seeded
from models.dev, a layered markup, and the $0.01 credit peg, with
creditsForChat/transcriptionCredits for the settle path to come. reserveAiChat
charges INTERIM_FIXED_CHAT_CREDITS (the prior 2/10/2 values, relocated
unchanged), so this lands behavior-preserving; the UI drops the now-obsolete
fixed per-model credit display in favor of the Fast/Best tier.

Typechecks: constants, api worker, api/ui, app-shell, vocab. Billing tests 22/0.

…ed usage

Grounding (tokenizer research) found no workable offline Gemini tokenizer in a
Worker and an under-count risk for GPT with tool schemas, so exact pre-charge
is infeasible. Chat now reserves a rough pessimistic hold (byte heuristic) and
settles on the provider's authoritative returned prompt+completion tokens via
override_value: fail-closed and concurrency-safe from the hold, profit-safe and
fair from the settle. Supersedes the earlier pre-charge-ceiling-no-settle
decision; updates the decision table, chat flow, phase 3, and open questions 1
and 5.

Locks the metering model after the full design pass: one pattern shared by chat
and STT (gate on positive balance, run, settle the provider's actual returned
usage), overflow for a bounded netted negative, and a generous Autumn spend-limit
as a backstop rather than the primary bound. Drops the reserve lock, the
override_value settle, and all pre-call token estimation (no tokenizer). Grounds
negative-balance and per-model-tracking against Autumn. Marks Phase 1 done and
rewrites phases 2-6, decisions, architecture, and open questions to match.

Phase 3. Replaces the chat reservation lock with the gate/run/settle pattern
STT already uses. The library gateway requests stream_options.include_usage
(a standard field both providers normalize to prompt_tokens/completion_tokens;
self-host may ignore it). The Cloud policy gates on a positive balance, tees the
SSE reply through meterSSE (forward bytes unchanged, keep the last non-null usage
per the OpenAI/Gemini research), and settles the real per-token charge on the
after-response queue. A stream that ends without usage (client abort or a
mid-stream error frame) charges a nominal fallback, so a read-then-abort cannot
get free inference.

Grounding correction: the pinned autumn-js has no per-call overageBehavior, so
the settle is a plain track; enabling negative balances (overageAllowed +
overageLimit on ai_usage) is plan-config, deferred to Phase 6. gateAiChat and
settleAiChat replace reserveAiChat; the lock/finalize path, LOCK_TTL_MS, and the
interim fixed credits are gone.

Verify: server + api typecheck clean; billing suite 28/0 (new meter-sse parser
tests + rewritten chat policy tests); inference route tests 6/0.

…front

`packages/constants/src/ai-providers.ts` read as neutral shared
infrastructure while its content is a commercial fact: the models
Epicenter Cloud sells and the vendor lane each routes to. The module doc
even claimed "shared: library route + self-host", inviting a future
reader to put something there that a self-hosted instance must never see.

Rename the module and its exports to name what they are:

  ai-providers.ts  -> hosted-catalog.ts
  AI_MODELS        -> HOSTED_MODELS
  AiModel          -> HostedCatalogEntry   (avoids colliding with
                                            app-shell's HostedModel)
  AiProvider       -> HostedProvider
  ServableModel    -> HostedModelId        ("servable" was the library's
                                            word; the deployment decides)
  SERVABLE_MODELS  -> HOSTED_MODEL_IDS
  MODELS_BY_ID     -> HOSTED_MODELS_BY_ID
  toHostedCatalog  -> hostedCatalogFor

Behavior-preserving. `providerLabel` keeps its name: it reads well at
call sites and still degrades an unrecognized historical id to raw text.

Typechecks clean across constants, server, api, api-ui, and vocab;
catalog tests 2/0 and billing tests 28/0.

An instance mounts the library's inference gateway, which validates
against Epicenter Cloud's catalog and routes through Cloud's two-vendor
table. So a self-hoster with their own OpenAI key can serve the three
ids Epicenter sells and nothing else: their key, our storefront.

The pass started on a seam that would let a deployment supply its own
routing table, and found the gateway has almost no justification left.
Reachability is Tailscale's. Model breadth is the client's connection
registry, which already holds N endpoints with per-connection keys and
/v1/models discovery. Multi-vendor routing is LiteLLM's. And browsers can
reach providers directly: preflight confirms OpenAI and Gemini reflect
the request origin and allow `authorization`, and Anthropic answers
`Access-Control-Allow-Origin: *`.

What remained was letting people spend the operator's key without
holding it, which providers now serve better with per-user keys.

ADR-0264 records the decision and the rule underneath it: metering exists
exactly where someone else's key is spent, which is why an instance has
no billing rather than merely lacking one.

Also corrects ADR-0054 in place. Its consequence "a browser cannot reach
a raw provider API cross-origin (CORS)" is false for every provider in
the catalog, and it was load-bearing for an accepted loss.

The spec sequences chat first and gates transcription on a client-side
STT connection, because the client has no STT equivalent of the inference
registry and deleting self-host STT first would leave a self-hoster with
no transcription path off the desktop host.

Both entries drop `mountInferenceApp` and `mountTranscriptionApp` with
their rate-limit policies, so an instance is identity, sync, and storage
(ADR-0264). It holds no provider house key and imports none of Cloud's
model catalog.

The gateway existed to let a group spend the operator's key without
holding it. Everything else it seemed to provide is served better
elsewhere: the client's connection registry already holds N endpoints
with per-connection keys and /v1/models discovery, Tailscale handles
reachability, LiteLLM and OpenRouter handle multi-vendor routing, and
browsers reach every provider in the catalog directly. Providers now
issue per-user keys with their own spend limits, which also gives the
payer per-person accounting an instance cannot (one principal, ADR-0075).

Transcription ships with chat rather than after it. An earlier draft
gated STT on building a client-side STT connection; that was wrong. One
Connection drives both, because the client appends /chat/completions or
/audio/transcriptions to the same base and `transcribe()` takes the same
ResolvedConnection. Vocab and Whispering already resolve their STT model
through that registry.

The parity test inverts both rows to `absent` with a `why` citing the
ADR, so the surfaces stay pinned on both runtimes. README and
wrangler.jsonc replace the house-key sections with how to configure a
client connection, and state the one accepted loss plainly: a shared
provider key works everywhere, not just against your box.

self-host typechecks on both runtimes; runtime-profile 4/0, self-host
suite 21/0, server and api typecheck clean. The one server-suite failure
(`cloudflare:test` in workers/e2e.test.ts) is pre-existing and
environmental, confirmed against a clean tree.

The inference-backend line listed 'a self-hosted gateway (your key or a
local model)' as one of the swappable base URLs, which reads as though an
Epicenter instance is one of them. It is not (ADR-0264): a gateway you
run means Ollama, LiteLLM, or Speaches.

A refusal without a reopen condition is hard to revisit honestly. Names
the two pieces of evidence that would invert it (per-user key
provisioning proving impractical for real shared instances; self-host
onboarding stalling at connection setup), scopes both to the dumb-pipe
alternative rather than the routing seam, and states that a request for
multi-provider routing is not a trigger.

ADR-0056 put every local engine behind the OpenAI wire and named three
routes, and it contemplates an engine running in-process (Core ML,
embedded whisper.cpp). But an in-process engine has no URL, so it was the
one inference lane reached by `invoke`, which forced every consumer to
carry a second shape for it: Whispering's dispatch has a `bespoke` kind
alongside `wire`, and each app hand-wires local models.

ADR-0265 gives it a URL without giving it a socket. The host registers a
Tauri custom URI scheme, which is fetchable from the app's own webviews
and, per Tauri's docs, is not registered with the system, so no other
process and no web page can reach it. No port, no bind lifecycle, no
CORS, no token to invent.

Rejected the two obvious alternatives: a loopback server (reachable by
every process, and the cases it uniquely serves already belong to Ollama
and Speaches), and a synthetic invoke-backed fetch (more code than the
bespoke closure it replaces, all of it simulating HTTP).

The scheme is a transport; its route list follows what the host actually
runs. Transcription plus /v1/models now, /v1/audio/speech if TTS ever
lands in-process, and never /v1/chat/completions, because ADR-0056
delegates local chat to Ollama.

Marked Unbuilt: apps/epicenter is an ADR-0227 rebuild in progress. The
point of recording it now is that the rebuild should not re-derive it or
reach for `invoke` by default.

`hosted.baseURL` is Cloud's regardless of the selected instance, while
`hosted.fetch` carries whatever credential that instance yields. So a
self-host session sent an instance bearer to Cloud and got a bare 401,
and `resolveOrHosted` fell back to that transport unconditionally. The
bug predates ADR-0264; deleting the instance gateways removed the local
fallback that was hiding it.

`hosted` is now optional. Omitted, its candidate leaves the resolution
list and `hostedModels` reads empty, so the picker's Epicenter group
disappears. Vocab passes it only when `instanceSetting.isDefault()`.

`resolveOrHosted` collapses into `resolve(model): ResolvedConnection |
null`. Two names for one predicate become one, and a fallback that
shipped an unservable id to Cloud to "error loudly" becomes an honest
null on a path `canServe` already blocks.

Found while removing the fallback: Vocab's dictation reached `whisper-1`
THROUGH it, because STT ids are not in the chat picker catalog, so
nothing "served" it. Removing the fallback would have silently broken
dictation. Added `hostedAlsoServes` so the hosted transport declares the
ids it serves that nobody picks, turning an accident into a statement.

Added `TranscribeError.NoConnection({ model })`, because a caller that
resolves a transport first needs to report "nothing serves this" without
inventing an HTTP status for a request that never happened.

The picker now names its empty state when a device has no hosted
transport and no connections, rather than showing a blank list whose only
signal is a failed send.

Typechecks clean across app-shell, client, vocab, constants, server, api,
api-ui, self-host; app-shell + client 46/0, billing + self-host 49/0.

The three wire functions took the connection in three different
positions: `complete(conn, opts)`, `transcribe(audio, conn, opts)`, and
`listModels(conn)`. Same argument, same wire, no reason for the
disagreement.

Normalize `transcribe` to take the connection first like its siblings,
then add `createInferenceClient(resolved)` so a call site that has
already resolved a transport does not repeat it:

  const client = createInferenceClient(connections.resolve(model))
  await client.listModels()
  await client.transcribe(audio, { model })

Deliberately thin. Every method forwards to the free function of the same
name, and those stay exported for callers holding a connection briefly.
The client spreads `fetch` and `baseURL` through, so anything consuming a
bare transport (the agent engine's `data()`) takes a client unchanged.

Uses it at the call sites that already resolve first: Vocab's dictation
and entry candidates, and the registry's discover/refresh.

client, app-shell, vocab, and whispering typecheck; client 40/0,
app-shell + client suites green.

Phase 3 was planned as deleting `rateLimit`, since all four call sites
were self-host's and ADR-0264 called it orphaned. Reading the billing
spec first inverted it.

That spec bounds bad debt as "the per-call cap times the rate-limit". But
Cloud mounted only `chargeOpenAiCreditsWithAutumn`, so no rate limit
existed and the stated bound was not true. Chat settles AFTER the call
with `overageBehavior: "overflow"`, so N concurrent calls at exhaustion
each run before any settles, with nothing bounding N. Deleting the
primitive would have removed the thing the design names as one of its
three bounds.

Mounts `rateLimit({ requests: 120, windowSeconds: 60 })` per principal on
both Cloud gateways, ahead of the Autumn policies.

Also rewrites the middleware doc for its actual consumer. It described
itself as the instance's backstop "sized for the small trusted group the
instance targets", and said durable limiting at scale was Cloud's concern
and not this primitive. An instance no longer does inference at all, and
Cloud is now the only caller, so the doc says what it really is: a bound
on one-time overshoot at exhaustion, per-isolate and therefore
approximate on Cloudflare, explicitly not a sustained-abuse defense.

120/60s matches what self-host used and is a starting value, not a
researched one. It belongs with markup, the output cap, and the spend
limit as a dial to set against real data.

api typechecks; billing + middleware suites 47/0.

Mounted this at 120/60s, then read the implementation. It is a fixed
window, so it permits its entire quota at once: 120/60s allows a
120-call burst. The failure being bounded is exactly a burst, calls fired
together at exhaustion each passing the gate before any settles, so the
window was doing almost nothing for its stated purpose.

10/5s keeps the same sustained rate and caps the burst at ten.

Also corrects the scale I claimed. Reading the plan config rather than
the spec's edge-case prose: Pro bills overage as revenue
(`overage: { priceUsd: 1, billingUnits: 100 }`), and Free sells no
overage and is gated to `freeEligible` models by `gateAiChat`. So real
bad debt is a free user's burst at exhaustion on a cheap model, which is
cents. This is cheap insurance, not a designed ceiling, and the comment
now says so.

Watch for 429s on legitimate agent loops; ten concurrent per principal
should be generous, since a loop is sequential per conversation.

`bun install` fails on a fresh clone, and has been failing on main: the
postinstall runs `build:declarations`, which runs `bun run --cwd
packages/data build`, and `packages/data` has no `build` script.

`build:declarations` exists to emit `.d.ts` for packages that publish
`./dist`. `packages/field` does, so it stays. `packages/data` exports
source (`"." : "./src/index.ts"`), so it has nothing to build and never
had a `build` script. It inherited the slot from `packages/lens`, which
did publish `./dist` before the rename.

Verified by running `bun install --frozen-lockfile`, the exact command CI
fails on.

Unrelated to the rest of this branch; it is here because it blocks CI on
every PR. Cherry-pick it out if you would rather land it on its own.