Skip to content

refactor(usage): capture hosted pricing once per operation - #157838

Merged
obviyus merged 4 commits into
mainfrom
fix/remote-catalog-without-restart
Sep 25, 2026
Merged

obviyus merged 4 commits into
mainfrom
fix/remote-catalog-without-restart

Conversation

@obviyus

@obviyus obviyus commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Related: #140086

What Problem This Solves

Makes each usage-cost operation retain one hosted pricing context, preparing usage accounting for catalog updates that can occur while an operation is running.

User Impact

No user-visible change in this PR. Catalog downloads still require a Gateway restart to activate. Existing recorded costs, configured overrides, local model prices, and tier handling keep their current behavior. There are no configuration, protocol, SDK, or storage changes.

This is the maintainer-requested accounting prerequisite for live catalog publication, not the activation change itself.

Why This Change Was Made

Usage refresh previously selected its pricing fingerprint and each lazily resolved model price independently. The refresh now captures the hosted pricing context once and uses it for both. Diagnostic readers retain lazy preparation and capture their context when estimation first becomes necessary; subsequent model lookups retain that context.

The ordinary synchronous pricing APIs still use the same resolver and preserve their existing lookup behavior. Fingerprint-only reads do not allocate an unused captured resolver.

Evidence

  • Built Gateway, base and candidate: downloaded a synthetic catalog through openclaw models refresh, imported two unpriced transcript records through openclaw doctor --session-sqlite import, then called the public sessions.usage.logs and sessions.usage methods. Base 05edd36e12 and current candidate c95b7d0942e6 both reported individual costs $1 and $4, and total cost $5. Each ran with separate state, home, configuration, profile, and loopback ports. No provider inference was performed.
  • Mid-operation regression: after selecting a new pricing context, an existing operation retains the old price for a model it has not looked up yet and retains its fingerprint; a new operation uses the new prices and fingerprint. Replacing the captured lookup with the previous per-lookup selection makes this test fail with expected 4, received 40. This is a controlled pricing-publication test, not a claim that this PR activates downloaded catalogs.
  • Focused suites: 240 tests across 13 pricing, usage-format, usage-reporting, and usage-worker files passed during preparation. The operation regression was then moved into its own usage-pricing suite to avoid growing the existing pricing suite beyond its line cap; that relocated case and focused lint passed. The current head passed the built Gateway parity check again.
  • Measured cost: the relocated session-cost-usage-pricing-context.test.ts regression took 36 ms on the current head (21.56 s shard including cold transform). The earlier three-file command took 108.00 s; the remaining ten-file command took 101.34 s. Exact-head CI passed, with all 147 jobs completed and no failures.

Documentation is unchanged because the existing restart contract remains in force.

@clawsweeper

clawsweeper Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@obviyus
obviyus marked this pull request as ready for review September 25, 2026 02:47
@clawsweeper clawsweeper Bot added P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 25, 2026
@clawsweeper

clawsweeper Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 25, 2026, 1:38 AM ET / 05:38 UTC (Revision 5).

ClawSweeper review

What this changes

Usage accounting captures one hosted pricing context for a refresh’s fingerprint and model cost lookups, while diagnostic readers capture it at their first estimate; a regression test covers a context change between lookups.

Merge readiness

✅ Ready for maintainer review

Current main still lacks the operation-scoped pricing invariant. The focused PR has no actionable correctness finding, and the author’s explicit sponsorship statement is backed by a verified repository maintain role.

Priority: P3
Reviewed head: c95b7d0942e6a8b1455ef59d51a715c9439698e0

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) The focused patch, controlled regression, Gateway parity evidence, and exact-head CI support a normal good review rating.
Proof confidence 🐚 platinum hermit (4/6) Not applicable: The repository maintain-role author is exempt from the external-contributor proof gate. The PR reports a built Gateway run through public session usage methods with unchanged $1 and $4 estimates totaling $5, while the controlled test exercises the new mid-operation capture invariant; downloaded catalog activation remains restart-based. No stored-data contract changes.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: The repository maintain-role author is exempt from the external-contributor proof gate. The PR reports a built Gateway run through public session usage methods with unchanged $1 and $4 estimates totaling $5, while the controlled test exercises the new mid-operation capture invariant; downloaded catalog activation remains restart-based. No stored-data contract changes.
Evidence reviewed 10 items Introduced refresh behavior: The refresh now uses one prepared pricing context for both the fingerprint and later model cost requests.
Existing pricing precedence: The captured hosted context enters the existing resolver, which still checks local models.json and configured prices before inherited catalog prices.
Controlled regression: The added test changes the selected context between model lookups and checks the old operation’s unlooked-up price and fingerprint against a new operation.
Findings None None.
Security None None.

How this fits together

Hosted catalog prices and configured or local prices feed estimates for session transcripts. The usage worker refreshes cost rollups, which the Gateway exposes through session usage reports.

flowchart LR
  A[Hosted catalog prices] --> C[Operation pricing capture]
  B[Configured and local prices] --> D[Cost resolver]
  C --> D
  C --> E[Pricing fingerprint]
  F[Session transcripts] --> G[Usage refresh and reports]
  D --> G
  E --> G
Loading

Before merge

None.

Agent review details

Security

None.

PR surface

Source +37, Tests +74. Total +111 across 5 files.

View PR surface stats
Area Files Added Removed Net
Source 4 67 30 +37
Tests 1 74 0 +74
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 5 141 30 +111

Review metrics

None.

Technical review

Best possible solution:

Keep the single-context accounting invariant in the existing pricing owner, then handle live catalog publication separately with proof that model rows and prices activate together.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR establishes an internal invariant rather than reporting a current user-visible failure. Its controlled regression changes the selected pricing context between lookups.

Is this the best way to solve the issue?

Yes: the change uses the existing pricing owner to align the refresh fingerprint with subsequent hosted price estimates while preserving local-price precedence.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 63944b8a1df1.

Labels

Label changes:

  • remove proof: sufficient: Current real behavior proof status is not_applicable, not sufficient.

Label justifications:

  • P3: This is a focused internal accounting prerequisite with no immediate user-visible behavior change.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The repository maintain-role author is exempt from the external-contributor proof gate. The PR reports a built Gateway run through public session usage methods with unchanged $1 and $4 estimates totaling $5, while the controlled test exercises the new mid-operation capture invariant; downloaded catalog activation remains restart-based. No stored-data contract changes.

Evidence

What I checked:

Likely related people:

  • unknown: The claimed source-line change could not be verified from bounded local history. (role: source history unknown; confidence: low)
  • obviyus: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • vincentkoc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (4 earlier review cycles)
  • reviewed 2026-09-25T02:54:10.671Z sha 0b862a4 :: needs maintainer review before merge. :: none
  • reviewed 2026-09-25T03:13:32.216Z sha b9c8e16 :: blocked before merge. :: none
  • reviewed 2026-09-25T04:33:00.400Z sha ba0edff :: blocked before merge. :: none
  • reviewed 2026-09-25T05:04:04.975Z sha c95b7d0 :: blocked before merge. :: none

@obviyus

obviyus commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@obviyus

obviyus commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Sep 25, 2026
@obviyus

obviyus commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

Maintainer sponsorship (Ayaan): approved. Each usage operation captures one immutable pricing context before any per-model lookup, so a later catalog/pricing publication cannot mix prices within that operation. This is the prerequisite for applying new remote catalogs without a Gateway restart (rows and prices must switch together). No visible behavior change intended.

@obviyus

obviyus commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Sep 25, 2026
@obviyus
obviyus merged commit 73d1fc7 into main Sep 25, 2026
208 checks passed
@obviyus
obviyus deleted the fix/remote-catalog-without-restart branch September 25, 2026 06:03
@obviyus

obviyus commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

Landed as 73d1fc7. Usage-cost operations now retain one hosted pricing context for their fingerprint and lazy estimates; catalog activation behavior is unchanged here. The isolated built-Gateway parity proof reported $1 and $4 estimates totaling $5, and the mid-operation regression preserves the earlier context after a pricing change. Exact-head CI and ClawSweeper review passed. Thanks for confirming the pricing invariant and sponsorship.

vincentkoc added a commit that referenced this pull request Sep 25, 2026
…4-prep

* commit '8fe893da516ebdc817692fb2d272e27bc3c88376': (31 commits)
  fix(test): wait for reload admission before restart polling (#157964)
  refactor(mattermost): reuse DM retry option selection (#157800)
  chore(i18n): refresh native locales (#157850)
  perf(sessions): reuse current participants across transcript appends (#157622)
  refactor(signal): share pending transport migration results (#157777)
  fix(tui): open model pickers without waiting for the Gateway (#157839)
  fix(gateway): apply config despite delayed file notifications (#157906)
  fix(auth): clear deleted account selections and support account recovery (#157745)
  fix: plugin reloads lose tool results and strand subagent settlement (#157339)
  fix(gateway): avoid preview errors during worker metadata updates (#157943)
  refactor(diffs): reuse native promisify for compression (#157835)
  fix(models): keep current choices on the first picker page (#157869)
  fix(plugins): retain existing quarantine across unrelated reloads (#157907)
  fix(openai): prioritize ChatGPT sign-in and clarify auth choices (#157792)
  chore(ui): refresh control ui locales (#157955)
  fix(ui): refresh connected accounts after auth changes (#157950)
  fix(scripts): avoid redundant npm version process in package checks (#157809)
  refactor(usage): capture hosted pricing once per operation (#157838)
  refactor(ui): share slash-command failure results (#157649)
  fix(cron): announce runs miss the channel's formatting rules (#157877)
  ...
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Sep 26, 2026
…157838)

Related: openclaw#140086

## What Problem This Solves

Makes each usage-cost operation retain one hosted pricing context, preparing usage accounting for catalog updates that can occur while an operation is running.

## User Impact

No user-visible change in this PR. Catalog downloads still require a Gateway restart to activate. Existing recorded costs, configured overrides, local model prices, and tier handling keep their current behavior. There are no configuration, protocol, SDK, or storage changes.

This is the maintainer-requested accounting prerequisite for live catalog publication, not the activation change itself.

## Why This Change Was Made

Usage refresh previously selected its pricing fingerprint and each lazily resolved model price independently. The refresh now captures the hosted pricing context once and uses it for both. Diagnostic readers retain lazy preparation and capture their context when estimation first becomes necessary; subsequent model lookups retain that context.

The ordinary synchronous pricing APIs still use the same resolver and preserve their existing lookup behavior. Fingerprint-only reads do not allocate an unused captured resolver.

## Evidence

- **Built Gateway, base and candidate:** downloaded a synthetic catalog through `openclaw models refresh`, imported two unpriced transcript records through `openclaw doctor --session-sqlite import`, then called the public `sessions.usage.logs` and `sessions.usage` methods. Base `7353f7e382` and current candidate `c95b7d0942e6` both reported individual costs **$1 and $4**, and total cost **$5**. Each ran with separate state, home, configuration, profile, and loopback ports. No provider inference was performed.
- **Mid-operation regression:** after selecting a new pricing context, an existing operation retains the old price for a model it has not looked up yet and retains its fingerprint; a new operation uses the new prices and fingerprint. Replacing the captured lookup with the previous per-lookup selection makes this test fail with **expected 4, received 40**. This is a controlled pricing-publication test, not a claim that this PR activates downloaded catalogs.
- **Focused suites:** 240 tests across 13 pricing, usage-format, usage-reporting, and usage-worker files passed during preparation. The operation regression was then moved into its own usage-pricing suite to avoid growing the existing pricing suite beyond its line cap; that relocated case and focused lint passed. The current head passed the built Gateway parity check again.
- **Measured cost:** the relocated `session-cost-usage-pricing-context.test.ts` regression took **36 ms** on the current head (21.56 s shard including cold transform). The earlier three-file command took 108.00 s; the remaining ten-file command took 101.34 s. [Exact-head CI passed](https://github.com/openclaw/openclaw/actions/runs/36095336705), with all 147 jobs completed and no failures.

Documentation is unchanged because the existing restart contract remains in force.

Co-authored-by: Ayaan Zaidi <hi@obviy.us>
obviyus added a commit that referenced this pull request Oct 1, 2026
…8000)

Related: #140086, #141885, #156535, #157838

## What Problem This Solves

A running Gateway doesn't see a newly downloaded hosted model catalog until it restarts. This PR publishes each accepted catalog through the existing prepared-runtime owner, without a restart. Model rows and their prices switch together as one generation, which keeps the invariant from #140086.

## User Impact

- Compatible downloads are adopted at the Gateway's background catalog check, or after an explicit `models.list` refresh. That refresh returns the currently accepted rows right away and runs adoption afterward.
- A turn admitted on catalog N keeps N's rows and prices until it finishes. New turns use N+1. Rows and prices are never mixed.
- A concurrent auth or config publication no longer postpones adoption to the next scheduled check (up to 6 h). Adoption waits for that publication to settle, then retries, up to 3 attempts.
- An owner whose build failed or timed out ends the adoption instead of waiting on unbounded work. Gateway shutdown cancels an adoption that is still preparing.
- Malformed, schema-invalid, too-new (`minVersion`) and older catalogs are rejected, and the previously accepted catalog stays in use.
- Changing `models.catalogRefresh.url` no longer needs a restart: the previous source's catalog stops applying, and the mirror's catalog is adopted at the next catalog check.

**Bad-catalog exposure:** with live apply, a *valid but wrong* published catalog reaches running Gateways at their next catalog check (at most every 6 h) or on the next explicit `models.list` refresh. It no longer waits for a restart. Recovery uses existing mechanisms only:
- Republish a corrected catalog with a newer `generatedAt`; Gateways adopt it the same way.
- Operators can set `models.catalogRefresh.enabled: false`, which withdraws remote rows and prices without a restart (covered by the Gateway integration test).

This PR adds no new kill switch, config option or env knob.

### Compatibility

No config keys, defaults, types, validation, stored rows, protocol or SDK contracts change. The only config-surface change is the `models.catalogRefresh.url` help text, which drops the stale "Changes apply after a Gateway restart" sentence, and its regenerated config-doc baseline hash. Existing configs validate unchanged and need no Doctor migration (maintainer confirmation: #158000 (comment)). Startup behavior is unchanged. Upgrade impact for existing installs: an accepted download activates at the next catalog check instead of the next restart.

## Why This Change Was Made

- `prepared-model-runtime.configured-refresh.ts` builds a complete candidate generation of the configured owners under the new catalog. One serialized commit then publishes rows, the accepted bundle, the pricing context and the reply-dispatch projection together.
- Adoption re-reads the stored catalog until the config it read under is still current, so a stale caller can't cancel a current adoption.
- Each preparation attempt has its own abort signal. A config advance restarts only the attempt; a newer catalog or shutdown ends the whole adoption, including pricing preparation.
- Between attempts, adoption waits only on publication gates: a pending replacement or an owner's pending publication.
- Adopted owners install the same plugin-retirement recovery as configured publication (#161267). After commit, a lost Gateway plugin loan republishes them through the normal recovery. Before commit, it restarts the adoption attempt, and the commit refuses any candidate whose plugin generation retired.

### Why downloads were restart-only, and what this keeps

Restart-only activation was a mechanism, not the goal. #140086 chose it to stop rows and prices from different catalog versions mixing, and #157838 was merged as "the prerequisite for applying new remote catalogs without a Gateway restart (rows and prices must switch together)". This PR is that follow-up. Every requirement those PRs set still holds:

| Original requirement | Source | How it holds here |
|---|---|---|
| Rows and prices from one catalog version; never mixed across reloads or new requests | #140086 | One serialized commit publishes owners, bundle, pricing context and dispatch; pricing contexts are keyed by the exact accepted catalog. Integration test: new rows appear only with new prices |
| Admitted work keeps its pair | #140086, #157838 | Runs carry their plugin generation's catalog; usage operations capture one pricing context. Integration test and live proof: the in-flight turn keeps the old price |
| Startup absence is a real state (no downloaded rows without prices) | #140086 | Absence → catalog goes through the same atomic commit; overlay absence tests unchanged |
| Worker replacement inherits the host's accepted pair, not a later download | #140086 | The commit updates the inherited pair; later workers and a worker-exit recovery keep it (overlay and integration tests) |
| Current enablement and source URL still gate eligibility | #140086, #156535 | Checked on every read and before adoption, including the default-install v1 fallback; disablement withdraws rows and prices together (integration test) |
| Bad or superseded downloads never replace the active pair | #140086, #141885 | Compatibility, `minVersion`, revision and `generatedAt` checks; stale reads can't cancel a current adoption (regression test) |
| Failed catalog checks retry at the remaining fresh interval, not a full TTL | #141885 | Unchanged scheduler behavior; the deleted notice test's retry case is restored for failed adoption (fails if the retry falls back to the full TTL) |
| Operators learn when a downloaded catalog is not yet active | #141885 | No longer needed: downloads activate at the next check. The restart notice and its tests are removed; `models refresh` says when a running Gateway applies the update |
| Billing-route prices switch with their rows | #156535 | `upstreamPricing` and `providerPricing` are part of the accepted catalog pair |

## Evidence

**Regressions.** Each fails with its fix reverted and passes with it:
- *Retries a scheduled adoption when its pending auth owner settles.* Runs through the real Gateway update scheduler. Reverted, it logs `remote model catalog check superseded; deferred to the next check`.
- *Does not let a read under a superseded config cancel the current adoption.* Reverted, both calls end `superseded`.
- *Ends adoption instead of joining a timed-out owner build.* Reverted, adoption never settles.
- *Does not hold Gateway shutdown on an adoption's pricing preparation.* Reverted, shutdown waits on the held preparation until the test times out.
- *Recovers adopted owners when their borrowed Gateway plugin retires after commit / before commit.* Without the recovery, both fail: `Prepared model runtime plugin generation retired` and `prepared reply dispatch runtime owner was not published`.
- *Uses the remaining stored TTL after a fresh startup check when adoption fails.* With the retry reverted to the full TTL, the second check doesn't run.

**Suites:**

| Suite | Result |
|---|---|
| `prepared-model-runtime.remote-publication.test.ts` | 10/10 |
| Gateway integration (`models-list.remote-catalog`) | v1 and v2 pass. Config and auth churn during preparation end `published` on the settled owners. Also covers retained admitted runs, rejected and stale bundles, worker replacement and disablement |
| `prepared-model-runtime*`, `server-plugin-reload*`, `update-startup`, and all PR-touched test files | pass |
| `tsgo:core`, all `tsgo:test:src` shards | pass |
| oxlint and oxfmt on changed files; `config:docs:check`, `config:schema:check`; max-lines, assertion-safety and test-timeout-race ratchets | pass |

Tests wait on owned completion signals (`withinTest`), not wall-clock deadlines.

**Live proof** on an isolated Gateway built from `cb8c9197fdc` (no provider mocks). Later commits add plugin-retirement recovery for adopted owners, covered by the regression tests above, and rebases onto `main`. It used a real OpenAI key through `openai/gpt-4.1-mini`, and the build stamp was set before the real catalog's publication date. A client polled `models.list` back to back over one WebSocket for the whole run (1023 polls, no errors). One Gateway process (PID unchanged) and no restart:
1. The stored catalog was seeded with an older revision of the real `catalog.openclaw.ai` v2 catalog: generated 2 days earlier, `gpt-4.1-mini` priced ×10, plus one extra kimi row. `models.list` listed the extra row, and a turn priced **$4.00 / $16.00 per M** input/output.
2. `openclaw models refresh` downloaded the real catalog (`updated`, 1039 models). The listed rows didn't change for the next 7.1 s, and a turn in that window still priced **$4.00 / $16.00 per M**: a download stays inactive until the Gateway adopts it.
3. A long turn was admitted on the older catalog, then `models.list {refresh:true}` returned the older rows (extra kimi row still listed) and started adoption. The new catalog was visible 0.9 s later, while the long turn was still running: the extra kimi row was gone.
4. The in-flight turn finished at **$4.00 / $16.00 per M** (older rows and prices). The next turn priced **$0.40 / $1.60 per M** (real catalog).
5. `models.catalogRefresh.url` was moved to a local mirror of the real catalog through `config.patch`, and the mirror's catalog was adopted without a restart. The mirror then served malformed JSON: `models refresh` failed with `SyntaxError`, the model list was unchanged, and the next turn still priced $0.40 / $1.60 per M.

**Model picker during republication (also on `main`).** Right after the new generation commits, `models.list` shows the new generation's configured and static rows until its full catalog loads, then the full list. In the live run this lasted 109 ms. The same poller against a `main` build shows the same window after a `models.*` config reload (20 → 6 → 16 rows for about 350 ms), so this PR adds a new trigger for an existing behavior. It doesn't change it. The short list comes from the new generation, so rows and prices stay paired.

**Published-driver upgrade cells.** Candidate tarball built from a fresh clone at `6f682c7944e` with the canonical Docker packaging script and no build-time overrides. sha256 `46d881a0…ef0ad`; embedded commit `6f682c7944e2`, version 2026.9.7. `6f682c7944e` already includes the shutdown-cancellation and pricing-deadline commits. The current head the current head differs from it only by rebases onto `main`: `main` had moved the scheduled catalog check into `update-startup-catalog.ts`, and this PR's adoption call moved there unchanged (`git range-diff` shows no other production change; a `remoteCatalog: null` test-fixture field moved to main's relocated `cli-compaction.test-support.ts`).

| Driver → candidate | Scenario | Result |
|---|---|---|
| `openclaw@2026.9.6` | base | passed (930 s); updater outcome success, no recovery |
| `openclaw@2026.9.6` | plugin-deps-cleanup | passed (923 s); updater outcome success, no recovery |
| `openclaw@2026.9.7` (latest) | base | passed (813 s); updater outcome success, no recovery |

In every cell:
- Migration, post-Doctor config validation, survival, plugin-dependency cleanup and runtime-deps repair checks passed.
- The candidate Gateway logged ready, then its catalog check fetched and saved the hosted catalog about 0.2 s after starting. The hosted catalog is older than the candidate's build stamp, so adoption ends `unchanged`, which isn't logged. After a 300 s settlement window, `/readyz` (`ready:true`, nothing failing) and a Gateway `status` RPC passed, and the Gateway shut down cleanly.
- To show a logged terminal outcome, each cell was repeated with a loopback mirror serving a newer copy of the same catalog. Each check logged `remote model catalog applied` about 0.26 s after it started, followed by `/readyz` and `status` passing.

Not run: `openclaw@2026.9.7` plugin-deps-cleanup. At the previous head it failed inside the 2026.9.7 driver's retained-runtime verification (`Retained runtime entry does not reference its inventoried file: dist/a2ui-…mjs`), identically for a merge-base control package with none of this PR's commits.

Harness note for the 2026.9.7 cell: unchanged, the upgrade harness can't run against 2026.9.7. It seeds the retired `tools.toolSearch {mode:"code"}` setting, which 2026.9.7 rejects. The 2026.9.7 cell used the harness's existing Tool Search "absent" mode, a one-line local change that skips only that seed and its check. The 2026.9.6 cells used the unchanged harness.

**CI:** fully green on the final head ([run 36818367970](https://github.com/openclaw/openclaw/actions/runs/36818367970)). Earlier heads hit failures that reproduce on `main` in code this PR doesn't touch: `update-cli.target-schema` (main reproduction: #158000 (comment)), the type-suppression inventory, the Windows partition owner test and the Windows backup-rename test. Main has since fixed the last three. Config compatibility confirmation: #158000 (comment).

No overlap with Pash/Sarah changes.

Co-authored-by: Ayaan Zaidi <hi@obviy.us>
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Oct 2, 2026
…nclaw#158000)

Related: openclaw#140086, openclaw#141885, openclaw#156535, openclaw#157838

## What Problem This Solves

A running Gateway doesn't see a newly downloaded hosted model catalog until it restarts. This PR publishes each accepted catalog through the existing prepared-runtime owner, without a restart. Model rows and their prices switch together as one generation, which keeps the invariant from openclaw#140086.

## User Impact

- Compatible downloads are adopted at the Gateway's background catalog check, or after an explicit `models.list` refresh. That refresh returns the currently accepted rows right away and runs adoption afterward.
- A turn admitted on catalog N keeps N's rows and prices until it finishes. New turns use N+1. Rows and prices are never mixed.
- A concurrent auth or config publication no longer postpones adoption to the next scheduled check (up to 6 h). Adoption waits for that publication to settle, then retries, up to 3 attempts.
- An owner whose build failed or timed out ends the adoption instead of waiting on unbounded work. Gateway shutdown cancels an adoption that is still preparing.
- Malformed, schema-invalid, too-new (`minVersion`) and older catalogs are rejected, and the previously accepted catalog stays in use.
- Changing `models.catalogRefresh.url` no longer needs a restart: the previous source's catalog stops applying, and the mirror's catalog is adopted at the next catalog check.

**Bad-catalog exposure:** with live apply, a *valid but wrong* published catalog reaches running Gateways at their next catalog check (at most every 6 h) or on the next explicit `models.list` refresh. It no longer waits for a restart. Recovery uses existing mechanisms only:
- Republish a corrected catalog with a newer `generatedAt`; Gateways adopt it the same way.
- Operators can set `models.catalogRefresh.enabled: false`, which withdraws remote rows and prices without a restart (covered by the Gateway integration test).

This PR adds no new kill switch, config option or env knob.

### Compatibility

No config keys, defaults, types, validation, stored rows, protocol or SDK contracts change. The only config-surface change is the `models.catalogRefresh.url` help text, which drops the stale "Changes apply after a Gateway restart" sentence, and its regenerated config-doc baseline hash. Existing configs validate unchanged and need no Doctor migration (maintainer confirmation: openclaw#158000 (comment)). Startup behavior is unchanged. Upgrade impact for existing installs: an accepted download activates at the next catalog check instead of the next restart.

## Why This Change Was Made

- `prepared-model-runtime.configured-refresh.ts` builds a complete candidate generation of the configured owners under the new catalog. One serialized commit then publishes rows, the accepted bundle, the pricing context and the reply-dispatch projection together.
- Adoption re-reads the stored catalog until the config it read under is still current, so a stale caller can't cancel a current adoption.
- Each preparation attempt has its own abort signal. A config advance restarts only the attempt; a newer catalog or shutdown ends the whole adoption, including pricing preparation.
- Between attempts, adoption waits only on publication gates: a pending replacement or an owner's pending publication.
- Adopted owners install the same plugin-retirement recovery as configured publication (openclaw#161267). After commit, a lost Gateway plugin loan republishes them through the normal recovery. Before commit, it restarts the adoption attempt, and the commit refuses any candidate whose plugin generation retired.

### Why downloads were restart-only, and what this keeps

Restart-only activation was a mechanism, not the goal. openclaw#140086 chose it to stop rows and prices from different catalog versions mixing, and openclaw#157838 was merged as "the prerequisite for applying new remote catalogs without a Gateway restart (rows and prices must switch together)". This PR is that follow-up. Every requirement those PRs set still holds:

| Original requirement | Source | How it holds here |
|---|---|---|
| Rows and prices from one catalog version; never mixed across reloads or new requests | openclaw#140086 | One serialized commit publishes owners, bundle, pricing context and dispatch; pricing contexts are keyed by the exact accepted catalog. Integration test: new rows appear only with new prices |
| Admitted work keeps its pair | openclaw#140086, openclaw#157838 | Runs carry their plugin generation's catalog; usage operations capture one pricing context. Integration test and live proof: the in-flight turn keeps the old price |
| Startup absence is a real state (no downloaded rows without prices) | openclaw#140086 | Absence → catalog goes through the same atomic commit; overlay absence tests unchanged |
| Worker replacement inherits the host's accepted pair, not a later download | openclaw#140086 | The commit updates the inherited pair; later workers and a worker-exit recovery keep it (overlay and integration tests) |
| Current enablement and source URL still gate eligibility | openclaw#140086, openclaw#156535 | Checked on every read and before adoption, including the default-install v1 fallback; disablement withdraws rows and prices together (integration test) |
| Bad or superseded downloads never replace the active pair | openclaw#140086, openclaw#141885 | Compatibility, `minVersion`, revision and `generatedAt` checks; stale reads can't cancel a current adoption (regression test) |
| Failed catalog checks retry at the remaining fresh interval, not a full TTL | openclaw#141885 | Unchanged scheduler behavior; the deleted notice test's retry case is restored for failed adoption (fails if the retry falls back to the full TTL) |
| Operators learn when a downloaded catalog is not yet active | openclaw#141885 | No longer needed: downloads activate at the next check. The restart notice and its tests are removed; `models refresh` says when a running Gateway applies the update |
| Billing-route prices switch with their rows | openclaw#156535 | `upstreamPricing` and `providerPricing` are part of the accepted catalog pair |

## Evidence

**Regressions.** Each fails with its fix reverted and passes with it:
- *Retries a scheduled adoption when its pending auth owner settles.* Runs through the real Gateway update scheduler. Reverted, it logs `remote model catalog check superseded; deferred to the next check`.
- *Does not let a read under a superseded config cancel the current adoption.* Reverted, both calls end `superseded`.
- *Ends adoption instead of joining a timed-out owner build.* Reverted, adoption never settles.
- *Does not hold Gateway shutdown on an adoption's pricing preparation.* Reverted, shutdown waits on the held preparation until the test times out.
- *Recovers adopted owners when their borrowed Gateway plugin retires after commit / before commit.* Without the recovery, both fail: `Prepared model runtime plugin generation retired` and `prepared reply dispatch runtime owner was not published`.
- *Uses the remaining stored TTL after a fresh startup check when adoption fails.* With the retry reverted to the full TTL, the second check doesn't run.

**Suites:**

| Suite | Result |
|---|---|
| `prepared-model-runtime.remote-publication.test.ts` | 10/10 |
| Gateway integration (`models-list.remote-catalog`) | v1 and v2 pass. Config and auth churn during preparation end `published` on the settled owners. Also covers retained admitted runs, rejected and stale bundles, worker replacement and disablement |
| `prepared-model-runtime*`, `server-plugin-reload*`, `update-startup`, and all PR-touched test files | pass |
| `tsgo:core`, all `tsgo:test:src` shards | pass |
| oxlint and oxfmt on changed files; `config:docs:check`, `config:schema:check`; max-lines, assertion-safety and test-timeout-race ratchets | pass |

Tests wait on owned completion signals (`withinTest`), not wall-clock deadlines.

**Live proof** on an isolated Gateway built from `cb8c9197fdc` (no provider mocks). Later commits add plugin-retirement recovery for adopted owners, covered by the regression tests above, and rebases onto `main`. It used a real OpenAI key through `openai/gpt-4.1-mini`, and the build stamp was set before the real catalog's publication date. A client polled `models.list` back to back over one WebSocket for the whole run (1023 polls, no errors). One Gateway process (PID unchanged) and no restart:
1. The stored catalog was seeded with an older revision of the real `catalog.openclaw.ai` v2 catalog: generated 2 days earlier, `gpt-4.1-mini` priced ×10, plus one extra kimi row. `models.list` listed the extra row, and a turn priced **$4.00 / $16.00 per M** input/output.
2. `openclaw models refresh` downloaded the real catalog (`updated`, 1039 models). The listed rows didn't change for the next 7.1 s, and a turn in that window still priced **$4.00 / $16.00 per M**: a download stays inactive until the Gateway adopts it.
3. A long turn was admitted on the older catalog, then `models.list {refresh:true}` returned the older rows (extra kimi row still listed) and started adoption. The new catalog was visible 0.9 s later, while the long turn was still running: the extra kimi row was gone.
4. The in-flight turn finished at **$4.00 / $16.00 per M** (older rows and prices). The next turn priced **$0.40 / $1.60 per M** (real catalog).
5. `models.catalogRefresh.url` was moved to a local mirror of the real catalog through `config.patch`, and the mirror's catalog was adopted without a restart. The mirror then served malformed JSON: `models refresh` failed with `SyntaxError`, the model list was unchanged, and the next turn still priced $0.40 / $1.60 per M.

**Model picker during republication (also on `main`).** Right after the new generation commits, `models.list` shows the new generation's configured and static rows until its full catalog loads, then the full list. In the live run this lasted 109 ms. The same poller against a `main` build shows the same window after a `models.*` config reload (20 → 6 → 16 rows for about 350 ms), so this PR adds a new trigger for an existing behavior. It doesn't change it. The short list comes from the new generation, so rows and prices stay paired.

**Published-driver upgrade cells.** Candidate tarball built from a fresh clone at `6f682c7944e` with the canonical Docker packaging script and no build-time overrides. sha256 `46d881a0…ef0ad`; embedded commit `6f682c7944e2`, version 2026.9.7. `6f682c7944e` already includes the shutdown-cancellation and pricing-deadline commits. The current head the current head differs from it only by rebases onto `main`: `main` had moved the scheduled catalog check into `update-startup-catalog.ts`, and this PR's adoption call moved there unchanged (`git range-diff` shows no other production change; a `remoteCatalog: null` test-fixture field moved to main's relocated `cli-compaction.test-support.ts`).

| Driver → candidate | Scenario | Result |
|---|---|---|
| `openclaw@2026.9.6` | base | passed (930 s); updater outcome success, no recovery |
| `openclaw@2026.9.6` | plugin-deps-cleanup | passed (923 s); updater outcome success, no recovery |
| `openclaw@2026.9.7` (latest) | base | passed (813 s); updater outcome success, no recovery |

In every cell:
- Migration, post-Doctor config validation, survival, plugin-dependency cleanup and runtime-deps repair checks passed.
- The candidate Gateway logged ready, then its catalog check fetched and saved the hosted catalog about 0.2 s after starting. The hosted catalog is older than the candidate's build stamp, so adoption ends `unchanged`, which isn't logged. After a 300 s settlement window, `/readyz` (`ready:true`, nothing failing) and a Gateway `status` RPC passed, and the Gateway shut down cleanly.
- To show a logged terminal outcome, each cell was repeated with a loopback mirror serving a newer copy of the same catalog. Each check logged `remote model catalog applied` about 0.26 s after it started, followed by `/readyz` and `status` passing.

Not run: `openclaw@2026.9.7` plugin-deps-cleanup. At the previous head it failed inside the 2026.9.7 driver's retained-runtime verification (`Retained runtime entry does not reference its inventoried file: dist/a2ui-…mjs`), identically for a merge-base control package with none of this PR's commits.

Harness note for the 2026.9.7 cell: unchanged, the upgrade harness can't run against 2026.9.7. It seeds the retired `tools.toolSearch {mode:"code"}` setting, which 2026.9.7 rejects. The 2026.9.7 cell used the harness's existing Tool Search "absent" mode, a one-line local change that skips only that seed and its check. The 2026.9.6 cells used the unchanged harness.

**CI:** fully green on the final head ([run 36818367970](https://github.com/openclaw/openclaw/actions/runs/36818367970)). Earlier heads hit failures that reproduce on `main` in code this PR doesn't touch: `update-cli.target-schema` (main reproduction: openclaw#158000 (comment)), the type-suppression inventory, the Windows partition owner test and the Windows backup-rename test. Main has since fixed the last three. Config compatibility confirmation: openclaw#158000 (comment).

No overlap with Pash/Sarah changes.

Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant