Repository navigation
fix(models): defer downloaded catalog changes until restart - #140086
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
|
Codex review: needs maintainer review before merge. Reviewed September 6, 2026, 9:22 AM ET / 13:22 UTC. ClawSweeper reviewWhat this changesThe PR keeps downloaded model descriptions and prices on one startup snapshot across Gateway worker replacements, activating newer downloads after a full restart. Merge readiness✅ Ready for maintainer review This remains a useful, focused repair: current main and v2026.9.2 retain the cache behavior behind the reported inconsistency. No blocking defect was found in the introduced changes, and the protected maintainer label also precludes cleanup closure. Priority: P2 Review scores
Verification
How this fits togetherThe hosted model catalog supplements installed plugins with model metadata and pricing. Gateway catalog workers and usage estimates consume that data, while background downloads update the stored catalog for the next restart. flowchart LR
A[Hosted catalog download] --> B[Stored catalog]
B --> C[Gateway startup snapshot]
C --> D[Current enablement and source checks]
D --> E[Catalog workers]
D --> F[Usage price estimates]
E --> G[Model lists]
Before mergeNone. Agent review detailsSecurityNone. PR surfaceSource +15, Tests +129. Total +144 across 6 files. View PR surface stats
Review metrics
Technical reviewBest possible solution: Keep one validated process-start catalog snapshot for both model rows and prices, with current configuration controlling eligibility and full restart controlling activation. Do we have a high-confidence way to reproduce the issue? Yes. Current-main source allows a replacement worker to read a newer stored catalog while another module retains older pricing; the body reports this through real Gateway operations on main and v2026.9.2. This review did not execute the scenario. Is this the best way to solve the issue? Yes. Capturing the shared metadata at kernel startup directly enforces the documented restart contract without adding configuration, changing storage, or duplicating catalog policy. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 60c66beed84d. LabelsLabel changes:
Label justifications:
EvidenceWhat I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (1 earlier review cycle)
|
|
Proof: Built Gateway model lists and fresh agent usage retain the startup rows/pricing pair through actual same-process worker replacement. Startup absence stays absent and unpriced. Full process restart activates the downloaded pair.
Failure/recovery proof preserves unavailable responses as failures; this change does not promise uninterrupted service through a refused reload. Earlier failed attempts and corrected proof scopes remain retained. |
…#140086) Retain one startup catalog and pricing pair across Gateway worker replacement, including startup absence. Activate later downloads only after a full process restart. Use an object transport record so nullable absence satisfies Node's worker-data contract without clearing the captured value or widening the dependency API. Verified with the built Gateway's public model lists, distinct successful usage operations, real same-process replacement workers, and full restart controls. Related: openclaw#136257 Co-authored-by: Ayaan Zaidi <hi@obviy.us>
Related: #140086 ## What Problem This Solves Makes each usage-cost operation retain one hosted pricing context, preparing usage accounting for catalog updates that can occur while an operation is running. ## User Impact No user-visible change in this PR. Catalog downloads still require a Gateway restart to activate. Existing recorded costs, configured overrides, local model prices, and tier handling keep their current behavior. There are no configuration, protocol, SDK, or storage changes. This is the maintainer-requested accounting prerequisite for live catalog publication, not the activation change itself. ## Why This Change Was Made Usage refresh previously selected its pricing fingerprint and each lazily resolved model price independently. The refresh now captures the hosted pricing context once and uses it for both. Diagnostic readers retain lazy preparation and capture their context when estimation first becomes necessary; subsequent model lookups retain that context. The ordinary synchronous pricing APIs still use the same resolver and preserve their existing lookup behavior. Fingerprint-only reads do not allocate an unused captured resolver. ## Evidence - **Built Gateway, base and candidate:** downloaded a synthetic catalog through `openclaw models refresh`, imported two unpriced transcript records through `openclaw doctor --session-sqlite import`, then called the public `sessions.usage.logs` and `sessions.usage` methods. Base `05edd36e12` and current candidate `c95b7d0942e6` both reported individual costs **$1 and $4**, and total cost **$5**. Each ran with separate state, home, configuration, profile, and loopback ports. No provider inference was performed. - **Mid-operation regression:** after selecting a new pricing context, an existing operation retains the old price for a model it has not looked up yet and retains its fingerprint; a new operation uses the new prices and fingerprint. Replacing the captured lookup with the previous per-lookup selection makes this test fail with **expected 4, received 40**. This is a controlled pricing-publication test, not a claim that this PR activates downloaded catalogs. - **Focused suites:** 240 tests across 13 pricing, usage-format, usage-reporting, and usage-worker files passed during preparation. The operation regression was then moved into its own usage-pricing suite to avoid growing the existing pricing suite beyond its line cap; that relocated case and focused lint passed. The current head passed the built Gateway parity check again. - **Measured cost:** the relocated `session-cost-usage-pricing-context.test.ts` regression took **36 ms** on the current head (21.56 s shard including cold transform). The earlier three-file command took 108.00 s; the remaining ten-file command took 101.34 s. [Exact-head CI passed](https://github.com/openclaw/openclaw/actions/runs/36095336705), with all 147 jobs completed and no failures. Documentation is unchanged because the existing restart contract remains in force. Co-authored-by: Ayaan Zaidi <hi@obviy.us>
…157838) Related: openclaw#140086 ## What Problem This Solves Makes each usage-cost operation retain one hosted pricing context, preparing usage accounting for catalog updates that can occur while an operation is running. ## User Impact No user-visible change in this PR. Catalog downloads still require a Gateway restart to activate. Existing recorded costs, configured overrides, local model prices, and tier handling keep their current behavior. There are no configuration, protocol, SDK, or storage changes. This is the maintainer-requested accounting prerequisite for live catalog publication, not the activation change itself. ## Why This Change Was Made Usage refresh previously selected its pricing fingerprint and each lazily resolved model price independently. The refresh now captures the hosted pricing context once and uses it for both. Diagnostic readers retain lazy preparation and capture their context when estimation first becomes necessary; subsequent model lookups retain that context. The ordinary synchronous pricing APIs still use the same resolver and preserve their existing lookup behavior. Fingerprint-only reads do not allocate an unused captured resolver. ## Evidence - **Built Gateway, base and candidate:** downloaded a synthetic catalog through `openclaw models refresh`, imported two unpriced transcript records through `openclaw doctor --session-sqlite import`, then called the public `sessions.usage.logs` and `sessions.usage` methods. Base `7353f7e382` and current candidate `c95b7d0942e6` both reported individual costs **$1 and $4**, and total cost **$5**. Each ran with separate state, home, configuration, profile, and loopback ports. No provider inference was performed. - **Mid-operation regression:** after selecting a new pricing context, an existing operation retains the old price for a model it has not looked up yet and retains its fingerprint; a new operation uses the new prices and fingerprint. Replacing the captured lookup with the previous per-lookup selection makes this test fail with **expected 4, received 40**. This is a controlled pricing-publication test, not a claim that this PR activates downloaded catalogs. - **Focused suites:** 240 tests across 13 pricing, usage-format, usage-reporting, and usage-worker files passed during preparation. The operation regression was then moved into its own usage-pricing suite to avoid growing the existing pricing suite beyond its line cap; that relocated case and focused lint passed. The current head passed the built Gateway parity check again. - **Measured cost:** the relocated `session-cost-usage-pricing-context.test.ts` regression took **36 ms** on the current head (21.56 s shard including cold transform). The earlier three-file command took 108.00 s; the remaining ten-file command took 101.34 s. [Exact-head CI passed](https://github.com/openclaw/openclaw/actions/runs/36095336705), with all 147 jobs completed and no failures. Documentation is unchanged because the existing restart contract remains in force. Co-authored-by: Ayaan Zaidi <hi@obviy.us>
…8000) Related: #140086, #141885, #156535, #157838 ## What Problem This Solves A running Gateway doesn't see a newly downloaded hosted model catalog until it restarts. This PR publishes each accepted catalog through the existing prepared-runtime owner, without a restart. Model rows and their prices switch together as one generation, which keeps the invariant from #140086. ## User Impact - Compatible downloads are adopted at the Gateway's background catalog check, or after an explicit `models.list` refresh. That refresh returns the currently accepted rows right away and runs adoption afterward. - A turn admitted on catalog N keeps N's rows and prices until it finishes. New turns use N+1. Rows and prices are never mixed. - A concurrent auth or config publication no longer postpones adoption to the next scheduled check (up to 6 h). Adoption waits for that publication to settle, then retries, up to 3 attempts. - An owner whose build failed or timed out ends the adoption instead of waiting on unbounded work. Gateway shutdown cancels an adoption that is still preparing. - Malformed, schema-invalid, too-new (`minVersion`) and older catalogs are rejected, and the previously accepted catalog stays in use. - Changing `models.catalogRefresh.url` no longer needs a restart: the previous source's catalog stops applying, and the mirror's catalog is adopted at the next catalog check. **Bad-catalog exposure:** with live apply, a *valid but wrong* published catalog reaches running Gateways at their next catalog check (at most every 6 h) or on the next explicit `models.list` refresh. It no longer waits for a restart. Recovery uses existing mechanisms only: - Republish a corrected catalog with a newer `generatedAt`; Gateways adopt it the same way. - Operators can set `models.catalogRefresh.enabled: false`, which withdraws remote rows and prices without a restart (covered by the Gateway integration test). This PR adds no new kill switch, config option or env knob. ### Compatibility No config keys, defaults, types, validation, stored rows, protocol or SDK contracts change. The only config-surface change is the `models.catalogRefresh.url` help text, which drops the stale "Changes apply after a Gateway restart" sentence, and its regenerated config-doc baseline hash. Existing configs validate unchanged and need no Doctor migration (maintainer confirmation: #158000 (comment)). Startup behavior is unchanged. Upgrade impact for existing installs: an accepted download activates at the next catalog check instead of the next restart. ## Why This Change Was Made - `prepared-model-runtime.configured-refresh.ts` builds a complete candidate generation of the configured owners under the new catalog. One serialized commit then publishes rows, the accepted bundle, the pricing context and the reply-dispatch projection together. - Adoption re-reads the stored catalog until the config it read under is still current, so a stale caller can't cancel a current adoption. - Each preparation attempt has its own abort signal. A config advance restarts only the attempt; a newer catalog or shutdown ends the whole adoption, including pricing preparation. - Between attempts, adoption waits only on publication gates: a pending replacement or an owner's pending publication. - Adopted owners install the same plugin-retirement recovery as configured publication (#161267). After commit, a lost Gateway plugin loan republishes them through the normal recovery. Before commit, it restarts the adoption attempt, and the commit refuses any candidate whose plugin generation retired. ### Why downloads were restart-only, and what this keeps Restart-only activation was a mechanism, not the goal. #140086 chose it to stop rows and prices from different catalog versions mixing, and #157838 was merged as "the prerequisite for applying new remote catalogs without a Gateway restart (rows and prices must switch together)". This PR is that follow-up. Every requirement those PRs set still holds: | Original requirement | Source | How it holds here | |---|---|---| | Rows and prices from one catalog version; never mixed across reloads or new requests | #140086 | One serialized commit publishes owners, bundle, pricing context and dispatch; pricing contexts are keyed by the exact accepted catalog. Integration test: new rows appear only with new prices | | Admitted work keeps its pair | #140086, #157838 | Runs carry their plugin generation's catalog; usage operations capture one pricing context. Integration test and live proof: the in-flight turn keeps the old price | | Startup absence is a real state (no downloaded rows without prices) | #140086 | Absence → catalog goes through the same atomic commit; overlay absence tests unchanged | | Worker replacement inherits the host's accepted pair, not a later download | #140086 | The commit updates the inherited pair; later workers and a worker-exit recovery keep it (overlay and integration tests) | | Current enablement and source URL still gate eligibility | #140086, #156535 | Checked on every read and before adoption, including the default-install v1 fallback; disablement withdraws rows and prices together (integration test) | | Bad or superseded downloads never replace the active pair | #140086, #141885 | Compatibility, `minVersion`, revision and `generatedAt` checks; stale reads can't cancel a current adoption (regression test) | | Failed catalog checks retry at the remaining fresh interval, not a full TTL | #141885 | Unchanged scheduler behavior; the deleted notice test's retry case is restored for failed adoption (fails if the retry falls back to the full TTL) | | Operators learn when a downloaded catalog is not yet active | #141885 | No longer needed: downloads activate at the next check. The restart notice and its tests are removed; `models refresh` says when a running Gateway applies the update | | Billing-route prices switch with their rows | #156535 | `upstreamPricing` and `providerPricing` are part of the accepted catalog pair | ## Evidence **Regressions.** Each fails with its fix reverted and passes with it: - *Retries a scheduled adoption when its pending auth owner settles.* Runs through the real Gateway update scheduler. Reverted, it logs `remote model catalog check superseded; deferred to the next check`. - *Does not let a read under a superseded config cancel the current adoption.* Reverted, both calls end `superseded`. - *Ends adoption instead of joining a timed-out owner build.* Reverted, adoption never settles. - *Does not hold Gateway shutdown on an adoption's pricing preparation.* Reverted, shutdown waits on the held preparation until the test times out. - *Recovers adopted owners when their borrowed Gateway plugin retires after commit / before commit.* Without the recovery, both fail: `Prepared model runtime plugin generation retired` and `prepared reply dispatch runtime owner was not published`. - *Uses the remaining stored TTL after a fresh startup check when adoption fails.* With the retry reverted to the full TTL, the second check doesn't run. **Suites:** | Suite | Result | |---|---| | `prepared-model-runtime.remote-publication.test.ts` | 10/10 | | Gateway integration (`models-list.remote-catalog`) | v1 and v2 pass. Config and auth churn during preparation end `published` on the settled owners. Also covers retained admitted runs, rejected and stale bundles, worker replacement and disablement | | `prepared-model-runtime*`, `server-plugin-reload*`, `update-startup`, and all PR-touched test files | pass | | `tsgo:core`, all `tsgo:test:src` shards | pass | | oxlint and oxfmt on changed files; `config:docs:check`, `config:schema:check`; max-lines, assertion-safety and test-timeout-race ratchets | pass | Tests wait on owned completion signals (`withinTest`), not wall-clock deadlines. **Live proof** on an isolated Gateway built from `cb8c9197fdc` (no provider mocks). Later commits add plugin-retirement recovery for adopted owners, covered by the regression tests above, and rebases onto `main`. It used a real OpenAI key through `openai/gpt-4.1-mini`, and the build stamp was set before the real catalog's publication date. A client polled `models.list` back to back over one WebSocket for the whole run (1023 polls, no errors). One Gateway process (PID unchanged) and no restart: 1. The stored catalog was seeded with an older revision of the real `catalog.openclaw.ai` v2 catalog: generated 2 days earlier, `gpt-4.1-mini` priced ×10, plus one extra kimi row. `models.list` listed the extra row, and a turn priced **$4.00 / $16.00 per M** input/output. 2. `openclaw models refresh` downloaded the real catalog (`updated`, 1039 models). The listed rows didn't change for the next 7.1 s, and a turn in that window still priced **$4.00 / $16.00 per M**: a download stays inactive until the Gateway adopts it. 3. A long turn was admitted on the older catalog, then `models.list {refresh:true}` returned the older rows (extra kimi row still listed) and started adoption. The new catalog was visible 0.9 s later, while the long turn was still running: the extra kimi row was gone. 4. The in-flight turn finished at **$4.00 / $16.00 per M** (older rows and prices). The next turn priced **$0.40 / $1.60 per M** (real catalog). 5. `models.catalogRefresh.url` was moved to a local mirror of the real catalog through `config.patch`, and the mirror's catalog was adopted without a restart. The mirror then served malformed JSON: `models refresh` failed with `SyntaxError`, the model list was unchanged, and the next turn still priced $0.40 / $1.60 per M. **Model picker during republication (also on `main`).** Right after the new generation commits, `models.list` shows the new generation's configured and static rows until its full catalog loads, then the full list. In the live run this lasted 109 ms. The same poller against a `main` build shows the same window after a `models.*` config reload (20 → 6 → 16 rows for about 350 ms), so this PR adds a new trigger for an existing behavior. It doesn't change it. The short list comes from the new generation, so rows and prices stay paired. **Published-driver upgrade cells.** Candidate tarball built from a fresh clone at `6f682c7944e` with the canonical Docker packaging script and no build-time overrides. sha256 `46d881a0…ef0ad`; embedded commit `6f682c7944e2`, version 2026.9.7. `6f682c7944e` already includes the shutdown-cancellation and pricing-deadline commits. The current head the current head differs from it only by rebases onto `main`: `main` had moved the scheduled catalog check into `update-startup-catalog.ts`, and this PR's adoption call moved there unchanged (`git range-diff` shows no other production change; a `remoteCatalog: null` test-fixture field moved to main's relocated `cli-compaction.test-support.ts`). | Driver → candidate | Scenario | Result | |---|---|---| | `openclaw@2026.9.6` | base | passed (930 s); updater outcome success, no recovery | | `openclaw@2026.9.6` | plugin-deps-cleanup | passed (923 s); updater outcome success, no recovery | | `openclaw@2026.9.7` (latest) | base | passed (813 s); updater outcome success, no recovery | In every cell: - Migration, post-Doctor config validation, survival, plugin-dependency cleanup and runtime-deps repair checks passed. - The candidate Gateway logged ready, then its catalog check fetched and saved the hosted catalog about 0.2 s after starting. The hosted catalog is older than the candidate's build stamp, so adoption ends `unchanged`, which isn't logged. After a 300 s settlement window, `/readyz` (`ready:true`, nothing failing) and a Gateway `status` RPC passed, and the Gateway shut down cleanly. - To show a logged terminal outcome, each cell was repeated with a loopback mirror serving a newer copy of the same catalog. Each check logged `remote model catalog applied` about 0.26 s after it started, followed by `/readyz` and `status` passing. Not run: `openclaw@2026.9.7` plugin-deps-cleanup. At the previous head it failed inside the 2026.9.7 driver's retained-runtime verification (`Retained runtime entry does not reference its inventoried file: dist/a2ui-…mjs`), identically for a merge-base control package with none of this PR's commits. Harness note for the 2026.9.7 cell: unchanged, the upgrade harness can't run against 2026.9.7. It seeds the retired `tools.toolSearch {mode:"code"}` setting, which 2026.9.7 rejects. The 2026.9.7 cell used the harness's existing Tool Search "absent" mode, a one-line local change that skips only that seed and its check. The 2026.9.6 cells used the unchanged harness. **CI:** fully green on the final head ([run 36818367970](https://github.com/openclaw/openclaw/actions/runs/36818367970)). Earlier heads hit failures that reproduce on `main` in code this PR doesn't touch: `update-cli.target-schema` (main reproduction: #158000 (comment)), the type-suppression inventory, the Windows partition owner test and the Windows backup-rename test. Main has since fixed the last three. Config compatibility confirmation: #158000 (comment). No overlap with Pash/Sarah changes. Co-authored-by: Ayaan Zaidi <hi@obviy.us>
…nclaw#158000) Related: openclaw#140086, openclaw#141885, openclaw#156535, openclaw#157838 ## What Problem This Solves A running Gateway doesn't see a newly downloaded hosted model catalog until it restarts. This PR publishes each accepted catalog through the existing prepared-runtime owner, without a restart. Model rows and their prices switch together as one generation, which keeps the invariant from openclaw#140086. ## User Impact - Compatible downloads are adopted at the Gateway's background catalog check, or after an explicit `models.list` refresh. That refresh returns the currently accepted rows right away and runs adoption afterward. - A turn admitted on catalog N keeps N's rows and prices until it finishes. New turns use N+1. Rows and prices are never mixed. - A concurrent auth or config publication no longer postpones adoption to the next scheduled check (up to 6 h). Adoption waits for that publication to settle, then retries, up to 3 attempts. - An owner whose build failed or timed out ends the adoption instead of waiting on unbounded work. Gateway shutdown cancels an adoption that is still preparing. - Malformed, schema-invalid, too-new (`minVersion`) and older catalogs are rejected, and the previously accepted catalog stays in use. - Changing `models.catalogRefresh.url` no longer needs a restart: the previous source's catalog stops applying, and the mirror's catalog is adopted at the next catalog check. **Bad-catalog exposure:** with live apply, a *valid but wrong* published catalog reaches running Gateways at their next catalog check (at most every 6 h) or on the next explicit `models.list` refresh. It no longer waits for a restart. Recovery uses existing mechanisms only: - Republish a corrected catalog with a newer `generatedAt`; Gateways adopt it the same way. - Operators can set `models.catalogRefresh.enabled: false`, which withdraws remote rows and prices without a restart (covered by the Gateway integration test). This PR adds no new kill switch, config option or env knob. ### Compatibility No config keys, defaults, types, validation, stored rows, protocol or SDK contracts change. The only config-surface change is the `models.catalogRefresh.url` help text, which drops the stale "Changes apply after a Gateway restart" sentence, and its regenerated config-doc baseline hash. Existing configs validate unchanged and need no Doctor migration (maintainer confirmation: openclaw#158000 (comment)). Startup behavior is unchanged. Upgrade impact for existing installs: an accepted download activates at the next catalog check instead of the next restart. ## Why This Change Was Made - `prepared-model-runtime.configured-refresh.ts` builds a complete candidate generation of the configured owners under the new catalog. One serialized commit then publishes rows, the accepted bundle, the pricing context and the reply-dispatch projection together. - Adoption re-reads the stored catalog until the config it read under is still current, so a stale caller can't cancel a current adoption. - Each preparation attempt has its own abort signal. A config advance restarts only the attempt; a newer catalog or shutdown ends the whole adoption, including pricing preparation. - Between attempts, adoption waits only on publication gates: a pending replacement or an owner's pending publication. - Adopted owners install the same plugin-retirement recovery as configured publication (openclaw#161267). After commit, a lost Gateway plugin loan republishes them through the normal recovery. Before commit, it restarts the adoption attempt, and the commit refuses any candidate whose plugin generation retired. ### Why downloads were restart-only, and what this keeps Restart-only activation was a mechanism, not the goal. openclaw#140086 chose it to stop rows and prices from different catalog versions mixing, and openclaw#157838 was merged as "the prerequisite for applying new remote catalogs without a Gateway restart (rows and prices must switch together)". This PR is that follow-up. Every requirement those PRs set still holds: | Original requirement | Source | How it holds here | |---|---|---| | Rows and prices from one catalog version; never mixed across reloads or new requests | openclaw#140086 | One serialized commit publishes owners, bundle, pricing context and dispatch; pricing contexts are keyed by the exact accepted catalog. Integration test: new rows appear only with new prices | | Admitted work keeps its pair | openclaw#140086, openclaw#157838 | Runs carry their plugin generation's catalog; usage operations capture one pricing context. Integration test and live proof: the in-flight turn keeps the old price | | Startup absence is a real state (no downloaded rows without prices) | openclaw#140086 | Absence → catalog goes through the same atomic commit; overlay absence tests unchanged | | Worker replacement inherits the host's accepted pair, not a later download | openclaw#140086 | The commit updates the inherited pair; later workers and a worker-exit recovery keep it (overlay and integration tests) | | Current enablement and source URL still gate eligibility | openclaw#140086, openclaw#156535 | Checked on every read and before adoption, including the default-install v1 fallback; disablement withdraws rows and prices together (integration test) | | Bad or superseded downloads never replace the active pair | openclaw#140086, openclaw#141885 | Compatibility, `minVersion`, revision and `generatedAt` checks; stale reads can't cancel a current adoption (regression test) | | Failed catalog checks retry at the remaining fresh interval, not a full TTL | openclaw#141885 | Unchanged scheduler behavior; the deleted notice test's retry case is restored for failed adoption (fails if the retry falls back to the full TTL) | | Operators learn when a downloaded catalog is not yet active | openclaw#141885 | No longer needed: downloads activate at the next check. The restart notice and its tests are removed; `models refresh` says when a running Gateway applies the update | | Billing-route prices switch with their rows | openclaw#156535 | `upstreamPricing` and `providerPricing` are part of the accepted catalog pair | ## Evidence **Regressions.** Each fails with its fix reverted and passes with it: - *Retries a scheduled adoption when its pending auth owner settles.* Runs through the real Gateway update scheduler. Reverted, it logs `remote model catalog check superseded; deferred to the next check`. - *Does not let a read under a superseded config cancel the current adoption.* Reverted, both calls end `superseded`. - *Ends adoption instead of joining a timed-out owner build.* Reverted, adoption never settles. - *Does not hold Gateway shutdown on an adoption's pricing preparation.* Reverted, shutdown waits on the held preparation until the test times out. - *Recovers adopted owners when their borrowed Gateway plugin retires after commit / before commit.* Without the recovery, both fail: `Prepared model runtime plugin generation retired` and `prepared reply dispatch runtime owner was not published`. - *Uses the remaining stored TTL after a fresh startup check when adoption fails.* With the retry reverted to the full TTL, the second check doesn't run. **Suites:** | Suite | Result | |---|---| | `prepared-model-runtime.remote-publication.test.ts` | 10/10 | | Gateway integration (`models-list.remote-catalog`) | v1 and v2 pass. Config and auth churn during preparation end `published` on the settled owners. Also covers retained admitted runs, rejected and stale bundles, worker replacement and disablement | | `prepared-model-runtime*`, `server-plugin-reload*`, `update-startup`, and all PR-touched test files | pass | | `tsgo:core`, all `tsgo:test:src` shards | pass | | oxlint and oxfmt on changed files; `config:docs:check`, `config:schema:check`; max-lines, assertion-safety and test-timeout-race ratchets | pass | Tests wait on owned completion signals (`withinTest`), not wall-clock deadlines. **Live proof** on an isolated Gateway built from `cb8c9197fdc` (no provider mocks). Later commits add plugin-retirement recovery for adopted owners, covered by the regression tests above, and rebases onto `main`. It used a real OpenAI key through `openai/gpt-4.1-mini`, and the build stamp was set before the real catalog's publication date. A client polled `models.list` back to back over one WebSocket for the whole run (1023 polls, no errors). One Gateway process (PID unchanged) and no restart: 1. The stored catalog was seeded with an older revision of the real `catalog.openclaw.ai` v2 catalog: generated 2 days earlier, `gpt-4.1-mini` priced ×10, plus one extra kimi row. `models.list` listed the extra row, and a turn priced **$4.00 / $16.00 per M** input/output. 2. `openclaw models refresh` downloaded the real catalog (`updated`, 1039 models). The listed rows didn't change for the next 7.1 s, and a turn in that window still priced **$4.00 / $16.00 per M**: a download stays inactive until the Gateway adopts it. 3. A long turn was admitted on the older catalog, then `models.list {refresh:true}` returned the older rows (extra kimi row still listed) and started adoption. The new catalog was visible 0.9 s later, while the long turn was still running: the extra kimi row was gone. 4. The in-flight turn finished at **$4.00 / $16.00 per M** (older rows and prices). The next turn priced **$0.40 / $1.60 per M** (real catalog). 5. `models.catalogRefresh.url` was moved to a local mirror of the real catalog through `config.patch`, and the mirror's catalog was adopted without a restart. The mirror then served malformed JSON: `models refresh` failed with `SyntaxError`, the model list was unchanged, and the next turn still priced $0.40 / $1.60 per M. **Model picker during republication (also on `main`).** Right after the new generation commits, `models.list` shows the new generation's configured and static rows until its full catalog loads, then the full list. In the live run this lasted 109 ms. The same poller against a `main` build shows the same window after a `models.*` config reload (20 → 6 → 16 rows for about 350 ms), so this PR adds a new trigger for an existing behavior. It doesn't change it. The short list comes from the new generation, so rows and prices stay paired. **Published-driver upgrade cells.** Candidate tarball built from a fresh clone at `6f682c7944e` with the canonical Docker packaging script and no build-time overrides. sha256 `46d881a0…ef0ad`; embedded commit `6f682c7944e2`, version 2026.9.7. `6f682c7944e` already includes the shutdown-cancellation and pricing-deadline commits. The current head the current head differs from it only by rebases onto `main`: `main` had moved the scheduled catalog check into `update-startup-catalog.ts`, and this PR's adoption call moved there unchanged (`git range-diff` shows no other production change; a `remoteCatalog: null` test-fixture field moved to main's relocated `cli-compaction.test-support.ts`). | Driver → candidate | Scenario | Result | |---|---|---| | `openclaw@2026.9.6` | base | passed (930 s); updater outcome success, no recovery | | `openclaw@2026.9.6` | plugin-deps-cleanup | passed (923 s); updater outcome success, no recovery | | `openclaw@2026.9.7` (latest) | base | passed (813 s); updater outcome success, no recovery | In every cell: - Migration, post-Doctor config validation, survival, plugin-dependency cleanup and runtime-deps repair checks passed. - The candidate Gateway logged ready, then its catalog check fetched and saved the hosted catalog about 0.2 s after starting. The hosted catalog is older than the candidate's build stamp, so adoption ends `unchanged`, which isn't logged. After a 300 s settlement window, `/readyz` (`ready:true`, nothing failing) and a Gateway `status` RPC passed, and the Gateway shut down cleanly. - To show a logged terminal outcome, each cell was repeated with a loopback mirror serving a newer copy of the same catalog. Each check logged `remote model catalog applied` about 0.26 s after it started, followed by `/readyz` and `status` passing. Not run: `openclaw@2026.9.7` plugin-deps-cleanup. At the previous head it failed inside the 2026.9.7 driver's retained-runtime verification (`Retained runtime entry does not reference its inventoried file: dist/a2ui-…mjs`), identically for a merge-base control package with none of this PR's commits. Harness note for the 2026.9.7 cell: unchanged, the upgrade harness can't run against 2026.9.7. It seeds the retired `tools.toolSearch {mode:"code"}` setting, which 2026.9.7 rejects. The 2026.9.7 cell used the harness's existing Tool Search "absent" mode, a one-line local change that skips only that seed and its check. The 2026.9.6 cells used the unchanged harness. **CI:** fully green on the final head ([run 36818367970](https://github.com/openclaw/openclaw/actions/runs/36818367970)). Earlier heads hit failures that reproduce on `main` in code this PR doesn't touch: `update-cli.target-schema` (main reproduction: openclaw#158000 (comment)), the type-suppression inventory, the Windows partition owner test and the Windows backup-rename test. Main has since fixed the last three. Config compatibility confirmation: openclaw#158000 (comment). No overlap with Pash/Sarah changes. Co-authored-by: Ayaan Zaidi <hi@obviy.us>
…odels (#167178) ## What Problem This Solves Fixes: the model picker and `models.list` drop every discovered model (for example all OpenRouter rows) while a running Gateway adopts a newly downloaded hosted catalog. If that provider's discovery then fails, the provider stays reduced to its built-in rows until someone runs an explicit refresh. A production Gateway showed this during a fleet catalog refresh: the list fell from 599 to 54 models for about 2.5 to 4 minutes. ## User Impact While a Gateway adopts a new catalog, pickers and `models.list` keep showing the current models and prices. The new catalog's models and prices appear together, once its discovery has finished. Adoption itself now takes as long as that discovery (seconds in practice). The previous catalog keeps serving during that time. ## Why This Change Was Made Adoption (#158000) built each replacement owner with static rows only and published it. It then started that owner's discovery fire-and-forget. Readers therefore saw only configured and catalog rows until discovery finished. A failed discovery left them that way, because failed providers carry no renewal deadline. Adoption now waits for each candidate's own discovery, inside the candidate's catalog scope, before the single commit. The wait can be cancelled through the adoption's existing abort signal. The post-commit fire-and-forget is removed. `loadFullModelCatalog` gets a `wait` option, so adoption awaits the acquisition instead of the 5 s foreground race that ordinary readers use. Rows and prices still switch in one commit, and admitted runs keep their captured pair (#140086). Maintainer decision (Ayaan, delegated): adoption waits for candidate discovery. The previous catalog's rows and prices keep serving coherently in the meantime, which is strictly safer than publishing a partial list. Maintainer decision (Ayaan, delegated): if a provider's discovery fails during adoption, that provider publishes with the new catalog's built-in rows. Rows are never carried across catalog versions (#140086). On `main` this is also the end state, but only after a window in which every discovered row was gone. No overlap with Pash/Sarah changes. ## Evidence Regression test `keeps discovered rows published until the adopted catalog's discovery completes` holds the candidate's discovery and drives the real `models.list` handler. While discovery is held, readers still see the old catalog's discovered row and the old pricing generation. After release, they see only the new catalog's row and generation. It fails on `main` (`expected [ 'remote-200' ] to include 'discovered-200'`) and passes with this change. The existing "bounds refresh with two agents…" test now asserts that adoption publishes only after the held discovery completes. All 10 tests in `prepared-model-runtime.remote-publication.test.ts` pass, and `pnpm check:changed` is green. Live, on an isolated Gateway (loopback, separate state): - A local catalog mirror serves a fresh bundle per step. Each version renames `kimi/k3` to `Kimi K3 vN` and sets the input price of `openai/gpt-5.4-mini` to N $/M. - OpenRouter's DNS resolution is delayed through a Node preload. - `models.list` is polled every 50 ms. - Adoption is triggered with `openclaw models refresh` followed by a `models.list` refresh. | Adoption | `main` (adoption code unchanged on current `main`) | This PR | | --- | --- | --- | | Warm (no delay) | 514 → **1** row (configured only) for 0.74–0.79 s, then 514 with the new marker | 514 rows throughout; marker switches 3.3 s after the trigger | | Candidate discovery delayed 3–4 s (its own DNS lookup) | 514 → **1** row for 5.36 s | 514 rows throughout; marker switches 4.98 s and 5.42 s after the trigger, right after the candidate's lookup completes | | Candidate discovery fails (60 s DNS, 5 s fetch timeout) | 514 → **1** row for 5.38 s, then 51 (OpenRouter 466 → 3 built-in rows) | 514 rows with the old marker for 7.2 s, then 51 with the new marker (decision above) | On the final diff (rebuilt), the result was the same: a warm adoption with 994 rows throughout switched 2.1 s after the trigger. A 3 s delayed candidate discovery kept 994 rows throughout and switched 5.4 s after the trigger. Prices follow the same commit. Usage for real `openai/gpt-5.4-mini` turns was computed at the bundled $0.75/M input before the marked catalogs. After adopting v28, v29 and v32, it was computed at exactly $28/$56, $29/$58 and $32/$64 per M (input/output). Real cheap turns (`openclaw agent … --model openai/gpt-5.4-mini`) completed with reply `ok` after each adoption. One turn on the final build hit a retryable agent-database admission error from the session-state layer, about 30 s after adoption. It is unrelated to the catalog, and the immediate retry succeeded. Local review: `codex review --base origin/main` (model gpt-6-astra), round 1: "No actionable defects were found." Merge danger: a two-way door. No stored state, schema, config or protocol changes, so a revert fully restores the previous behavior. Blast radius: every Gateway that adopts a downloaded hosted catalog. If this is wrong, adoption could stall behind a hung provider discovery. That wait is still bounded by each provider's own discovery timeouts and is cancelled by supersession, config reload or shutdown. Meanwhile the previous catalog keeps serving. Co-authored-by: Ayaan Zaidi <hi@obviy.us>
…odels (openclaw#167178) ## What Problem This Solves Fixes: the model picker and `models.list` drop every discovered model (for example all OpenRouter rows) while a running Gateway adopts a newly downloaded hosted catalog. If that provider's discovery then fails, the provider stays reduced to its built-in rows until someone runs an explicit refresh. A production Gateway showed this during a fleet catalog refresh: the list fell from 599 to 54 models for about 2.5 to 4 minutes. ## User Impact While a Gateway adopts a new catalog, pickers and `models.list` keep showing the current models and prices. The new catalog's models and prices appear together, once its discovery has finished. Adoption itself now takes as long as that discovery (seconds in practice). The previous catalog keeps serving during that time. ## Why This Change Was Made Adoption (openclaw#158000) built each replacement owner with static rows only and published it. It then started that owner's discovery fire-and-forget. Readers therefore saw only configured and catalog rows until discovery finished. A failed discovery left them that way, because failed providers carry no renewal deadline. Adoption now waits for each candidate's own discovery, inside the candidate's catalog scope, before the single commit. The wait can be cancelled through the adoption's existing abort signal. The post-commit fire-and-forget is removed. `loadFullModelCatalog` gets a `wait` option, so adoption awaits the acquisition instead of the 5 s foreground race that ordinary readers use. Rows and prices still switch in one commit, and admitted runs keep their captured pair (openclaw#140086). Maintainer decision (Ayaan, delegated): adoption waits for candidate discovery. The previous catalog's rows and prices keep serving coherently in the meantime, which is strictly safer than publishing a partial list. Maintainer decision (Ayaan, delegated): if a provider's discovery fails during adoption, that provider publishes with the new catalog's built-in rows. Rows are never carried across catalog versions (openclaw#140086). On `main` this is also the end state, but only after a window in which every discovered row was gone. No overlap with Pash/Sarah changes. ## Evidence Regression test `keeps discovered rows published until the adopted catalog's discovery completes` holds the candidate's discovery and drives the real `models.list` handler. While discovery is held, readers still see the old catalog's discovered row and the old pricing generation. After release, they see only the new catalog's row and generation. It fails on `main` (`expected [ 'remote-200' ] to include 'discovered-200'`) and passes with this change. The existing "bounds refresh with two agents…" test now asserts that adoption publishes only after the held discovery completes. All 10 tests in `prepared-model-runtime.remote-publication.test.ts` pass, and `pnpm check:changed` is green. Live, on an isolated Gateway (loopback, separate state): - A local catalog mirror serves a fresh bundle per step. Each version renames `kimi/k3` to `Kimi K3 vN` and sets the input price of `openai/gpt-5.4-mini` to N $/M. - OpenRouter's DNS resolution is delayed through a Node preload. - `models.list` is polled every 50 ms. - Adoption is triggered with `openclaw models refresh` followed by a `models.list` refresh. | Adoption | `main` (adoption code unchanged on current `main`) | This PR | | --- | --- | --- | | Warm (no delay) | 514 → **1** row (configured only) for 0.74–0.79 s, then 514 with the new marker | 514 rows throughout; marker switches 3.3 s after the trigger | | Candidate discovery delayed 3–4 s (its own DNS lookup) | 514 → **1** row for 5.36 s | 514 rows throughout; marker switches 4.98 s and 5.42 s after the trigger, right after the candidate's lookup completes | | Candidate discovery fails (60 s DNS, 5 s fetch timeout) | 514 → **1** row for 5.38 s, then 51 (OpenRouter 466 → 3 built-in rows) | 514 rows with the old marker for 7.2 s, then 51 with the new marker (decision above) | On the final diff (rebuilt), the result was the same: a warm adoption with 994 rows throughout switched 2.1 s after the trigger. A 3 s delayed candidate discovery kept 994 rows throughout and switched 5.4 s after the trigger. Prices follow the same commit. Usage for real `openai/gpt-5.4-mini` turns was computed at the bundled $0.75/M input before the marked catalogs. After adopting v28, v29 and v32, it was computed at exactly $28/$56, $29/$58 and $32/$64 per M (input/output). Real cheap turns (`openclaw agent … --model openai/gpt-5.4-mini`) completed with reply `ok` after each adoption. One turn on the final build hit a retryable agent-database admission error from the session-state layer, about 30 s after adoption. It is unrelated to the catalog, and the immediate retry succeeded. Local review: `codex review --base origin/main` (model gpt-6-astra), round 1: "No actionable defects were found." Merge danger: a two-way door. No stored state, schema, config or protocol changes, so a revert fully restores the previous behavior. Blast radius: every Gateway that adopts a downloaded hosted catalog. If this is wrong, adoption could stall behind a hung provider discovery. That wait is still bounded by each provider's own discovery timeouts and is cancelled by supersession, config reload or shutdown. Meanwhile the previous catalog keeps serving. Co-authored-by: Ayaan Zaidi <hi@obviy.us>
What Problem This Solves
After downloading a hosted catalog, config reload could expose new model rows while fresh requests still used old prices. Startup without a catalog had the same mismatch.
Why This Change Was Made
Capture one rows/pricing snapshot, including absence, at Gateway startup and retain it across later catalog-process replacements. Current enablement and source checks still determine eligibility; download notices no longer imply activation.
User Impact
Rows and price estimates become active together after a full restart. Downloads and config/plugin replacement retain the startup pair. Failed reloads can still interrupt availability.
Evidence
Built Gateway model lists, new agent requests, usage, and real CLI reload reproduced the split on main and a published release. Candidate checks retained the original pair through actual process replacement and activated the new pair only after restart. Controls covered startup absence, source changes, disabled refresh, bad downloads, and failed reload recovery. All 285 focused tests passed. Prices used fixed synthetic usage, not actual charges.
Related: #136257.