feat(run): enforce the Fan-out Limit inside the Run Owner by sauldhernandez · Pull Request #2571 · withcoral/coral

sauldhernandez · 2026-10-20T20:23:26Z

Slice 5 of #2378 (ADR 0018, docs/adr/0018-run-owner-coordinates-each-run.md). Closes #2384. Starts a new gh-stack from main, since slices 0 to 4 merged.

Intent

Bound how many of one Run's Invocations are upstream at once, without hiding queued calls from Reef or history and without changing which Approval Requests become visible. The ADR's Fan-out Limit bullet: at most 32 Invocations of one Run dispatching at once, deployment-wide and bootstrap-configured; admission stays eager; excess ready calls wait as pending and dispatch in readiness order; approval and account waits hold no slot.

What it enables

[runtime] fan_out_limit = 32 in the server configuration, resolved at process bootstrap into RuntimeLimits beside the Applied cap, validated (at least 1) and handed to the Run module whole. No per-Workspace override.
The Run Owner decides each dispatch start in its pure apply against the slots it holds (one per fence request it has batched whose dispatch has not ended). Below the limit a ready Invocation takes a slot and joins the open chunk's fence batch; at the limit it waits in owner memory, pending, with no Event, no required action and no chunk of its own, and joins a batch in readiness order (mailbox arrival order) when an earlier dispatch ends. The count is owner memory only; a recovered owner never dispatches (slice 9), so nothing recomputes it from rows.
Waits for an account connection or an approval happen before any fence request and hold no slot; a refresh wait the fence parks frees its slot before the next fence. Every Approval Request a Run needs is therefore visible at once.
No separate deadline, state, Event or failure code: a stop, the Run deadline, a lost claim or a retiring owner answers a waiting request exactly as it answers an uncommitted fence batch, and termination ends the call as run_terminated.
The slot is a guard the Invocation task holds from the fence's decision until the dispatch it decided is over. An upstream result the owner applies carries the release: the owner frees the slot as it pushes the result's row work, so the Invocation that has waited longest fences in the same chunk the ended row rides, after it, and persistence never shows more than the limit dispatching; a result the owner refuses (over the Run Output Budget) leaves the slot with the task until the row's end is recorded. Every other end (denied, superseded, parked on a refresh, a failure, the Run ending) frees the slot on drop, which tells the owner through Message::DispatchEnded. A decision comes only from a batch that committed and every request of a batch holds a slot, so the count is exact by construction. Once the Run takes no more Program requests (a stop, the deadline, the Program's end, a lost claim), every waiting request is answered at once, as one batch.
The load harness reports invocations.max_dispatching_at_once per fan-out cell (from the rows' dispatch start and end instants) and scripts/load.mjs shows it as max dispatching.

ADR contract lines kept: 3 (dispatch starts only after one authoritative Dispatch Fence; a waiting call fences when it takes its slot, never from a cached decision) and 6 (an Approval binds one exact Invocation and every Approval Request a Run needs is visible at once, since approval waits hold no slot). Lines 1, 2, 4, 5, 7 and 8 are untouched: the wait is owner memory, writes no Event, and the owner code is identical on both backends.

Usage examples

# coral.toml
[runtime]
applied_output_cap_bytes = 8388608
fan_out_limit = 32   # Invocations of one Run upstream at once; at least 1

A Program that fans out past the limit sees nothing new: every call is admitted at once, every approval it needs is asked at once, and calls above the limit start their upstream send in the order they became ready as earlier ones end.

const results = await Promise.all(
  Array.from({ length: 101 }, () => coral.providers.evidence.read({ body: {} })),
);

Reef and the Run's history show the waiting calls as pending Invocations with no dispatch start; a slot wait is never a required action, and a Run canceled while a call waits ends that call as run_terminated.

Tests

crates/coral-run/tests/domain/fan_out.rs, both backends, limit 2: three/four ready calls (two dispatch, the rest stay pending with no Event, no fence attempt and no required action, and dispatch in readiness order as earlier ones end); a call parked on approval holds no slot (two later ready calls dispatch while its Approval Request stays visible; approved, it waits for a slot, then dispatches); a Run terminated while a call waits for a slot ends it as run_terminated with no dispatch start; a refresh wait the fence parks holds no slot (limit 1).
The apply table (tests/domain/owner.rs): slot rows for the queue and readiness order, the result-carried handover riding the same chunk, a slot freed through DispatchEnded with nobody waiting, a stop, the deadline, the Program's end and a lost claim.
coral-app: the configured limit reaches the shared Run module over MCP; a limit of 0, a cap below the minimum and an unknown [runtime] field fail closed.

Validation

Gates on the tree as submitted: make rust-checks (fmt, clippy, migrations, 1449 tests) and make postgres-tests (1449 tests) pass. ./dev test: scripts/load.test.mjs passes; scripts/dev/fs-security.test.mjs fails in this worktree only because apps/reef/node_modules is not installed (it imports Reef's vite.config.ts), unrelated to this change.

Harness (release, Postgres 17 + toxiproxy, --calls 30,101 --latency 0,5,20, and SQLite --calls 30,101), compared cell by cell against the slice 4 final run on the same Mac; zero coral_run=warn lines in every cell:

cell	slice 4 final	slice 5
fan-out 101 calls +20 ms, all visible / admit p95 / dispatch p95 / max dispatching	47.7 s / 2875 ms / 5698 ms / —	49.1 s / 2862 ms / 5195 ms / 4
fan-out 101 calls +5 ms	13.9 s / 829 ms / 1110 ms / —	13.9 s / 816 ms / 1159 ms / 4
fan-out 101 calls +0 ms	0.7 s / 48 ms / 40 ms / —	0.6 s / 52 ms / 21 ms / 32
fan-out 30 calls +0 / +5 / +20 ms, all visible	0.3 / 4.6 / 15.5 s	0.2 / 4.6 / 15.6 s (16 / 3 / 4 dispatching at most)
emit 1000 small +0 / +5 / +20 ms	0.1 / 0.8 / 3.4 s	0.1 / 0.8 / 3.0 s, every Emission observed
concurrent 20 Runs +0 / +5 / +20 ms, all ended	0.5 / 4.6 / 16.8 s	0.3 / 4.7 / 16.4 s; ready failed 0/1, 0/5, 2/12 (s4: 0/1, 0/6, 2/12)
SQLite fan-out 30 / 101 calls, all visible	0.2 / 0.2 s	0.2 / 0.2 s (28 / 32 dispatching at most)

Every slice 4 cell keeps its outcome within noise (fan-out cells from the final tree; emit and concurrent cells from the run before the simplify pass, which touched none of their paths). The 101-call cells show at most 32 dispatching at once and no change to the all-requests-visible time: at +0 ms the dispatches burst and the rows show exactly 32 at once, with dispatch p95 down to 21 ms as the freed slot's fence rides the ended row's chunk; at +5 and +20 ms the harness decides one approval per round trip, so at most 4 dispatch at once and the limit never binds. The concurrent-Runs cell at +5 and +20 ms is unchanged and still outside the readiness budget at +20 ms: each of its Runs makes one call, so the limit cannot reach it; its pressure is twenty Runs' admission chunks and fences on one 10-connection pool (#2391 item 3). Full comparison on #2384.

Bound how many of one Run's Invocations are upstream at once, without hiding queued calls from Reef or history and without changing which Approval Requests become visible (ADR 0018, slice 5 of #2378, #2384). A new bootstrap setting, `[runtime] fan_out_limit` (default 32), is resolved at process bootstrap into `RuntimeLimits` beside the Applied cap and reaches the Run module whole; it is deployment-wide, with no per-Workspace override. The Run Owner decides each dispatch start in `apply` against its in-memory count of dispatching Invocations: a ready Invocation below the limit takes a slot and joins the open chunk's fence batch; one at the limit waits in owner memory, `pending`, with no Event, no required action and no chunk of its own, and joins a batch in readiness order (mailbox arrival order) when an earlier dispatch ends. Waits for an account connection or an approval happen before any fence request and hold no slot; a refresh wait the fence parks frees its slot before the next fence, so every Approval Request a Run needs is visible at once. There is no separate deadline: a stop, the Run deadline, a lost claim or a retiring owner answers a waiting request as it answers an uncommitted batch, and termination ends the call as `run_terminated`. The slot is a guard the Invocation task holds from the fence's decision until the dispatch it decided is over. An upstream result carries the release: the owner frees the slot as it applies the result, so the Invocation that has waited longest fences in the same chunk the ended row rides, after it. Every other end (denied, superseded, parked on a refresh, a failure, the Run ending) frees the slot on drop, which tells the owner through `Message::DispatchEnded`. A decision comes only from a batch that committed and every request of a batch holds a slot, so the count is exact; nothing recomputes it from rows, since a recovered owner never dispatches. Once the Run takes no more Program requests (a stop, the deadline, the Program's end, a lost claim), every waiting request is answered at once, as one batch. The load harness reports `invocations.max_dispatching_at_once` per fan-out cell, from the rows' dispatch start and end instants.

cubic-dev-ai · 2026-10-20T20:26:12Z

Running ultrareview automatically — this PR appears to change concurrency-sensitive code such as locks, queues, or retries, where a missed race may only surface under production load, so a deeper multi-pass review is worth running. I'll post findings when complete.

cubic-dev-ai

Ultrareview ran automatically — this PR appears to change concurrency-sensitive code such as locks, queues, or retries, where a missed race may only surface under production load, so a deeper multi-pass review is worth running.

All reported issues were addressed across 21 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

…end the scripted Program on a stop Cubic's ultrareview of #2571, three findings: - `FENCE_BATCH_CAP`'s comment claimed a batch reaches the cap only under a Fan-out Limit raised above the default; at the default a batch reaches it, and closes its chunk, whenever every slot's request joined one chunk. The comment and the apply table's batch-cap row say so now. - `SequencedUpstream` released sends by ordinal, so a send canceled by the Run's stop burned the permit meant for the next held send. Releases now go to a queue of the sends still waiting (a canceled send has left it), and a release with nobody waiting is banked for the next send. - `ScriptedCallsEngine` read the stop only while waiting for commands; a stop requested after `finish` is now honored before each call is awaited, as the type's doc says.

… back The doc comment on `Message::DispatchEnded` listed the cases in a shorthand nobody could read; it now names them and points at the result message that frees a slot otherwise.

cubic-dev-ai

All reported issues were addressed across 1 file (changes from recent commits).

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

Authorization and Built-in Operation failures after the fence free the slot too, not only a failed send.

Saul's review of #2571, items 1 to 5: - A result the owner refused (over the Run Output Budget, or a Run no longer taking work) still freed its slot, while its row stayed `dispatching` until the Invocation task recorded the failure; the next waiter's fence could commit first, and persistence showed limit + 1 `dispatching` for one commit. The owner now frees the slot only on the branch that pushes the result's row work, and the task keeps the slot when the owner answers with an error, freeing it after the row's end is recorded. The apply table has the row. - The owner's count is `slots_held`, not `dispatching`: it counts batched fence requests from the moment they join a batch, which is not the glossary's `dispatching` row state. The queue is `waiting`, as its accessor already said. - `FanOutLimit::new` rejecting zero is tested in coral-run; the bootstrap test asserts only the setting each error names. - The stop, Program-end and lost-claim rows of the apply table are one table over the ending message and the error it makes. - `FENCE_BATCH_CAP` is a literal again: a chunk bound, not a config default; at the default limit a batch never exceeds it.

cubic-dev-ai

All reported issues were addressed across 5 files (changes from recent commits).

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

🤖 I have created a release *beep* *boop* --- ## [1.2.0](v1.1.0...v1.2.0) (2026-10-22) ### Features * **activity:** record authored source locations for calls and outputs ([#2535](#2535)) ([2d735ca](2d735ca)) * **activity:** show persistent Program source beside selected records ([#2569](#2569)) ([f403819](f403819)) * add the Claude recipe and read gzipped OpenAPI documents ([#2513](#2513)) ([07c27a2](07c27a2)) * **catalog:** share named TypeScript types across a Provider ([#2482](#2482)) ([256e37d](256e37d)) * **desktop:** run the runtime on a Postgres database named in desktop.json ([#2485](#2485)) ([96465a0](96465a0)) * **functions:** type coral.providers, coral.workspace and coral.functions at Saved Function save ([#2526](#2526)) ([a465850](a465850)) * **provider:** create the Coral Built-in Provider at coral.workspace in every new Workspace ([#2349](#2349)) ([06e65ca](06e65ca)) * **provider:** store an agent's createProvider request on its Setup ([#2593](#2593)) ([92847d2](92847d2)) * **provider:** update every Workspace's Built-in Provider catalog at boot ([#2355](#2355)) ([24615d9](24615d9)) * **reef:** add NextDNS provider recipe ([#2509](#2509)) ([e44880e](e44880e)) * **reef:** calmer providers table and a flush app shell ([#2462](#2462)) ([ffe279c](ffe279c)) * **reef:** improve the UI on small screens ([#2531](#2531)) ([75e020a](75e020a)) * **reef:** make the providers rail vertical tabs with an All providers entry ([#2552](#2552)) ([08d007a](08d007a)) * **reef:** providers page UI tweaks ([#2562](#2562)) ([7ca218e](7ca218e)) * **reef:** put the list pages' main action back in the top bar as a primary button ([#2546](#2546)) ([761ea6d](761ea6d)) * **reef:** read ^, $ and * in every filter field ([#2520](#2520)) ([d322356](d322356)) * **reef:** refuse a provider display name another provider uses ([#2518](#2518)) ([fdcd473](fdcd473)) * **reef:** review provider configuration changes in a dialog ([#2579](#2579)) ([24417e9](24417e9)) * **reef:** show search indexing failures on a Provider and its Operations ([#2504](#2504)) ([6cffed0](6cffed0)) * **reef:** show your recent activity summary on the workspace overview ([#2574](#2574)) ([0d1dba5](0d1dba5)) * **reef:** title list pages in their toolbar, with the main action beside it ([#2525](#2525)) ([fd44b72](fd44b72)) * **reef:** UI tweaks to provider's page ([#2566](#2566)) ([62204ca](62204ca)) * **reef:** unify Activity detail panels and action reviews ([#2544](#2544)) ([409d062](409d062)) * return only declared successes from Operation calls ([#2517](#2517)) ([0270bf1](0270bf1)) * **run:** enforce the Fan-out Limit inside the Run Owner ([#2571](#2571)) ([cb53419](cb53419)) * **run:** introduce the Run Owner for Program requests ([#2524](#2524)) ([245d19d](245d19d)) * **run:** let coral.workspace.createProvider configure and publish the Provider itself ([#2352](#2352)) ([088faaf](088faaf)) * **run:** let createProvider configure sign-in and link to Reef's sign-in step ([#2353](#2353)) ([8d5495c](8d5495c)) * **run:** list a workspace's providers with coral.workspace.listProviders ([#2374](#2374)) ([179c43a](179c43a)) * **run:** make the Run Owner survive uncertain commits and lost leases ([#2541](#2541)) ([3a1fad1](3a1fad1)) * **run:** name Tasks and Runs with four-word identifiers ([#2558](#2558)) ([c82259a](c82259a)) * **run:** prepare a Provider from a Program with coral.workspace.createProvider ([#2351](#2351)) ([e7e56fd](e7e56fd)) * **run:** reshape the Dispatch Fence into one batched check per chunk commit ([#2550](#2550)) ([120c9d3](120c9d3)) * **run:** route approval decisions and cancellation through the Run Owner ([#2577](#2577)) ([77bcbf5](77bcbf5)) * **run:** summarize your recent activity in a Workspace ([#2573](#2573)) ([bf4c75e](bf4c75e)) * **search:** show a Saved Function's result fields in its signature line ([#2522](#2522)) ([41365f6](41365f6)) * **search:** trace each reconcile pass and its steps ([#2456](#2456)) ([0594094](0594094)) * **ui:** add option to create a correctly-configured Slack app during Slack provider setup ([#2585](#2585)) ([8c3a883](8c3a883)) * **ui:** move Add provider to the providers toolbar ([#2591](#2591)) ([e1ee4cb](e1ee4cb)) ### Bug Fixes * **app:** sweep device authorization leases every 15 s ([#2452](#2452)) ([1a4e4ae](1a4e4ae)) * **auth:** drop the unverified client name note from the approval page ([#2578](#2578)) ([ddf5aa4](ddf5aa4)) * **catalog:** derive each Built-in Operation's contract from the types Coral parses ([#2582](#2582)) ([534e636](534e636)) * **catalog:** import OpenAPI Providers whose Operations request different OAuth scopes ([#2401](#2401)) ([3acfe55](3acfe55)) * **catalog:** keep keyword names in Operation Call Paths ([#2506](#2506)) ([b20795b](b20795b)) * **catalog:** never make an Operation unavailable for search material ([#2465](#2465)) ([f83c909](f83c909)) * **catalog:** stop rejecting createProvider request headers by name ([#2589](#2589)) ([39e54b9](39e54b9)) * **provider:** ask for an OAuth client ID when Coral cannot register one ([#2580](#2580)) ([baefe9e](baefe9e)) * **provider:** bind each Setup to its starter and hide Coral from Members ([#2480](#2480)) ([6011a64](6011a64)) * **reef:** align the catalog table with the first rail entry ([#2484](#2484)) ([fb59758](fb59758)) * **reef:** calm the Activity badge and keep it while the stream reconnects ([#2468](#2468)) ([1b66639](1b66639)) * **reef:** frame the providers table again and lower the top bar ([#2472](#2472)) ([87e41ee](87e41ee)) * **reef:** pin third-party recipe OpenAPI documents to a commit ([#2510](#2510)) ([b4e9280](b4e9280)) * **reef:** replace loose help text with info tips and banners ([#2461](#2461)) ([695500a](695500a)) * **reef:** round the page's corners beside the sidebar ([#2503](#2503)) ([a1fef64](a1fef64)) * **reef:** shorten OAuth scope chips that share a URL prefix ([#2455](#2455)) ([5ae67ec](5ae67ec)) * **reef:** suggest a provider's namespace from its document name, not its host ([#2565](#2565)) ([8d77893](8d77893)) * **run:** name every argument to correct in an invalid_input failure ([#2354](#2354)) ([07bd727](07bd727)) * **run:** reject a credential beside createProvider's bearer method and report unreachable sources as upstream_incomplete ([#2542](#2542)) ([590eab7](590eab7)) * **run:** report an MCP JSON-RPC rejection as invalid_input or upstream_rejected ([#2402](#2402)) ([749770b](749770b)) * **search:** load each Generation's schema graph once per search ([#2448](#2448)) ([445d945](445d945)) * **search:** retry a failed Saved Function only after one interval ([#2460](#2460)) ([a61ab3a](a61ab3a)) * **search:** return no hits for a query without a letter or digit ([#2561](#2561)) ([bb0a4f0](bb0a4f0)) * **search:** start the write limit at the lock and build rows before the write ([#2575](#2575)) ([06ad55d](06ad55d)) * **search:** stop a self-referential Promise result from overflowing the stack ([#2543](#2543)) ([1d4f257](1d4f257)) * **search:** stop retrying an index write that fails the same way every time ([#2548](#2548)) ([e3b6d15](e3b6d15)) ### Performance Improvements * **catalog:** dedup credential destinations once per build, not per Operation ([#2555](#2555)) ([0afbcf5](0afbcf5)) * **catalog:** project each named TypeScript type once per Provider ([#2443](#2443)) ([153c273](153c273)) * **search:** drop schema text from the phrase test ([#2559](#2559)) ([cfdbafa](cfdbafa)) * **search:** index each word of schema text once ([#2547](#2547)) ([d1123fb](d1123fb)) * **search:** prepare search entries in the indexer after commit ([#2467](#2467)) ([1ab6765](1ab6765)) * **search:** recount only Workspaces whose rows changed ([#2457](#2457)) ([777cfb3](777cfb3)) * **search:** skip unchanged rows when an index write runs again ([#2459](#2459)) ([43b5188](43b5188)) * **search:** write only the statistics rows that changed ([#2458](#2458)) ([8e2832c](8e2832c)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: coral-release-bot[bot] <269256696+coral-release-bot[bot]@users.noreply.github.com> Co-authored-by: release-please[bot] <release-please[bot]@users.noreply.github.com>

sauldhernandez changed the title ~~sauldhernandez/2026 10 16 run owner s5 fanout~~ feat(run): enforce the Fan-out Limit inside the Run Owner Oct 20, 2026

sauldhernandez marked this pull request as ready for review October 20, 2026 20:25

cubic-dev-ai Bot reviewed Oct 20, 2026

View reviewed changes

Comment thread crates/coral-run/src/owner/chunk.rs Outdated

Comment thread crates/coral-run/tests/domain/support/upstreams.rs Outdated

Comment thread crates/coral-run/tests/domain/support/engines.rs

sauldhernandez added 2 commits October 20, 2026 22:41

docs(run): say in plain words when a fenced Invocation gives its slot…

28ba92c

… back The doc comment on `Message::DispatchEnded` listed the cases in a shorthand nobody could read; it now names them and points at the result message that frees a slot otherwise.

cubic-dev-ai Bot reviewed Oct 21, 2026

View reviewed changes

Comment thread crates/coral-run/src/owner/handle.rs Outdated

sauldhernandez added 2 commits October 21, 2026 09:54

docs(run): name every dispatch failure that frees a slot

7d45f2f

Authorization and Built-in Operation failures after the fence free the slot too, not only a failed send.

cubic-dev-ai Bot reviewed Oct 21, 2026

View reviewed changes

Comment thread crates/coral-app/src/bootstrap/server_config.rs

sauldhernandez merged commit cb53419 into main Oct 21, 2026
46 of 48 checks passed

sauldhernandez deleted the sauldhernandez/2026-10-16-run-owner-s5-fanout branch October 21, 2026 08:34

coral-release-bot Bot mentioned this pull request Oct 21, 2026

chore(main): release 1.2.0 #2451

Merged

sauldhernandez mentioned this pull request Oct 22, 2026

Implement the Run Owner coordination model (ADR 0018) #2378

Closed

Navigation Menu

Repository navigation

feat(run): enforce the Fan-out Limit inside the Run Owner - #2571

feat(run): enforce the Fan-out Limit inside the Run Owner#2571
sauldhernandez merged 5 commits into
mainfrom
sauldhernandez/2026-10-16-run-owner-s5-fanout

sauldhernandez commented Oct 20, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot commented Oct 20, 2026

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

sauldhernandez commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

What it enables

Usage examples

Tests

Validation

Uh oh!

cubic-dev-ai Bot commented Oct 20, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

sauldhernandez commented Oct 20, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading