Skip to content

feat(activity): record authored source locations for calls and outputs - #2535

Merged
lazybean merged 10 commits into
mainfrom
activity-record-locations
Oct 21, 2026
Merged

lazybean merged 10 commits into
mainfrom
activity-record-locations

Conversation

@lazybean

@lazybean lazybean commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Operation call / Program Log / Program Emission
  → capture the runtime caller
  → resolve through the existing authored source map
  → identify the Run's accepted Program or Saved Function snapshot
  → persist optional location with the record
  → expose through authorized Activity summaries and details

Record 1-based lines and UTF-16 columns without UI guessing. Error frames and Activity locations share the same source-map lookup. Saved Function linking preserves authored text rather than reconstructing positions from reprinted tokens. Private or unmapped code and legacy records have no location.

Backend foundation only: no source-pane UI, Live controls, transitions, or Middleware. Additive migration: crates/coral-db/migrations/0030_record_source_locations.sql; existing migrations are unchanged.

Evidence

  • Before: runtime caller information used for error reporting was not durably attached to successful Operation calls, Logs, or Emissions. Reprinting a Saved Function containing a trailing comma could also lose its authored mapping; the new regression failed against the previous linker.
    After: runtime, domain, Activity projection, and server restart tests cover recorded locations and accepted Saved Function snapshots; the source-preserving linker passes the trailing-comma regression. Inline-default-export tests also verify that authored calls before inserted private code retain exact locations, including UTF-16 columns.
  • Current reviewed head: e56537969dadb2f77db5e30db84573df43124071, rebased on 87209e520b0e02130393f0de8de5620d471c1e8c. Independent project Standards and Spec review found no blocking correctness defect.
  • Local validation on this head: Rust checks and PostgreSQL tests each passed with 1,446 passed and 6 skipped; compiler tests, embedded JavaScript checks, and migration guard passed.
  • Published-head CI: CI run 39426087241 is green, including Required, Rust/Postgres, Rust checks, Embedded JS, Reef, and Reel suites. The first Postgres attempt hit a 40P01 deadlock in saved_function_descriptions_survive_replacement_and_deletion; the rerun passed without code changes. The initial deadlock's cause has not been established.

Merge Danger

Door: one-way for the durable schema/history change; two-way for the application implementation.

Migration 0030 adds nullable columns and consistency checks to Operation Invocation and Run Event storage. No destructive migration or backfill is introduced; legacy rows remain readable without a location. However, reverting application code does not undo an applied migration or erase recorded evidence. Treat schema rollback as a separate, deliberate migration/compatibility decision—not a simple Git revert. Once consumers rely on the new API evidence, retain its compatibility contract.

Blast Radius: runtime

Touches Saved Function compilation/source mapping (including existing error frames), host-call capture, Invocation and Run Event persistence on SQLite/Postgres, and Activity API projections. Incorrect mapping can misidentify authored source; incorrect persistence can reject writes or lose optional evidence. Capture stays synchronous at each call, accepted snapshots remain authoritative, and locations use existing authorization paths. No live database or deployed service was modified during validation.

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Running ultrareview automatically — this PR appears to change a database schema or migrate data, where a missed bug can corrupt or lose records, so a deeper multi-pass review is worth running. I'll post findings when complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultrareview ran automatically — this PR appears to change a database schema or migrate data, where a missed bug can corrupt or lose records, so a deeper multi-pass review is worth running.

3 issues found across 33 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/coral-app/tests/grpc/record_locations_tests.rs">

<violation number="1" location="crates/coral-app/tests/grpc/record_locations_tests.rs:142">
P3: The restart assertions validate only internal consistency: the SavedFunction document in each event is built from the same response's `snapshot.saved_function`, so a lost or placeholder snapshot identity on both sides would still pass. Since the PR's contract is preserving exact snapshot identity across restart, pin the returned snapshot against the recorded source: `assert_eq!(snapshot.source, SAVED_FUNCTION);` (and optionally check `snapshot.path == "coral.functions.report.forecast"`).</violation>
</file>

<file name="crates/coral-grpc/tests/grpc/activity_evidence.rs">

<violation number="1" location="crates/coral-grpc/tests/grpc/activity_evidence.rs:1294">
P3: The expected positions (1,9 / 2,13 / 2,13 / 3,37 / 4,32) are bare literals with no note of which record each belongs to or that columns point at the called member name. Name the expected locations (e.g. `let console_warn = program(1, 9); let emit = program(2, 13); let info = in_snapshot(2, 13); let read_in_snapshot = in_snapshot(3, 37); let read_in_program = program(4, 32);`) so the mapping to the two sources is auditable when the assertion fails.</violation>
</file>

<file name="crates/coral-v8/src/js/compiler/saved_function_linking.js">

<violation number="1" location="crates/coral-v8/src/js/compiler/saved_function_linking.js:47">
P3: `printNode` does not terminate function or class declarations with semicolons, so this overstates the printer behavior; describe only the statement forms where it inserts them.</violation>
</file>

Tip: Get a Slack DM when cubic finishes reviewing your PRs. Connect Slack

Re-trigger cubic

Comment thread crates/coral-v8/src/lib.rs Outdated
.await
.unwrap()
.into_inner();
let [snapshot] = detail.linked_functions.as_slice() else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The restart assertions validate only internal consistency: the SavedFunction document in each event is built from the same response's snapshot.saved_function, so a lost or placeholder snapshot identity on both sides would still pass. Since the PR's contract is preserving exact snapshot identity across restart, pin the returned snapshot against the recorded source: assert_eq!(snapshot.source, SAVED_FUNCTION); (and optionally check snapshot.path == "coral.functions.report.forecast").

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At crates/coral-app/tests/grpc/record_locations_tests.rs, line 142:

<comment>The restart assertions validate only internal consistency: the SavedFunction document in each event is built from the same response's `snapshot.saved_function`, so a lost or placeholder snapshot identity on both sides would still pass. Since the PR's contract is preserving exact snapshot identity across restart, pin the returned snapshot against the recorded source: `assert_eq!(snapshot.source, SAVED_FUNCTION);` (and optionally check `snapshot.path == "coral.functions.report.forecast"`).</comment>

<file context>
@@ -0,0 +1,179 @@
+        .await
+        .unwrap()
+        .into_inner();
+    let [snapshot] = detail.linked_functions.as_slice() else {
+        panic!("one linked snapshot")
+    };
</file context>

column,
)
};
let outputs = vec![program(1, 9), program(2, 13), in_snapshot(2, 13)];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The expected positions (1,9 / 2,13 / 2,13 / 3,37 / 4,32) are bare literals with no note of which record each belongs to or that columns point at the called member name. Name the expected locations (e.g. let console_warn = program(1, 9); let emit = program(2, 13); let info = in_snapshot(2, 13); let read_in_snapshot = in_snapshot(3, 37); let read_in_program = program(4, 32);) so the mapping to the two sources is auditable when the assertion fails.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At crates/coral-grpc/tests/grpc/activity_evidence.rs, line 1294:

<comment>The expected positions (1,9 / 2,13 / 2,13 / 3,37 / 4,32) are bare literals with no note of which record each belongs to or that columns point at the called member name. Name the expected locations (e.g. `let console_warn = program(1, 9); let emit = program(2, 13); let info = in_snapshot(2, 13); let read_in_snapshot = in_snapshot(3, 37); let read_in_program = program(4, 32);`) so the mapping to the two sources is auditable when the assertion fails.</comment>

<file context>
@@ -1220,3 +1220,161 @@ async fn invocation_wire_rejects_an_http_error_status_with_its_evidence_and_keep
+            column,
+        )
+    };
+    let outputs = vec![program(1, 9), program(2, 13), in_snapshot(2, 13)];
+
+    // InspectRun: authored output Events carry a location; others never do.
</file context>

Comment on lines +47 to +51
// The printer also ends every statement it prints with a semicolon, including
// ones the author left to automatic semicolon insertion, so a printed semicolon
// without an authored match is skipped. Authored tokens are never skipped, and
// any other difference stops the alignment, so earlier tokens stay unmapped
// rather than guessed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: printNode does not terminate function or class declarations with semicolons, so this overstates the printer behavior; describe only the statement forms where it inserts them.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At crates/coral-v8/src/js/compiler/saved_function_linking.js, line 47:

<comment>`printNode` does not terminate function or class declarations with semicolons, so this overstates the printer behavior; describe only the statement forms where it inserts them.</comment>

<file context>
@@ -35,6 +44,11 @@ function equivalentPrintedToken(left, right) {
 // Saved Function linking changes only each retained statement's export prefix.
 // Align the unchanged token suffix so printer whitespace and blank-line removal
 // cannot move runtime frames away from the original authored token positions.
+// The printer also ends every statement it prints with a semicolon, including
+// ones the author left to automatic semicolon insertion, so a printed semicolon
+// without an authored match is skipped. Authored tokens are never skipped, and
</file context>
Suggested change
// The printer also ends every statement it prints with a semicolon, including
// ones the author left to automatic semicolon insertion, so a printed semicolon
// without an authored match is skipped. Authored tokens are never skipped, and
// any other difference stops the alignment, so earlier tokens stay unmapped
// rather than guessed.
// The printer can add a semicolon to a statement the author left to automatic
// semicolon insertion, so an unmatched printed semicolon is skipped. Authored
// tokens are never skipped, and any other difference stops the alignment, so
// earlier tokens stay unmapped rather than guessed.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/coral-v8/src/js/compiler/saved_function_linking.js">

<violation number="1" location="crates/coral-v8/src/js/compiler/saved_function_linking.js:21">
P2: Custom agent: **Unsupported claims**

A split-line edit also leaves the authored prefix unmapped, despite this comment saying only private lines and the return are unmapped. Record a mapping for the prefix or document that it also remains unmapped; otherwise code before an inline export can lose its authored runtime location.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

printedIndex -= 1;
// Copies the authored source with each edit's span blanked and its private
// line, if any, on generated lines of its own before the span. Every other
// generated line holds one authored line, or the rest of one, at its authored

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Custom agent: Unsupported claims

A split-line edit also leaves the authored prefix unmapped, despite this comment saying only private lines and the return are unmapped. Record a mapping for the prefix or document that it also remains unmapped; otherwise code before an inline export can lose its authored runtime location.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At crates/coral-v8/src/js/compiler/saved_function_linking.js, line 21:

<comment>A split-line edit also leaves the authored prefix unmapped, despite this comment saying only private lines and the return are unmapped. Record a mapping for the prefix or document that it also remains unmapped; otherwise code before an inline export can lose its authored runtime location.</comment>

<file context>
@@ -1,88 +1,65 @@
-  let authoredIndex = authoredTokens.length - 1;
+// Copies the authored source with each edit's span blanked and its private
+// line, if any, on generated lines of its own before the span. Every other
+// generated line holds one authored line, or the rest of one, at its authored
+// columns, so one column-zero mapping per line is exact and TypeScript's own
+// map supplies the column. Private lines and the return stay unmapped.
</file context>

@lazybean
lazybean force-pushed the activity-record-locations branch 2 times, most recently from 1abfd9a to e565379 Compare October 20, 2026 12:55
@blacksmith-sh

This comment has been minimized.

@lazybean
lazybean added this pull request to stack #2570 October 20, 2026 15:25
@lazybean
lazybean force-pushed the activity-record-locations branch 2 times, most recently from 1df3d15 to 2fe5e8d Compare October 21, 2026 11:17
lazybean and others added 10 commits October 21, 2026 13:18
…, log and emission

Capture the innermost authored frame synchronously inside each V8 host
binding (Operation calls, console.debug/info/log/warn/error, coral.emit),
map it through the Run's retained authored source map, and persist it with
the Invocation or Event. A Saved Function location names the Run's
snapshot, so later edits or deletion cannot move it. Records without a
location, including all earlier ones, read back as unavailable.

Expose the location as ActivitySourceLocation on ActivityInvocation,
ActivityRunEvent and ActivityEventSummary. Lines follow ECMAScript line
terminators; columns count UTF-16 code units.

Count authored lines with TypeScript's own line starts, so a CR, U+2028 or
U+2029 in accepted source no longer drops later Program locations and
error frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…semicolons

Saved Function linking reprints each statement and maps it back to the
authored source by aligning tokens. The printer ends every statement with
a semicolon, so a body the author left to automatic semicolon insertion
stopped the alignment at its first statement and lost every location and
error frame before it. Skip only those inserted semicolons; any other
difference still stops alignment, leaving positions unavailable rather
than guessed.

Take both token lists from the parse tree instead of a context-free
scanner, which split template continuations and regular expressions
wrongly and shifted the column of a later call on the same line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…earch's steps

Universal Search moved its migrations into the shared chain as versions
0030 to 0033, so the record locations migration shared version 0030 with
search_text_search. Use 0035, after credential_token_expiry. The SQL is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ields

The Dispatch Fence fixtures build an admitted Invocation and an accepted
Run directly. Give them no source location and no linked snapshots, as an
Invocation without a captured location and a Run without Saved Functions
have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…index revision steps

Main now uses version 0035 for the search index indexer revision and 0036 for
the paired search entities fillfactor steps. Move the record locations
migration, which is not on main, to the next free version 0037. Its SQL is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's Fan-out Limit fixture invokes the Program host without an authored
location argument. Give it None, like the other synthetic calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nversion

Recording each Invocation's source location took the invoke path one line past
Clippy's length limit. Return the Run's termination through From<Error>, which
gives the same Fatal error, in one line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lazybean
lazybean force-pushed the activity-record-locations branch from 2fe5e8d to 21a9793 Compare October 21, 2026 11:19
@lazybean
lazybean merged commit 2d735ca into main Oct 21, 2026
25 checks passed
@lazybean
lazybean deleted the activity-record-locations branch October 21, 2026 11:38
lazybean added a commit that referenced this pull request Oct 21, 2026
#2569)

## Summary

```text
Activity drawer (one persistent split per Run)
├── Selected record: evidence and review decisions in its own footer
└── Accepted source: read-only Program / Saved Function snapshot
    ├── highlight the selected operation, log or emission's recorded location
    ├── always reveal the selected line; preserve the split and source DOM
    └── missing-location banner only for operations, logs and emissions
```

Adds the persistent, resizable Program source pane, recorded-location
lookup and annotated code highlighting to Activity. Removes the
redundant “Accepted program source” / “Read-only snapshot” labels and
Follow selection switch. Run-scoped story records resolve to a Run and a
Program containing their recorded call; intentional missing-location
cases remain covered. Keeps review decisions below their record evidence
and omits the redundant history link for program logs.

Stack parent: #2544 (`activity-side-panels`), above #2535. Intended
source-only delta: four commits above local parent
`9ef47ba5a085f83fd90dab2eae718bdcb294cde7`.

**Draft dependency:** parent PR #2544 still publishes pre-restack head
`860ec2c57c128e0d0b76cae3f4396b9e0c728648`. Until its owner publishes
the restacked parent, GitHub's diff includes inherited parent changes.
This PR does not authorize publishing or modifying parent/sibling
branches. Source head: `fa3182961c6189050fb4ca1d6074307c687a7dca`.

## Evidence

- **Before:** focused regression run on the previous UI failed four
split tests: redundant labels/toggle remained, and Runs, out-of-source
Run error frames and non-call Run events displayed the missing-location
banner. A source-location regression also failed because a non-call Run
event inherited a call location.
- **After:** the implementation report records those regressions
passing. Story plays assert the exact recorded highlighted line, its
“Line N” status and matching call in the Run's Program.
- Historical local validation on pre-restack snapshot
`6fa548d4df6220cb8f19ecd8c98dce0d253cae87` (committed as
`571518678b630546c73f3728c6bcb473fb20278f`): `npm run check`, `npm run
typecheck`, `npm test`, `npm run build`, and `npm run test:server` all
exited 0. Test report: 1,896 unit tests and 814 Chromium story tests.
- **Not yet validated:** these results are from before the restack, not
a fresh run on this PR head. The Reel `activity/source-locations`
SQLite/Postgres/local-mode journeys have not been rerun after their
helper changed. Current-head CI and post-restack gates remain pending;
this is not a landability claim.

## Merge Danger

**Door:** two-way

The intended delta above the restacked parent changes Reef UI, loaders,
tests and fixtures; no new migration or durable-data write is introduced
by this layer. Revert this layer to undo its presentation and
location-resolution behavior. Parent migrations and inherited history
are outside that rollback boundary.

**Blast Radius:** Activity

Source scrolling now always follows a newly selected record when its
line is out of view. Missing-location warnings disappear for Runs and
non-call events. Split resizing, code annotation, Saved Function
snapshots, approval footer placement and mobile layout need review.
Story snapshots change across Activity, side panels, pending and rail.
Shared fixtures reject call paths absent from their Program. Do not
merge until the parent PR is updated and final-head validation is
complete.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
sauldhernandez added a commit that referenced this pull request Oct 21, 2026
…Owner (#2577)

Slice 6 of #2378 (ADR 0018). Closes #2385.

## Intent

Approval decisions and cancellation were the last Program-era writers of
a nonterminal Run's row and Events from outside its Run Owner, each
under `hold_run`, and the owner's own chunk commit took `hold_run` first
to tolerate them. This slice makes both Run Requests the owner records,
so the Run Ledger has one writer while a Run is nonterminal, the chunk
commit holds no Run row, and the in-memory head predicate switches on.

## What it enables

- **Decide Approval** (`ApprovalModule::decide`,
`RunModule::decide_approval`, `answer_action_prompt`, Reef over gRPC,
the MCP prompt) and **Cancel** (`RunModule::cancel`, MCP `cancel`, Reef)
are asks to the Run's owner, answered at Recorded. A request is keyed by
the entity it decides (`RequestKey::Approval(id)`,
`RequestKey::Termination`): the first wins and rides the next chunk; a
repeat waits for that chunk's answer or, once the entity is recorded, is
answered from owner memory with no chunk; the requester then reads the
record (the decision, who made it, when), so a loser learns it lost.
- The decision chunk reads no policy and takes no policy lock:
`DecideWork` records Approved or Declined (Expired at the request
deadline; Closed when the Run no longer accepts actions) and ends a
declined Invocation. The read before the write decides every outcome the
request can have (a request no longer pending answers from the record
with no write), so a write that fails with a definite answer fails the
chunk as any row work's does; the Dispatch Fence remains the authority
on policy when the Invocation dispatches. Two consequences: a decline
made after the Operation was disabled is recorded as the decline and the
Invocation ends `approval_declined` (before: Closed with the policy
failure); and **a persistence fault in the decision's own write now ends
the Run** (the owner stops, the Program is terminated, the Run ends
Interrupted with the request Closed by the terminal write, and the
approver is answered `OwnerLost`), where before it answered the approver
with the database error and left the Run waiting for a retry. A definite
failure there is a bug or a corrupt row, and the chunk's other work is
not worth a savepoint per decision to isolate it.
- `cancellation.requested` stays an Event: the owner pushes it in
arrival order and writes the intent columns (kind, instant, requester)
in that chunk's head update, so the gRPC field, the Event decoder and
Reef's timeline are unchanged. The owner terminates the Program at once
(Applied), the request is answered when the chunk lands, and
`RunModule::cancel` then waits for the owner to end the Run within the
shutdown grace before returning the record.
- The Run deadline reaches the owner through its mailbox
(`Message::Deadline`): the owner stops the Program and the timeout is
the intent the terminal write records (the Run row refuses chunks past
its deadline, so none carries it). A cancel past the deadline is
answered once that write has run. **There is no outside deadline writer
beside a live owner**: `begin_deadline_shutdown` and its `hold_run` are
gone. Once the owner retired, the deadline task runs recovery's pass for
its Run (`recover_run`, idempotent: a Run the owner ended is nothing to
recover), so an owner that retired on a failed write still leaves a Run
that ends at its deadline, as the outside writer guaranteed before.
Slice 7 moves the timer itself into the owner's `apply` beside the
approval and connection deadlines.
- A stopped Program that never reports its end (a Program blocked in a
wait that is not stop-aware) is ended once the shutdown grace is out
(`Effect::GraceTimer` → `Message::GraceExpired`), as the force-settle
after the grace did before.
- The owner ends every nonterminal Invocation and the Run itself: the
settle of a recorded termination is one attempt at a time between the
owner's commits (`Effect::Settle { again }`,
`Message::SettleAttempted`), never beside a chunk commit, with the grace
judged on the host's clock. `end_nonterminal_invocations` stays inside
the terminal chunk. Recovery's `complete_shutdown` waits for a live
owner in this process to retire before it settles anything (the owner
ends its Run within its own grace; the force after the grace applies
only to the settle that follows), so no two terminal writers run beside
each other here; and should two still meet (another process's owner
during a rolling update, until slice 9), an Invocation one of them ended
first reads as `HeadMismatch` to the other, which reads the Run again,
as it would after its own head update mismatched. Found by the Postgres
gate: `recovery_settles_a_lost_publication_from_its_publication` failed
about once in five runs because recovery's forced settle ran beside the
owner's and both ended the same Invocation, which the hold on the Run
row used to serialize.
- `OwnerLost` when the owner stopped before answering (outcome unknown:
read again). New `Error::OwnedElsewhere` for a nonterminal Run with no
owner in this process (its lease held by another process during a
rolling update, or lapsed and not yet swept): retryable, nothing
written; slice 9's sweep ends such Runs. A terminal Run answers every
request from its record. `owner::local_owner` is the one lookup, and
`ApprovalModule::new` takes the Run Owners it decides through
(`RunModule::approvals()` builds the production module); a module over
an empty registry sees every nonterminal Run as owned elsewhere, which
the owned-elsewhere test uses on purpose.
- Adapters change only their error mapping: coral-grpc maps `OwnerLost`
to `ABORTED` and `OwnedElsewhere` to `UNAVAILABLE`; coral-mcp projects
`run_owner_lost` and `run_owned_elsewhere`, and the prompt answer treats
`OwnerLost` like a request resolved elsewhere (the reread shows the
state). Both keep reading Recorded state.

### The chunk commit and the head

`commit_once` reads the Run row without holding it and asserts the head
the owner tracks in memory (set by the start mark, moved by every commit
and by the terminal write). A row that disagrees is classified in
evidence order: terminal → `RunEnded`; another execution claim →
`ClaimMismatch`; a head below the owner's → `Corrupt`; a released worker
claim → `Refused` (recovery's write, until slice 9); a head the owner
did not move → `Corrupt` (slice 3's rule takes effect;
`HEAD_MOVES_OUTSIDE_THE_OWNER` is gone). A head update that matches no
row is reclassified from a reread. The owner's other writes (the start
mark, the terminal write, the Interrupted record, the settle) read the
row without a hold too and are serialized by the owner running one write
at a time.

### `hold_run` inventory (on `main` before this PR: 11 call sites)

| caller | now |
| --- | --- |
| `approval.rs` `ApprovalModule::decide` | gone (owner ask) |
| `owner/host.rs` chunk commit | gone (no hold; in-memory head) |
| `termination.rs` `cancel` | gone (owner ask) |
| `termination.rs` `begin_deadline_shutdown` | gone (deadline through
the mailbox) |
| `termination.rs` `settle_termination` | gone (read without a hold;
owner-serialized, and recovery's settle waits for a live owner here to
retire) |
| `termination.rs` `interrupt_owned_run` | gone (read without a hold;
the write's claim-and-head predicate fences it) |
| `worker.rs` `start_execution`, `complete_execution` | gone (read
without a hold; the owner's own serialized writes; the start mark is
also predicated on the worker claim, which recovery may release before
the start) |
| `termination.rs` `recover_run` | **stays, slice 9** (the expired-lease
sweep replaces it; it is the one outside writer left, explicit host
maintenance only) |
| `retention.rs` `hold_in_transaction` | **stays, slice 9** (terminal
Runs only) |

## Contract (ADR 0018)

Lines 1 (one writer, serial chunk commits), 4 (no Event after
`run.ended`; no writer other than the owner touches Run-scoped rows
while the Run is nonterminal), 5 (every ledger commit predicated on the
claim id and the head, never the clock), 6 (approvals independent; the
first durable decision wins). Readers never read owner memory: the
winning decider's view is the authorizing snapshot's row with the
chunk's recorded state, instant and approver filled in (the Run's own
fields in that view are as the snapshot read them); a loser, and
`cancel`, read a fresh snapshot after the owner's answer.

Two narrow cases to know: a decision whose request an earlier chunk
closed between the authorizing snapshot and the owner's chunk still
commits that chunk (its work writes nothing; the head moves once), and
`ApprovalModule::new` is now crate-private with `RunModule::approvals()`
the public constructor, since a module built without the Run Owners
would see every nonterminal Run as owned elsewhere.

## Usage examples

Nothing changes for callers that already decide or cancel:
`RunModule::decide_approval`, `RunModule::cancel`, the gRPC
`DecideApproval` / `CancelActivityRun` methods and the MCP `cancel` tool
keep their shapes. Two answers are new and both mean "read the record
again":

```text
gRPC   ABORTED      "request outcome unknown; read the run again"   (OwnerLost)
gRPC   UNAVAILABLE  "run is owned by another process; retry"        (OwnedElsewhere)
MCP    run_owner_lost / run_owned_elsewhere
```

A domain caller:

```rust
match runs.cancel(&actor, &workspace, &run, &observation).await {
    Ok(update) => update,                       // the record after the owner ended the Run
    Err(Error::OwnedElsewhere) => retry_later(),   // another process holds the lease
    Err(Error::OwnerLost) => runs.observe(..),     // outcome unknown: the record decides
    Err(other) => return Err(other),
}
```

## Tests

`crates/coral-run/tests/domain` on both backends:
- `approvals.rs`:
`two_deciders_on_one_request_are_answered_in_mailbox_order_and_the_loser_learns_the_winner`
(the first's chunk held; the second parked and answered with the first's
record; a third from memory; one lifecycle event, one dispatch),
`a_decision_on_a_terminated_run_closes_the_obsolete_request_and_records_no_approval`
(during the settle, and once terminal),
`an_owner_that_dies_after_recording_a_decision_before_answering_reports_owner_lost`
(`OwnerLost`, reread Approved, supervisor interrupts),
`a_decision_for_a_run_owned_elsewhere_is_refused_without_a_write`,
`a_decision_persistence_fault_ends_the_run_and_records_no_partial_decision`
(replaces
`approval_event_failure_rolls_back_decision_and_invocation_settlement`:
the fault is a partial unique index on decided lifecycle events, so only
the decision's write fails);
`a_decision_at_the_request_deadline_expires` and the two policy-change
tests adapted to a decision that reads no policy.
- `cancellation.rs`:
`a_cancel_during_an_in_flight_fence_is_applied_at_once_and_run_ended_follows_the_fence`,
`a_second_cancel_on_an_ending_run_joins_no_chunk`,
`a_cancel_for_a_run_owned_elsewhere_is_refused_without_a_write`
(replaces the independent-worker cancel), the queued cancel and queued
deadline tests on the owner's order.
- `resilience.rs`:
`a_run_whose_owner_retired_on_a_failed_settle_is_ended_by_its_deadline`
(the deadline task's recovery pass; fails without it),
`a_foreign_head_read_before_the_write_stops_the_owner_without_a_write`
(the chunk commit's own read refuses a foreign head, one attempt,
nothing written).
- `ledger.rs`: `a_start_mark_on_a_released_worker_claim_is_refused` (the
start mark is predicated on the worker claim acceptance wrote; fails
without the predicate).
- `owner.rs` `apply` table:
`apply_records_outside_requests_once_in_mailbox_order_and_applies_a_cancel_at_once`
(two deciders; a cancel during an in-flight fence and its repeats; the
deadline against a decision; a cancel past the deadline; a Program that
never reports its end; a lost claim answering every requester
`OwnerLost`; a cancel before the start mark; a decision during the
terminal write and during the settle). The settling block of the
existing table follows the serialized settle.
- coral-grpc: `OwnerLost` / `OwnedElsewhere` rows in the status table.
coral-mcp:
`owner_lost_and_owned_elsewhere_project_stable_retryable_codes`.

## Validation

Gates on the final tree (2026-10-21, this Mac, after the simplify pass,
the removal of the decision savepoint, the required `owners` argument
and the recovery-settle fix): `make rust-checks` (fmt, clippy `-D
warnings`, migrations check, workspace tests on SQLite) 1460 passed;
`make postgres-tests` 1460 run, 1459 passed, the one failure
`coral-search
a_timeout_after_the_workspace_lock_waits_for_a_newer_coral_version`, a
search-index timing test this branch does not touch (main fixed that
path in #2575 after this branch's base), green on three isolated reruns.
The previously flaky
`recovery_settles_a_lost_publication_from_its_publication` passed 12 of
12 isolated Postgres runs after the fix (it failed about 1 in 5 before).
coral-run alone: 211 tests on both backends. None of these passes
changed a database round trip on a measured path (owner-memory shapes;
three statements fewer per decision chunk; recovery's wait), so the
harness below was not rerun on them.

After merging `main` (ed9fbf3: #2535's source locations on Emitted and
Logged Events, three mechanical hunks in `run_repository/events.rs` and
the ledger tests): `make rust-checks` 1477 passed; `make postgres-tests`
1477 run, 1476 passed, the one failure `coral-run
credentials::a_path_argument_cannot_send_shared_credentials_to_an_undeclared_endpoint`
timing out its one-second terminal-state wait under the full-workspace
load (6.6 s in that run), green on three isolated reruns and in every
earlier Postgres run of this branch.

Harness on the final head (2026-10-21, release, Postgres 17 + toxiproxy,
`--calls 30,101 --latency 0,5,20`; SQLite `--calls 30,101`; the cancel
path with `--cells fanout --mode hold` on both backends), compared cell
by cell against the slice 5 runs; the full summary is on #2385, the
reports under the worktree's `.context/load-s6/2026-10-21/`. Zero
`coral_run=warn` lines in every cell.

- **Every slice 4/5 burst cell keeps its outcome**: all Approval
Requests visible 0.2 / 4.6 / 15.6 s for 30 calls and 0.6 / 14.1 / 49.4 s
for 101 at +0 / +5 / +20 ms (s5: 0.2 / 4.6 / 15.6 and 0.6 / 13.9 /
49.1); admit p95 44 / 381 / 1405 ms and 47 / 818 / 2812 ms (s5: 38 / 395
/ 1392 and 52 / 816 / 2862); lock-wait ceiling 0 / 1 / 1 unchanged;
dispatch p95 lower in every latency cell (the fence chunk no longer
waits behind a decision's `hold_run`). Emit cells within noise (1000
emits 0.1 / 0.8 / 3.2 s, every Emission observed, same heads).
Concurrent 20 Runs 0.2 / 4.5 / 17.1 s (s5: 0.3 / 4.7 / 16.4), `ready`
failed 0/1, 0/5, 2/12 unchanged (still outside budget at +20 ms: #2391
item 3).
- **The decide phase moved, up**: decide p50 / p95 at +5 ms 359 / 1157
ms (s5: 342 / 618) and at +20 ms 3314 / 8894 ms (s5: 1928 / 4958) for
101 calls; the 101-call +20 ms lifetime 474 s (s5: 299 s). A decision is
now two pool acquisitions (the authorizing snapshot and the owner's
chunk) where it was one, and the decide phase's pool is saturated by the
parked waits' polling (`BEGIN` p50 0.7 s in that cell, before and after
this slice), so each acquisition pays the pool wait. The first run read
the record back in a third acquisition (decide p95 9.7 s, lifetime 508
s); the final head answers the first decision from what its chunk
recorded. Slice 8 removes the saturation.
- **Cancel path** (no earlier slice ran it): every call parked on its
Approval Request, the Run cancelled after the steady phase: cancel
returned in 0.06 / 1.1 / 3.9 s for 30 calls and 0.11 / 3.2 / 17.6 s for
101 at +0 / +5 / +20 ms, every Invocation `run_terminated`, the Run
`canceled`; the 101-call +20 ms figure is the terminal chunk ending 101
Invocations at one round trip each. SQLite: 12 / 15 ms.
- **SQLite parity**: every cell runs; fan-out visible 0.2 s (29 / 32 at
once), 1000 emits 0.1 s, 20 Runs 0.1 s.
simonwhitaker pushed a commit that referenced this pull request Oct 22, 2026
🤖 I have created a release *beep* *boop*
---


## [1.2.0](v1.1.0...v1.2.0)
(2026-10-22)


### Features

* **activity:** record authored source locations for calls and outputs
([#2535](#2535))
([2d735ca](2d735ca))
* **activity:** show persistent Program source beside selected records
([#2569](#2569))
([f403819](f403819))
* add the Claude recipe and read gzipped OpenAPI documents
([#2513](#2513))
([07c27a2](07c27a2))
* **catalog:** share named TypeScript types across a Provider
([#2482](#2482))
([256e37d](256e37d))
* **desktop:** run the runtime on a Postgres database named in
desktop.json ([#2485](#2485))
([96465a0](96465a0))
* **functions:** type coral.providers, coral.workspace and
coral.functions at Saved Function save
([#2526](#2526))
([a465850](a465850))
* **provider:** create the Coral Built-in Provider at coral.workspace in
every new Workspace
([#2349](#2349))
([06e65ca](06e65ca))
* **provider:** store an agent's createProvider request on its Setup
([#2593](#2593))
([92847d2](92847d2))
* **provider:** update every Workspace's Built-in Provider catalog at
boot ([#2355](#2355))
([24615d9](24615d9))
* **reef:** add NextDNS provider recipe
([#2509](#2509))
([e44880e](e44880e))
* **reef:** calmer providers table and a flush app shell
([#2462](#2462))
([ffe279c](ffe279c))
* **reef:** improve the UI on small screens
([#2531](#2531))
([75e020a](75e020a))
* **reef:** make the providers rail vertical tabs with an All providers
entry ([#2552](#2552))
([08d007a](08d007a))
* **reef:** providers page UI tweaks
([#2562](#2562))
([7ca218e](7ca218e))
* **reef:** put the list pages' main action back in the top bar as a
primary button ([#2546](#2546))
([761ea6d](761ea6d))
* **reef:** read ^, $ and * in every filter field
([#2520](#2520))
([d322356](d322356))
* **reef:** refuse a provider display name another provider uses
([#2518](#2518))
([fdcd473](fdcd473))
* **reef:** review provider configuration changes in a dialog
([#2579](#2579))
([24417e9](24417e9))
* **reef:** show search indexing failures on a Provider and its
Operations ([#2504](#2504))
([6cffed0](6cffed0))
* **reef:** show your recent activity summary on the workspace overview
([#2574](#2574))
([0d1dba5](0d1dba5))
* **reef:** title list pages in their toolbar, with the main action
beside it ([#2525](#2525))
([fd44b72](fd44b72))
* **reef:** UI tweaks to provider's page
([#2566](#2566))
([62204ca](62204ca))
* **reef:** unify Activity detail panels and action reviews
([#2544](#2544))
([409d062](409d062))
* return only declared successes from Operation calls
([#2517](#2517))
([0270bf1](0270bf1))
* **run:** enforce the Fan-out Limit inside the Run Owner
([#2571](#2571))
([cb53419](cb53419))
* **run:** introduce the Run Owner for Program requests
([#2524](#2524))
([245d19d](245d19d))
* **run:** let coral.workspace.createProvider configure and publish the
Provider itself
([#2352](#2352))
([088faaf](088faaf))
* **run:** let createProvider configure sign-in and link to Reef's
sign-in step ([#2353](#2353))
([8d5495c](8d5495c))
* **run:** list a workspace's providers with
coral.workspace.listProviders
([#2374](#2374))
([179c43a](179c43a))
* **run:** make the Run Owner survive uncertain commits and lost leases
([#2541](#2541))
([3a1fad1](3a1fad1))
* **run:** name Tasks and Runs with four-word identifiers
([#2558](#2558))
([c82259a](c82259a))
* **run:** prepare a Provider from a Program with
coral.workspace.createProvider
([#2351](#2351))
([e7e56fd](e7e56fd))
* **run:** reshape the Dispatch Fence into one batched check per chunk
commit ([#2550](#2550))
([120c9d3](120c9d3))
* **run:** route approval decisions and cancellation through the Run
Owner ([#2577](#2577))
([77bcbf5](77bcbf5))
* **run:** summarize your recent activity in a Workspace
([#2573](#2573))
([bf4c75e](bf4c75e))
* **search:** show a Saved Function's result fields in its signature
line ([#2522](#2522))
([41365f6](41365f6))
* **search:** trace each reconcile pass and its steps
([#2456](#2456))
([0594094](0594094))
* **ui:** add option to create a correctly-configured Slack app during
Slack provider setup
([#2585](#2585))
([8c3a883](8c3a883))
* **ui:** move Add provider to the providers toolbar
([#2591](#2591))
([e1ee4cb](e1ee4cb))


### Bug Fixes

* **app:** sweep device authorization leases every 15 s
([#2452](#2452))
([1a4e4ae](1a4e4ae))
* **auth:** drop the unverified client name note from the approval page
([#2578](#2578))
([ddf5aa4](ddf5aa4))
* **catalog:** derive each Built-in Operation's contract from the types
Coral parses ([#2582](#2582))
([534e636](534e636))
* **catalog:** import OpenAPI Providers whose Operations request
different OAuth scopes
([#2401](#2401))
([3acfe55](3acfe55))
* **catalog:** keep keyword names in Operation Call Paths
([#2506](#2506))
([b20795b](b20795b))
* **catalog:** never make an Operation unavailable for search material
([#2465](#2465))
([f83c909](f83c909))
* **catalog:** stop rejecting createProvider request headers by name
([#2589](#2589))
([39e54b9](39e54b9))
* **provider:** ask for an OAuth client ID when Coral cannot register
one ([#2580](#2580))
([baefe9e](baefe9e))
* **provider:** bind each Setup to its starter and hide Coral from
Members ([#2480](#2480))
([6011a64](6011a64))
* **reef:** align the catalog table with the first rail entry
([#2484](#2484))
([fb59758](fb59758))
* **reef:** calm the Activity badge and keep it while the stream
reconnects ([#2468](#2468))
([1b66639](1b66639))
* **reef:** frame the providers table again and lower the top bar
([#2472](#2472))
([87e41ee](87e41ee))
* **reef:** pin third-party recipe OpenAPI documents to a commit
([#2510](#2510))
([b4e9280](b4e9280))
* **reef:** replace loose help text with info tips and banners
([#2461](#2461))
([695500a](695500a))
* **reef:** round the page's corners beside the sidebar
([#2503](#2503))
([a1fef64](a1fef64))
* **reef:** shorten OAuth scope chips that share a URL prefix
([#2455](#2455))
([5ae67ec](5ae67ec))
* **reef:** suggest a provider's namespace from its document name, not
its host ([#2565](#2565))
([8d77893](8d77893))
* **run:** name every argument to correct in an invalid_input failure
([#2354](#2354))
([07bd727](07bd727))
* **run:** reject a credential beside createProvider's bearer method and
report unreachable sources as upstream_incomplete
([#2542](#2542))
([590eab7](590eab7))
* **run:** report an MCP JSON-RPC rejection as invalid_input or
upstream_rejected
([#2402](#2402))
([749770b](749770b))
* **search:** load each Generation's schema graph once per search
([#2448](#2448))
([445d945](445d945))
* **search:** retry a failed Saved Function only after one interval
([#2460](#2460))
([a61ab3a](a61ab3a))
* **search:** return no hits for a query without a letter or digit
([#2561](#2561))
([bb0a4f0](bb0a4f0))
* **search:** start the write limit at the lock and build rows before
the write ([#2575](#2575))
([06ad55d](06ad55d))
* **search:** stop a self-referential Promise result from overflowing
the stack ([#2543](#2543))
([1d4f257](1d4f257))
* **search:** stop retrying an index write that fails the same way every
time ([#2548](#2548))
([e3b6d15](e3b6d15))


### Performance Improvements

* **catalog:** dedup credential destinations once per build, not per
Operation ([#2555](#2555))
([0afbcf5](0afbcf5))
* **catalog:** project each named TypeScript type once per Provider
([#2443](#2443))
([153c273](153c273))
* **search:** drop schema text from the phrase test
([#2559](#2559))
([cfdbafa](cfdbafa))
* **search:** index each word of schema text once
([#2547](#2547))
([d1123fb](d1123fb))
* **search:** prepare search entries in the indexer after commit
([#2467](#2467))
([1ab6765](1ab6765))
* **search:** recount only Workspaces whose rows changed
([#2457](#2457))
([777cfb3](777cfb3))
* **search:** skip unchanged rows when an index write runs again
([#2459](#2459))
([43b5188](43b5188))
* **search:** write only the statistics rows that changed
([#2458](#2458))
([8e2832c](8e2832c))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: coral-release-bot[bot] <269256696+coral-release-bot[bot]@users.noreply.github.com>
Co-authored-by: release-please[bot] <release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant