Skip to main content
← Back to list
01Issue
BugShippedSwamp CLI
Assigneesstack72

Relationships

#1621 data.latest() is memoized per run, contradicting the documented "sees current-run data" contract

Opened by jamesakeech · 8/12/2026· Shipped 8/12/2026

Summary

data.latest() results are memoized in a Map that lives for the entire workflow run. The first evaluation of a given (namespace, modelName, dataName) triple wins, and every later evaluation anywhere in the run — task inputs, guards, assert steps — reuses that value, even after the run itself has written a new version of that data. Misses are cached too: a null read taken before the data exists is pinned for the rest of the run.

This contradicts the workflow reference. references/workflow/references/data-chaining.md documents data.latest() as a shortcut for data.query() and states that both see current-run data:

Expression Sees current-run data?
data.query('<predicate>') Yes — sync catalog query
data.latest("<name>", "<spec>") Yes — shortcut for query

data.query() does. data.latest() does not. The shortcut has diverged from the primitive it is documented as sugar for.

The most visible consequence is that composing two documented idioms — the "skip if data exists" guard and data.latest() chaining, both from that same document — produces a hard failure on the first run.

Nothing invalidates the cache, and no test covers the behaviour.

Reproduction A — two documented idioms, composed, fail on first run

Clean repo, command/shell models only.

# models/command/shell/<id>.yaml   (name: virgin)
methods:
  execute:
    arguments:
      run: "echo virgin-output"
# workflows/<id>.yaml
name: guard-then-consume
jobs:
  - name: repro
    steps:
      # Documented idiom 1: skip if the output already exists.
      - name: produce
        guard: ${{ data.latest("virgin", "result") }}
        task:
          type: model_method
          modelIdOrName: virgin
          methodName: execute

      # Documented idiom 2: chain that output into the next step.
      - name: consume
        dependsOn:
          - step: produce
            condition: { type: completed }
        task:
          type: model_method
          modelIdOrName: consumer
          methodName: execute
          inputs:
            run: >-
              echo consumed=${{ data.latest("virgin", "result").attributes.stdout }}

With no prior data for virgin:

produce   succeeded
consume   failed: Invalid expression: No such key: attributes

>    1 | data.latest("virgin", "result").attributes.stdout
                                         ^

The guard reads (virgin, result) before anything has written it, caches null, and produce then writes the data. consume re-reads the same coordinates, gets the pinned null, and the dereference fails.

Deleting the guard — same models, same clean state — makes both steps pass. The guard is the cause, and it is the guard the manual recommends.

Reproduction B — read ordering decides correctness

Two workflows differing only by a read-only assert step placed before the write. stamper is a command/shell model; executedAt is stamped fresh by every method run.

# with-early-read
steps:
  - name: early-read              # touches the coordinates, writes nothing
    task:
      type: assert
      expr: >-
        data.latest("stamper", "result").?attributes.?executedAt.orValue("") != "sentinel"

  - name: restamp
    dependsOn: [{ step: early-read, condition: { type: succeeded } }]
    task: { type: model_method, modelIdOrName: stamper, methodName: execute }

  - name: late-read
    dependsOn: [{ step: restamp, condition: { type: succeeded } }]
    task:
      type: assert
      expr: >-
        data.latest("stamper", "result").attributes.executedAt > run.startedAt
Workflow Steps Result
without-early-read restamp → late-read succeeded
with-early-read early-read → restamp → late-read failed — late read saw a pre-run executedAt

late-read asserts only that the data written moments earlier in this run is newer than the run's own start time. It fails because an earlier step in the same run looked at those coordinates.

The same shape reproduces with a guard in place of the assert, and inside a forEach iteration — the expansion path spreads the same context object.

Root cause

src/domain/expressions/model_resolver.ts:764 — buildDataNamespace() creates const latestCache = new Map<string, DataRecord | null>(). The latest implementation checks it at :797 and populates it at :824, :841 and :845 — the last of which caches a miss as null. The key is ${namespace}\0${modelName}\0${dataName} (:796), i.e. the (model, name) pair and not the expression, so data.latest("planner", "plan-summary").attributes.clean and ...attributes.warnings share one entry.

That Map lives as long as the ExpressionContext that closes over it, and the context is built once per run:

  • src/domain/workflows/execution_service.ts:1648 — normal run
  • :1639 — --last-evaluated (light context, same data namespace)
  • :2090 — resume

Each step then takes a shallow copy (:2658), and guard evaluation spreads that copy into its own context (:2690), so the data object — and its latestCache — is one shared instance across every job, step, guard, assert and deferred task input for the whole run. expandArrayItem / expandObjectItem spread the same context, so forEach iterations share it too. There is no invalidation code anywhere.

The cache was added for performance in 2d793440 ("perf(expressions): resolve data.latest() via direct filesystem lookup"), which cut model evaluate on 20K data items from ~17s to ~1.7s. That win is worth keeping — the problem is only its lifetime. The commit describes a lookup-path change, so the run-long lifetime reads as incidental rather than intended.

Why "document the caching instead" is not a way out

Three things rule out retro-documenting the current behaviour as the contract:

  1. It contradicts the data-chaining.md table above, which is the reference authors are pointed at, and which ties data.latest to data.query.
  2. It is not the semantic the other docs claim either. Two references describe a "snapshot taken at workflow start"; this is lazy first-touch memoization, which is a different thing. The shipped behaviour matches no documented semantic.
  3. First-touch is not specifiable across a DAG. Steps within a job run concurrently, so which read populates the cache is order-dependent in a serial DAG and racy in a parallel one. There is no contract to write down.

Nor is there a clean workaround to point authors at. version, listVersions, findByTag, findBySpec and query are uncached, and every write upserts the catalog synchronously in-process (src/infrastructure/persistence/unified_data_repository.ts:168), so they do observe writes made earlier in the same run — data.query was moved to step-execution-stage resolution in f9ffa6fe. But the CEL query helper passes no loadAttributes (model_resolver.ts:908), and DataQueryService only loads attributes when the predicate or the select expression references them (src/domain/data/data_query_service.ts:372) — so the obvious form

data.query('modelName == "m" && name == "n"')[0].attributes.field

silently yields empty attributes. A projection is required, and nothing documents that. model.*.resource does see in-run updates, but it is deprecated and warns.

Real-world case

A plan/apply pair of workflows over a homelab estate. A planner method stamps one plannedAt across every plan-<device> entry and a plan-summary, writing the summary last as a commit marker. Each mutating step guards on entry.plannedAt == data.latest("planner", "plan-summary").attributes.plannedAt, so an entry left over from an earlier run cannot drive an action against a world that has since moved. That interlock is correct only by accident of step ordering: the first read of plan-summary happens to fall after the planner step. Insert any step that reads it earlier and every fresh entry mismatches, every mutating step skips, and the run reports success having done nothing — fail-closed, but silent. Separately, a verify gate at the end of the same workflow (re-plan, then assert the summary is clean) judged the pre-converge plan while the summary on disk said clean=true; that is what first surfaced the behaviour. It was worked around by moving the check inside the method so it no longer depends on a CEL read.

Expected behaviour

data.latest() observes writes made earlier in the same run, matching the data-chaining.md contract and the behaviour of data.query().

Suggested scope of work

  • Core: narrow the lifetime of the latest memo in buildDataNamespace (src/domain/expressions/model_resolver.ts:764) so it does not outlive a single evaluation stage, or invalidate affected keys from the data write path that already updates the catalog synchronously (unified_data_repository.ts:168). The perf goal of 2d793440 is a per-evaluation concern. Cover the chosen semantics with a test — there is none today.
  • Docs: two references currently say data.latest reads a "snapshot taken at workflow start" (.claude/skills/swamp/references/data/references/expressions.md:108 and .../data/references/examples.md:26), which contradicts .../workflow/references/data-chaining.md:87 and should be corrected whichever way the fix lands. A third calls it a "sync disk read" (.../model/references/examples.md:16). And .../workflow/references/execution-semantics.md:57 says the step's expression context "is built" per step, which reads as if each step gets a fresh data namespace; it is a shallow copy sharing one. The guard reference should state plainly whether a guard sees writes made earlier in the same run — that is the single thing a guard author most needs to know, and it is currently unstated.

Environment

  • swamp 20260811.005918.0-sha.c8953e9e, macOS (arm64)
  • Both reproductions run against that build in a clean swamp repo init repository, using only command/shell models
  • Source line references are against a swamp-club/swamp checkout at c8953e9e; latestCache is unchanged on main at 278864d1 (20260812.013400.0-sha.278864d1), so this is not already fixed
  • Observed live on 2026-08-08; reduced to the reproductions above on 2026-08-12
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 4 MOREREVIEW+ 4 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/12/2026, 5:47:39 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/12/2026, 3:06:15 PM
Editable. Press Enter to edit.

stack72 commented 8/12/2026, 5:47:47 PM

Thanks @jamesakeech for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.