Skip to main content
← Back to list
01Issue
BugOpenSwamp CLI
AssigneesNone

Relationships

#1639 workflow resume truncates the run's log file, losing every earlier attempt's records

Opened by stevendc · 8/13/2026

What happens

swamp workflow resume overwrites the run's log file instead of appending to it, so swamp workflow history logs <run> for a resumed run shows only the records of the last attempt. Everything the earlier attempts logged is gone from disk, and nothing in the output says so — a run that failed at step 12 and was resumed from step 12 shows a log that begins at step 12, which reads as a short, clean run.

Observed directly: one log line before the resume, a different single line after, same file, same run id, the first line gone. Full standalone reproduction below.

Why it matters to us: we drive a PostgreSQL major-version upgrade where a failed run is resumed with --from. An operator reading the log mid-incident — database down — sees a handful of lines ending on a success and has to infer which step failed from DAG order and the absence of a line. We first read this as "history logs truncates to ten lines", and it took a while to work out that the file itself had been rewritten.

Reproduction

Self-contained: a fresh repo, one throwaway model, one two-step workflow. No credentials, no datastore configuration, no external services. Tested on 20260813.015841.0-sha.687b5fc4, Linux x86_64.

mkdir /tmp/resumeprobe && cd /tmp/resumeprobe
swamp init
mkdir -p extensions/models

extensions/models/manifest.yaml:

name: resumeprobe
description: Throwaway probe asking what a resume does to a workflow run's log file
version: "2026.08.13.1"
models:
  - resumeprobe.ts

extensions/models/resumeprobe.ts — second fails until a sentinel file exists, so the same run can be failed and then resumed to success:

import { z } from "npm:zod@3";

export const model = {
  type: "@probe/resumeprobe",
  version: "2026.08.13.1",
  globalArguments: z.object({}),
  resources: {},
  methods: {
    first: {
      description: "Log a line and succeed",
      arguments: z.object({}),
      execute: (
        _args: unknown,
        context: { logger: { info(msg: string): void } },
      ) => {
        context.logger.info("FIRST step ran");
        return Promise.resolve({});
      },
    },
    second: {
      description: "Fail unless /tmp/resumeprobe-ok exists, then log and succeed",
      arguments: z.object({}),
      execute: async (
        _args: unknown,
        context: { logger: { info(msg: string): void } },
      ) => {
        try {
          await Deno.stat("/tmp/resumeprobe-ok");
        } catch {
          throw new Error("SECOND step failed on purpose");
        }
        context.logger.info("SECOND step ran");
        return {};
      },
    },
  },
};

workflows/workflow-probe.yaml — replace the id with any valid v4 UUID:

id: 3f8c1a92-5d4e-4b7a-9c1f-2e6d8a4b5c7d
name: probe
description: Two steps; the second fails until a sentinel file exists.
tags: {}
jobs:
  - name: run
    description: Two steps.
    steps:
      - name: step_one
        description: Logs and succeeds.
        task:
          type: model_method
          modelIdOrName: probe
          methodName: first
          inputs: {}
        dependsOn: []
      - name: step_two
        description: Fails until the sentinel exists.
        task:
          type: model_method
          modelIdOrName: probe
          methodName: second
          inputs: {}
        dependsOn:
          - step: step_one
            condition:
              type: succeeded

Then:

swamp model create @probe/resumeprobe probe
swamp workflow validate probe

rm -f /tmp/resumeprobe-ok
swamp workflow run probe                      # step_one succeeds, step_two fails
cat .swamp/workflow-runs/*/workflow-run-*.log
# 2026-08-13T15:21:57.930Z [INF] model·method·run·probe·first: FIRST step ran

touch /tmp/resumeprobe-ok
swamp workflow resume probe --from step_two    # step_two now succeeds
cat .swamp/workflow-runs/*/workflow-run-*.log
# 2026-08-13T15:22:08.421Z [INF] model·method·run·probe·second: SECOND step ran

One line before, one line after, same file, same run id — FIRST step ran is gone. swamp workflow history logs probe shows the same single line, since it prints that file.

Keep the repo fresh for this: each workflow run starts a new run with its own log file, so on a repo where the workflow has been run more than once that cat glob spans several files and the overwrite is much harder to see. With one run there is exactly one file, before and after — ls .swamp/workflow-runs/*/*.log | wc -l returns 1 either side. (A second failed run also makes resume --from refuse until you pass --run <run-id>.)

Likely mechanism

Offered as a lead rather than a finding — I read this out of the compiled binary, not the source, so please verify against the real thing:

  • RunFileSink.register() opens its target with { write: true, create: true, truncate: true }.
  • The resume path appears to call it with the existing run's own log path (existingRun.logFile ?? join(swampPath(repoDir, SWAMP_SUBDIRS.workflowRuns), workflow.id, "workflow-run-" + existingRun.id + ".log")), under a comment about re-registering the sink so resume output is captured.

Truncating is right for a new run and wrong for a resumed one, which is consistent with what the file shows.

Ruled out

So nobody re-treads these:

  • Not a display cap. history logs --tail has no default — the code path is input.tail ? allEntries.slice(-input.tail) : allEntries — and omitting --tail prints the whole file.
  • Not category filtering. The sink is registered with the catch-all [] prefix, and the surviving records are the same category as the vanished ones.
  • Not our models. The reproduction above logs from a throwaway model with no dependencies.

Expected

A resume should append to the run's log, so the file is the whole run across attempts. A marker line at the join would make the attempt boundary legible.

If truncation on resume is deliberate, one file per attempt (workflow-run-<run-id>.<attempt>.log) with history logs concatenating them in order would also solve it, and would additionally make "what did attempt 1 do?" answerable.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

8/13/2026, 3:22:40 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.