@swamp/s3-datastore push failed (code=ConditionalRequestConflict): S3
putObjectConditional failed HTTP 409 ConditionalRequestConflict — The
conditional request cannot succeed due to a conflicting operation
against this resource.

Environment: @swamp/s3-datastore on OVH S3 ([HOST-1], region us-east-va), one namespace; the workflow runs serve-side, so every step completion pushes from the same process.

Mechanism: each step completion triggers its own datastore push, and both pushes rewrite shared namespace files (observed on .catalog-export.json). The writes go out as conditional PUTs; the loser of the race gets HTTP 409, the datastore has no retry for it, and the conflict surfaces as a push failure that fails an otherwise successful step.

Evidence: the journal shows two "Preparing push" lines inside the same second immediately before each failure. The "failed" push's objects verifiably land — correct content, Last-Modified at the failure second, index shards and _root.json updated. Direct S3 probes with the same bucket and credentials are clean: 25/25 PUT→HEAD→GET loops immediately consistent, unconditional PUT and DELETE fine. swamp run doctor clean. Reproduced 4/4 runs on the same workflow; earlier runs passed on luckier step timing.

Impact: any workflow whose sibling steps complete close together flaps — a step reports failure and downstream asserts never run, while the pushed data is intact in the bucket. A false negative on the run, not data loss.

Suggested fix: retry a conditional PUT on 409/ConditionalRequestConflict with a re-read of the current object and backoff, or serialize pushes per namespace inside the serve process so concurrent step completions queue instead of racing.

Workaround in use: serializing the formerly-sibling steps in the workflow DAG. That removes the deterministic pair but cannot close the race for steps on independent DAG roots, so intermittent flaps remain possible.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

Extension: @swamp/s3-datastore@2026.08.12.1
swamp: 20260820.072114.0-sha.05d20b6a
OS: linux (x86_64)
Deno: 2.8.3
Shell: /bin/bash

02Bog Flow

Open

8/20/2026, 2:15:31 PM

No activity in this phase yet.

03Sludge Pulse

data prune still misclassifies live records as orphans on 20260820 (after #1396) — serve-minted token/grant records, and report data of every live workflow

wss:// direct to serve's own TLS listener still fails on 20260817 — client ALPN lands on h2, WS-over-h2 unimplemented (follow-up to #1363)

Concurrent pushes to one namespace fail with HTTP 409 ConditionalRequestConflict — the conditional PUT is never retried

serve: triggerSource not persisted to workflow run summary

dataOutputOverrides not applied during remote execution — ephemeral resources GC'd before override takes effect

extension rm also deletes the lockfile entry, and install on an empty lockfile is a silent success — rm/install is not a round trip

vault: audit trail should record put, delete, and annotate operations

ExtensionWorkflowRepository scans manifest.yaml as a workflow, emitting 'Unknown key manifestVersion' warning

--max-run-duration only applies to detached runs, not attached runs

Cancel endpoint reports status: cancelled for runs it did not actually cancel

@swamp/git: no way to amend a commit, but push can --force

Installation with new account creation times out

pre-flight checks run on orchestrator, not on worker — breaks remote execution with filesystem-dependent checks

Docs: update cancel endpoint response statuses in REST API reference and workflows page

docs: document --remote-only flag for swamp serve

Swamp serve daemon doesn't run with --hot-reload

serve: include workflowName in HealthSnapshotSchedule

Retyping a model instance strands catalog rows no command can reclaim — prune matches on model_id, delete resolves the current type (follow-up to #1009)

knowledge-base retrieve persists query output into the state resource, with lifetime infinite and the query text in the instance name

Method-name collision between a base model type and an extending extension silently drops the whole extension file — no error, doctor reports pass

Skip 50MB monolithic index rewrite on v2 shard-first path

Share single AwsS3Client instance across lock, sync, and verifier

Double shard assembly per command lifecycle (pull + push both read all shards)

pullChanged never records commitSeq for shard-first (v2) index

localHasAllRemoteEntries() permanently blocks sidecar update on partial caches

serve: workflow.history.get cannot find CLI-created runs or look up by run ID

"TRIAL ENDED" modal shown to team plan user when toggling extension visibility

Concurrent suspended workflow runs can't be disambiguated from 'approvals'; fresh runs should supersede matching-input suspended runs

I _think_ I might like a guard in the trigger yaml

swamp cmds don't realise the user is sshed into the system

serve dashboard: add cluster.instances and serve.config endpoints

CEL expressions have timestamp/duration but no current-time value, so no guard can express "older than N"

Support direct type execution (@type) with --server

access token mint writes to user vault, not serve's _token-secrets control plane in HA mode

Generic OTLP endpoint appends signal path after query or fragment

Issue lifecycle start drops top-level issue comments

serve dashboard: add instanceId, triggerSource, and step progress to WorkflowRunSummary

Document worker affinity (affinity: true) for workflows and jobs

Datastore MongoDB driver leaks connections — 5960 accumulated over 8 days

workflow approvals displays the latest run's prompt for every suspended run

@swamp/git: add a read-only working-tree diff for pre-commit review

docs: document --oauth-client-name flag for swamp serve

Docs: update workflow reference to include trigger in JSON output shapes

@swamp/git branch and push methods fail data naming when branch contains '/'

Docs: update workflow-placement.md for job/workflow-level placement inheritance

feat(serve): use identifiable OAuth client_name instead of random UUID

@swamp/git does not pass AbortSignal to Git subprocesses

Global Secret Manager methods build an invalid regional parent

Document model.method() support in assert expressions

No way to store code-review results — verdicts, findings, and cost are not persisted for any repo

Redaction regex boundary bug: label_base64 secrets with an unrecognized label pass through unredacted (e.g. swamp_ API keys)

User token page (/u/<user>) shows only "swamp_..." for every API key, making leaked-key identification impossible

upstream_state treats sparse-fetch branches as having no upstream

Add native support for the Pi coding harness

Cloud Build create supplies project instead of required projectId path parameter

workflows: support labels/target/platform at job and workflow level

swamp access grant create never registers the grant — a --deny created this way silently fails open

Support credentials in datastore config (vault-resolvable)

swamp serve OOMs on repos with 2000+ model definitions

Serve telemetry attribution for collectives

Extension workflows: no way to override trigger configuration

Add a list/discover method to the generated @swamp/aws types — no way to enumerate resources in a region

Document @vault= prefix for webhook secrets in serve.yaml

swamp access grant create installs an unrelated extension from the registry

docs: document SWAMP_API_KEY headless OAuth bootstrap for swamp serve

workflow-succeeded selects the globally-latest run, so siblings block each other under parallel use

swamp serve: webhook secrets should support @vault= resolution

Misleading collective_not_trusted auto-resolve error when a local source extension transiently fails to index during workflow validate

datastore setup: symlinks not counted in bytesCopied, causing misleading '(0B)' output

vault.get() CEL error embeds raw newline when wrapped by model resolver, producing broken output

Feature: user-configurable redaction rules (.swamp/redaction.yaml)

Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit

Sync every signup to Attio, and backfill the existing population (#809 follow-up)

@swamp/issue-lifecycle probes /api/health, which swamp-club stopped serving when the Fresh scaffold stubs were removed

Lab lifecycle POST rejects any payload string that starts with a dollar sign, not just operator keys

Feed: detect and gracefully handle dead (404) post links (link rot)

/feed: per-post unread indicators and mark-as-read to track reading progress

/feed: filter and search by author, title/text, and topic

Add a :micdrop: emoji / reaction shortcode (swamp-native art included)

Push down the two cheap leaderboard scans: ghost-gate dictionary and eventCounts owner filter