Skip to main content
← Back to list
01Issue
BugClosedSwamp CLITeam
Assigneesstack72

Relationships

#1600 swamp serve: data.query fails permanently with "UNIQUE constraint failed: catalog.*" — one duplicate catalog 5-tuple aborts every catalog rebuild

Opened by magistr · 8/11/2026

Summary

On a swamp serve node (MongoDB datastore, SQLite _catalog.db), every data.query (CEL predicate scan) fails with:

UNIQUE constraint failed: catalog.namespace, catalog.type_normalized, catalog.model_id, catalog.data_name, catalog.version

after ~75s. data.get / data.latest (lookup by name) keep working. Any workflow with a data.query assert gate therefore breaks: the assert errors (rather than evaluating to false), so downstream steps blocked on condition: succeeded never run and the whole workflow fails — even for a benign, low-severity, allowFailure-style gate.

Impact

A single poisoned local record wedges all data.query on the node, persistently, across restarts (local data persists in the HOME volume). In our case it silently broke a daily digest workflow whose gate is size(data.query('modelName == "reading" && specName == "message" && tags.workflowRunId == run.id')) > 0 — the compose step succeeded, but the gate threw and blocked the send.

Root cause (observed)

There is one duplicate 5-tuple in the node's local _catalog.db: the same (model, name, version) exists as a local record with one id and as a datastore record with a different id — i.e. a version number was reused by two writes that raced during a boot-reconciliation cron-replay storm (serve replays every missed cron on boot; overlapping replays wrote the same next-version twice).

The catalog rebuild does a plain INSERT per record scanned (local files + datastore sync). The second insert of the duplicate 5-tuple aborts the entire build, so the query fails. A different node (a laptop CLI) pointed at the same datastore rebuilds fine, because it never materialized the duplicate local record — confirming the datastore itself is clean and the corruption is node-local.

Notes / non-fixes

  • --log-level debug gives no per-row trace — the rebuild is a single SQL operation, ~5 log lines then the error.
  • Deleting _catalog.db + restart does not fix it — the rebuild re-hits the duplicate among local files.
  • Removing one model's local dir does not fix it — any data.query triggers a full repo catalog build, so the dup can be in any model.
  • Recovery required moving all local model data aside and letting the node re-sync from the clean datastore.

Repro sketch

On a serve node, induce two writes of the same (model, name) that both compute the same next version (e.g. boot-reconciliation replaying overlapping scheduled runs of a workflow that writes a versioned step-output), then run any data.query.

Asks

  1. A read (data.query) must not fail on a catalog INSERT. The catalog build should dedupe / upsert (INSERT … ON CONFLICT DO NOTHING/UPDATE) or skip-and-log a duplicate 5-tuple, rather than aborting the whole rebuild. One bad record should not wedge every query on the node.
  2. Prevent the version reuse at write time — serialize next-version assignment per (model, name), or detect and reject a reuse, so overlapping boot-replay runs can't create the duplicate in the first place.
02Bog Flow
✓OPEN○TRIAGED○IN PROGRESS◉CLOSED+ 1 MOREASSIGNED+ 2 MOREREVIEWAPPROVAL

Closed

8/11/2026, 4:53:38 PM

No activity in this phase yet.

03Sludge Pulse
stack72 assigned stack728/11/2026, 3:20:21 PM
Editable. Press Enter to edit.

stack72 commented 8/11/2026, 4:53:37 PM

@magistr — thanks for the detailed report, especially the root cause analysis and repro sketch.

The core symptom (catalog rebuild crashing with UNIQUE constraint failed) was fixed earlier today in #1581 — the backfill switched from bulkReplaceAll (plain INSERT that aborts on any duplicate 5-tuple) to bulkUpsert (INSERT OR REPLACE), which is in release v20260811.024318.0. Updating your serve node to that version should resolve the persistent data.query failures.

Sign in to post a ripple.