Skip to main content
← Back to list
01Issue
BugOpenExtensions
AssigneesNone

Relationships

#1584 migrate-index silently drops entries with shallow paths — all of workflows-evaluated/ and any root-level file

Opened by sntxrr · 8/10/2026

Summary

swamp datastore migrate-index silently drops index entries whose paths are shallower than partitionKeyFromPath expects. Nothing is logged, the reported shard count looks right, and the resulting v2 sharded index is quietly incomplete.

workflows-evaluated/ is affected 100% of the time, not as an edge case — the layout is one level shallower than the guard requires. Root-level entries such as .catalog-export.json are dropped too.

Observed

Migrating a real datastore (18,791 entries):

[s3-sync] Migrating monolithic index to shard-first: 18791 entries → 102 shard(s)
[s3-sync] Migration complete: 102 shard(s) written, _meta.json v2 with commitSeq=1

Summing the entries actually written across all 102 shards gives 18,788. Three are missing:

.catalog-export.json
workflows-evaluated/workflow-063a0cb1-....yaml
workflows-evaluated/workflow-e14e03ef-....yaml

No warning, and no discrepancy between "18791 entries" and what was written is reported.

Cause

datastores/_lib/s3_cache_sync.ts, partitionKeyFromPath:

static partitionKeyFromPath(rel: string): string | undefined {
  const segments = rel.split("/");
  if (segments.length < 2) return undefined;          // (a)
  const subdir = segments[0];
  switch (subdir) {
    ...
    case "workflow-runs":
    case "workflows-evaluated": {
      if (segments.length < 3) return undefined;      // (b)
      return `${subdir}--${segments[1]}`;
    }

and groupEntriesByPartition:

const key = S3CacheSyncService.partitionKeyFromPath(rel);
if (!key) continue;                                   // silent

workflow-runs and workflows-evaluated share a case arm, but they are not the same depth. Measured across the whole index:

subdir actual segment counts guard result
workflow-runs {3: 1334} >= 3 ok
workflows-evaluated {2: 2} >= 3 all dropped
root file {1: 1} >= 2 at (a) dropped

workflow-runs/<workflowId>/<file> is 3 segments; workflows-evaluated/<file>.yaml is 2. Applying the former's requirement to the latter drops every entry.

For reference, the other depths in the same index: data 5-8, outputs 5-6, definitions-evaluated 4-5, auto-definitions 4.

It does not self-heal

Still missing after four subsequent commits (_meta.json now commitSeq=4). The entries never reappear, because nothing re-derives them.

Severity: low, and it fails safe — but please do not close on that basis

  • The S3 objects still exist. Verified by head-object; this is index-only, not data loss. .catalog-export.json is 5.8 MB and intact.
  • It fails in the safe direction. toDelete is computed from index entries, so a smaller index yields fewer deletion candidates, never more. This cannot trigger a mass delete.
  • Practical cost is churn: affected files look absent to the index and get re-pushed.

The reason I would still fix it: the silent continue is the same shape as the silently-skipped push that hid swamp-club#1557 here for seven days. And .catalog-export.json is the catalog, not an incidental file — an index that does not know it exists is a poor foundation for anything that later trusts the index as authoritative.

Suggested fix

  1. Split the case arm so workflows-evaluated requires >= 2 and partitions on segments[0] (or its own scheme), while workflow-runs keeps >= 3.
  2. Give root-level files a partition (a _root shard) rather than dropping them at check (a).
  3. Do not continue silently. Either count and log unpartitionable entries, or fail the migration — a migration that drops entries without saying so is worse than one that refuses to run.
  4. Assert sum(shard entries) == entryCount before writing _meta.json, so this class of bug cannot ship again.

Environment

  • CLI 20260809.004828.0-sha.b61c9de2
  • @swamp/s3-datastore@2026.08.07.1
  • Remote datastore with prefix + namespace, 18,791 index entries, 102 shards

Migration was otherwise clean: additive, monolith retained as fallback, zero objects removed, and doctor passes on both writers afterwards.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.08.07.1
  • swamp: 20260809.004828.0-sha.b61c9de2
  • OS: darwin (aarch64)
  • Deno: 2.8.3
  • Shell: /bin/zsh
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

8/10/2026, 9:26:05 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.