Skip to main content
← Back to list
01Issue
BugShippedSwamp CLI
Assigneeskeeb

Relationships

#1580 Catalog backfill deletes rows for any model whose data names are numeric

Opened by keeb · 8/10/2026· Shipped 8/10/2026

Summary

The catalog backfill's directory walk misclassifies a type directory as a model-id directory whenever a model's data names are purely numeric. Every data item under that model is silently dropped from the walk, and because backfill commits its results with bulkReplaceNamespace (a DELETE + reinsert), the model's catalog rows are then deleted.

The effect: for an affected model, any swamp data query — regardless of predicate — wipes that model's rows from _catalog.db. Writes repopulate them, the next query deletes them again. It never self-heals, so the model is permanently invisible to data query, data search, and queryData() in extensions.

Data on disk is never touched — only catalog rows. data get and data list keep working, which is what makes this so quiet: the model looks fine until you try to query it.

Root cause

src/infrastructure/persistence/unified_data_repository.ts, isModelIdDirectory() / isModelIdDirectorySync():

for await (const subEntry of Deno.readDir(childPath)) {
  if (subEntry.isDirectory && /^\d+$/.test(subEntry.name)) {
    return true;
  }
}

The heuristic reads a numeric grandchild directory as a version dir (1/, 2/, 3/). But a numeric directory name is ambiguous — it is equally a legitimate data name. A model keying resources by an external numeric id (TMDB id, issue number, SKU) produces:

data/@scope/type/{model-id}/207333/2/metadata.yaml
                            ^^^^^^ data name, not a version

So isModelIdDirectory("data/@scope/type") sees grandchild 207333, returns true, and collectAllData takes the model-id branch with pathSegments = ["@scope"]:

const typeStr = typeSegments.join("/");   // "@scope"
try {
  const modelType = ModelType.create(typeStr);   // throws
  ...
} catch {
  // Skip invalid model types      <-- every record under the model is dropped here
}

ModelType.create("@scope") throws, the bare catch {} swallows it, and the walk returns zero records for that model.

backfillAsync then commits the RT-less row set:

const ns = this.dataRepo.namespace;
if (ns && ns.length > 0) {
  this.catalogStore.bulkReplaceNamespace(ns, rows);   // DELETE FROM catalog WHERE namespace = ?
} else {
  this.catalogStore.bulkReplaceAll(rows);             // DELETE FROM catalog
}

Anything the walk missed is deleted.

Worth noting: collectByTaggedName in the same file already does this correctly, using the existing looksLikeModelId() UUID check rather than the structural heuristic. The two walkers have drifted.

Reproduction

Real-world case: a @keeb/rottentomatoes model writing one resource per show, keyed by TMDB id (207333, 124364, ...). 314 resources on disk, 0 rows in the catalog since May.

$ swamp model method run rottentomatoes browse --input sort=popular
  [rottentomatoes/browse] wrote 28 show resources

$ sqlite3 .swamp/data/_catalog.db \
    "SELECT COUNT(*) FROM catalog WHERE type_normalized='@keeb/rottentomatoes';"
30

# any query at all — note the predicate does not mention this model
$ swamp data query 'modelName == "tmdb"'

$ sqlite3 .swamp/data/_catalog.db \
    "SELECT COUNT(*) FROM catalog WHERE type_normalized='@keeb/rottentomatoes';"
0

A predicate matching literally nothing (name == "__nonexistent__") wipes them just the same — backfill runs before the predicate is applied.

Only this one model was affected out of 21 in the repo, because it is the only one whose data names are bare integers. Sibling models keyed by slug/title (mal, tmdb, jellyfin, subsplease-schedule) catalog normally.

Every query also emits one debug line, which is the visible symptom:

[DBG] data·repository: findAllForModel called with model name "rottentomatoes"
      instead of a UUID — use context.readModelData("rottentomatoes") for
      cross-model access by name

Minimal synthetic repro — save two data items under one model and walk:

await repo.save(testType, "model-1", makeData("207333"), enc("x"));
await repo.save(testType, "model-1", makeData("recommendations"), enc("y"));

const found = await repo.findAllGlobal();
// expected: ["207333", "recommendations"]
// actual:   []                              <-- both dropped

Note both items vanish, not just the numeric one — the misclassification happens one level up, so the whole model goes.

Impact

  • Any model keyed by an external numeric id is permanently unqueryable.
  • Failure is silent: an empty result set is indistinguishable from "no matching data", so a negative query result cannot be trusted for these models. I originally hit this while trying to confirm a show was absent from a catalogue.
  • Extensions that use queryData() against such a model silently see nothing. The affected extension here had already grown a workaround with the comment "queryData() filters via the local sqlite catalog, which does not see rottentomatoes records on this datastore. Bypass it" — the bug was being routed around rather than reported.

Patch

Only a directory that actually contains metadata.yaml is a version directory. 207333/ does not (its metadata lives in 207333/2/); a real version dir does. Applied to both the async and sync walkers.

--- a/src/infrastructure/persistence/unified_data_repository.ts
+++ b/src/infrastructure/persistence/unified_data_repository.ts
@@ -67,6 +67,36 @@ function looksLikeModelId(name: string): boolean {
   return UUID_RE.test(name);
 }
 
+/**
+ * A purely numeric directory name is ambiguous: it is the version directory of
+ * a data item, but it is equally a legitimate *data name* — models that key
+ * resources by an external numeric id (a TMDB id, an issue number) produce
+ * `{model-id}/{207333}/{2}/`. Depth alone cannot tell the two apart, so the
+ * tree walk has to confirm that a numeric directory actually holds a version's
+ * `metadata.yaml` before treating its grandparent as a model-id directory.
+ *
+ * Without this check the walk mistakes the *type* directory for a model-id
+ * directory, derives an invalid ModelType from the truncated path, and silently
+ * drops every data item under that model — which the catalog backfill then
+ * treats as "this model has no data" and deletes from the catalog.
+ */
+async function hasVersionMetadata(versionDir: string): Promise<boolean> {
+  try {
+    const stat = await Deno.stat(join(versionDir, "metadata.yaml"));
+    return stat.isFile;
+  } catch {
+    return false;
+  }
+}
+
+function hasVersionMetadataSync(versionDir: string): boolean {
+  try {
+    return Deno.statSync(join(versionDir, "metadata.yaml")).isFile;
+  } catch {
+    return false;
+  }
+}
+
 /**
  * File system implementation of UnifiedDataRepository.
  *
@@ -387,7 +417,8 @@ export class FileSystemUnifiedDataRepository implements UnifiedDataRepository {
         const childPath = join(dir, entry.name);
         try {
           for await (const subEntry of Deno.readDir(childPath)) {
-            if (subEntry.isDirectory && /^\d+$/.test(subEntry.name)) {
+            if (!subEntry.isDirectory || !/^\d+$/.test(subEntry.name)) continue;
+            if (await hasVersionMetadata(join(childPath, subEntry.name))) {
               return true;
             }
           }
@@ -1681,7 +1712,8 @@ export class FileSystemUnifiedDataRepository implements UnifiedDataRepository {
         const childPath = join(dir, entry.name);
         try {
           for (const subEntry of Deno.readDirSync(childPath)) {
-            if (subEntry.isDirectory && /^\d+$/.test(subEntry.name)) {
+            if (!subEntry.isDirectory || !/^\d+$/.test(subEntry.name)) continue;
+            if (hasVersionMetadataSync(join(childPath, subEntry.name))) {
               return true;
             }
           }

An alternative fix is to key the walk on looksLikeModelId() (the UUID test) as collectByTaggedName already does, which would also unify the two walkers. I went with the structural check because it does not assume model ids are always UUIDs.

Verification

Three regression tests added to unified_data_repository_test.ts (numeric names async, numeric names sync, numeric + named coexisting under one model). Against unpatched source all three fail returning 0 records; all pass with the patch.

unified_data_repository_test.ts          42 passed | 0 failed
src/infrastructure/persistence/ + src/domain/data/
                                       1282 passed | 0 failed

Verified against the real repo after deno run compile — cleared the populated flag to force a rebuild:

  • 316 RT resources catalogued (830 rows across versions); total catalog 3362 → 4192
  • rows survive an unrelated query, a neutral query, and a targeting query
  • a previously dead query now returns 105 shows

backfillAsync sources its rows from dataRepo.findAllGlobal() — a local disk walk — and then hard-deletes anything absent from that walk, even when a datastore is configured. Under hydrationStrategy: lazy that means data present in the datastore but not yet materialized locally gets deleted from the catalog. Arguably backfill should reconcile against the datastore rather than the local cache, which is a cache by definition.

That is what turns this bug from "incomplete index" into "destructive index". Filing separately if you'd prefer — flagging it here because it is the reason the numeric-name bug destroys rows instead of merely failing to add them.

Environment

  • swamp: dev build at 19b64706 (fix(workflows): fix regressions in workflow-name filename PR (#2108))
  • Deno 2.8.3, Linux 6.19.13-arch1-1
  • Repo datastore: @keeb/mongodb-datastore, hydrationStrategy: lazy, namespace swamp-media
  • Reproduced on both the namespaced (bulkReplaceNamespace) path and by inspection on the solo (bulkReplaceAll) path
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 9 MOREFINDINGS+ 7 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/10/2026, 9:45:26 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
keeb assigned keeb8/10/2026, 8:26:13 PM

Sign in to post a ripple.