Both pullChanged (line 1672) and pushChanged (line 2009) independently call assembleIndexFromShards(), each reading _meta.json + N partition shard files from S3. When the fast path is broken (see companion issues), this happens on every command — the shards are fetched twice.

pullChanged populates this.index with the assembled entries, but pushChanged ignores it and re-fetches everything from S3 to get a fresh view. For read-only commands where nothing was dirtied between pull and push, this is wasted work.

Impact

With the fast path broken, every command does 2x the S3 reads for shard assembly. For a repo with ~20-30 shards, that's 40-60+ S3 GetObject calls instead of 20-30.

Suggested fix

Two options:

Cache-aware assembly: If this.index is already populated from a recent pullChanged in the same process lifecycle and localDirty is false (nothing was written between pull and push), skip the push-side assembly and reuse the pull's index.
Two-phase sync refactor: Core already supports preparePush/commitPush (two-phase sync). If core uses two-phase, the same assembly can serve both pull verification and push preparation.

References

s3_cache_sync.ts:1672 (pullChanged shard assembly)
s3_cache_sync.ts:2009 (pushChanged shard assembly)
assembleIndexFromShards() reads _meta.json + all partition files

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

Extension: @swamp/s3-datastore@2026.08.12.1
swamp: 20260819.011806.0-sha.a9c7ee9a
OS: darwin (x86_64)
Deno: 2.8.3
Shell: /bin/zsh

02Bog Flow

Open

8/19/2026, 5:25:14 PM

No activity in this phase yet.

03Sludge Pulse

vault: audit trail should record put, delete, and annotate operations

ExtensionWorkflowRepository scans manifest.yaml as a workflow, emitting 'Unknown key manifestVersion' warning

--max-run-duration only applies to detached runs, not attached runs

dataOutputOverrides not applied during remote execution — ephemeral resources GC'd before override takes effect

Cancel endpoint reports status: cancelled for runs it did not actually cancel

@swamp/git: no way to amend a commit, but push can --force

Installation with new account creation times out

pre-flight checks run on orchestrator, not on worker — breaks remote execution with filesystem-dependent checks

Docs: update cancel endpoint response statuses in REST API reference and workflows page

serve: triggerSource not persisted to workflow run summary

docs: document --remote-only flag for swamp serve

Swamp serve daemon doesn't run with --hot-reload

serve: include workflowName in HealthSnapshotSchedule

Retyping a model instance strands catalog rows no command can reclaim — prune matches on model_id, delete resolves the current type (follow-up to #1009)

extension rm also deletes the lockfile entry, and install on an empty lockfile is a silent success — rm/install is not a round trip

knowledge-base retrieve persists query output into the state resource, with lifetime infinite and the query text in the instance name

Method-name collision between a base model type and an extending extension silently drops the whole extension file — no error, doctor reports pass

Skip 50MB monolithic index rewrite on v2 shard-first path

Share single AwsS3Client instance across lock, sync, and verifier

Double shard assembly per command lifecycle (pull + push both read all shards)

pullChanged never records commitSeq for shard-first (v2) index

localHasAllRemoteEntries() permanently blocks sidecar update on partial caches

serve: workflow.history.get cannot find CLI-created runs or look up by run ID

"TRIAL ENDED" modal shown to team plan user when toggling extension visibility

Concurrent suspended workflow runs can't be disambiguated from 'approvals'; fresh runs should supersede matching-input suspended runs

I _think_ I might like a guard in the trigger yaml

swamp cmds don't realise the user is sshed into the system

serve dashboard: add cluster.instances and serve.config endpoints

CEL expressions have timestamp/duration but no current-time value, so no guard can express "older than N"

Support direct type execution (@type) with --server

access token mint writes to user vault, not serve's _token-secrets control plane in HA mode

Generic OTLP endpoint appends signal path after query or fragment

Issue lifecycle start drops top-level issue comments

serve dashboard: add instanceId, triggerSource, and step progress to WorkflowRunSummary

Document worker affinity (affinity: true) for workflows and jobs

Datastore MongoDB driver leaks connections — 5960 accumulated over 8 days

workflow approvals displays the latest run's prompt for every suspended run

@swamp/git: add a read-only working-tree diff for pre-commit review

docs: document --oauth-client-name flag for swamp serve

Docs: update workflow reference to include trigger in JSON output shapes

@swamp/git branch and push methods fail data naming when branch contains '/'

Docs: update workflow-placement.md for job/workflow-level placement inheritance

feat(serve): use identifiable OAuth client_name instead of random UUID

@swamp/git does not pass AbortSignal to Git subprocesses

Global Secret Manager methods build an invalid regional parent

Document model.method() support in assert expressions

No way to store code-review results — verdicts, findings, and cost are not persisted for any repo

Redaction regex boundary bug: label_base64 secrets with an unrecognized label pass through unredacted (e.g. swamp_ API keys)

User token page (/u/<user>) shows only "swamp_..." for every API key, making leaked-key identification impossible

upstream_state treats sparse-fetch branches as having no upstream

Add native support for the Pi coding harness

Cloud Build create supplies project instead of required projectId path parameter

workflows: support labels/target/platform at job and workflow level

swamp access grant create never registers the grant — a --deny created this way silently fails open

Support credentials in datastore config (vault-resolvable)

swamp serve OOMs on repos with 2000+ model definitions

Serve telemetry attribution for collectives

Extension workflows: no way to override trigger configuration

Add a list/discover method to the generated @swamp/aws types — no way to enumerate resources in a region

Document @vault= prefix for webhook secrets in serve.yaml

swamp access grant create installs an unrelated extension from the registry

docs: document SWAMP_API_KEY headless OAuth bootstrap for swamp serve

workflow-succeeded selects the globally-latest run, so siblings block each other under parallel use

swamp serve: webhook secrets should support @vault= resolution

Misleading collective_not_trusted auto-resolve error when a local source extension transiently fails to index during workflow validate

datastore setup: symlinks not counted in bytesCopied, causing misleading '(0B)' output

vault.get() CEL error embeds raw newline when wrapped by model resolver, producing broken output

Feature: user-configurable redaction rules (.swamp/redaction.yaml)

Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit

Sync every signup to Attio, and backfill the existing population (#809 follow-up)

@swamp/issue-lifecycle probes /api/health, which swamp-club stopped serving when the Fresh scaffold stubs were removed

Lab lifecycle POST rejects any payload string that starts with a dollar sign, not just operator keys

Feed: detect and gracefully handle dead (404) post links (link rot)

/feed: per-post unread indicators and mark-as-read to track reading progress

/feed: filter and search by author, title/text, and topic

Add a :micdrop: emoji / reaction shortcode (swamp-native art included)

Push down the two cheap leaderboard scans: ghost-gate dictionary and eventCounts owner filter

$identify events are never consumed — anonymous web history never binds to the account

Finish the swamp_* attribute set on Attio users, and settle the two unprefixed ones (#809 follow-up)

aria-expanded is a raw boolean in 12 places — collapsed state is unannounced to screen readers