Problem
Running two independent built instances against the same repo (e.g. one local, one on a
separate host) works fine in isolation, but the two orchestrators have zero visibility into
each other. Each has its own sqlite DB, its own Card rows/ids, and its own Curator — so both
sides can independently propose and start working the same (or overlapping) task, racing to
push conflicting branches/commits against the same shared git remote.
built already solves this exact problem within one process:
Card.claimed_by_worker_id + Card.lease_expires_at, enforced by claim_next_card's atomic
conditional UPDATE (orchestrator/worker.py). The proposal is to promote that same
lease-with-TTL pattern to a small shared service both instances call, instead of inventing a
new mechanism from scratch.
Proposed shape
A small serverless lease service (Lambda + DynamoDB) both orchestrators hit before claiming a
card or before a Curator proposes a new task:
- Table: partition key
lease_key (string), attributes owner_id, acquired_at,
expires_at (epoch seconds — doubles as the DynamoDB TTL attribute for auto-cleanup).
acquire(lease_key, owner_id, ttl) — PutItem with
ConditionExpression: attribute_not_exists(lease_key) OR expires_at < :now. Returns a lease
token on success, 409 if another owner holds an unexpired lease.
renew(lease_key, owner_id) — conditional UpdateItem extending expires_at, gated on
owner_id = :self.
release(lease_key, owner_id) — conditional DeleteItem, same ownership check.
Function URLs are enough for two callers; no need for full API Gateway.
The open design question: what is lease_key?
Each built instance has its own independent card ids — node A's card and node B's card for
"add HP bar" are unrelated rows with unrelated ids. A lease service only dedupes them if
lease_key is a content fingerprint computed the same way on both sides (e.g. a normalized/
slugified title, or a hash of the request text) — not either side's internal card id. This
fingerprinting scheme is the actual crux of the problem; the lease store just enforces whatever
key it's given.
Alternatives considered (from discussion)
- Single-writer via existing
paused_at — zero new code, pause one instance's claiming
for the shared project at a time. Simplest, but gives up real parallelism (time-slicing, not
coordination).
BUILT.md-style shared file with card state — cheap, fits the existing AGENTS.md-reading
convention, but purely advisory (no enforcement), and ironically prone to the exact
simultaneous-write conflict it's meant to prevent.
- Git ref as a distributed lock — push a claim marker to a dedicated ref namespace
(refs/built-claims/<fingerprint>); ref creation is atomic on the remote (non-fast-forward
rejection = built-in compare-and-swap), reusing infrastructure both nodes already have (the
shared git remote). No new service, but no TTL/expiry either — a crashed node's claim-ref
needs manual/periodic cleanup, and it's invisible in the normal GitHub branch UI.
- Full GitHub Issues two-way sync — real atomicity via issue assign/label, visible to
humans for free, but requires building a genuine sync layer between built's own Card model
and GitHub's issue semantics (closer in shape to the existing CI-watcher's Checks polling than
a quick integration).
The Lambda/Dynamo lease service (this issue) sits between options 3 and 4: real TTL-backed
atomicity like a purpose-built coordinator, without the GitHub API impedance mismatch of full
Issues sync.
Scope, if picked up
- New: a small standalone Lambda + DynamoDB service (lives outside
src/built/, its own
deploy).
- New: a client module in
built's orchestrator (e.g. orchestrator/lease_client.py) wired
into claim_next_card and into curation's propose step (agent/curation.py), gated behind
its own BUILT_*_ENABLED toggle per existing convention (config.py) so single-instance
deployments pay zero cost.
- Needs: an auth story between
built nodes and the Lambda (shared key or IAM SigV4, analogous
to BUILT_API_KEY), and a decided fingerprinting scheme for lease_key.
Not started — filed to capture the design discussion, not a commitment to build it yet.
Problem
Running two independent
builtinstances against the same repo (e.g. one local, one on aseparate host) works fine in isolation, but the two orchestrators have zero visibility into
each other. Each has its own sqlite DB, its own Card rows/ids, and its own Curator — so both
sides can independently propose and start working the same (or overlapping) task, racing to
push conflicting branches/commits against the same shared git remote.
builtalready solves this exact problem within one process:Card.claimed_by_worker_id+Card.lease_expires_at, enforced byclaim_next_card's atomicconditional UPDATE (
orchestrator/worker.py). The proposal is to promote that samelease-with-TTL pattern to a small shared service both instances call, instead of inventing a
new mechanism from scratch.
Proposed shape
A small serverless lease service (Lambda + DynamoDB) both orchestrators hit before claiming a
card or before a Curator proposes a new task:
lease_key(string), attributesowner_id,acquired_at,expires_at(epoch seconds — doubles as the DynamoDB TTL attribute for auto-cleanup).acquire(lease_key, owner_id, ttl)—PutItemwithConditionExpression: attribute_not_exists(lease_key) OR expires_at < :now. Returns a leasetoken on success, 409 if another owner holds an unexpired lease.
renew(lease_key, owner_id)— conditionalUpdateItemextendingexpires_at, gated onowner_id = :self.release(lease_key, owner_id)— conditionalDeleteItem, same ownership check.Function URLs are enough for two callers; no need for full API Gateway.
The open design question: what is
lease_key?Each
builtinstance has its own independent card ids — node A's card and node B's card for"add HP bar" are unrelated rows with unrelated ids. A lease service only dedupes them if
lease_keyis a content fingerprint computed the same way on both sides (e.g. a normalized/slugified title, or a hash of the request text) — not either side's internal card id. This
fingerprinting scheme is the actual crux of the problem; the lease store just enforces whatever
key it's given.
Alternatives considered (from discussion)
paused_at— zero new code, pause one instance's claimingfor the shared project at a time. Simplest, but gives up real parallelism (time-slicing, not
coordination).
BUILT.md-style shared file with card state — cheap, fits the existing AGENTS.md-readingconvention, but purely advisory (no enforcement), and ironically prone to the exact
simultaneous-write conflict it's meant to prevent.
(
refs/built-claims/<fingerprint>); ref creation is atomic on the remote (non-fast-forwardrejection = built-in compare-and-swap), reusing infrastructure both nodes already have (the
shared git remote). No new service, but no TTL/expiry either — a crashed node's claim-ref
needs manual/periodic cleanup, and it's invisible in the normal GitHub branch UI.
humans for free, but requires building a genuine sync layer between
built's own Card modeland GitHub's issue semantics (closer in shape to the existing CI-watcher's Checks polling than
a quick integration).
The Lambda/Dynamo lease service (this issue) sits between options 3 and 4: real TTL-backed
atomicity like a purpose-built coordinator, without the GitHub API impedance mismatch of full
Issues sync.
Scope, if picked up
src/built/, its owndeploy).
built's orchestrator (e.g.orchestrator/lease_client.py) wiredinto
claim_next_cardand into curation's propose step (agent/curation.py), gated behindits own
BUILT_*_ENABLEDtoggle per existing convention (config.py) so single-instancedeployments pay zero cost.
builtnodes and the Lambda (shared key or IAM SigV4, analogousto
BUILT_API_KEY), and a decided fingerprinting scheme forlease_key.Not started — filed to capture the design discussion, not a commitment to build it yet.