Skip to content

Cross-instance card coordination: shared lease service for multiple orchestrators on one repo #4

Description

@ehippy

Problem

Running two independent built instances against the same repo (e.g. one local, one on a
separate host) works fine in isolation, but the two orchestrators have zero visibility into
each other. Each has its own sqlite DB, its own Card rows/ids, and its own Curator — so both
sides can independently propose and start working the same (or overlapping) task, racing to
push conflicting branches/commits against the same shared git remote.

built already solves this exact problem within one process:
Card.claimed_by_worker_id + Card.lease_expires_at, enforced by claim_next_card's atomic
conditional UPDATE (orchestrator/worker.py). The proposal is to promote that same
lease-with-TTL pattern to a small shared service both instances call, instead of inventing a
new mechanism from scratch.

Proposed shape

A small serverless lease service (Lambda + DynamoDB) both orchestrators hit before claiming a
card or before a Curator proposes a new task:

  • Table: partition key lease_key (string), attributes owner_id, acquired_at,
    expires_at (epoch seconds — doubles as the DynamoDB TTL attribute for auto-cleanup).
  • acquire(lease_key, owner_id, ttl)PutItem with
    ConditionExpression: attribute_not_exists(lease_key) OR expires_at < :now. Returns a lease
    token on success, 409 if another owner holds an unexpired lease.
  • renew(lease_key, owner_id) — conditional UpdateItem extending expires_at, gated on
    owner_id = :self.
  • release(lease_key, owner_id) — conditional DeleteItem, same ownership check.

Function URLs are enough for two callers; no need for full API Gateway.

The open design question: what is lease_key?

Each built instance has its own independent card ids — node A's card and node B's card for
"add HP bar" are unrelated rows with unrelated ids. A lease service only dedupes them if
lease_key is a content fingerprint computed the same way on both sides (e.g. a normalized/
slugified title, or a hash of the request text) — not either side's internal card id. This
fingerprinting scheme is the actual crux of the problem; the lease store just enforces whatever
key it's given.

Alternatives considered (from discussion)

  1. Single-writer via existing paused_at — zero new code, pause one instance's claiming
    for the shared project at a time. Simplest, but gives up real parallelism (time-slicing, not
    coordination).
  2. BUILT.md-style shared file with card state — cheap, fits the existing AGENTS.md-reading
    convention, but purely advisory (no enforcement), and ironically prone to the exact
    simultaneous-write conflict it's meant to prevent.
  3. Git ref as a distributed lock — push a claim marker to a dedicated ref namespace
    (refs/built-claims/<fingerprint>); ref creation is atomic on the remote (non-fast-forward
    rejection = built-in compare-and-swap), reusing infrastructure both nodes already have (the
    shared git remote). No new service, but no TTL/expiry either — a crashed node's claim-ref
    needs manual/periodic cleanup, and it's invisible in the normal GitHub branch UI.
  4. Full GitHub Issues two-way sync — real atomicity via issue assign/label, visible to
    humans for free, but requires building a genuine sync layer between built's own Card model
    and GitHub's issue semantics (closer in shape to the existing CI-watcher's Checks polling than
    a quick integration).

The Lambda/Dynamo lease service (this issue) sits between options 3 and 4: real TTL-backed
atomicity like a purpose-built coordinator, without the GitHub API impedance mismatch of full
Issues sync.

Scope, if picked up

  • New: a small standalone Lambda + DynamoDB service (lives outside src/built/, its own
    deploy).
  • New: a client module in built's orchestrator (e.g. orchestrator/lease_client.py) wired
    into claim_next_card and into curation's propose step (agent/curation.py), gated behind
    its own BUILT_*_ENABLED toggle per existing convention (config.py) so single-instance
    deployments pay zero cost.
  • Needs: an auth story between built nodes and the Lambda (shared key or IAM SigV4, analogous
    to BUILT_API_KEY), and a decided fingerprinting scheme for lease_key.

Not started — filed to capture the design discussion, not a commitment to build it yet.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions