Skip to content

docs(process): a mutation's node ID is checkable before the write (closes #1916) - #1922

Merged
cagataycali merged 1 commit into
strands-labs:mainfrom
cagataycali:fix/graphql-node-id-targeting
Aug 4, 2026
Merged

cagataycali merged 1 commit into
strands-labs:mainfrom
cagataycali:fix/graphql-node-id-targeting

Conversation

@cagataycali

Copy link
Copy Markdown
Member

Closes #1916.

What changed

AGENTS.md > PR Workflow > step 8 covers a mutation reporting the wrong thing - a mergePullRequest that says "not mergeable" on a merge that landed - and not a mutation addressing the wrong thing, which is the direction with no undo.

A mutation names its subject by node ID and by nothing else: createIssue takes a repositoryId, not an owner and a name. So a well-formed ID that is wrong does not fail - it succeeds against whatever object it does name. A repositoryId carried over from an earlier response rather than queried, R_kgDOD1WOFw for R_kgDORUMiZg, created issue #1 in an unrelated third-party repository and returned success. The only clue was the url in the response, and deleteIssue needs admin on the target, so it could not be undone.

This lands as the last bullet of step 8, whose subject is already verifying a mutation by reading it back.

The issue's prescribed location does not exist

#1916 proposed recording the rule "next to the existing use_github guidance". There is none - AGENTS.md has no use_github, node-ID or GraphQL passage at all:

grep -n "use_github\|node ID\|GraphQL\|repositoryId" AGENTS.md   ->  0 hits

Worth stating because the absence is why nothing caught the write: the rule was not weakly documented, it was undocumented.

The premise the incident was written up under is measurably false

The write-up says IDs are "unreadable by design". They are not. A node ID is <TypePrefix>_<urlsafe-base64(msgpack array)>, where a repository is [0, databaseId] and anything a repository owns is [0, repository databaseId, own databaseId] - so the type and the target repository are readable with no network call, before the write:

node ID decodes to published databaseId
R_kgDORUMiZg [0, 1162027622] 1162027622 (this repository)
I_kwDORUMiZs8AAAABLT9z4g [0, 1162027622, 5054100450] 5054100450 (#1916)
PR_kwDORUMiZs76PCIu [0, 1162027622, 4198244910] 4198244910 (#1920)

Every right-hand column is what repository(owner: "strands-labs", name: "robots") returns beside the ID, so the decoder is checked against GitHub's own answer rather than against a reimplementation of itself.

That matters because it changes the remedy. "Always query the ID" is advice a stale value defeats - which is exactly what happened three times in one run. A check runnable on the value in hand does not depend on remembering.

One finding the write-up did not have

All three guessed IDs in that run carried one wrong repository:

guessed ID decodes to outcome
R_kgDOD1WOFw [0, 257265175] wrote to another repository
PR_kwDOD1WOF87DdSjQ [0, 257265175, 3279235280] Could not resolve to a node

So a single stale value contaminated every mutation, and the two that failed did so only because their own databaseId happened not to exist under that repository. Failing closed was luck about the guess, not a property of the API - and the guess that got lucky the other way is the one that wrote.

Test surface

tests/test_graphql_node_id_targeting.py, 15 tests, two classes with different jobs.

TestTheNodeIdEnvelopeIsCheckableOffline (10) executes the claim. A pin that only asserted AGENTS.md says the ID is checkable would keep passing against a future ID format that had stopped being checkable, leaving the guidance reading plausibly while advising something impossible. This decodes the three real IDs above against their published databaseIds, asserts the stray ID is distinguishable from the intended one, and asserts both incident IDs resolve to the same wrong repository.

TestTheGuidanceNamesTheDecodableEnvelope (5) pins the prose, because the prose is the deliverable - an agent reads AGENTS.md, not this module. It asserts adjacency rather than vocabulary: the fail-open property, the decodable envelope, the missing undo and the response-url check have to stay in the same breath as the rule, since each alone is unactionable. A future edit tightening the passage back to "resolve IDs with a query" is exactly the regression, it looks like an improvement, and nothing else in the tree would notice. Same structural reason tests/test_merge_gate_viewer_scope.py and tests/test_codeql_query_filters.py exist, and the text-assertion shape follows those modules.

Negative control - with origin/main's AGENTS.md restored, all 5 guidance tests fail and all 10 offline-decoding tests pass unchanged:

5 failed, 10 passed        # every failure in TestTheGuidanceNamesTheDecodableEnvelope

The split is the point: the envelope is a property of GitHub's IDs, not of this change, so only the guidance half is new.

Writing the decoder surfaced a second permissiveness

base64.urlsafe_b64decode discards bytes outside the alphabet and everything after the padding, so "kgDORUMiZg==extra" decodes to this repository's id - a corrupted ID yielding a plausible answer, in a value used to decide whether a write is safe. The trailing-garbage case caught it. The decoder now re-encodes and compares rather than the test being relaxed.

Verification

MUJOCO_GL=egl python -m pytest tests -q --no-cov -p no:randomly
  18485 passed, 257 skipped in 585.34s

ruff check strands_robots tests tests_integ          All checks passed!
ruff format --check strands_robots tests tests_integ  1046 files already formatted
mypy strands_robots tests tests_integ                Success: no issues found in 1043 source files

python3 scripts/assemble_changelog.py --check         changelog fragments OK (146 pending)
python3 scripts/check_merge_base_overlap.py           No overlap (base 2f2f1911, unmoved)

Scope

Documentation and test only; no production code changes. Nothing here touches a policy, simulation behaviour, rendering, recording or a robot asset, so there is no visual artifact to attach - the measurements above are the evidence.

The stray issue itself cannot be cleaned up: deleteIssue needs admin on the target repository. It is closed as NOT_PLANNED with an apology, which is the whole reason this is a check-before rather than a verify-after.

…oses strands-labs#1916)

AGENTS.md > PR Workflow > step 8 covers a mutation reporting the wrong thing - a
mergePullRequest that says "not mergeable" on a merge that landed - and not a
mutation addressing the wrong thing, which is the direction with no undo. A
mutation names its subject by node ID and by nothing else: createIssue takes a
repositoryId, not an owner and a name, so a well-formed ID that is wrong does not
fail. It succeeds against whatever object it does name. A repositoryId carried
over from an earlier response rather than queried, R_kgDOD1WOFw for R_kgDORUMiZg,
created issue #1 in an unrelated third-party repository and returned success; the
only clue was the url in the response, and deleteIssue needs admin on the target,
so it could not be undone.

The issue proposed recording the rule "next to the existing use_github guidance".
There is none - AGENTS.md has no use_github, node-ID or GraphQL passage at all -
so this lands as the last bullet of step 8, whose subject is already verifying a
mutation by reading it back.

It also corrects the premise the incident was written up under. The ID is not
opaque: it is <TypePrefix>_<urlsafe-base64(msgpack array)>, a repository being
[0, databaseId] and anything a repository owns [0, repository databaseId, own
databaseId], so the type and the target repository are both readable with no
network call. That turns "always query the ID" - advice a stale value defeats,
which is what happened - into a check runnable on the value in hand.

One finding the write-up did not have: all three guessed IDs in that run carried
one wrong repository (257265175 against this repository's 1162027622), so a
single stale value contaminated every mutation, and the two that failed did so
only because their own databaseId happened not to exist there. Failing closed was
luck about the guess rather than a property of the API.

tests/test_graphql_node_id_targeting.py executes the claim instead of asserting
the prose says it, decoding this repository's own node ID and those of an issue
and a pull request in it against the databaseIds the API publishes beside them,
so guidance that has become impossible fails loudly rather than reading
plausibly. Writing that decoder surfaced a second permissiveness worth pinning:
base64.urlsafe_b64decode discards bytes outside the alphabet and everything after
the padding, so "kgDORUMiZg==extra" decodes to this repository's id. The decoder
re-encodes and compares rather than relaxing the test.

Documentation and test only; no production code changes.

Closes strands-labs#1916

@yinsong1986 yinsong1986 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Docs-and-test-only change closing #1916. Adds a new bullet to AGENTS.md > PR Workflow > step 8 recording that a GraphQL mutation names its subject by node ID alone, that a well-formed but wrong ID fails open (writes to whatever object it does name, with no undo via deleteIssue), and that the ID envelope is decodable offline (<TypePrefix>_<urlsafe-base64(msgpack array)>), so the target repository is checkable before the write. Ships tests/test_graphql_node_id_targeting.py (15 tests) that both executes the offline-decode claim against GitHub-published databaseIds and pins the prose adjacency of the rule's qualifiers, plus a changelog fragment. No production code is touched.

What's good

  • Every factual claim in the diff is independently verifiable: I re-decoded all five node IDs (this repository, issue #1916, PR #1920, and both stray incident IDs) with a separate decoder and every databaseId in the table matches.
  • The negative control reproduces exactly as described: with main's AGENTS.md restored, the 5 guidance tests fail and the 10 offline-decode tests pass unchanged (5 failed, 10 passed).
  • The decoder fails closed - unknown msgpack tags, non-round-tripping base64 (the "==extra" trailing-garbage permissiveness of urlsafe_b64decode is caught by re-encode-and-compare), and length-mismatched arrays all raise ValueError rather than returning a guessed value, consistent with the no-silent-defaults rule.
  • Changelog fragment used instead of editing CHANGELOG.md directly, named per changelog.d/README.md (issue number is explicitly allowed). No host paths, no non-ASCII, ruff check/format clean on the new module.

@cagataycali
cagataycali merged commit ab14119 into strands-labs:main Aug 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants