Description
simulate() still returns broken for same-domain successes that merely rhyme with a rule's trigger. The #723 domain gate compares only coarse tool groups (git -> vcs), so it cannot distinguish git push from git status/git pull, and BROKEN_MARGIN is a fixed absolute score offset. When a successful trajectory in the same domain happens to contain push-failure wording anywhere in its haystack (task or any step input/output/error), the rule is counted as having broken a success.
This is the structural gap flagged in a reader comment on the "Killed by the Word 'git'" article (dev.to comment 3f3ob, article https://dev.to/debashish_ghosal/killed-by-the-word-git-one-token-of-coincidence-40-points-of-pass-rate-140f):
"did you consider keying the domain on the trigger command itself rather than lexical similarity? Even after the margin, I'd worry a push-specific rule still over-fires on git status runs in corpora where push and pull failures both show up often."
The #723 fix (domain-gate broken + match-strength margin) is necessary but not sufficient: the gate is keyed on the trigger token (git), not the trigger command (push), so a git status success is "same domain" and the only remaining guard is the 0.10 margin.
Steps to Reproduce
- Build the F-001 rule from
field-test/corpus/curated/failures/positive/F-001-git--001-git-push-non-ff.jsonl (the near-verbatim extracted candidate).
- Replay it against a successful
git trajectory whose haystack mentions an earlier push failure (e.g. a runbook/notes step), as happens in corpora where push and pull failures co-occur.
- Call
simulate(candidate, success_trajectory).
Expected Behavior
no_effect (or near_miss): a successful git status/git pull/documentation run is not interference from a push-specific rule.
Actual Behavior
broken — a spurious broken success that demotes a correct rule (the same failure class #723 was meant to eliminate, just within-domain instead of cross-domain).
Environment
- CauterRule version: 0.3.1 (
pyproject.toml version = "0.3.1")
- Python version: 3.11+ (reproduced on 3.14)
- OS: macOS
- LLM provider: None (deterministic replay)
- Model: n/a
Minimal Reproducible Example
from cauterule.models.candidate import CandidateRule
from cauterule.models.rule import RuleWhen, RuleDo
from cauterule.models.trajectory import Trajectory, Step
from cauterule.replay.matcher import match_score, extract_trigger_domain
from cauterule.replay.simulator import simulate
c = CandidateRule(
when=RuleWhen(trigger="when git push fails with non-fast-forward"),
do=RuleDo(directive="pull latest changes before pushing"),
confidence=1.0,
)
s = Trajectory(
id="S-git-status-rhymes",
timestamp="2026-09-16T00:00:00Z",
task="Check git status, then document the earlier push failure for the runbook",
steps=(
Step(1, "bash", "git status", "nothing to commit, working tree clean"),
Step(2, "bash", "cat notes.md",
"Earlier: git push origin main was rejected non-fast-forward; resolved by pulling."),
),
success=True,
domain="git",
failure_class=None,
)
assert extract_trigger_domain(c.when.trigger) == "git"
assert match_score(c, s) == 0.70 # threshold 0.60 + BROKEN_MARGIN 0.10
assert simulate(c, s) == "broken" # expected "no_effect"
Trajectory or Corpus (if applicable)
- Rule:
field-test/corpus/curated/failures/positive/F-001-git--001-git-push-non-ff.jsonl
- Same-domain successes that expose the gap:
field-test/corpus/curated/successes/S-023-git-status.jsonl, S-022-git-pull.jsonl, NM-044-git-commit-hook.jsonl
Output
trigger domain : git
match_score : 0.7
rule_matches(0.6) : (True, 0.7)
simulate -> : broken (expected: no_effect)
Relevant code:
src/cauterule/replay/simulator.py:135-138 — check_domain_mismatch then score < threshold + BROKEN_MARGIN
src/cauterule/replay/matcher.py:749-787 — extract_trigger_domain / check_domain_mismatch operate on the coarse _KNOWN_DOMAINS/_DOMAIN_GROUPS token, not the command
src/cauterule/replay/matcher.py:438-450 — _build_haystack includes every step input/output, letting unrelated wording raise the score
Severity
Acceptance Criteria
Suggested direction
Key the domain on the trigger command (e.g. derive push/pull/status sub-domains for git), or gate broken on a signature/grounding check (as is_grounded already does for prevented) so a same-domain success must share the failure signature, not just the tool token, before it counts as interference.
Description
simulate()still returnsbrokenfor same-domain successes that merely rhyme with a rule's trigger. The #723 domain gate compares only coarse tool groups (git->vcs), so it cannot distinguishgit pushfromgit status/git pull, andBROKEN_MARGINis a fixed absolute score offset. When a successful trajectory in the same domain happens to contain push-failure wording anywhere in its haystack (task or any stepinput/output/error), the rule is counted as having broken a success.This is the structural gap flagged in a reader comment on the "Killed by the Word 'git'" article (dev.to comment
3f3ob, article https://dev.to/debashish_ghosal/killed-by-the-word-git-one-token-of-coincidence-40-points-of-pass-rate-140f):The #723 fix (domain-gate
broken+ match-strength margin) is necessary but not sufficient: the gate is keyed on the trigger token (git), not the trigger command (push), so agit statussuccess is "same domain" and the only remaining guard is the 0.10 margin.Steps to Reproduce
field-test/corpus/curated/failures/positive/F-001-git--001-git-push-non-ff.jsonl(the near-verbatim extracted candidate).gittrajectory whose haystack mentions an earlier push failure (e.g. a runbook/notes step), as happens in corpora where push and pull failures co-occur.simulate(candidate, success_trajectory).Expected Behavior
no_effect(ornear_miss): a successfulgit status/git pull/documentation run is not interference from a push-specific rule.Actual Behavior
broken— a spuriousbrokensuccess that demotes a correct rule (the same failure class #723 was meant to eliminate, just within-domain instead of cross-domain).Environment
pyproject.tomlversion = "0.3.1")Minimal Reproducible Example
Trajectory or Corpus (if applicable)
field-test/corpus/curated/failures/positive/F-001-git--001-git-push-non-ff.jsonlfield-test/corpus/curated/successes/S-023-git-status.jsonl,S-022-git-pull.jsonl,NM-044-git-commit-hook.jsonlOutput
Relevant code:
src/cauterule/replay/simulator.py:135-138—check_domain_mismatchthenscore < threshold + BROKEN_MARGINsrc/cauterule/replay/matcher.py:749-787—extract_trigger_domain/check_domain_mismatchoperate on the coarse_KNOWN_DOMAINS/_DOMAIN_GROUPStoken, not the commandsrc/cauterule/replay/matcher.py:438-450—_build_haystackincludes every stepinput/output, letting unrelated wording raise the scoreSeverity
Acceptance Criteria
ruff check .+mypy src/ tests/)Suggested direction
Key the domain on the trigger command (e.g. derive
push/pull/statussub-domains forgit), or gatebrokenon a signature/grounding check (asis_groundedalready does forprevented) so a same-domain success must share the failure signature, not just the tool token, before it counts as interference.