Skip to content

Bug: same-domain git over-fire — push-specific rule still marks git status/git pull successes as broken #815

Description

@deghosal-2026

Description

simulate() still returns broken for same-domain successes that merely rhyme with a rule's trigger. The #723 domain gate compares only coarse tool groups (git -> vcs), so it cannot distinguish git push from git status/git pull, and BROKEN_MARGIN is a fixed absolute score offset. When a successful trajectory in the same domain happens to contain push-failure wording anywhere in its haystack (task or any step input/output/error), the rule is counted as having broken a success.

This is the structural gap flagged in a reader comment on the "Killed by the Word 'git'" article (dev.to comment 3f3ob, article https://dev.to/debashish_ghosal/killed-by-the-word-git-one-token-of-coincidence-40-points-of-pass-rate-140f):

"did you consider keying the domain on the trigger command itself rather than lexical similarity? Even after the margin, I'd worry a push-specific rule still over-fires on git status runs in corpora where push and pull failures both show up often."

The #723 fix (domain-gate broken + match-strength margin) is necessary but not sufficient: the gate is keyed on the trigger token (git), not the trigger command (push), so a git status success is "same domain" and the only remaining guard is the 0.10 margin.

Steps to Reproduce

  1. Build the F-001 rule from field-test/corpus/curated/failures/positive/F-001-git--001-git-push-non-ff.jsonl (the near-verbatim extracted candidate).
  2. Replay it against a successful git trajectory whose haystack mentions an earlier push failure (e.g. a runbook/notes step), as happens in corpora where push and pull failures co-occur.
  3. Call simulate(candidate, success_trajectory).

Expected Behavior

no_effect (or near_miss): a successful git status/git pull/documentation run is not interference from a push-specific rule.

Actual Behavior

broken — a spurious broken success that demotes a correct rule (the same failure class #723 was meant to eliminate, just within-domain instead of cross-domain).

Environment

  • CauterRule version: 0.3.1 (pyproject.toml version = "0.3.1")
  • Python version: 3.11+ (reproduced on 3.14)
  • OS: macOS
  • LLM provider: None (deterministic replay)
  • Model: n/a

Minimal Reproducible Example

from cauterule.models.candidate import CandidateRule
from cauterule.models.rule import RuleWhen, RuleDo
from cauterule.models.trajectory import Trajectory, Step
from cauterule.replay.matcher import match_score, extract_trigger_domain
from cauterule.replay.simulator import simulate

c = CandidateRule(
    when=RuleWhen(trigger="when git push fails with non-fast-forward"),
    do=RuleDo(directive="pull latest changes before pushing"),
    confidence=1.0,
)

s = Trajectory(
    id="S-git-status-rhymes",
    timestamp="2026-09-16T00:00:00Z",
    task="Check git status, then document the earlier push failure for the runbook",
    steps=(
        Step(1, "bash", "git status", "nothing to commit, working tree clean"),
        Step(2, "bash", "cat notes.md",
             "Earlier: git push origin main was rejected non-fast-forward; resolved by pulling."),
    ),
    success=True,
    domain="git",
    failure_class=None,
)

assert extract_trigger_domain(c.when.trigger) == "git"
assert match_score(c, s) == 0.70          # threshold 0.60 + BROKEN_MARGIN 0.10
assert simulate(c, s) == "broken"          # expected "no_effect"

Trajectory or Corpus (if applicable)

  • Rule: field-test/corpus/curated/failures/positive/F-001-git--001-git-push-non-ff.jsonl
  • Same-domain successes that expose the gap: field-test/corpus/curated/successes/S-023-git-status.jsonl, S-022-git-pull.jsonl, NM-044-git-commit-hook.jsonl

Output

trigger domain      : git
match_score         : 0.7
rule_matches(0.6)   : (True, 0.7)
simulate ->         : broken (expected: no_effect)

Relevant code:

  • src/cauterule/replay/simulator.py:135-138 — check_domain_mismatch then score < threshold + BROKEN_MARGIN
  • src/cauterule/replay/matcher.py:749-787 — extract_trigger_domain / check_domain_mismatch operate on the coarse _KNOWN_DOMAINS/_DOMAIN_GROUPS token, not the command
  • src/cauterule/replay/matcher.py:438-450 — _build_haystack includes every step input/output, letting unrelated wording raise the score

Severity

  • Critical — crashes, data loss, or security vulnerability
  • High — incorrect results or broken core feature
  • Medium — unexpected behavior with workaround
  • Low — cosmetic or minor inconvenience

Acceptance Criteria

  • Bug reproduced and root cause identified
  • Fix implemented
  • Test added that catches the regression
  • Code review passed
  • Lint strict clean (ruff check . + mypy src/ tests/)
  • Code coverage > 95%
  • All necessary documents updated

Suggested direction

Key the domain on the trigger command (e.g. derive push/pull/status sub-domains for git), or gate broken on a signature/grounding check (as is_grounded already does for prevented) so a same-domain success must share the failure signature, not just the tool token, before it counts as interference.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions