Skip to content

fix(kernel): restore checkpoint verify_max_retries on paused goal readout - #8359

Merged
houko merged 2 commits into
librefang:mainfrom
DaBlitzStein:fix/paused-readout-verify-budget
Sep 16, 2026
Merged

houko merged 2 commits into
librefang:mainfrom
DaBlitzStein:fix/paused-readout-verify-budget

Conversation

@DaBlitzStein

Copy link
Copy Markdown
Contributor

Summary

  • GoalRunner::state() (crates/librefang-kernel/src/goal_runner.rs) reconstructs a paused goal run from its persisted checkpoint once the loop task has exited and self-cleaned its registry slot. The checkpoint carries verify_max_retries for exactly this reason: it is the one loop-engineering value the goal document never holds, because it is a per-run number the operator sets on the start body rather than part of the goal's own configuration.
  • On main, state() reads the retry budget as DEFAULT_VERIFY_MAX_RETRIES.max(1) instead of the checkpoint, so a run started with {"verify_max_retries": 8} reports the compiled default once paused. The bodyless POST /api/goals/{id}/resume that follows a readout resolves its own budget the same way state() does, so the operator's number silently disappears on resume with nothing failing.
  • Fixed by resolving it exactly as start() resolves it on the way back in: checkpoint.verify_max_retries.unwrap_or(DEFAULT_VERIFY_MAX_RETRIES).max(1).
  • Added the two regression tests this behaviour was missing: a_paused_readout_reports_the_checkpoints_verify_max_retries and its sibling a_paused_readout_reports_no_verifier_budget_without_loop_engineering, which pins that a goal without loop engineering still reports no verifier budget from the same checkpoint-reconstruction path.

How this regressed

This fix and both regression tests already shipped once, on the branch (feat/goal-pause-resume) that introduced pause/resume for loop-engineered goals. #7973 merged that work into main by squash while the branch was still live, and the squashed commit that landed did not carry this particular fix or its tests — a known failure mode of squashing a still-advancing branch, not anyone's mistake. Closes the gap: #7973

Verification

Ported both tests first with state() still in main's form and confirmed the RED:

FAIL [0.056s] librefang-kernel goal_runner::tests::a_paused_readout_reports_the_checkpoints_verify_max_retries
assertion `left == right` failed: a paused run must report the retry budget it was actually running under, not the compiled default the resume will not use
  left: 3
 right: 8

(a_paused_readout_reports_no_verifier_budget_without_loop_engineering passed even against the broken code, since that branch's _ => 0 arm is unaffected — confirming it tests a distinct path.)

Applied the fix, both tests went GREEN, then ran the full gate:

  • cargo nextest run -p librefang-kernel --no-fail-fast → 2132 passed, 3 skipped
  • cargo clippy -p librefang-kernel --all-targets -- -D warnings → 0 warnings
  • cargo fmt --all --check → clean

…dout

GoalRunner::state() reconstructs a paused run from its checkpoint once
the loop task has exited, and the checkpoint carries verify_max_retries
for exactly this reason: it is the one loop-engineering value the goal
document never holds, since it is a per-run number the operator sets on
the start body rather than part of the goal's own configuration.

Reading it from the goal document instead meant a run started with
{"verify_max_retries": 8} reported the compiled default once paused,
and the bodyless /resume that follows a readout resolves its own
budget the same way state() does, so a paused goal's operator-set
retry budget silently disappeared on resume with nothing failing.

This fix and its two regression tests shipped once already on the
branch that introduced pause/resume for loop-engineered goals, but
librefang#7973 was merged into main by squash while that branch was still
live, and the squashed commit that landed did not carry them.
@github-actions github-actions Bot added area/kernel Core kernel (scheduling, RBAC, workflows) size/M 50-249 lines changed labels Sep 14, 2026
The fragment was written before opening the pull request; GitHub
assigned librefang#8359, not the librefang#8358 anticipated.
DaBlitzStein added a commit to DaBlitzStein/librefang that referenced this pull request Sep 14, 2026
@houko

houko commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Correct fix for a defect that is on main right now. Checked rather than taken on trust, since the changelog attributes the loss to a squash merge and I merged #7973 earlier today.

The defect is real. ResumePoint::verify_max_retries carries its own justification:

restored on resume for the same reason max_iterations is: without it, GET /api/goals/{id}/run reports the compiled default for a checkpoint that never had one, and a bodyless /resume silently re-budgets it to that default

start() honours that — verify_max_retries.or_else(|| resume…r.verify_max_retries).unwrap_or(DEFAULT).max(1) — and state() did not. So the readout and the resume disagreed, and the resume was right.

The fix matches start() exactly for the case that matters. A readout cannot know about a future explicit argument, so the right thing to mirror is the bodyless resume path: checkpoint, then compiled default, then .max(1). That is what checkpoint.verify_max_retries.unwrap_or(DEFAULT_VERIFY_MAX_RETRIES).max(1) gives.

The test discriminates. A checkpoint at 8 read back as 8, with max_iterations at 100 asserted alongside as the control — the sibling field that already did this. Before the change it returns DEFAULT_VERIFY_MAX_RETRIES.

On the attribution

The changelog says the fix "shipped once already … but a squash merge on a long-lived branch left them out of the squashed commit that reached main". I checked whether that squash was mine, because the conflict resolution I pushed to #7973 took this branch's side of goal_runner.rs:

                     verify_max_retries   DEFAULT_   struct ResumePoint
#7785 head                   19              2              0
main after #7785             19              2              0
#7973 head (518107373)       33              3              1
main now                     33              3              1

ResumePoint and the whole checkpoint-reconstruction path arrive with #7973 — #7785 has none of it — so taking this branch's side could not have dropped anything, and 518107373, the commit that was merged, already carried the compiled-default form. Whatever squash lost it happened earlier in the branch's own history.

Which is worth saying out loud rather than leaving implied: the loss is not visible in any conflict, so no resolution would have caught it. What catches this class is the test, and that is what the PR adds.

Nothing blocking.

@houko houko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed, and I verified the load-bearing claim rather than taking it from the comment.

The fix says it resolves the budget "exactly as start() resolves it on the way back in". That is the whole argument — if it were off, the readout would just be wrong in a new way — so I read the other side:

// goal_runner.rs:1081, start()
verify_max_retries: if loop_engineering {
    verify_max_retries
        .or_else(|| resume.as_ref().and_then(|r| r.verify_max_retries))
        .unwrap_or(DEFAULT_VERIFY_MAX_RETRIES)
        .max(1)
} else { 0 }

Explicit argument, then checkpoint, then compiled default, then .max(1). A bodyless /resume supplies no explicit argument, so it collapses to checkpoint → default → .max(1), which is what state() now does. The _ => 0 arm matches the else { 0 }. The claim holds.

Worth noting the one case the comment scopes correctly and I would have flagged otherwise: a /resume carrying an explicit verify_max_retries still overrides, so the readout and that resume can differ. That is right — the operator's new number should win — and the comment says "the bodyless /resume", not "resume", so it is not overclaiming.

The two tests are the right pair. The second is the one I would have asked for if it were missing: a checkpoint carrying 8 read against a goal whose loop_engineering was switched off while suspended must report 0, because reporting the number would advertise a gate the resume will not apply. Asserting max_iterations == 100 in the first test as the stated reference behaviour is a nice touch — it makes the fix "be consistent with the field two lines up" rather than a bare assertion.

One question, not a defect

The changelog says this shipped once already and a squash merge on a long-lived branch dropped it. I believe it, and it matches the shape of the bug.

If that is right, the interesting question is not this fix but whether anything would catch the next one. A regression that lands, then silently un-lands in a squash, leaves no trace: the tests went with it, so main is green and the behaviour is gone. Nothing here is wrong for not addressing that — it is out of this PR's scope by any reading — but it is the kind of thing worth a tracking issue if it has happened more than once, because the cost is paid a second time in full each time.

No blocking findings. This is a correct restore with the tests that should have been on main all along.

@houko
houko merged commit 9d906c8 into librefang:main Sep 16, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kernel Core kernel (scheduling, RBAC, workflows) size/M 50-249 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants