Skip to content

Replay shrunk failures via PropTestConfig.shrinkPaths + Eval index (#3076) - #6098

Open
PreAgile wants to merge 4 commits into
kotest:masterfrom
PreAgile:3076-shrink-replay
Open

PreAgile wants to merge 4 commits into
kotest:masterfrom
PreAgile:3076-shrink-replay

Conversation

@PreAgile

Copy link
Copy Markdown
Contributor

Part 2 of #3076, builds on top of #6097 (path recording). This PR adds the consumer side: a
PropTestConfig.shrinkPaths field that, paired with seed and skipTo, replays a previously
discovered failure directly to its shrunk value without re-running the search.

Until #6097 merges, the diff here shows both PRs' changes. After #6097 lands the diff will
narrow to just the replay infrastructure. Reviewing #6097 first is easiest.

What it does

After a failure, the message now looks like:

Property failed after 47 attempts

	Arg 0: -1 (shrunk from -32891)

Repeat this test by using seed 1234567890
Eval index: 47
Shrink paths: [[0, 2, 1]]

Caused by AssertionError: ...

To replay that exact case without re-shrinking:

forAll(PropTestConfig(
   seed = 1234567890L,
   skipTo = 47,
   shrinkPaths = listOf(listOf(0, 2, 1)),
), Arb.int()) { n -> n >= 0 }

The shrink phase skips its search and walks children[0] -> children[2] -> children[1] directly.

Pieces

  • PropTestConfig.shrinkPaths: List<List<Int>>? — one path per arg. Like skipTo (added in
    Add skipTo for skipping N test cases (#4280) #4307) this lives on PropTestConfig only; the higher-level PropTest DSL does not expose it.
  • doReplay walks the recorded path and runs the test once on the final node. Strict — if a
    path index is out of bounds (Shrinker changed), the final value passes (property changed), or
    it's rejected by assume(), it throws an internal ReplayShrinkPathException. Silent fallback
    would label a non-failing value as the "shrunk" case, which defeats the feature.
  • handleException catches that exception and prints a Note: shrink replay was skipped — ...
    line to stdout, then continues through the normal property-failure path. Users see a regular
    property failure (seed, eval index, cause), never a leaked internal exception. The same path
    handles requireShrinkPathsArity when the supplied shrinkPaths size doesn't match the
    property's arity.
  • Eval index: N is PropertyContext.evals() at failure time — the value skipTo consumes.
    The existing Property failed after N attempts line uses successes + failures which drifts
    away from evals when there are assume() skips or earlier skipTo'd iterations, so it
    can't drop into skipTo cleanly. The two values coexist now.
  • shrinkfn (22 overloads) and the matching proptest call sites thread shrinkPaths through.

maxFailure > 0 lazy shrink

While integrating the catch I noticed I'd moved shrinkfn() outside the throw branches in
handleException and fixed it before pushing. The shrink/replay now runs only when the test
is actually about to fail terminally, matching pre-#3076 behavior. There's a regression test
for it ("with maxFailure > 0, the shrinker is not invoked for allowed failures...").

API / binary compatibility

  • PropTestConfig gains the new field. Its JVM all-args constructor signature changes — this
    matches how skipTo was added in Add skipTo for skipping N test cases (#4280) #4307. @JvmOverloads would balloon the generated overload
    set on a data class with this many defaulted params, so I didn't add it.
  • doShrinking, the public throwPropertyTestAssertionError, and all 22 shrinkfn overloads
    get @JvmOverloads so their old JVM signatures stay alongside the new ones. The api dump
    shows both.
  • ReplayShrinkPathException and requireShrinkPathsArity are internal — they don't enter
    the public ABI.

Tests

  • doReplay strict failure modes: out-of-bounds, negative index, final-value-passes, assume-skip
  • end-to-end: extract Eval index + Shrink paths from a real failure message, feed them back
    via PropTestConfig with ShrinkingMode.Off, verify the same shrunk Arg 0 is reached. (Mode
    Off proves replay is following the path rather than searching — without the path the run would
    report the un-shrunk initial value.)
  • arity mismatch + stale path both produce regular property failures, not raw exceptions
  • maxFailure > 0 lazy-shrink regression
  • regression updates to ForAll2Test / ForNoneTest expected messages for the new lines

Open question

The Shrink paths: [[2, 0, 1]] format was chosen so it copy/pastes cleanly into a
PropTestConfig(shrinkPaths = ...) literal. Happy to switch to per-arg lines or a different
label if you'd rather — easier to settle now than later.

PreAgile added 2 commits May 31, 2026 01:06
…otest#3076)

ShrinkResult and StepResult gain a `path: List<Int>` field that records
the raw child indices traversed during the shrink search. The failure
message now prints a `Shrink paths: ...` line whenever any arg actually
shrunk; non-shrinking runs (Exhaustive, ShrinkingMode.Off) stay quiet.

This is the first half of kotest#3076. The path is recorded here but no
replay machinery yet — that lands in a follow-up. The path uses raw
indices into RTree.children rather than positions in the dedup-filtered
sequence, so future replay can just walk `children[path[i]]` at each
level without reconstructing the search's `tested` state.

@jvmoverloads on both data class constructors keeps the old 3-arg and
2-arg JVM signatures, so callers that don't touch the new field stay
ABI-compatible.
…ting (kotest#3076)

Builds on the previous PR which recorded the shrink path on ShrinkResult.
This one wires the path through so users can replay a known failure
directly without re-running the shrink search.

What's new:

- `PropTestConfig.shrinkPaths: List<List<Int>>?` — one path per arg.
  When supplied, the shrink phase calls `doReplay` instead of `doStep`
  and walks `RTree.children[path[i]]` at each level. Like `skipTo`,
  this lives on PropTestConfig only; the higher-level `PropTest` DSL
  does not expose it.
- `doReplay` is strict: out-of-bounds indices, a final value that
  passes, or a value rejected by `assume()` all throw a (package-
  internal) `ReplayShrinkPathException`. Silent fallback would label
  a non-failing value as the shrunk counterexample, which defeats the
  feature's whole point.
- `handleException` catches the replay exception and prints
  `Note: shrink replay was skipped — ...` to stdout, then continues
  through the normal property-failure path. Users see a regular
  property failure (seed, eval index, original cause) — never a raw
  exception leak. Same path handles size-mismatched shrinkPaths
  (arity guard).
- Failure messages gain an `Eval index: N` line. It's the
  pre-assumption iteration counter (`PropertyContext.evals()`),
  which is the value `PropTestConfig.skipTo` consumes. The existing
  `Property failed after N attempts` is `successes + failures`,
  so it doesn't line up with `skipTo` when there are assumption
  skips or earlier skipTo'd iterations.
- `shrinkfn` (22 overloads) and the corresponding `proptest` call
  sites thread `shrinkPaths` through. Each shrinkfn guards with
  `requireShrinkPathsArity(shrinkPaths, N)` so a size mismatch
  flows through the same replay-failure path as a stale path.

API notes:

- `PropTestConfig` gains the new field. Like `skipTo` (added in
  kotest#4307) the JVM constructor signature changes; `@JvmOverloads`
  is intentionally not added since it'd balloon the generated
  overload set on a data class with this many defaulted params.
- `@JvmOverloads` is on the public `throwPropertyTestAssertionError`,
  on `doShrinking`, and on the 22 `shrinkfn` overloads, so their
  old JVM signatures are preserved alongside the new ones.
- `ReplayShrinkPathException` and `requireShrinkPathsArity` are
  internal — they don't appear in the API dump.

Tests cover:
- doReplay strict failure modes (out-of-bounds, negative index,
  test passes, assume skip)
- end-to-end: extract Eval index + Shrink paths from a real failure
  message, feed them back via PropTestConfig with ShrinkingMode.Off,
  and verify the same shrunk Arg 0 is reached (proves path-following,
  not searching)
- arity mismatch + stale path both surface as regular property
  failures, not raw exceptions
- maxFailure > 0: shrink runs only on the failure that crosses the
  threshold, not on every allowed failure (regression test for the
  lazy shrinkfn() invocation behavior)
- regression test message format updates in ForAll2Test/ForNoneTest

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants