Conversation
…otest#3076) ShrinkResult and StepResult gain a `path: List<Int>` field that records the raw child indices traversed during the shrink search. The failure message now prints a `Shrink paths: ...` line whenever any arg actually shrunk; non-shrinking runs (Exhaustive, ShrinkingMode.Off) stay quiet. This is the first half of kotest#3076. The path is recorded here but no replay machinery yet — that lands in a follow-up. The path uses raw indices into RTree.children rather than positions in the dedup-filtered sequence, so future replay can just walk `children[path[i]]` at each level without reconstructing the search's `tested` state. @jvmoverloads on both data class constructors keeps the old 3-arg and 2-arg JVM signatures, so callers that don't touch the new field stay ABI-compatible.
…ting (kotest#3076) Builds on the previous PR which recorded the shrink path on ShrinkResult. This one wires the path through so users can replay a known failure directly without re-running the shrink search. What's new: - `PropTestConfig.shrinkPaths: List<List<Int>>?` — one path per arg. When supplied, the shrink phase calls `doReplay` instead of `doStep` and walks `RTree.children[path[i]]` at each level. Like `skipTo`, this lives on PropTestConfig only; the higher-level `PropTest` DSL does not expose it. - `doReplay` is strict: out-of-bounds indices, a final value that passes, or a value rejected by `assume()` all throw a (package- internal) `ReplayShrinkPathException`. Silent fallback would label a non-failing value as the shrunk counterexample, which defeats the feature's whole point. - `handleException` catches the replay exception and prints `Note: shrink replay was skipped — ...` to stdout, then continues through the normal property-failure path. Users see a regular property failure (seed, eval index, original cause) — never a raw exception leak. Same path handles size-mismatched shrinkPaths (arity guard). - Failure messages gain an `Eval index: N` line. It's the pre-assumption iteration counter (`PropertyContext.evals()`), which is the value `PropTestConfig.skipTo` consumes. The existing `Property failed after N attempts` is `successes + failures`, so it doesn't line up with `skipTo` when there are assumption skips or earlier skipTo'd iterations. - `shrinkfn` (22 overloads) and the corresponding `proptest` call sites thread `shrinkPaths` through. Each shrinkfn guards with `requireShrinkPathsArity(shrinkPaths, N)` so a size mismatch flows through the same replay-failure path as a stale path. API notes: - `PropTestConfig` gains the new field. Like `skipTo` (added in kotest#4307) the JVM constructor signature changes; `@JvmOverloads` is intentionally not added since it'd balloon the generated overload set on a data class with this many defaulted params. - `@JvmOverloads` is on the public `throwPropertyTestAssertionError`, on `doShrinking`, and on the 22 `shrinkfn` overloads, so their old JVM signatures are preserved alongside the new ones. - `ReplayShrinkPathException` and `requireShrinkPathsArity` are internal — they don't appear in the API dump. Tests cover: - doReplay strict failure modes (out-of-bounds, negative index, test passes, assume skip) - end-to-end: extract Eval index + Shrink paths from a real failure message, feed them back via PropTestConfig with ShrinkingMode.Off, and verify the same shrunk Arg 0 is reached (proves path-following, not searching) - arity mismatch + stale path both surface as regular property failures, not raw exceptions - maxFailure > 0: shrink runs only on the failure that crosses the threshold, not on every allowed failure (regression test for the lazy shrinkfn() invocation behavior) - regression test message format updates in ForAll2Test/ForNoneTest
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of #3076, builds on top of #6097 (path recording). This PR adds the consumer side: a
PropTestConfig.shrinkPathsfield that, paired withseedandskipTo, replays a previouslydiscovered failure directly to its shrunk value without re-running the search.
What it does
After a failure, the message now looks like:
To replay that exact case without re-shrinking:
The shrink phase skips its search and walks
children[0] -> children[2] -> children[1]directly.Pieces
PropTestConfig.shrinkPaths: List<List<Int>>?— one path per arg. LikeskipTo(added inAdd skipTo for skipping N test cases (#4280) #4307) this lives on
PropTestConfigonly; the higher-levelPropTestDSL does not expose it.doReplaywalks the recorded path and runs the test once on the final node. Strict — if apath index is out of bounds (Shrinker changed), the final value passes (property changed), or
it's rejected by
assume(), it throws an internalReplayShrinkPathException. Silent fallbackwould label a non-failing value as the "shrunk" case, which defeats the feature.
handleExceptioncatches that exception and prints aNote: shrink replay was skipped — ...line to stdout, then continues through the normal property-failure path. Users see a regular
property failure (seed, eval index, cause), never a leaked internal exception. The same path
handles
requireShrinkPathsAritywhen the suppliedshrinkPathssize doesn't match theproperty's arity.
Eval index: NisPropertyContext.evals()at failure time — the valueskipToconsumes.The existing
Property failed after N attemptsline usessuccesses + failureswhich driftsaway from
evalswhen there areassume()skips or earlierskipTo'd iterations, so itcan't drop into
skipTocleanly. The two values coexist now.shrinkfn(22 overloads) and the matchingproptestcall sites threadshrinkPathsthrough.maxFailure > 0 lazy shrink
While integrating the catch I noticed I'd moved
shrinkfn()outside the throw branches inhandleExceptionand fixed it before pushing. The shrink/replay now runs only when the testis actually about to fail terminally, matching pre-#3076 behavior. There's a regression test
for it ("with maxFailure > 0, the shrinker is not invoked for allowed failures...").
API / binary compatibility
PropTestConfiggains the new field. Its JVM all-args constructor signature changes — thismatches how
skipTowas added in Add skipTo for skipping N test cases (#4280) #4307.@JvmOverloadswould balloon the generated overloadset on a data class with this many defaulted params, so I didn't add it.
doShrinking, the publicthrowPropertyTestAssertionError, and all 22shrinkfnoverloadsget
@JvmOverloadsso their old JVM signatures stay alongside the new ones. The api dumpshows both.
ReplayShrinkPathExceptionandrequireShrinkPathsArityare internal — they don't enterthe public ABI.
Tests
doReplaystrict failure modes: out-of-bounds, negative index, final-value-passes, assume-skipEval index+Shrink pathsfrom a real failure message, feed them backvia
PropTestConfigwithShrinkingMode.Off, verify the same shrunk Arg 0 is reached. (ModeOff proves replay is following the path rather than searching — without the path the run would
report the un-shrunk initial value.)
ForAll2Test/ForNoneTestexpected messages for the new linesOpen question
The
Shrink paths: [[2, 0, 1]]format was chosen so it copy/pastes cleanly into aPropTestConfig(shrinkPaths = ...)literal. Happy to switch to per-arg lines or a differentlabel if you'd rather — easier to settle now than later.