Skip to content

fix(common): contain jaq parser panics in jq::parse_file, and wire the fuzz dictionaries - #1209

Merged
bryan-minimal merged 3 commits into
mainfrom
fix/jq-panic-containment-and-dicts
Aug 12, 2026
Merged

fix(common): contain jaq parser panics in jq::parse_file, and wire the fuzz dictionaries#1209
bryan-minimal merged 3 commits into
mainfrom
fix/jq-panic-containment-and-dicts

Conversation

@bryan-minimal

@bryan-minimal bryan-minimal commented Aug 12, 2026

Copy link
Copy Markdown
Member

From the latest fuzzing bundle, rebased onto current main (base was 4 commits behind) with authorship normalised.

The fix

jaq/hifijson panics on adversarial JSON — an Ord total-order violation in std sort on numbers that overflow f64 to ±inf, which aborts the process. jq::parse_file is supply-chain-influenced (upstream project data files, via decode::stacks) and its contract is to return Err on bad input, never to abort. Both parse branches are now wrapped in catch_unwind and a caught unwind becomes a JqError.

Verified two ways:

  • No profile sets panic = "abort", so catch_unwind genuinely applies to shipped builds rather than only to tests.
  • The regression test is non-vacuous: with the guard neutralised the checked-in minimised repro panics with user-provided comparison function does not correctly implement a total order. It also asserts the panic path specifically, so it cannot pass if the reproducer ever stops panicking.

Worth noting the target could not prove this itself — jq_parse_json builds panic = "abort" under libfuzzer-sys, so containment is invisible to it. An ordinary #[test] carries the proof instead.

The dictionaries — and one gap I closed

The bundle added four libFuzzer dictionaries. As shipped they were dead files: libFuzzer never discovers a dictionary on its own, it has to be handed -dict=, and neither the just fuzz recipe nor the docs passed one. Same failure mode as a fuzz target that stops compiling — it reads as coverage that is not there.

So each dict is now named after the target it feeds (args.dict -> arg_schema_parse.dict, etc.) and the recipe passes -dict= when fuzz/<target>.dict exists. Convention does the wiring, so the next dict needs no recipe change.

Verified: just fuzz common jq_parse_json reports Dictionary: 14 entries, where it previously reported none.

Also ships the two minimised jq crash inputs as regression seeds, and updates docs/fuzzing.md (which still listed dictionaries as a future idea).

cargo test -p common 51 passing, clippy and fmt clean.

Note

Contain jaq parser panics in jq::parse_file and wire fuzz dictionaries

  • Wraps TOML and JSON parse branches in jq::parse_file with a new guard function that uses catch_unwind to catch panics from the jaq/hifijson stack, converting them into structured JqError values instead of unwinding.
  • Replaces .unwrap() on path conversion with to_string_lossy() to handle non-UTF-8 paths.
  • Adds libFuzzer dictionaries for four fuzz targets (arg_schema_parse, graph_from_bytes, jq_parse_json, mfile_from_toml) and updates the just fuzz recipe to auto-pass a dictionary when a matching .dict file exists.
  • Adds a regression test using a minimized crash artifact to verify panics are surfaced as JqError.

Macroscope summarized 617e95e.

bryan-minimal and others added 3 commits August 12, 2026 09:01
jaq/hifijson can panic on adversarial JSON — a fuzzer found an Ord
total-order violation in std's sort on numbers that overflow f64 to
±inf, aborting the process. parse_file's input is supply-chain-influenced
(upstream project data files, via decode::stacks), and the module already
contracts to return Err on bad input, never to abort. Wrap both the JSON
and TOML parse branches in catch_unwind and convert a caught unwind to
JqError.

Proven by parse_file_contains_jaq_panic, an ordinary #[test]: the
jq_parse_json fuzz target builds panic=abort (libfuzzer-sys installs an
abort hook) and so cannot observe the containment. Minimized crash
fixture checked in.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dictionaries for the structured targets — graph wire tag bytes + JSON
keys, minimal.toml section/key vocabulary, jq JSON tokens, and the
ArgSchema grammar — steer the mutator toward structurally valid inputs
instead of rediscovering framing from scratch. Measured overnight:
graph_from_bytes edge coverage rose 6.5k → 8.3k with graph.dict live,
and arg_schema_parse moved off a multi-billion-exec plateau.

Seeds preserve both jq_parse_json Ord-violation crash variants for
regression, alongside the checked-in minimized fixture.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dictionaries landed as files nothing read. libFuzzer does not discover a
dictionary on its own — it has to be handed `-dict=` — and neither the `just
fuzz` recipe nor the docs passed one, so all four were inert. That is the
same failure mode as a fuzz target that no longer compiles: it reads as
coverage that is not there.

Name each dict after the target it feeds (`args.dict` ->
`arg_schema_parse.dict`, and so on) so the recipe can find it by convention,
and pass `-dict=` when `fuzz/<target>.dict` exists. A target without one is
unaffected; a dictionary only biases mutation.

Verified: `just fuzz common jq_parse_json` now reports `Dictionary: 14
entries` where it previously reported none.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bryan-minimal
bryan-minimal requested a review from a team as a code owner August 12, 2026 16:12
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 27 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 300254bf-200a-4fb6-8d2b-aac90a940338

📥 Commits

Reviewing files that changed from the base of the PR and between 536fc29 and 617e95e.

📒 Files selected for processing (10)
  • crates/args/fuzz/arg_schema_parse.dict
  • crates/common/fuzz/jq_parse_json.dict
  • crates/common/fuzz/seeds/jq_parse_json_regressions/ord_violation
  • crates/common/fuzz/seeds/jq_parse_json_regressions/ord_violation_variant2
  • crates/common/src/jq.rs
  • crates/common/src/testdata/jq_ord_violation_repro
  • crates/graph/fuzz/graph_from_bytes.dict
  • crates/mfile/fuzz/mfile_from_toml.dict
  • docs/fuzzing.md
  • justfile

Comment @coderabbitai help to get the list of available commands.

@twitchyliquid64 twitchyliquid64 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also report this upstream ?

@bryan-minimal

Copy link
Copy Markdown
Member Author

Yes — and it turns out to be a clean upstream bug, so thanks for asking. I dug in.

It is not our call site. jaq_json::read::parse_single panics on its own, with none of our code in the picture and no jq program involved — just parsing a document.

Root cause is in jaq-json src/num.rs: Num::cmp compares exactly within a variant pair but lossily through f64 across variants.

(Self::Int(x),    Self::Int(y))    => x.cmp(y),                          // exact
(Self::Int(i),    Self::Float(f))  => float_cmp(*i as f64, *f),          // lossy
(Self::BigInt(x), Self::BigInt(y)) => x.cmp(y),                          // exact
(Self::BigInt(x), Self::Float(y))  => float_cmp(x.to_f64().unwrap(), *y) // saturates to inf

So equality is not transitive, and it needs nothing exotic to show:

let a = Num::Int(1_000_000_000_000_000_001);
let b = Num::Int(1_000_000_000_000_000_002);
let f = Num::Float(1e18);      // f64 spacing at 1e18 is 128
// a == f, b == f, yet a < b

The BigInt arm is the same defect with a wider mouth: every integer above f64::MAX is Equal to Float(inf). Our fuzz artifact hits it because Map<Val, Val> means object keys are Vals, and both Hash for Val and Ord for Val sort an object entries by key — so hashing an object-as-key runs Num::cmp over numbers straddling the boundary. Delta-minimised 1461 -> 799 bytes.

Separately, Float(NAN).cmp(&Float(NAN)) is Less — a value strictly less than itself. The comment there explains the jq-compatible nan-sorts-smallest intent, which seems reasonable, but Ord is also what BTreeMap and binary_search stand on. Returning Equal for that one pair keeps the jq behaviour and restores reflexivity.

Not reported upstream (checked the tracker; #232 is unrelated jq-compat stuff) and 2.0.2 is the latest release, so it is live.

I have an issue drafted with both reproducers and a suggested fix — compare Int/BigInt against Float exactly, since every finite f64 is a dyadic rational, which is contract-correct and more accurate than today. Will file it against 01mf02/jaq unless you would rather send it.

Worth being explicit that this does not change the case for this PR: catch_unwind here is containment, not a fix. Even with an upstream release we would want to keep it, since the guard is what makes the Err contract true regardless of what the parser does.

@bryan-minimal

Copy link
Copy Markdown
Member Author

Went back and checked the jaq tracker properly — my last comment said "not reported upstream" off a keyword search, which was too thin an answer to give you. Enumerating all 460 issues+PRs (state=all) instead of relying on GitHub fuzzy search turned up two things that change the report.

Still nothing tracking it, confirmed properly this time: no issue or PR mentioning total order, Ord, transitivity or a sort panic. I also cloned the repo — master (6e82c3c, Aug 7) is byte-identical to 2.0.2 in the Ord impl, so it is live rather than fixed-but-unreleased.

There is a sharper bug than the one I described. PR #343 ("Simplify integer-float equality", merged Oct 2025) gave PartialEq an f.is_finite() guard. Ord::cmp, twenty lines below, never got it:

// PartialEq (num.rs:336)
(BigInt(i), Float(f)) | (Float(f), BigInt(i)) => f.is_finite() && float_eq(i.to_f64().unwrap(), *f),
// Ord (num.rs:364)
(BigInt(x), Float(y)) => float_cmp(x.to_f64().unwrap(), *y),

So a == inf is false while a.cmp(&inf) is Equal, and Ord requires those to agree. That is one containment rule with two implementations and only one of them hardened — the same shape as the tar and path issues we have been working through, which I did not expect to find waiting for me in a dependency. It now leads the report, because unlike the precision question it needs no design debate and the fix is one line.

#343 also saw this coming and priced it wrong, which is the most useful thing to tell an upstream maintainer. Its own description anticipates the collision — "if you store many very large integers that are quite close to each other and that fit into f64 as object keys... many of them map to the same hash, and therefore looking up such entries will be rather slow" — and weighs it purely as a performance risk. The same collision is what makes Eq non-transitive and Ord non-total. So the report is not "you missed this", it is "you considered this and the cost was larger than it looked".

One correction to my last comment: I said the nan < nan behaviour looked like a reasonable jq-compat intent. It is more than intent — PR #248 ("NaN compatibility") was merged specifically to enforce it. So I have demoted that from a bug to a note about what the decision costs in std-library terms, and I am not proposing to change the observable semantics.

Worth noting for our own purposes: jaq does have fuzz targets (#262), but they cover jaq-core lexer/parser, not the jaq-json reader. That is the gap our parse_single harness landed in.

Draft is rewritten and ordered by how unarguable each point is. Nothing here changes this PR — the guard is still the right call regardless of what upstream decides.

@bryan-minimal

Copy link
Copy Markdown
Member Author

going to merge and deal with the notification separately

@bryan-minimal
bryan-minimal merged commit 0375b20 into main Aug 12, 2026
30 checks passed
@bryan-minimal
bryan-minimal deleted the fix/jq-panic-containment-and-dicts branch August 12, 2026 23:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants