Independent review of main at 932ab52, done on 2026-09-18. Line numbers below refer to src/jev_lint/cli.py on main unless stated. Every fix landed on the review-fixes branch with a test that fails on main; the offline suite ran after each commit. Evidence came from running the CLI against the brain snapshot (99 pages), a 2-page sample vault, two generated 2,020-page vaults, and the fixture, plus pipx and uv installs from a clean directory.
Counts: 5 high, 12 medium (11 fixed, M8 withdrawn), 9 low, 13 recorded but not fixed, 4 proposed fixes rejected. Two oracle passes (bottom) added H6, M10–M12 and L8, downgraded H3 to L9, and overturned two of my fixes.
cli.py:85-86: data.setdefault(current, []).append(...) when data[current] is already a string (for example tags: work followed by - home). AttributeError, traceback, no report. Real Obsidian vaults contain this form. Fix: only append when the current value is a list. Test: test_frontmatter_bullet_under_scalar_does_not_crash.
cli.py:166-210: for every page pair the loop recomputed linked_names(), claims() and words() of both pages, then sorted all overlaps. Measured 274 s on a generated 2,020-page vault. Fix: precompute per-page claims, word sets and link names once, resolve explicit links through a name-to-index map, keep the same pair order and max() instead of sorted()[-1]. Output verified identical on the brain snapshot (1,267 questions, same order and content). Now 15 s on the dense vault, 5 s on a realistic one.
pyproject.toml on main declares license = "MIT" (PEP 639) with requires = ["setuptools>=68"]. setuptools 68 rejects the field (verified in a setuptools-68 venv). With isolated builds pip fetches the newest setuptools, so pipx install git+… normally succeeds; the failure only appears with --no-build-isolation, offline wheels, or a pinned setuptools. Fix: setuptools>=77, the first version that supports the field.
cli.py:224,231: unresolved-marker and missing-page findings are constructed with 1.0, 1.0. The README promises "Mechanical findings never receive model probabilities"; the HTML hides them but the JSON output and any downstream consumer see a Jev-looking probability that Jev never produced. Fix: probability/confidence are None for rule findings (Finding.probability: float | None). Tests: test_finds_all_four_classes, test_report_labels_rule_and_jev_findings_differently. Verified on the snapshot run: all 10 rule findings now carry null.
cli.py:275-284 catches only HTTPError. DNS failure, connection refused, TLS errors and read timeouts raise URLError/TimeoutError inside worker threads and print a multi-thread traceback. A response without answers.check raised KeyError. Fix: those become RuntimeError with one line; 401 gets a "check TYPESAFE_API_KEY" hint; retries cover 429, 529 and other 5xx. Tests: test_retry_on_429_then_success, test_401_and_network_errors_become_clear_messages_without_the_key.
http.client.putheader raises ValueError("Invalid header value 'Bearer …'") when the key contains a newline or other control character, and nothing caught it: the full key went to the terminal in a traceback. A copied key with a trailing newline in Amp settings is a realistic mistake. Fix: both key sources are stripped; a key that is still not a printable ASCII token without spaces is rejected with a fixed message that never includes it. Test: test_key_is_stripped_and_never_echoed.
cli.py:221-231 scans raw visible lines. On the brain snapshot this produced 3 false positives: graph-legibility.md:15 (`[[wikilinks]]` reported as a missing page) and two pages that document the `[?]` convention. Fix: strip inline code spans before the marker and link scan. Test: test_markers_inside_inline_code_are_ignored. Note: the README's recorded run (16 findings) was made with 0.1.0; the same vault now yields 13. The README says so rather than rewriting the recorded run.
LINK_RE (cli.py:29) captured page\ as the target of [[page\|alias]] (the form Obsidian writes inside tables), so those links were reported as missing and did not count as links when nominating contradiction pairs. [[folder/page]] only matched a page whose full relative path was exactly that, whereas Obsidian's shortest-path setting resolves any trailing path segment. [[diagram.png]] was reported as a missing page (embeds ![[…]] were skipped entirely, including embedded notes). Fix: regex accepts \|, qualified names include every folder suffix, links whose extension is one of Obsidian's accepted attachment formats are skipped, embedded notes are checked like links, .md/.MD is stripped case-insensitively after the attachment check so [[missing.pdf.md]] is a page. The rules are my reading of Obsidian's documented behaviour, checked against the test cases, not against Obsidian itself. Test: test_wikilink_resolution.
cli.py:366-374: the client (and thus the cache) was only constructed when not dry-running, so the estimate for an already-scored vault was the full price, and --dry-run raised the budget error instead of showing the estimate. A user with a 2,000-page vault could not see the cost that --dry-run exists to show. Fix: dry-run reads the cache, reports cached_questions, always prints the estimate, and exits 2 when the estimate exceeds the budget. Tests: test_cache_hits_skip_network_and_budget, test_budget_estimate_excludes_cache_hits_only, test_cli_exit_codes_and_report. Note: staleness questions embed the run date, so they miss the cache on a new day by design (194 of 1,267 on the snapshot); the README now says so.
cli.py:290-295 serialized the whole cache (indent=2, sorted) under the lock after each of the 1,267 responses. O(n²) disk writes; on a 58k-question vault that is gigabytes. Fix: mark dirty, write once in run()'s finally (also on Ctrl-C and errors), compact JSON. Test: test_cache_hits_skip_network_and_budget spies on atomic_write and asserts exactly one write for the whole run.
cli.py:425-428 checked whether report.html is a foreign file only after run() finished. A user with a hand-written report.html in the working directory paid for the run and got an error. Fix: check before run(), and again immediately before the write (oracle: a file can appear during a two-minute run). Test: test_cli_exit_codes_and_report patches JevClient.ask to fail if scoring starts, and has a fake client that creates the output file mid-run.
cli.py:76-78: text.startswith("---\n") fails with a BOM (files saved by Windows editors and some sync tools) and text.find("\n---\n") misses a file that is only frontmatter or ends with --- without a trailing newline. Frontmatter then leaks into claims and the title/aliases are lost. Fix: read with utf-8-sig, accept --- at EOF. Test: test_frontmatter_without_trailing_newline_and_with_bom.
cli.py:111 only toggles on backtick fences. Content inside tilde fences was sent to Jev and scanned for markers. Fix: treat ~~~ as a fence. Test: test_line_numbers_skip_frontmatter_and_fences.
A single long paragraph (minified JSON, a pasted log) becomes one claim and is sent in full, repeated in every contradiction pair it takes part in. I capped claims at 2,000 characters; the oracle pointed out that truncation runs before DATE_RE and words(), so it can change which pairs are nominated and can cut a negation or qualifier out of the text Jev judges. Reverted. The budget estimate already bounds the cost; the limitation is recorded below.
The key was resolved lazily on the first ask(); with an empty cache the user saw the error after threads spun up and after deterministic_findings ran. Fix: client.api_key is checked before scoring when there is anything billable; the message names both sources (env and Amp settings) and never includes any key material. Test: test_missing_key_fails_before_scoring patches JevClient.ask to fail, proving scoring never starts.
ask() only required answers.check to exist. An answer like {} or {"noul": "high"} was cached, then raised KeyError/ValueError inside score_questions on this and every later run until the cache was deleted; a score answer without probabilities produced a finding with an invented probability of 0.0; NaN passed both thresholds. Fix: answer_values() reads exactly the fields the tool uses (noul; or score, confidence, probabilities["2"]), requires finite numbers, and runs before caching. Test: test_malformed_answers_are_errors_and_never_cached (5 shapes).
My first fence fix toggled on either marker, so ~~~ inside a ``` block ended the block and the rest of the code became claim text. Main did not have this bug. Fix: visible_lines records the opening run (character and length) and closes only on a run of the same character at least as long, which also handles fences that contain ````. After the second oracle pass it also follows CommonMark on two more points: a line indented four or more spaces is never a fence, and a backtick opener whose info string contains a backtick (````x``` ````) is an inline span, not a fence. Inline spans are found by a small run scanner (mask_code_spans): whole backtick runs pair only with the next run of exactly the same length, an escaped opener does not count, and spans are masked with spaces instead of removed, so `TO`x`DO` no longer collapses into a marker and link positions stay aligned with the original line. Multi-line spans are not handled (Recorded 12). Tests: `test_line_numbers_skip_frontmatter_and_fences`, `test_fence_edge_cases_follow_commonmark`, `test_markers_inside_inline_code_are_ignored`, `test_code_span_runs_must_match_exactly`, `test_wikilink_resolution` line 8.
---\n---\n has its closing delimiter at index 3, but the search started at 4, so the next horizontal rule in the body was taken as the end of frontmatter and everything before it vanished from linting. Fix: search from 3. Test: test_empty_frontmatter_does_not_swallow_body.
Added -o/--output FILE (parent directory created). README updated.
Added; version is read from jev_lint.__version__ and pyproject uses it dynamically.
Help text and README now state: 0 no findings, 1 findings, 2 error, 130 interrupted. Ctrl-C printed a traceback; it now prints "interrupted".
A first-time user pointing at the wrong path saw "0 pages · 0 findings" and a report file. Now: "no Markdown pages found under …", exit 2, nothing written.
--budget 0 on a small vault printed a contradiction. usd() shows six decimals below $0.00005.
The reference run takes 114 s with no output. Now prints one stderr line with the billable question count and estimate before scoring, and a progress counter when stderr is a terminal.
The overwrite guard ran outside the try that maps OSError to exit 2, and read the whole existing file to check 500 bytes. Fix: main() wraps the whole CLI body (argument parsing, lint(), a final stdout flush) in one error boundary; the guard reads only 500 characters; report and cache go through atomic_write() (tempfile.mkstemp in the target directory, so a stale or symlinked .tmp is never truncated; on failure the temp file is removed and the destination untouched). A closed stdout (jev-lint … | head) previously ended with "Exception ignored … BrokenPipeError" and exit 120; it now exits 2 quietly. Tests: test_cli_exit_codes_and_report (-o <dir> exits 2 with "Is a directory"), test_atomic_write_failure_leaves_destination_and_no_temp, test_closed_stdout_exits_cleanly_without_traceback (subprocess with a closed pipe; exit 120 on main).
"configured threshold" replaced by the actual numbers (contradiction ≥ 0.65; stale score ≥ 1.5 on the 0–2 legend with confidence ≥ 0.5, verified against the TypeSafe score-answer docs where legend keys are 0-based). Cache path documented. Page.path now uses POSIX separators so --json paths are stable on Windows.
- Symlink escape:
load_pagesskips symlinks and anything whose resolved path leaves the resolved vault (cli.py:132-137). Test:test_symlinks_and_hidden_dirs_are_not_scanned. - Atomic report write via temp file and
os.replace; the overwrite guard uses the generator meta tag. Kept. Executor.mapcancels queued futures when the consumer stops iterating after an observed exception, so a failing question does not run the rest of the vault first. I initially addedshutdown(cancel_futures=True);test_failure_cancels_queued_questionspassed on main without it, so the change was dropped. The oracle notes the remaining gap (see Recorded 8).- Numeric claims in README and PRODUCT.md match
vault-run-final.json(99 pages, 1,267 questions, 579,006 tokens, $0.0243, 113.9 s, 2 contradiction, 1 stale, 1 missing-page, 12 unresolved). A rerun on 2026-09-18 with this branch and the owner's cache gave 10 unresolved, 1 stale (0.89/0.79 instead of 0.91/0.83, re-asked because the date changed), and 3 contradictions: the two recorded pairs at 0.83 and 0.68 (recorded 0.84 and 0.71, so the cache holds a later run than the recorded one) pluswiki/decisions/2026-09-amp-agent-routing.md:18at exactly 0.65. Jev answers vary by a few hundredths between runs; the README now says a pair near the threshold can flip. - Budget math: the estimate uses ⅔ token per serialized character, which overestimates the reference run by 1.16× (673,537 estimated against 579,006 measured tokens for 1,267 questions; the
estimated_costinsidevault-run-final.jsonpredates the conservative ratio in 932ab52). Conservative by design; left as is.
- Site copy mismatch (
site/index.html, out of scope for this branch). The terminal mock nameswiki/decisions/agent-routing.md:18, which does not exist invault-run-final.json, andnetwork-guard.md:14, whose real path iswiki/decisions/2026-08-watchdog-network-guard.md:14. The 91 % Amsterdam/Frankfurt example matchesfixture-run.json. Another thread ownssite/today. - Line-fragment claims. Claims are single visible lines of at least 28 characters, so a sentence wrapped across lines is sent to Jev as fragments and the report quotes fragments. Product decision; changing it changes every recorded number.
- Frontmatter
up/relatedlinks are not checked for missing pages, only body wikilinks are. Cheap to add but changes the recorded finding count; left for the owner. - No budget enforcement during the run. The cap is a preflight estimate; if Jev's real token usage exceeds the estimate the run continues. The estimate is conservative (see above), so this is theoretical.
- Question count on
up-structured vaults. Pages that share a parent are all paired, so a vault of 2,000 pages under a handful of hubs produces 58k–111k questions (estimated $0.79–$1.45) and would take hours at the observed ~11 questions/s.--dry-runnow shows this honestly; a per-page pair cap would be the fix. - Windows path in the vault argument.
pathlibhandles separators; not tested on Windows. [[Next.js]]-style names end in what looks like an attachment suffix (.jsis not in the list, but.md-less names with dots are fine). Only the extensions in Obsidian's accepted-formats list are skipped (.oga/.opus, which are not on that list, were removed after the oracle pass).- Fail-fast has a gap (oracle).
Executor.mapyields in submission order: if question 0 is slow (up to the 60 s timeout, longer with retries) while question 1 fails, the other 7 workers keep working through the queue until question 0 returns, so in the worst case most of the vault is still billed. Ctrl-C during the submission phase also waits for queued work. Failures that would trigger this (401, network down) fail every request quickly, so the realistic cost is small; the fix (as_completedwith explicit cancellation) is more code than it saves for a patch release. - Very long lines are sent to Jev in full (see M8). A 50 KB pasted line in ten pairs is about $0.02; the preflight estimate includes it.
- Duplicate concurrent misses (oracle). Two workers can miss the same cache key at once and both pay for it. The preflight counts both, so the budget is not underestimated.
- Stale questions send the page title and
updatedvalue with the claim. The README wording "only selected claim text" was corrected to say so. - Inline code spans are matched per line, so a span that continues on the next line is not masked. Fences cover the multi-line case; the per-line limit is documented here rather than guessed at across paragraphs.
- Cache entries written by 0.1.0 are not re-validated on load; a malformed one (only possible if the API returned garbage) would fail with "unexpected answer" until the cache file is deleted. 0.1.1 never writes such entries.
pool.shutdown(cancel_futures=True)inscore_questions: redundant for the observed-exception case,Executor.map's result iterator cancels the remaining futures in itsfinallywhen the consumer stops iterating. Verified by runningtest_failure_cancels_queued_questionsagainst main's code. Dropped to keep the diff minimal; the residual gap is Recorded 8.- Oracle: "confine the corrected
LINK_REto rule checks so pair nomination stays byte-compatible with main." Rejected. Main's regex capturedpage\for[[page\|alias]], which is a parsing bug, not a nomination rule; the corrected target is what Obsidian links to. On the recorded vault the question list is identical either way (verified: 1,267 questions, same order and content), and keeping two link parsers would be the wrong abstraction. - Oracle: rewrite
score_questionsaroundas_completedwith explicit cancellation. Rejected as Recorded 8: bounded cost, more code. - Oracle: coalesce duplicate in-flight cache misses. Rejected as Recorded 10.
The oracle reviewed REVIEW.md, the diff against main, and the intended behaviour. It kept the question-building rewrite, null rule probabilities, cache-aware dry runs, budget-before-key ordering and end-of-run cache flushing. It added H6, M10, M11, M12 and L8 (all fixed above) and asked for the REVIEW corrections now applied: H3 downgraded and its evidence narrowed; M4/M5/M9 tests gained spies so a wrong ordering or per-answer writes fail them; M2's description of main corrected; the Obsidian-parity wording narrowed; the privacy sentence in the README corrected (Recorded 11). It overturned two fixes: the claim truncation (M8, reverted) and the boolean fence toggle (M11, replaced). Three suggestions were rejected with reasons above.
A second, narrower pass reviewed only the code written after the first pass. It found the indented-closer and backtick-info-string fence cases, the partial-run and escaped-opener weaknesses of the code-span regex, and the broken-pipe exit path (all fixed, see M11 and L8); it asked for the key test to use the characters urllib actually chokes on (done), for OverflowError in the answer parser (done), for an atomic_write failure test (done), and for the fail-fast rationale in Recorded 8 to stop claiming a per-minute bound (corrected). It judged the three rejections defensible for a patch release and confirmed that the answer validation cannot reject a documented TypeSafe HTTP answer (string-keyed probabilities; noul without confidence).