Property-based and differential tests for Nix evaluators, written with Hegel (hegel-rust).
The goal is to check alternative evaluators such as fix for equivalence with Nix, and to catch bugs in Nix itself along the way.
Each property draws inputs with Hegel, builds a Nix expression, and calls
check(expr, expect). That evaluates the expression with every
configured evaluator and fails the test case if:
- an evaluator crashed or timed out,
- an evaluator disagrees with the first ("reference") one, or
- the result differs from the Rust model of the builtin, if there is one.
Evaluation errors are compared by whether they happened, never by message.
The model can be Expect::Value(json), Expect::Error, or
Expect::Unspecified (differential only). check returns the reference's
Outcome, so a property can build on it (e.g. check that a canonical form
is a fixed point).
Hegel shrinks failing inputs, so a report looks like this:
let expr = "(map (assert false; (v1: v1 + 0)) [ ])".to_string();
expression:
(map (assert false; (v1: v1 + 0)) [ ])
model:
<unspecified, differential only>
cppnix:
[]
lix:
[]
fix:
<evaluation error> error: assertion failed
problems:
fix disagrees with cppnix
# CppNix evaluation server (needs a Nix checkout with the C API; or omit
# --arg nixFlake to build against nixpkgs' Nix)
nix-build -A server-capi --arg nixFlake ../nix -o result-server-capi
# fix main with the server patch (--arg fixSrc ../fix for a checkout)
nix-build -A fix -o result-fix
# Lix main
nix build git+https://git.lix.systems/lix-project/lix -o result-lix
. ./evaluators.env # CppNix (reference), Lix, fix; long-lived sessions
nix-shell --run 'cargo test --no-fail-fast'
HEGEL_TEST_CASES=2000 cargo test --test flakes # more examples
cargo test --test strings substring # one property
NIX_PBT_SKIP=circular-import cargo test --test flakes # get past known divergences
# evaluate by hand with every configured evaluator
cargo run -q --bin nix-pbt-eval -- 'builtins.splitVersion "__"'
cargo run -q --bin nix-pbt-eval -- --bench 1000 '1 + 1'Without any configuration, the tests use nix-instantiate from $PATH.
evaluators-exec.env is the same three-way setup in exec mode (no
server needed, ~100× slower).
| Variable | Meaning |
|---|---|
NIX_PBT_EVALUATORS |
;-separated name=[mode:]command… entries, see below. The first is the reference. |
NIX_PBT_FLAKE_LOCK |
;-separated name=command… entries that lock the flake in the current directory (for flakes::lock_files_are_interchangeable). |
NIX_PBT_SKIP |
Comma-separated known-divergence tags to skip, see below. |
NIX_PBT_TIMEOUT |
Per-evaluation timeout in seconds (default 10). |
NIX_PBT_MAX_SESSIONS |
At most this many sessions per evaluator (default: one per test thread). |
NIX_PBT_LOG |
Append every evaluation and each evaluator's outcome to this file, as JSON lines. Useful when Hegel reports a flaky test. |
NIX_PBT_TMPDIR |
Where generated flakes and file trees go (default $TMPDIR; nix-shell deletes that on exit, so set this to keep the trees of failing tests). |
HEGEL_TEST_CASES, HEGEL_SEED, HEGEL_STATISTICS, … |
Hegel settings, see the hegel docs. |
-
exec(default): one process per expression, which is appended as the last argument. The command must print the strict value as JSON and exit non-zero on errors (nix-instantiate --eval --strict --json -E,fix eval --strict --json -E). About 30 evaluations/s. -
repl:: a long-lived REPL. Each expression is sent asbuiltins.toJSON (EXPR), followed bybuiltins.trace "NONCE" "NONCE"to find the end of its output on stdout and stderr. Nix-style and JSON-style string literals and ANSI colours are all handled. Works withnix repl(CppNix, Lix) andfix repl --json. -
server:: a long-lived process speaking a tiny protocol:request: <byte length>\n<expression> response: ok <byte length>\n<JSON value> | error <byte length>\n<error message>servers/nix-capi/server.cimplements it on the CppNix C API (~100 lines), andservers/fix/pbt-server.patchadds it to fix asfix repl --pbt-server(~60 lines of Zig;default.nixapplies it). It's the most robust option and the easiest one to add to another evaluator: unlike a REPL it takes multi-line expressions and any byte. The CppNix server evaluates in read-write mode, likenix repl: sources and derivations are written to the store. The fix server is read-only unless given--read-write-mode(likefix eval; the patch also letsfix repltake that flag), and keeps its parallel evaluator in read-write mode, likefix instantiate(fix eval --read-write-modeturns speculation off). It uses one daemon connection instead of fix's usual pool of 8.evaluators.envruns fix three ways: read-write with one worker (fix), read-write with 4 workers (fixpar), and read-only (fixro).
Sessions are pooled (one per concurrently running test, at most
NIX_PBT_MAX_SESSIONS), restarted after a crash or timeout, and recycled
every 5000 evaluations. A REPL that exits after printing an ordinary
error: counts as that error, not as a crash (Lix's REPL exits on stack
overflows). Lix ≥ 2.96 prints its prompt with TERM=dumb and applies
backspaces in piped input; the REPL driver strips prompts and sends \b
as an interpolation.
Throughput with evaluators.env (nix-pbt-eval --bench):
| evaluator | mode | evaluations/s |
|---|---|---|
| CppNix | server (C API) | ~60,000 |
| Lix | repl | ~30,000 |
| fix | server, read-write, --workers 1 |
~45,000 |
| fix | server, read-write, --workers 4 |
~25,000 |
| fix | server, read-only, --workers 1 |
~47,000 |
| fix | repl, --workers 1 |
~1,450 |
| any | exec | ~30 |
A full run of the suite takes under a minute.
Hegel reports one (minimal) failure per property. To look past a known
divergence, tests mark the inputs that trigger it with a tag, and
NIX_PBT_SKIP=tag1,tag2 rejects those inputs (or stops generating them):
| Tag | Test | Divergence |
|---|---|---|
circular-import |
flakes::* |
fix accepts some circular flake imports |
lock-no-check |
flakes::lock_files_are_interchangeable |
fix flake lock writes lock files with follow cycles / dangling follows |
root-input-hash |
flakes::lock_files_are_interchangeable |
fix: an input pointing at the root flake's own directory fails its lock file's narHash check |
lock-node-names |
flakes::lock_files_are_interchangeable |
fix names duplicate input nodes (a, a_2) in a different order (lock files are compared with canonical node names) |
override-original |
flakes::lock_files_are_interchangeable |
Lix: the original of an input overridden with a URL is the declared ref, not the override (lock files are compared without original) |
override-follows |
flakes::lock_files_are_interchangeable |
Lix follows a follows that an input override replaced (stack overflow) |
empty-list-laziness |
exprs::*, builtins::* |
fix doesn't force what map/filter/sort/all/… don't need on empty and one-element lists |
invalid-utf8-json |
exprs::* |
Lix < 2.95 crashes printing invalid UTF-8 as JSON |
convert-hash |
store::convert_hash, builtins::builtin_names |
Lix and fix have no convertHash |
error-order |
builtins::*, exprs::error_kinds_agree |
which of two errors wins depends on evaluation order, which differs (see Findings); only plant throws in callbacks |
context-compare |
builtins::* |
fix: < on a string with context and one without is a type error |
no-context |
builtins::* |
fix accepts string context where CppNix rejects it |
lax-types |
builtins::* |
fix doesn't type-check some arguments like CppNix |
function-equality |
builtins::* |
Lix ≥ 2.94 and fix: f == f is true; CppNix: false (and so [ f ] < [ f ] is an error) |
throw-coercion |
builtins::* |
fix: throw of a path or a set with __toString/outPath is a type error, which tryEval doesn't catch |
number-select |
builtins::operators_on_arbitrary_operands |
0.a: CppNix selects from 0, Lix and fix reject it (tokens-no-whitespace) |
tojson-outpath |
builtins::* |
fix: toJSON of a set with a non-string outPath is a type error |
generic-closure-operator |
builtins::* |
fix: genericClosure { startSet = [ ]; } requires operator |
foldl-thunk |
builtins::* |
fix: foldl' op nul [ ] returns nul unforced, and applying it fails |
regex |
builtins::*, regex::* |
match/split regular expressions: fix has its own (accepts some that CppNix rejects, rejects some it accepts, picks other capture groups) |
fetchtree-revcount |
fetchers::*git* |
Lix returns revCount from fetchTree (CppNix only from fetchGit) |
tarball-top-level |
fetchers::fetch_tarball |
Lix wants exactly one top-level entry in a tarball |
fetch-path-name |
fetchers::*path* |
fix: fetchTree { type = "path"; } is named after the directory, not source |
tarball-last-modified |
fetchers::*tarball* |
fix: tarballs have lastModified = 0 |
git-dirty-rev |
fetchers::fetch_git |
fix: a dirty repository gets rev = HEAD |
canonpath-signed-char |
fetchers::*git* |
CppNix drops a directory src from local git trees when there's a srcü next to it |
git-stale-mount |
fetchers::stale_git_mount |
CppNix reads a fetched git tree from a deleted repository |
store-write-desync |
store::store_errors_dont_poison_the_session, and the harness |
fix: after the daemon rejects a write, the connection is out of sync. With the tag, the harness also restarts a session after a daemon error, so that it doesn't spill into other tests |
versions |
versions::*, builtins::* |
fix: splitVersion/compareVersions of versions with _ or other punctuation, non-ASCII, or components ≥ 2³¹ |
float-to-int-range |
arith::floor, arith::ceil |
fix: out-of-range results are -9223372036854775808 |
subnormal-literals |
arith::* |
fix accepts subnormal float literals |
to-string-float |
strings::to_string_float |
fix rounds a float's shortest digits, not its exact value |
drv-validation |
store::derivation_paths |
fix accepts an empty system and names too long for the store |
drv-output-split |
store::derivation_outputs |
fix doesn't split outputs at whitespace ([ "a b" ] is outputs a and b) |
output-hash |
store::fixed_output_derivations |
fix parses outputHash its own way: any prefix is an algorithm (foo-bar), no empty hash, no outputHashMode = "nar", and a nix32 hash of only hex digits is read as base-16 |
drvpath-output |
store::derivation_outputs |
Lix, like Nix 2.18, accepts an output named drvPath, and derivationStrict then returns a set with two drvPath attributes |
hash-mode-nar |
store::fixed_output_derivations |
Lix, like Nix 2.18, doesn't know outputHashMode = "nar", newer Nix's name for recursive |
blake3 |
store::fixed_output_derivations |
outputHashAlgo = "blake3": CppNix main requires the blake3-hashes feature (2.31 didn't); Lix, like 2.18, doesn't know it, so it counts as no algorithm |
relative-inputs |
flakes::relative_inputs, flakes::subdir_flakes |
fix can't read relative inputs (path:./sub); Lix, like Nix 2.18, fetches one as a tree of its own instead of a part of its parent's source |
flake-subdir |
flakes::subdir_flakes |
fix: a flake in a subdirectory (?dir=a) has the source root as its outPath, not …-source/a |
fetchtree-url-shallow |
fetchers::fetch_git |
fix: fetchTree "git+file:…" has no revCount (it applies the attrset form's shallow = true default to a URL too) |
drv-unparsable |
store::derivation_paths |
CppNix and Lix write a .drv they can't read back for a system with " or \: an error in read-write mode, the path in read-only mode (fixro) |
drv-output-name |
store::derivation_* |
Lix rejects an output named drv |
context-nonexistent |
store::append_context_nonexistent_path |
fix accepts context for paths that don't exist |
split-context |
store::context_through_builtins |
fix: split drops context |
parse-drv-name |
store::parse_drv_name_model |
fix: leading and trailing dashes |
flakeref |
flakeref::* |
fix's (and a bit of Lix's) flakeref parser; see Findings |
read-only-outpath |
fetchers::* |
read-only fix: fetched trees' outPath isn't in the store |
path-lexing |
builtins::operators_on_arbitrary_operands |
fix: -./x and -/x are negations, not paths |
toxml-primop |
builtins::* |
fix: toXML prints primops as <function /> |
read-only-names |
store::to_file, store::derivation_paths |
read-only fix doesn't validate store path names |
failed-copy-crash |
store::store_errors_dont_poison_the_session, and the harness |
Lix main segfaults on the first store copy after copying a missing path fails. With the tag, the harness restarts a session after such an error |
known.env sets all of them (for CppNix main, Lix main and fix main), and
then the suite passes: every failure is a new finding.
. ./evaluators.env; . ./known.env
nix-shell --run 'cargo test --no-fail-fast'src/eval.rs: evaluator configuration, the three modes, session pool.src/value.rs:NixValue, rendering to Nix literals, JSON conversion, a model of==.src/generators.rs: Hegel generators for strings, attribute names, ints, floats, nested values.src/store.rs: exact models of store path computations: nix32,compressHash,makeStorePath, text paths, the ATerm.drvformat,hashDerivationModulo,placeholder.src/lib.rs:check,check_true,skip_known,commands,tmp_dir.src/bin/nix-pbt-eval.rs: evaluate or benchmark by hand.servers/nix-capi/: the CppNix evaluation server;default.nixbuilds it.servers/fix/pbt-server.patch:fix repl --pbt-server;default.nixbuilds fix main (or--arg fixSrc) with it.
| Test file | What | Ported from (Nix repo) |
|---|---|---|
strings.rs |
stringLength, substring, concatStringsSep, replaceStrings, toString, typeOf, literals, interpolation, invalid UTF-8 |
|
lists.rs |
length, elemAt, head/tail, sort (incl. stability), genList, concatLists, filter, elem, foldl', == |
|
attrs.rs |
attrNames, attrValues, hasAttr/getAttr, removeAttrs, intersectAttrs, //, listToAttrs, mapAttrs, catAttrs |
|
arith.rs |
integer ops and overflow, bit ops, float arithmetic, mixed comparisons, floor/ceil, literals |
|
json.rs |
toJSON/fromJSON |
|
versions.rs |
splitVersion, compareVersions |
libstore/names.cc |
regex.rs |
match/split of patterns built from the pieces where libstdc++'s std::regex (Nix's regular expressions) differs from POSIX and other engines: escapes, brackets, collating elements, intervals, capture groups under alternation and repetition; known answers |
|
flakes.rs |
random flake graphs with follows, nested overrides, flake = false: resolved graph vs a model of computeLocks; lock files from each implementation read by every other one; trees of flakes with relative inputs, in subdirectories (?dir=), in git repositories (differential: values, outPaths, attribute names, lock files) |
functional/flakes/follow-paths.sh, inputs.sh, non-flake-inputs.sh |
flakeref.rs |
parseFlakeRef/flakeRefToString: the unit test table, attrs round trip, canonical form is a fixed point, parsing URL-ish strings |
libflake-tests/flakeref.cc |
store.rs |
toFile, placeholder, derivation (exact drvPath/outPath, context), fixed-output derivations (every outputHash encoding, outputHashAlgo/outputHashMode edge cases, against a model of Hash::parseAny and makeFixedOutputPath), outputs split at whitespace, appendContext/getContext round trip, context propagation, parseDrvName, hashString, convertHash |
libexpr-tests/value/context.cc, libstore/derivations, names.cc |
exprs.rs |
random well-typed expressions (let, lambdas, formals with defaults, if, assert, rec sets, //, or, ?, tryEval, builtins) with throws in lazy positions; laziness laws; error kinds (tryEval (deepSeq e true)) |
|
builtins.rs |
every builtin and operator applied to arbitrary arguments (mostly of the right type, with throw, failed assertions, missing <paths>, abort and type errors planted in them) inside tryEval (deepSeq e true), which tells catchable errors from uncatchable ones; string context rejection; the set of builtins |
|
fetchers.rs |
generated "angry" file trees (every kind of byte in names, invalid UTF-8, 255-byte names, symlinks, executable bits, empty directories) through builtins.path (+ filter), filterSource, fetchTree (path, git clean and dirty and as a URL, tarball with three layouts), fetchTarball, fetchurl, and a readDir/hashFile walk, against a model of NAR hashes and store paths; the angryfiles corpus (every one-byte name, every name length) |
libfetchers-tests, functional/fetchGit.sh, tarball.sh |
#[hegel::test]
fn string_length(tc: TestCase) {
let s = tc.draw(nix_strings());
check(
&format!("builtins.stringLength {}", nix_string_literal(&s)),
Expect::value(s.len()),
);
}Build expressions with nix_string_literal / nix_int_literal /
nix_float_literal / NixValue::to_nix: they handle ${, \r, negative
numbers, i64::MIN, and floats (1e3 isn't a float literal in Nix).
Expressions must be single-line for repl mode.
Some tests write to the Nix store: toFile, derivation, and the flake
tests (which copy the generated flakes into the store). The generated flake
trees live in $TMPDIR/nix-pbt-flakes-*, are made read-only (fix writes
lock files during getFlake otherwise), and are kept when a test fails.
Against CppNix 2.36.0pre20260925 (../nix), Lix 2.96.0-dev (main,
0c16765) and fix 0.3.0 (main, 3b2ffbb2, unchanged since 2026-09-18).
"Model" means the behaviour of CppNix, which all models are written
against.
| Area | Input | Nix | fix |
|---|---|---|---|
| laziness | map (throw "f") [ ], same for filter, sort, all, any; sort of one element, filterSource of a file |
forces the function | doesn't |
| laziness | sort lessThan [ (throw "x") ] |
forces the element | [ … ] |
| parsing | -./x, -/x (e.g. builtins.typeOf -./x, 1 -./x) |
a path literal: "path", calls 1 |
negation of a path: type error |
toXML |
toXML builtins.add |
<unevaluated /> |
<function /> |
| laziness | genericClosure { startSet = [ ]; } |
[ ] |
error: missing attribute (operator) |
foldl' |
builtins.foldl' (x: x) (let f = a: a; in f) [ ] 1 |
1 |
error: expected function, got thunk |
toJSON |
toJSON { outPath = 1; }, { outPath = [ 1 ]; }, … |
the outPath, whatever it is: "1" |
type error |
| laziness | removeAttrs { } [ (throw "t") ] |
error | { } |
toString |
toString 3.002399751580331e16, toString 1.0078125, toString 6.71088640127945e7 |
"30023997515803312.000000", "1.007812", "67108864.012794": the exact value, rounded to even |
"30023997515803310.000000", "1.007813", "67108864.012795": the shortest round-trip digits, rounded half up |
splitVersion |
"__" / "a_b" / "é" |
["__"] / ["a_b"] / ["é"] |
["_","_"] / ["a","_","b"] / type error |
compareVersions |
"0" "2147483648" |
1 (32-bit component quirk) |
-1 |
floor/ceil |
floor 9.223372036854776e18 |
error | -9223372036854775808 |
| float literals | 1.1125369292536007e-308 |
error: invalid float | accepted |
parseDrvName |
"a-", "-0" |
{ name = "a-"; version = ""; }, { name = ""; version = "0"; } |
{ name = "a"; }, { name = "-0"; version = ""; } |
toFile, derivation |
toFile ".-" "", an output named "a b" |
error: invalid store path name | accepted without --read-write-mode (with it, the daemon rejects it) |
derivation |
name such that name.drv > 211 chars |
error | accepted |
derivation |
system = "\\" |
error (see Nix below) | accepted (same, unescaped, drv hash) |
| string context | appendContext for a path that doesn't exist |
error | accepted |
| string context | head (split "b" s) |
keeps s's context |
drops it |
| string context | "${drv}" < "z", lessThan, sort lessThan, [ s ] < [ "z" ]: one string with context, one without |
compares | error: type error |
| string context | getEnv, placeholder, match, split, parseDrvName, compareVersions, splitVersion, removeAttrs, listToAttrs, catAttrs, groupBy, unsafeGetAttrPos, derivation's outputs/outputHash/outputHashMode, of a string with context |
error | accepted |
| types | storePath "/nonexistent", storePath { outPath = "/x"; } |
error | returns it |
| types | appendContext ./p { … }, appendContext "a" { a = …; }, hasContext ./p, toFile "a" ./p, toFile "a" { __toString = …; }, removeAttrs { } [ 1 ] |
error | accepted |
| types | toPath { outPath = "/x"; }, unsafeDiscardOutputDependency { __toString = …; } |
coerces | error |
| types | unsafeDiscardOutputDependency ./p |
copies ./p to the store |
returns the path |
convertHash |
converts | doesn't exist (neither in Lix) | |
fetchTree, flakes |
outPath of a fetched tree or flake (also self), in fix eval without --read-write-mode |
a store path, with context | the source directory or ~/.cache/fix/…/source, without context: drvPaths of derivations using them (src = self;) differ from Nix's |
fetchTree git |
a dirty working tree | dirtyRev, no rev (fetchGit: rev = "000…", revCount = 0) |
rev = HEAD with the dirty tree's narHash: fetching that rev gives another narHash |
fetchTree path |
store path name | …-source |
named after the directory |
fetchTree tarball |
lastModified |
newest mtime in the tarball | 0 |
== |
let f = x: x; in f == f, builtins.add == builtins.add |
false |
true, like Lix ≥ 2.94 |
| store writes | after the daemon rejects a write (toFile "..-x" ""), in the same session |
later writes work | the connection is out of sync for good: the next write gets the earlier write's error, then ReadFailed, then WriteFailed. With the default pool of 8 connections this looks random |
derivation |
system = "" |
error: required attribute system missing |
accepted |
getFlake |
any path: flake whose lock is incomplete |
doesn't write flake.lock |
writes flake.lock into the source tree during evaluation |
| flakes | circular import via input overrides | error: circular import | evaluates |
| flakes | an input path:/…/f0 in /…/f0/flake.nix (the flake's own directory), locked by CppNix |
uses the lock file | error: NAR hash mismatch (the lock file changed the directory) |
flake lock |
inputs.a.follows = "a", follows to a missing input |
error (LockFile::check) |
writes the lock file |
flake lock |
two inputs b and c from the same flake with an input a each, and inputs.a.follows = "c" |
b's is node a, c's is a_2 |
the other way round: the same graph, but the lock file differs |
fix repl |
builtins.readFile ./f again after f changed |
the new contents | the old contents: the file cache lives as long as the session ("${./d}" and fetched trees are cached in nix repl too) |
parseFlakeRef |
"nixpkgs", "flake:nixpkgs/branch" |
{ type = "indirect"; id = "nixpkgs"; … } |
resolves through the registry to a channels.nixos.org tarball |
parseFlakeRef |
"github:a/b#frag", "/foo/bar#bla", "git+https://#" |
error: unexpected fragment / drops # |
fragment kept in repo/path/url |
parseFlakeRef |
"github:foo/bar?xyzzy=1", "/foo/bar?xyzzy=1" |
error: unknown parameter | ignored |
parseFlakeRef |
"github://////owner%42/////repo%41///branch%43////" |
github:ownerB/repoA/branchC |
github:/ |
parseFlakeRef |
"github:nixos/nix//master///something/" |
ref master/something |
ref dropped |
parseFlakeRef |
"git://somewhere/repo?ref=branch" |
git://… |
git+git://… |
flakeRefToString |
{ type = "path"; path = "/x y"; }, http://…/+3d.tar.gz |
percent-encodes (%20, %2B) |
doesn't |
flakeRefToString |
{ type = "indirect"; id = "a"; } |
"flake:a" |
error: InvalidFlakeRef |
fix fails 21 of the 31 cases from Nix's own flakeref.cc.
tryEval. builtins.rs looks for errors that tryEval catches in
one implementation and not in another: every builtin and operator, with
throws, failed assertions and missing <paths> in their arguments and
callbacks, over hundreds of thousands of cases. One is a fix bug:
throw of a path, a derivation, or a set with __toString or outPath
is a type error in fix, which tryEval doesn't catch; Nix coerces the
argument to a string like interpolation does, so it's an ordinary throw
(builtins::throw_coerces_its_argument):
nix-repl> builtins.tryEval (throw ./foo)
{ success = false; value = false; } # fix: error: expected a string, got path
In every other case where the kinds of error differ, fix catches a
throw that Nix never gets to, because Nix fails on an earlier argument
first with an error tryEval doesn't catch (the error-order tag):
| Expression | Nix | fix |
|---|---|---|
null + (throw "t") |
coerces null to a string first: type error |
forces both operands: throw |
substring (-1) (throw "t") "" |
negative start position | throw |
hashFile "" (throw "t") |
unknown hash algorithm | throw |
findFile [ ] "${ctx}" |
string has context | not found (a throw) |
addErrorContext { } (throw "t") |
the throw, then coercing the context message fails, which replaces it |
throw (the message is never coerced) |
partition (x: throw "t") [ (abort "a") ] |
forces the element first: abort |
calls the function first: throw |
Read-only mode and the parallel evaluator. Read-only fix (fixro,
like plain fix eval) gives the same results as read-write fix, except
that it doesn't validate store path names (toFile ".-" "" works) and
that fetched trees' outPath is the source directory or fix's cache
instead of a store path, which also changes the drvPaths of derivations
that use them. fix with 4 workers and speculation (fixpar, read-write)
agreed with fix with 1 worker in every test, over the whole suite at
20,000 cases per property (the slow fetch_tarball and
lock_files_are_interchangeable at a few thousand).
What agrees: all other builtins covered here, over thousands of cases;
drvPath/outPath for random derivations, bit for bit; follows and
override resolution over thousands of random flake graphs; lock files,
which are identical to CppNix's and interchangeable with them; random
expressions once the empty-list laziness issue is skipped; and NAR
hashes and store paths of builtins.path, filterSource and fetchTree
(git and tarball, read-write mode) for thousands of angry file trees
and the angryfiles corpus.
A stack of 39 jj changes on fix main (3b2ffbb2), in the fix checkout
next to this one, fixes the fix findings above except where fix arguably
does better. Each change has unit tests for the cases these tests found.
Against it, the suite passes with only these tags (fix-stack.env):
| Tag | Why it stays |
|---|---|
error-order |
which of two errors wins depends on evaluation order |
function-equality |
fix keeps f == f being true, like Lix |
subnormal-literals |
fix accepts subnormal float literals; accepting more can't break code written for Nix |
context-nonexistent |
Nix only checks this in read-write mode, by substituting the path; fix records .drvs and toFile outputs lazily, so a store query would reject paths fix made itself. Accepting more can't break code written for Nix |
number-select |
0.a is a parse error, as in Lix |
canonpath-signed-char, git-stale-mount, drv-unparsable |
CppNix bugs |
Fixing these turned up more fix bugs, also fixed in the stack:
unsafeDiscardOutputDependencydropped output context;- a tarball without a single top-level directory was unpacked wrongly;
- a git tree with a relative symlink crashed fix (
symLinkAbsolute); - in a long-lived session, a tree fetched again after its first source was deleted was read from the deleted directory;
- regular expressions: fix's own matcher accepted patterns Nix rejects
(
\d,a{,[z-a]), rejected ones it accepts (a*?,a**), picked other capture groups ((a|ab)(c|bcd)againstabcd) and didn't match(a*)+(b)againstbat all. The stack ports libstdc++'sstd::regex, which is what Nix's regular expressions are, fuzzed against it (580k cases without a difference); derivation: without structured attrs, Nix splitsoutputsat whitespace; output store path names weren't checked without a daemon;outputHashtook any prefix as the algorithm (foo-bar), rejected an empty hash andoutputHashMode = "nar", andoutputHashModewasn't checked without anoutputHash;- flakes, found by comparing real flakes rather than by these tests at
first:
fix eval --flake .#xin a git repository used apath:flake, where Nix uses the repository (git+file:, soselfhas another store path and arev); a flake in a subdirectory (?dir=sub) had the source root as itsoutPath, not…/sub; relative inputs (path:./sub,./sub) failed;flake = falseinputs had nosourceInfo. All of these change drvPaths of packages withsrc = self.flakes::relative_inputsnow covers relative inputs, which CppNix evaluates differently depending on where the lock comes from: nodes of a lock computed in the same evaluation get only the tree's store path assourceInfoand a normalizedoutPath(…-source/sub); nodes read fromflake.lockget their parent'ssourceInfoand the path as written (…-source/./sub).
The properties added after that (regex.rs, fixed-output derivations,
outputs, relative inputs in git repositories, subdirectory flakes) found
more, also fixed in the stack: outputs = [ "out" "" "" ] is one output
(fix rejected the duplicate "" in the declared list, which derivation
turns into attributes with listToAttrs); a fixed-output derivation's one
output must be out; fetchTree "git+file:…" has a revCount; a git
flake whose lock getFlake computed has no rev (Nix's forceDirty);
a flake's inputs don't include self; git+file:/x (one slash) is a
local repository (fix hung fetching it as a remote).
The value derivation returns follows derivation.nix for declared
outputs the .drv doesn't have ([ "a b" ], [ "" "out" ]) too: they
have attributes, whose outPath fails only when it's used
(store::derivation_outputs checks the value).
Not fixed: readFile in a long fix repl session (see the table above).
With the stack, drvPaths of nixpkgs packages (hello to firefox) and of
flakes with src = self, subdirectories, and relative and non-flake inputs
are the same as CppNix's, and fix flake lock writes byte-identical lock
files.
-
Local git trees lose directories. A directory
Xis left out offetchTree/fetchGitof a local git repository (so also out of git flakes and theirnarHash) when a sibling is namedXfollowed by a non-ASCII byte:src/next tosrcü,b/next tobé. Since 2.2x (2.18 is fine; 2.31, 2.34 and main aren't); Lix is fine. The cause:CanonPath::operator<=>compareschars, which are signed on x86-64, sosrcüsorts betweensrcandsrc/x, breaking its documented invariant that children directly follow their directory.CanonPath::isAllowedrelies on it (allowed.lower_bound(path)should find a child ofsrc, findssrcü), so the allow-list accessor over the working tree (git-utils.cc) hidessrc. Comparing asunsigned charshould fix it; on aarch64charis unsigned, so it presumably doesn't happen there (fetchers::git_signed_char_names). -
Fetched git trees are read from a stale repository. CppNix doesn't copy a
fetchTreegit result to the store; it serves the store path from the repository it fetched it from. If the same tree (same NAR hash) is later fetched from another repository after the first one is gone, its contents are unreadable:nix-repl> builtins.readDir (builtins.fetchTree { type = "git"; url = "file:///tmp/a"; }).outPath { sub = "directory"; } # rm -rf /tmp/a; /tmp/b is a clone of it nix-repl> builtins.readDir "${(builtins.fetchTree { type = "git"; url = "file:///tmp/b"; }).outPath}/sub" error: path '/nix/store/7paqfyacpbg57ax0vban7cm25rgkv792-source/sub' does not exist(
fetchers::stale_git_mount.) Another evaluator copying the same tree to the store hides the bug, which made this one show up as a flaky test. -
a // bevaluatesbfirst since 2.32 (97ce7759d, "Use same naive iterative merging but withevalForUpdate"); 2.3 to 2.31 evaluateafirst. Evaluation order is unspecified, but it's visible:tryEval ((abort "a") // (throw "t"))is caught by CppNix and aborts in Lix and fix. -
fetchTree { type = "git"; }doesn't returnrevCountany more (fetchGitdoes); Lix still does. -
derivation { system = "\\"; … }(or a"insystem) writes the platform into the.drvunescaped (printUnquotedStringinderivation/aterm.cc, meant for restricted alphabets), then fails to parse its own.drv:error parsing derivation '…': expected string ','.systemshould probably be validated when the derivation is created. Lix has the same bug. -
compareVersionsparses components withstring2Int<int>, so a component ≥ 2³¹ counts as a word and sorts before every number:compareVersions "0" "2147483648" == 1. -
Subnormal float literals are rejected (
invalid float '5.0e-324'). -
A relative flake input (
path:./sub) evaluates differently depending on where its lock comes from. WhengetFlakecomputes the lock (there's noflake.lock, or it's out of date), the input is an override incall-flake.nix: itssourceInfois only{ outPath = <tree>; }and itsoutPathis normalized (…-source/sub). Read fromflake.lock, it has its parent'ssourceInfo(narHash,rev, …) and the path as written (…-source/./sub), which ends up in drv hashes withsrc = inputs.sub. -
toXMLof a path in a flake shows the path of the local checkout (/home/…/flake/data.txt), not of the flake's store path. -
In the harness only: with read-write fix adding a fresh git repository to the store at the same time, a long-running CppNix session sometimes reads an empty
flake.nixfrom it ("syntax error, unexpected end of file"). Never with CppNix alone or read-only fix, not reproduced with anix repland concurrentfix eval --read-write-modeoutside the harness.check_one_at_a_timeevaluates git trees one evaluator after the other.
On main (2.96.0-dev):
-
Segfault after a failed store copy. Once copying a path that doesn't exist to the store has failed, the next copy crashes in
RemoteStore::addCAToStore:nix-repl> "${/nonexistent}" error: … No such file or directory nix-repl> "${./file}" Segmentation fault (core dumped)Same after
fetchTree { type = "path"; path = "/nonexistent"; }orgetFlake "path:/nonexistent". A regression: 2.94.2 and 2.95.2 are fine. Anything that keeps evaluating after an error with the same store connection (the REPL,nix-eval-jobs) is affected (store::store_errors_dont_poison_the_session). -
builtins.toXMLof a cyclic value (let x = { a = x; }; in x) never finishes (2.95.2 too); CppNix and fix report a stack overflow. -
The REPL prints its prompt to stdout when
TERM=dumb, even when stdin isn't a terminal, applies backspaces in piped input ("a<BS>b"is read as"b"), and colours values despiteNO_COLOR. -
0 ++ (throw "t")checks the left operand's type before forcing the right one: an uncatchable type error, where CppNix and fix evaluate both andtryEvalcatches thethrow. -
Since 2.94, functions compare equal to themselves:
f == f,builtins.add == builtins.addandlet s.f = f; in s.f == s.faretrue. This is deliberate (cl/4556, "Function equality semantics are more consistent, but still bad"). CppNix (2.3 to main) and Lix ≤ 2.93 sayfalse, except where==on lists and sets compares elements by pointer first ([ f ] == [ f ]istrue). -
Tarballs must have exactly one top-level entry ("contains an unexpected number of top-level files"); CppNix unpacks the rest as they are.
-
fetchTree { type = "git"; }returnsrevCount; noconvertHash. -
Rejects an output named
drv(CppNix allows it). -
Fails several of the 31
flakeref.cccases (mostly newer CppNix behaviour: percent-encoding,%23in refs,revCounton paths). -
Lock files: the
originalof an input that an override replaced with a URL is the input's declared ref (2.94 to main), where CppNix writes the override:inputs.a = { url = "path:/f1"; inputs.a.url = "path:/f1"; }withf1declaringinputs.a.url = "path:/f2"gives a node withlocked/f1andoriginal/f2. -
With input overrides, Lix still follows an overridden
follows: the root overridesf1's inputawith a URL,f1declaresa.follows = "a". CppNix and fix use the override; Lix overflows the stack, innix flake locktoo. -
Two
drvPathattributes. An output nameddrvPathis accepted, andbuiltins.attrNames (builtins.derivationStrict { …; outputs = [ "drvPath" ]; })is[ "drvPath" "drvPath" ]. Nix 2.18 did the same; newer CppNix rejects the name (invalid derivation output name 'drvPath'). -
Like Nix 2.18, and unlike newer CppNix:
outputHashMode = "nar"is rejected; a relative flake input is fetched as a tree of its own, not a part of its parent's source (another store path, its ownnarHash);outputHashAlgo = "blake3"is an unknown algorithm, so it counts as none.
Fixed since 2.94.2: printing invalid UTF-8 as JSON crashed Lix (fixed in
2.95); floor/ceil of out-of-range floats returned
-9223372036854775808 (fixed on main). Not re-checked on main: the error
printer dropping everything after a raw ESC ] in a quoted source line.
with-daemon.sh DAEMON -- CMD runs CMD against a private daemon of any
version, without root: in user and mount namespaces, with a fresh
/nix/store that holds only the closures the command needs. With it:
- The whole suite gives the same results with private Lix 2.95.2, Lix main and CppNix main daemons as with the host's Lix 2.95.2 daemon. None of the findings depend on the daemon version.
- The Lix main segfault, fix's out-of-sync connection and CppNix's stale git mount happen with each of those daemons: they're client bugs. The first two look like the same bug: after a streaming store operation fails, the client keeps using the connection (CppNix clients drop it).
- In read-write mode, fix uses a pool of 8 daemon connections per process
(
default_pool_workersinstore/daemon/runtime.zig), independent of--workers;fix repl --pbt-serveropened all 8 at startup, before any request. A Lix daemon runs ~18 threads per connection (CppNix ~2), so 13 fix processes make a Lix daemon use ~1,900 threads. On the host daemon here that ran into a task limit: new connections were reset, and the daemon's workers crashed (Lix crashed. This is a bug … std::system_error: Resource temporarily unavailableinstead of an error). The same happens with a private daemon given a lowRLIMIT_NPROC. Curiously only Lix main clients failed; Lix 2.95 and CppNix clients got through. The fix server now uses one connection, which avoids all this.
- A
serverfor Lix (on its C++ API): its REPL is the least robust session type. - More ports: relative
path:./subflake inputs, lock file updates (nix flake updateof one input),fromTOML,toXML, regexes. - Angrier tarballs (hard links,
..and absolute entries, pax headers, duplicate entries) with thetarcrate; git submodules,ref/revcombinations,.gitattributesexport-ignore. - Compare error messages where implementations agree on them.