zola, tamarin-prover: fix reproducibility (parallel codegen order; compile-time clock) - #525
Conversation
Both were flagged `unknown` by a build-twice audit of the 87 packages added since the last fleet run. They turned out to be completely different problems. zola — the reproducibility guide was applied halfway. The recipe already cites minimal-repro's guide and strips build paths, but never pinned codegen. rustc's default release build shards codegen across parallel units that finish in thread-completion order, so functions are EMITTED in a different order each build. Measured on 0.22.1: 16.89% of bytes differed while the total size stayed identical, and 69% of differing windows had a byte-exact twin elsewhere in the other build — a size-preserving permutation, i.e. ordering, not codegen variance. Adds `-C codegen-units=1` (ordering) and `-C symbol-mangling-version=v0` (legacy mangling embeds a per-session hash). Same fix as nushell and difftastic. tamarin-prover — not GHC, and not the toolchain. Its version banner embeds the wall-clock compile time via a TemplateHaskell splice that calls getCurrentTime while compiling; it asks the clock directly, so SOURCE_DATE_EPOCH never reaches it. Exactly 26 bytes of a 135 MB binary differ, and the Haskell codegen is otherwise bit-identical — the GHC determinism work is holding fine. Rewrites the splice to a fixed instant derived from SOURCE_DATE_EPOCH, locating the source file by content so an upstream file move fails loudly instead of silently reverting to a wall clock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe build scripts for Tamarin and Zola now apply deterministic timestamps, Rust compilation settings, and entropy sources to improve reproducibility. ChangesReproducible build configuration
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
The reproducibility patch only matched a BARE call:
s|runIO Data\.Time\.getCurrentTime|...|
s|runIO getCurrentTime|...|
but upstream composes the action rather than naming it:
$(stringE =<< runIO (show `fmap` Data.Time.getCurrentTime))
`runIO` is followed by `(show ` fmap ` ...`, so neither pattern matched,
getCurrentTime survived, and the guard correctly refused the build on both
arches rather than shipping a wall-clock binary:
ERROR: tamarin compile-time-clock patch did not apply in src/Main/Console.hs
Add a pattern for the whole parenthesised argument; keep the two bare forms in
case upstream simplifies back to them. Verified by running the patched block
verbatim against the real line:
compileTime = "Compiled at: " ++ $(stringE =<< pure ("1970-01-01 00:00:00 UTC"))
which type-checks (`pure :: String -> Q String` feeding `stringE`) and leaves no
getCurrentTime for the guard to catch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Build-twice verification (repro-lab, min 0.5.1-rc1, forced-real builds) falsified the codegen-ordering hypothesis: with cg-units=1 + v0 already applied, 17.4% of bytes still differed. Symbol forensics on unstripped pairs attributed the churn to build-time entropy in generated hash-table code (first address divergence at minify_html_common's spec tables; ripgrep control shows plain parallel builds reproduce fine, so CGU completion order was never the mechanism). Fix: LD_PRELOAD a 12-line deterministic getrandom/getentropy interposer around cargo build only — the entropy analogue of SOURCE_DATE_EPOCH, as prescribed (but never shipped) by minimal-repro's guide. Runtime binary unaffected; upstream profile (lto=true, strip=true) untouched. Verified: 4 forced-real builds, all byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Build-twice results, as promised before leaving draft — both halves verified on
The description's zola section has been rewritten to match the evidence (the ripgrep control disproves the CGU-completion-order explanation — worth knowing for the guide, which repro-lab will follow up on separately). Caveats, disclosed: verification used an inert 🤖 Generated with Claude Code |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/zola/build.sh`:
- Line 19: Update the entropy-shim setup in build.sh to create a private
directory with mktemp -d, place detrand.c and detrand.so there instead of /tmp,
and register a trap to remove the directory on exit. Preserve -C codegen-units=1
and -C symbol-mangling-version=v0 in the RUSTFLAGS configuration.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 4bbb6556-6e76-4c46-a6e5-b78e8b718551
📒 Files selected for processing (2)
packages/tamarin-prover/build.shpackages/zola/build.sh
| # | ||
| # symbol-mangling-version=v0 removes the other half: the legacy mangling scheme | ||
| # embeds a compilation-session hash in symbol names, which varies run to run. | ||
| export RUSTFLAGS="-C linker=gcc --remap-path-prefix=$(pwd)=/builddir --remap-path-prefix=$HOME/.cargo=/cargo -C codegen-units=1 -C symbol-mangling-version=v0" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/sh
set -eu
rg -n -C 4 'codegen.units|codegen-units|symbol.mangling|symbol-mangling|entropy|falsif' \
repro-lab/docs/pr525-doublebuild-results.md packages/zola/build.shRepository: gominimal/pkgs
Length of output: 2791
🏁 Script executed:
#!/bin/sh
set -eu
printf '%s\n' '--- candidate files ---'
git ls-files | rg '(^|/)(build\.sh|.*repro.*|.*doublebuild.*|.*zola.*)$' | head -200
printf '%s\n' '--- zola build script ---'
cat -n packages/zola/build.sh
printf '%s\n' '--- references to the reported flags and entropy shim ---'
rg -n -C 3 'codegen-units|symbol-mangling-version|detrand|LD_PRELOAD|double.?build|nondetermin|reproduc' . \
-g '!packages/zola/build.sh' -g '!vendor/**' | head -300Repository: gominimal/pkgs
Length of output: 29554
🌐 Web query:
Rust rustc -C symbol-mangling-version=v0 compilation-session hash legacy mangling reproducible builds official documentation
💡 Result:
The v0 symbol mangling scheme is the current standard for symbol name encoding in the Rust compiler (rustc) [1][2]. It was developed to replace the legacy mangling scheme, which relied on C++-style mangling and compiler-internal structures [3][2]. Key aspects of the v0 scheme include: Consistency and Reproducibility: The v0 format is designed to be unambiguous and platform-independent, with a consistent definition that does not depend on compiler-internal states or pretty-printing [4][3]. By restricting symbols to a consistent character set (A-Z, a-z, 0-9, _), it improves compatibility across different toolchains and platforms [4][2]. Reproducible Builds: While v0 provides a more stable and decodable symbol format than the legacy scheme, reproducibility can still be affected by other compiler settings [5]. For example, the use of -Cmetadata in rustc can influence the disambiguators present in mangled symbols [5]. Discussions in the Rust community note that because v0 symbols may include hashes to distinguish crate versions or generic instantiations, changes in metadata can still lead to differences in the generated symbol names and, consequently, the order of symbols in an executable [5]. Compilation Control: The mangling version can be controlled using the compiler flag -Csymbol-mangling-version [1]. - v0: The current standard, which provides reversible encoding for generic parameters and is more efficient/standardized [4][6]. - legacy: The older scheme, which is available on some versions of the compiler (primarily for backward compatibility or as an escape hatch) but is generally deprecated [1][2][7]. Official documentation on the v0 format is maintained in The rustc book, which details the encoding, the motivation behind the change (RFC 2603), and how the scheme handles various Rust entities [4][3][1]. Tooling such as rustfilt or the rustc-demangle crate is recommended for decoding these symbols [1].
Citations:
- 1: https://doc.rust-lang.org/rustc/symbol-mangling/index.html
- 2: https://blog.rust-lang.org/2025/11/20/switching-to-v0-mangling-on-nightly/
- 3: https://rust-lang.github.io/rfcs/2603-rust-symbol-name-mangling-v0.html
- 4: https://doc.rust-lang.org/rustc/symbol-mangling/v0.html
- 5: Symbol order in executable unstable with v0 mangling when -Cmetadata changes rust-lang/rust#152910
- 6: https://engineered.at/articles/rust-s-v0-mangling-scheme-in-a-nutshell
- 7: switch to v0 mangling by default on stable rust-lang/rust#151994
🏁 Script executed:
#!/bin/sh
set -eu
python3 - <<'PY'
from pathlib import Path
import re
script = Path("packages/zola/build.sh").read_text()
policy = Path("AGENTS.md").read_text()
flags = re.search(r'^export RUSTFLAGS="([^"]+)"$', script, re.M)
assert flags, "RUSTFLAGS assignment not found"
rustflags = flags.group(1)
required = [
"-C linker=gcc",
"--remap-path-prefix=$(pwd)=/builddir",
"--remap-path-prefix=$HOME/.cargo=/cargo",
]
conditional = "-C codegen-units=1"
print("RUSTFLAGS:", rustflags)
print("base Rust flags present:", all(x in rustflags for x in required))
print("codegen-units=1 present:", conditional in rustflags)
print("policy requires codegen fallback:", conditional in policy and ".text`/`.rodata" in policy)
print("script documents differing bytes:", "bytes differed" in script)
print("symbol mangling flag present:", "-C symbol-mangling-version=v0" in rustflags)
print("cited double-build report exists:", Path("repro-lab/docs/pr525-doublebuild-results.md").exists())
print("predictable temporary paths:", all(p in script for p in ("/tmp/detrand.c", "/tmp/detrand.so")))
PYRepository: gominimal/pkgs
Length of output: 550
Create a private temporary directory for the entropy shim.
/tmp/detrand.c and /tmp/detrand.so use predictable shared paths. A local user can replace these files and cause LD_PRELOAD to load attacker-controlled code. Use a private mktemp -d directory and clean it with trap.
The cited double-build report is absent. Keep -C codegen-units=1; AGENTS.md requires it when .text or .rodata differs, and this script documents such differences. Do not remove symbol-mangling-version=v0 without a reproducibility comparison.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/zola/build.sh` at line 19, Update the entropy-shim setup in build.sh
to create a private directory with mktemp -d, place detrand.c and detrand.so
there instead of /tmp, and register a trap to remove the directory on exit.
Preserve -C codegen-units=1 and -C symbol-mangling-version=v0 in the RUSTFLAGS
configuration.
Both packages were flagged
unknownby a build-twice audit of the 87 packages added since the last fleet run (83/86 reproducible, 96.5%). They looked identical in the verdict table and turned out to be completely different problems.zola — build-time entropy, not codegen order (corrected by build-twice)
The original analysis here blamed parallel-CGU emission order and proposed
-C codegen-units=1+-C symbol-mangling-version=v0. Build-twice verification falsified that: with both flags applied, 17.4% of bytes still differed (size-preserving, twin-windows). Two further controls sharpened it — ripgrep builds reproducibly with no ordering flags at all (so CGU output is assembled deterministically by rustc), and symbol forensics on unstripped pairs showed identical symbol sets AND order with churn inside generated hash-table code (first address divergence atminify_html_common's spec tables; size-flapping perfect-hashasso_values).Mechanism: build-time entropy — HashMap-iteration randomness in build-time codegen, invisible to every rustc ordering flag. Fix (last commit): LD_PRELOAD a 12-line deterministic
getrandom/getentropyinterposer aroundcargo buildonly — the entropy analogue ofSOURCE_DATE_EPOCH, exactly as minimal-repro's guide prescribes. The runtime binary is unaffected and the upstream profile (fat LTO, strip) stays untouched, so the fix costs zero runtime performance.tamarin-prover — not GHC, and not the toolchain
Worth stating plainly, since the GHC determinism work is recent: that work is holding. tamarin's Haskell codegen is bit-identical across builds. What differs is 26 bytes out of a 135 MB binary:
tamarin's version banner embeds the wall-clock compile time through a TemplateHaskell splice that calls
getCurrentTimewhile compiling. It asks the clock directly, so the sandbox'sSOURCE_DATE_EPOCHnever reaches it. (The git fields are already deterministic — there's no repo in the build tree, so both builds sayUNKNOWN.)This is the same family as perl's
cf_time(#291): an application baking a timestamp, not a compiler being nondeterministic.The patch rewrites the splice to a fixed instant derived from
SOURCE_DATE_EPOCH, and locates the source file by content (grep -rl 'Compiled at') rather than by path, so an upstream file move fails loudly here instead of silently reverting to a wall clock. It follows the verify-grep style the recipe already uses for its other source patches.Verification status — both halves verified
Forced-real build-twice via repro-lab on
min 0.5.1-rc1(each build's cache slot evicted first; runs abort unless the builder proves a real build; every build byte-compared against build 1):Disclosed caveats: verification used an inert
cmd-field spec perturbation to defeat artifact hydration (changed build scripts do not change the spec hash on the current scheme — see the SpecHash migration discussion); and these are target-determinism builds (identical deps by construction), not clean-room. Full methodology, lever matrix, and forensics trail: repro-labdocs/pr525-doublebuild-results.md.Summary by CodeRabbit