Repository navigation
Benchmark not fair #40
Description
Activity
Fair critique @BjarkeHautop. The benchmark was measuring "first install (resolution + install) for everyone except uvr, which was install-only because uvr separates
lockandsync." That's a real asymmetry.On
main:- Added a
pak_lockfilevariant tobenchmarks/bench.sh— pre-generates a pak lockfile withpak::lockfile_create()outside the timer, then timespak::lockfile_install()only. That's apples-to-apples withuvr sync. - Updated the marketing page to clarify that the published numbers are the first-install scenario (resolution + install), and to point at
pak::lockfile_install()/renv::restore()for like-for-like lockfile-restore numbers. - Will rerun and publish the lockfile-install numbers alongside the first-install ones.
A GH Actions workflow for reproducible benchmarks is a great ask — filing that as a follow-up. It's a chunk of work (consistent runner spec matters for wall-time comparisons), but worth doing.
Commit: will link after push
- Added a
- added a commit that references this issue
on Apr 24, 2026 A GH Actions workflow for reproducible benchmarks is a great ask — filing that as a follow-up. It's a chunk of work (consistent runner spec matters for wall-time comparisons), but worth doing.
Mainly just because I think stuff is still leaking/cache used. E.g. the renv seems unreasonable fast compared to pak/install.packages(). A docker file/GA workflow makes this easier to debug (for me at least).
@BjarkeHautop — landed on `main` (commits 33f7ff5 + 8b25ba8). The reproducible bench you asked for is now live.
What you can do today:
```sh
Local — match CI exactly (linux/amd64 even on Apple Silicon):
bash benchmarks/run-in-docker.sh
Customize:
R_VERSION=4.6.0 PPM_SNAPSHOT=2025-05-01 BENCH_RUNS=3
bash benchmarks/run-in-docker.sh
```The image pins R version (rocker/r-ver tag), debian:bookworm base, Rust toolchain (1.83.0 exact patch), and the CRAN mirror (PPM dated snapshot — versions of pak / renv / every package the bench installs are locked to a known date). Same image runs on every tag push via `.github/workflows/benchmark.yml`; workflow uploads `bench-results.json` and the raw run log as artifacts on every run, and the workflow now exits non-zero if no JSON was produced (so a green badge means the numbers actually exist).
Specifically addressing your "renv looks unreasonably fast" point: the prior numbers were measured on my local machine with whatever caches were lying around. With the Docker bench, every run starts from a clean container — caches are exactly what `bench.sh`'s warm/cold tier semantics specify, no host residue. If the renv numbers stay close to the published table after re-running in this controlled environment, the original measurement was correct; if they don't, we've located the cache contamination and can fix it.
I haven't re-run the published table in the Docker bench yet — the workflow runs on the next tag push, OR you can dispatch it manually from https://github.com/nbafrank/uvr/actions/workflows/benchmark.yml right now. Either way, the artifact is the source of truth going forward.
A note on what's NOT auto-committed: per #69's cadence rule, the workflow uploads results as an artifact rather than auto-committing them back to the repo. The maintainer pulls the artifact and updates the README table deliberately. Less churn, same evidence trail.
The bench's own internals also got a fix: `uvr run` previously isolated the R library so aggressively that pak/renv installed in rocker's system library were invisible to the bench's tool detection — meaning the Docker bench would have silently dropped pak and renv from the comparison. New `UVR_EXTRA_LIBS` env var fixes that without un-isolating in general. Caught by post-implementation code review.
Reach out if anything in the artifact format or the workflow inputs would be more useful for your debugging.
Closing for now — reopen if it's causing issues.
- added a commit that references this issue
on Aug 4, 2026
uvr uses a precomputed lockfile to install, so pak should probably do the same? See
pak::lockfile_create()andpak::lockfile_install().The renv timing also seems wrong, I think?
Can you make a GitHub Action workflow that runs the benchmark? Then it would be much easier to reproduce and debug.