bench(install): record tool versions in results; fix --cwd; re-measure bun 1.3.14

[colinhacks]

What

Three changes to the install-benchmark harness (tests/bench/install/):

1. Provenance — new provenance.sh, sourced by run.sh, run-warm-gvs.sh, and cold-cas.sh. Every results JSON now carries a top-level env object: platform, arch, timestamp_utc, load_1min, and a per-tool versions map. Added as a new key, keyed by tool so a new tool needs no schema change; the per-tool timing blocks are unchanged.
2. --cwd placement — nub's install no longer accepts --cwd after the subcommand (it is top-level: nub --cwd <dir> install …; the subcommand form is -C/--dir). run-warm-gvs.sh and run.sh's cold leg used the old form, so their nub leg exited non-zero and hyperfine aborted the run. Fixed at every nub call site; run.sh's warm leg already used the correct order, and the bun … --cwd uses are bun's own flag and left alone.
3. Re-measured warm results with bun 1.3.14 for simple, t3, tanstack-start, and the gvs-eligible/ineligible pair, 12 runs each.

Why

A committed result recorded no tool versions, so a benchmark could not be audited for what it measured. On the bench host, bare bun resolved to 0.5.7 (a 2023 npm-global install), which ignores the harness's --cwd and installed the repo's own dependencies instead of the fixture — a fixture-independent ~150–360 ms exit-0 "install" that never touched the fixture (reproduced across four fixtures). That stale install has since been removed from the host, so bare bun is now 1.3.14 and installs the fixture as intended. The pre-provenance results still cannot be attributed to a bun version, which is exactly what the env block prevents from recurring.

Warm bun means — committed (no provenance) vs re-measured (bun 1.3.14)

fixture	committed (unverifiable)	bun 1.3.14
tanstack-start	~660 ms	834 ms
simple	~1094 ms	1617 ms
t3	~1365 ms	1656 ms
gvs-eligible	2227 ms	2226 ms
gvs-ineligible	2952 ms	3060 ms

The committed numbers cannot be trusted — no version recorded, and some (e.g. monorepo at 266 ms) are indistinguishable from today's broken 0.5.7. The re-measured numbers are auditable.

Caveats

• The host is shared and contended; these files were taken at load 38–93 (env.load_1min per file), so absolute wall-clock still carries some inflation and is not a substitute for the quiet-runner CI numbers. The recorded load makes the conditions explicit.
• nub was built with LTO disabled (the host could not finish the LTO link under memory pressure). Install is I/O-bound, so this is within noise for these numbers.
• Pre-provenance mac-local results (cold-*, monorepo, large) are also unverifiable; they are left in place and can be pruned or re-measured on a quiet host.

Refs #500

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nub	Ready	Preview, Comment	Jul 24, 2026 4:35am

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[pullfrog]

✅ No new issues found.

Reviewed changes — a benchmark-harness-only PR (tests/bench/install/, no runtime code) that adds provenance stamping, fixes --cwd placement, and re-measures warm results with bun 1.3.14.

• Add provenance.sh with bench_env_json / bench_inject_env — dependency-free helpers that emit a compact env object (platform/arch/timestamp/load_1min/per-tool versions) and prepend it as the first key of hyperfine's JSON by rewriting the leading { via tail -c +2. Empty and (not installed) tools are dropped; \ and " are JSON-escaped. Injection verified to produce valid JSON.
• Fix --cwd placement across the nub install legs — --cwd is a top-level global consumed pre-verb (cli.rs:1514); the subcommand equivalent is -C/--dir, so the old nub install --cwd <dir> form exited non-zero and aborted the hyperfine run. Corrected at every nub call site in run-warm-gvs.sh and run.sh's cold leg. The bun ... --cwd post-subcommand uses are bun's own flag and are correctly left alone.
• Re-measure warm result JSONs with bun 1.3.14 for simple, t3, tanstack-start, and the gvs-eligible/ineligible pair, each now carrying the env provenance block.

I verified the --cwd-is-top-level premise directly against cli.rs and confirmed the byte-rewrite injection composes into valid JSON against the committed result files. load_1min parsing handles both the macOS (load averages:) and Linux (load average:) uptime formats.

ℹ️ Committed result JSONs were measured on a heavily loaded host

The re-measured warm result files were taken on a host at load ~131, and some contain only 6 timed samples rather than the harness's RUNS=12 (the run was cut short). The PR body already discloses the load and the LTO-disabled build and frames these as auditable-but-not-comparable to the quiet-runner CI numbers, so this is a scope note rather than a defect — worth a follow-up re-measure on a quiet host if these files are ever intended to be cited as headline numbers.

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}