rc/0.4.4 → main: 0.4.4 reproduce-trustworthiness line (+ #262 pulled in) - #271
Merged
Conversation
…he whole tree `pipeline_executor` hard-coded `subprocess.run(..., shell=True, timeout=3600)`. Two defects: - **Not configurable** — a 1h wall-clock cap makes a row reproducible only on hardware at least as fast as the machine that made it, with no override. - **Orphans the workload** — with `shell=True`, a timeout kills only the shell, so the grandchild (e.g. `train.py`) keeps running past the declared failure — a false failure plus a silent GPU-cost leak on someone else's bill. Now: `--step-timeout <seconds>` / `ROAR_REPRODUCE_STEP_TIMEOUT`, **default none (no timeout)**. The step runs in its own session (`start_new_session=True`); on timeout the whole process group is SIGKILLed and reaped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…an't install `install_pip_packages` demoted every un-installable pin to a *warning* and returned True; `setup()` then ignored that boolean and returned a healthy EnvironmentInfo. Result: "Pip package installation complete" → "Environment ready" → a dead run (ModuleNotFoundError). The AMI re-bake only removed the torch-family instance; the class (yanked version, private package, extra-index pin) still produced a green banner + dead run. - installers: return False when a recorded pin is unresolved (fallback failed, or skipped because --pip-any-version wasn't given). - environment_setup.setup(): raise RuntimeError on a failed install instead of ignoring `success` — the reproduce service already reports RuntimeError as "Environment setup failed" (not "Environment ready"). Bypass + debug (the two points raised): - **Bypass** the pin check with the existing `--pip-any-version` (installs available versions; recorded as warnings). Default is now honest-fail. - **Debug/export** with new `--export-requirements <path>`: writes the recorded pip pins to a pip-native requirements.txt (no uv assumption) so you can `pip install --dry-run -r <file>` to see exactly which pins don't resolve. Complements `--script` (which emits the shell, not the packages). Header notes the index-url/extra-index-url isn't replayed yet (that's the deeper [93]-class capture gap — a follow-up). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ate declines-fallback contract Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…process group Also use contextlib.suppress for the post-kill reap (ruff). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…the grandchild The prior version's grandchild ran via a multi-line `python -c` payload whose single-quoted literal spanned physical lines — a SyntaxError, so the grandchild never started and the assertion was silently inconclusive. It also blocked on a fixed 3s sleep, adding dead wall-clock to the slow macOS lane. Rewrite: grandchild is a real script that heartbeats a counter file every 50ms. After the step times out we assert the counter is frozen (killed) rather than waiting out a sleep, and a `heartbeat.exists()` guard proves the grandchild actually started. ~3.0s -> ~1.5s, stable, and now a real test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The macOS job ran the entire default suite (~1180 tests) — the same OS-independent pure-Python logic the Linux `test` job already covers across 5 Python versions. On slow macOS runners that adds wall-clock and intermittent job timeouts without adding signal: the only thing macOS uniquely exercises is the OS-specific surface (the DYLD preload tracer, sitecustomize/runtime injection, the native `_hash_native` extension, and the real `roar run` product path). Scope the macOS job to those platform-dependent trees, selected by PATH rather than by marker: tests/execution/runtime tests/integration tests/happy_path tests/application/run tests/backends/local/integration plus two hashing-value files (test_hashing_backend, test_canonical_session_hash) so a macOS-specific _hash_native ABI/endianness regression still can't slip through. Path selection is deliberate over a `macos` marker: with --strict-markers and 100+ platform-relevant files, per-file tagging is high-churn and easy to under-apply — which is exactly how a macOS-only tracer test would be silently dropped. Directory selection fails safe: the Linux-only tracer regressions that live inside these dirs already carry skipif(platform != "Linux") and simply skip on macOS. Collected macOS tests drop from ~2355 to 286 (~88% fewer) with no reduction in platform coverage. The macOS-only preload smoke step and the native-hash import verify step are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI runs `ruff format --check .` in addition to `ruff check`; the timeout message fits on one line under the limit. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The batch-install-failed recovery path re-installs the individually-resolvable pins together but discarded that install's return code. Because each pin was probed with a per-package `pip install --dry-run` (resolvable ALONE) rather than jointly, a set that resolves individually but conflicts in combination produced an empty `failed_packages`, so the honest-fail guard never fired: pip left an incomplete venv (MiniMind-O: 35 of 105 packages) and reproduce still printed "Environment ready". Capture the combined install's return code; on non-zero, treat those pins as unresolved (they are not jointly installable) and fail honestly with a message that names the conflict. `unresolved_packages` is now accumulated with extend() so the failed-pin branches don't clobber a combined-conflict result. Scope note: this closes the observed false-green but does NOT make env-setup bulletproof — the robust fix is to verify the rebuilt venv against the recorded package set (post-install `pip freeze`/`pip check` diff) rather than trusting install return codes at all. That verification redesign is tracked as a separate roar-core P0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(reproduce): configurable step timeout (default none) + kill the whole tree
Without uv, `roar reproduce` can only build the venv with the interpreter roar
itself runs under. When that differs from the recorded interpreter at the
major.minor level it used to build the wrong-version venv silently, warn once,
and continue — then the recorded (e.g. cp312) wheels fail to install and the
error blamed the package list.
Now, on a major.minor mismatch we:
- warn loudly (recorded vs building), explaining the ABI-tag risk,
- recommend uv and link its install docs (uv provisions the EXACT recorded
interpreter — the deterministic fix; no PATH-guessing among multiple
pythons),
- ask "continue anyway?" (default no); `-y/--yes` overrides to continue.
Declining aborts the reproduction rather than silently using the wrong Python.
Patch-level differences (3.12.9 vs 3.12.10) still pass silently — not
reproducibility-relevant. Many pure-Python repos reproduce fine on a different
minor, which is why this warns-and-asks rather than hard-failing.
`auto_confirm` is threaded setup -> setup_in_place -> _create_venv ->
_create_venv_uv; the CLI already maps `-y/--yes` to it.
Tests: tests/integration/test_reproduce_python_mismatch.py builds REAL venvs
(no subprocess mock) to exercise the whole non-uv path end-to-end — declined
aborts (+ uv link shown), --yes overrides and builds, confirmed continues,
matching-minor stays silent. Existing env-setup tests updated to the new
prompt/abort contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every process in a traced tree inherits the same ROAR_LOG_FILE, and write_log() opened it "w" — so each process truncated the others and the surviving record was whichever wrote LAST. With multiprocessing (litdata/DataLoader workers, HF datasets num_proc, torchrun ranks) that was often a worker with a subset of the imports — or none. Same command, different record: 45 packages (parent) or 10 (a litdata worker), decided only by scheduling. This is distinct from #264 (which changes how packages are derived FROM a record, not WHICH record survives), so it needs its own fix. write_log now writes a per-PID shard (`{ROAR_LOG_FILE}.<pid>`), and the tracer unions the shards into the canonical inject log after the run (merge_inject_logs): set/dict activity (opened_files, imported_modules, modules_files, used_packages, installed_packages, ...) is unioned across the tree; scalar identity (argv, python_version, ...) is taken from the richest shard — the workload imports everything, a `python -c` worker a subset. The collector reads the merged file unchanged; cleanup sweeps stray shards. Tests (tests/execution/runtime/test_inject_log_merge.py): write_log writes a shard not the shared file; a sparse `['-c']` worker shard does not clobber the workload's argv or packages (the exact litdata parent/worker shape); concrete versions beat None on union; no-op without shards. Existing tracker tests updated to merge the shard before asserting. Note: full multi-process `roar run` confirmation requires the built tracer binaries (not present in an editable checkout); the merge is covered here with faithful shards and confirmed end-to-end by the row-009 re-certification. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ow-up) test_writer_reader_roundtrip_carries_python_identity calls write_log then reads the canonical inject-log path — but write_log now writes a per-PID shard, so the canonical file was empty and python_version came back ''. Merge the shard first, matching the fix already applied to test_runtime_tracker. Caught by CI across all Python versions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-on-missing-pin # Conflicts: # roar/application/reproduce/requests.py # roar/cli/commands/reproduce.py
reproduce: fail honestly on un-installable pins + pin-debug tooling
ci(macos): run only the tracer / platform-dependent test subset
…0-14) roar makes itself importable in a traced child by putting entries on ROAR_RUNTIME_PYTHONPATH, which sitecustomize applied by prepending ALL of them. On a cross-interpreter run (roar under system 3.10, workload venv 3.12) that put roar's host dist-packages at sys.path[0], shadowing the recorded pins — the run executed against host packages. It crashed loudly only because the host's pyparsing was 5 years stale; a merely-different host package would import fine and certify GREEN for the wrong reason (and the venv-vs-manifest guard can't see it — the venv is right; the child's sys.path is wrong). Fix: split ROAR_RUNTIME_PYTHONPATH by precedence. - roar's ABI-matched runtime CACHE (~/.cache/roar/runtime/<tag>/…) stays PREPENDED — it must beat the child's wrong-ABI/stale system copies (the original typing_extensions-4.15-vs-system-4.4 fix). - everything else (roar's host site-packages) is now APPENDED, so the workload's own venv always wins. roar's core injection is pure-Python, so it's still importable via the appended path; ABI-specific backend deps are handled separately by the existing runtime gate. Cache-root detection is inlined (matching lazy_install.runtime_cache_root) because this runs before roar is importable. Tests: cache entry prepended (must-win preserved); host entry appended, not at front (P0-14); early-return unchanged. Plus an integration test running a roar-less child that confirms a workload package beats a host one (non-vacuous: fails without the fix) and the cache still beats the workload. Harness note: installing roar under the recorded interpreter (`uv tool install --python <ver>`) sidesteps this by making roar's interpreter == the child's; this fix removes the silent-false-pass regardless. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
reproduce: warn + confirm on a Python major.minor mismatch (P0-4)
tracker: per-PID inject-log shards + union merge (P0-9)
…0-14) roar makes itself importable in a traced child by putting entries on ROAR_RUNTIME_PYTHONPATH, which sitecustomize applied by prepending ALL of them. On a cross-interpreter run (roar under system 3.10, workload venv 3.12) that put roar's host dist-packages at sys.path[0], shadowing the recorded pins — the run executed against host packages. It crashed loudly only because the host's pyparsing was 5 years stale; a merely-different host package would import fine and certify GREEN for the wrong reason (and the venv-vs-manifest guard can't see it — the venv is right; the child's sys.path is wrong). Fix: split ROAR_RUNTIME_PYTHONPATH by precedence. - roar's ABI-matched runtime CACHE (~/.cache/roar/runtime/<tag>/…) stays PREPENDED — it must beat the child's wrong-ABI/stale system copies (the original typing_extensions-4.15-vs-system-4.4 fix). - everything else (roar's host site-packages) is now APPENDED, so the workload's own venv always wins. roar's core injection is pure-Python, so it's still importable via the appended path; ABI-specific backend deps are handled separately by the existing runtime gate. Cache-root detection is inlined (matching lazy_install.runtime_cache_root) because this runs before roar is importable. Tests: cache entry prepended (must-win preserved); host entry appended, not at front (P0-14); early-return unchanged. Plus an integration test running a roar-less child that confirms a workload package beats a host one (non-vacuous: fails without the fix) and the cache still beats the workload. Harness note: installing roar under the recorded interpreter (`uv tool install --python <ver>`) sidesteps this by making roar's interpreter == the child's; this fix removes the silent-false-pass regardless. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`get_used_packages` attributed a package to a job only via each loaded module's
`__file__`. An aliasing logging shim (`sys.modules["wandb"] = trackio`) makes
`import wandb` resolve to trackio's file, so wandb was never recorded in the
job's package freeze — yet the job genuinely needs it (diffusers/accelerate gate
on `importlib.metadata.version("wandb")`, and its install is required). On
reproduce, the wandb-less freeze was reinstalled and the workload died at
`import wandb`. The record lied by omission.
Fix: also attribute packages the workload imported by NAME. `tracking_import`
wraps `builtins.__import__`, so `import wandb` is captured even when
`sys.modules["wandb"]` is pre-populated (Python still calls
`__import__("wandb", ...)`). write_log now passes `imported_modules` into
`get_used_packages`, which unions in each imported top-level name's distribution.
No false positives — a name is added only if the workload actually imported it
AND it maps to an INSTALLED distribution (there is no unknown-name fallback on
this path), and the tracer's own package (`roar`) is never attributed. A package
that was never imported can never appear.
Tests (tests/execution/runtime/test_used_packages_by_name.py): aliased import is
attributed by name; a never-imported package is never added; an imported-but-not-
installed name and roar itself are both skipped; and an end-to-end pass through
the real `tracking_import` -> `write_log` -> log `used_packages`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
So the end-to-end test stays correct once P0-9's per-PID sharding lands (write_log
then writes {log}.<pid> rather than the canonical path). No behavior change to the
fix itself.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… of the freeze Two ways a package that isn't a real third-party dependency was landing in the recorded freeze: P0-11 — roar records ITSELF. The file pass mapped site-packages/roar -> roar-cli. Harmless noise on a PyPI release (roar-cli==0.4.3 resolves) but fatal on an unpublished build: the freeze pins roar-cli==0.4.4.dev0, which can't resolve, so reproduce can never rebuild a row captured on a dev build. _install_roar installs roar-cli separately and unpinned, so the pin is always redundant. The name pass already skipped `roar`; the file pass now does too. P0-12 — a #264 regression. `pip install -e .` (and the <pkg>.egg-info a later `pip uninstall` leaves behind, which importlib.metadata still reports installed) made the name pass re-pin the workload's OWN package from PyPI. The name pass now skips any dist whose metadata resolves inside the workload repo (workload_root, threaded from write_log as os.getcwd()); a genuine site-packages dep outside the repo is still recorded. Tests: roar-cli never enters the freeze via the file pass; an editable/egg-info self-package is skipped; a real out-of-repo dep is still pinned. All non-vacuous. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sitecustomize: don't let roar's env shadow the workload's sys.path (P0-14)
tracker: attribute imported packages by name; keep roar & self-package out of the freeze (P0-6/P0-11/P0-12)
#264's name pass attributed every imported name that mapped to an installed distribution. On the HF stack that over-attributes: accelerate *probes* optional integrations (`import sagemaker`), and on a SageMaker AMI those happen to be installed, so 13 substrate packages (sagemaker-core/-train/…, transformer_engine, …) landed in the freeze — mutually unsatisfiable, so row 008's env setup died. The name pass's only legitimate job is recovering an import the file pass mis-attributed because it was ALIASED (`sys.modules["wandb"] = trackio`). Scope it to exactly that: attribute a name only when it was imported AND the module actually loaded for it lives in a *different* site-packages package than the name (detected via a new loaded_files map: name -> loaded module __file__, built from sys.modules in write_log). This keeps wandb (loaded as trackio) and drops: - normally-loaded imports (name == loaded package) -> the file pass's job; - merely-probed optional imports (loaded as themselves, or not loaded) -> P0-13; - the workload's own editable package (loaded from the repo, not site-packages). Tests: aliased import attributed; probed import NOT attributed (loaded-as-self and not-loaded, non-vacuous vs the old pass); never-imported not added; not- installed / roar / self-package excluded; end-to-end via write_log. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tracker: scope the name pass to aliased imports only (P0-13)
…safe-runtime-paths # Conflicts: # roar/execution/runtime/inject/sitecustomize.py # roar/execution/runtime/inject/tracker.py # tests/execution/runtime/test_runtime_tracker.py # tests/execution/runtime/test_sitecustomize_path_order.py # tests/integration/test_no_crossenv_syspath_shadow.py
fix(runtime): preserve workload package precedence
christophergeyer
temporarily deployed
to
testpypi
August 7, 2026 19:34 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Two more caller-assumption gaps the 007 (lerobot) capture hit, both in the
existing init()/log() wrappers:
- init(): wandb's `resume` default is None ("do not resume"); trackio accepts
only must/allow/never and raises ValueError on None. lerobot always passes the
kwarg (`resume="must" if cfg.resume else None`), so it died in init(). Drop a
None resume before trackio sees it.
- log(): wandb's first parameter is named `data`; trackio names it `metrics`, so
`wandb.log(data=..., step=...)` is a TypeError. Forward `data=` positionally.
With #279's get_url(), these were the three blockers between lerobot and the
trackio shim. Verified against trackio 0.34.0 and installed rc4 (Carl), and here
with regression tests asserting the metrics are actually forwarded — not merely
that no exception was raised (the HTTP-200 gate is worthless: a Gradio Space
returns 200 for any project, even a nonexistent one).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
trackio shim: provide run.get_url() so wandb callers don't crash
#278 merged with a ruff-format violation (ruff check passed but ruff format --check did not), failing the CI lint job for every branch off rc/0.4.4. Pure formatting; no behavior change. Verified: location + used_packages tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
trackio shim: adapt lerobot's wandb calls (resume=None, log(data=))
… reopened) glaas keys job inputs/outputs on (job_id, artifact_hash) and stages with skipDuplicates, so byte-identical outputs written to several names (timm os.link last/best/checkpoint = one inode, three names) collapse to ONE stored edge. roar asserted the raw per-path count at finalize, so those rows 400'd with "Staged lineage counts did not match" after a full paid GPU run — and roar's canonical session hash (per-path) would diverge from glaas's (per-content) for the same reason. Align roar to glaas's content-addressed model, no schema migration: - build_staged_lineage_counts counts DISTINCT content hashes per job. - build_canonical_session_payload dedups edges by hash (keeping the smallest path deterministically) so roar's session hash matches glaas's. This function is used by BOTH publish and reproduce lookup, so both stay consistent with each other and now with glaas. Both are a no-op for every currently-passing row (no duplicate-content edges -> distinct == path), so existing session hashes are unchanged — verified: the canonical-hash unit tests pass untouched. Also fixes the integration fake (fake_glaas) to model glaas's (job_id, artifact_hash) dedup in its staged-count and canonical-hash paths, so it no longer asserts a per-path count the real server never has. This is why my first attempt (#281) broke CI: the fake, not glaas, was the per-path contract. Why Option B over a glaas PK migration: glaas has never recorded duplicate paths (the PK has always dropped them), and reproduction/AI-BOM key on content + DAG structure, not filesystem-name multiplicity. So this loses nothing vs today and avoids a production PK migration + reproduce rework. Scope note (draft): the finalize-count fix is fully safe and is the P0-22 unblock. End-to-end hash agreement on a real duplicate-byte row also depends on glaas's skipDuplicates keeping the same representative path roar picks (smallest) — i.e. staging in sorted order — to be confirmed on a real dup-byte capture (nerf/012). compute_canonical_jobs_session_hash is intentionally left unchanged (no callers). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
chore(lint): ruff format the #278 files (unbreak CI lint)
The root process's argv was read back from /proc by every tracer. /proc reports what the kernel ran rather than what was asked for, so the two differ in ways that matter: roar run ./train.sh #!/usr/bin/env python3 recorded: ['/usr/bin/env', 'python3', './train.sh'] That value is not incidental -- runtime_collector feeds the root process's argv to RuntimeInfo.command, so it is the run's recorded command in provenance. Recording the kernel's rewrite instead of the user's command is wrong for a tool whose job is to say what ran. It is also nondeterministic. A process that exits before the read is a zombie, whose /proc entry survives while its memory is torn down, so cmdline reads back empty. The same run then records two different commands depending on machine load. roar launches the workload, so the root's argv is known exactly and needs no discovery. Prefer it in all three tracers. Descendants have no such source and keep using /proc, which is also what build_pip_collector needs. The eBPF path had no way to know it: Register now carries root_command. The field is #[serde(default)] and the wire format is field-named MessagePack, so a long-lived roard and a client of a different version still interoperate -- an older daemon ignores the extra key, and an older client's message decodes as empty, which every caller reads as "fall back to /proc". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
P0-22 (Option B): dedup job edges by content hash to match glaas (W2 reopened)
On Apple Silicon the system launchers are arm64e platform binaries, and dyld refuses to insert the arm64 preload dylib into them: incompatible architecture (have 'arm64', need 'arm64e') so the process aborts with 134 rather than running untraced. Both macOS jobs failed on the shebang case for that reason, via /usr/bin/env. Name the interpreter directly in the shebang instead. That keeps the case covered on macOS, and still exercises the kernel's cmdline rewrite on Linux -- verified it fails against a tracer built without the fix. The wrapper and short-lived cases need `env` and /usr/bin/true, which are protected with no substitute, so they carry the skip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ci: pin bpf-linker 0.10.4
fix: record the command the user ran, not the one /proc reports
TrevorBasinger
force-pushed
the
rc/0.4.4
branch
from
August 19, 2026 14:55
8fe3308 to
0c5de2f
Compare
Review found the finalizer covered only half the teardown paths.
`Pool.__exit__` is `terminate()`, which SIGTERMs every worker, and
`util._exit_function` does the same to surviving daemon children --
which is what DataLoader creates. SIGTERM's default disposition kills
the process outright, so neither the Finalize callback nor atexit runs.
Measured under real injection, 4 workers:
with ctx.Pool(4) as p: p.map(...) 1 shard (0 of 4 workers)
p.close(); p.join() 5 shards (4 of 4)
So the common idiom -- and the `num_proc` case the finalizer's own
docstring cites -- reported nothing at all. It now reports 4 of 4, and a
daemon child is captured too.
The handler is installed only in a fork child, and only where nothing
else owns the signal, so a workload's own SIGTERM handling is never
displaced. It restores the default disposition and re-raises, so the
process still dies of SIGTERM and still reports exit status -15. One of
our own handlers may be superseded, so a re-install or a second tracker
does not leave a stale one writing the wrong shard.
`signal` is imported in the parent so the child's import is a
sys.modules hit; importing for the first time inside a fork child can
deadlock on the import lock.
Two tests, each verified to fail against the code without its fix:
- the wiring test goes through `install()` rather than the private
method. Deleting the single line that wires this into `install()`
previously left all 125 tests passing -- and that line sits in the
merge-conflict region with #287, so a conflict resolution could drop
it silently.
- the Pool test pins the terminated case.
A third asserts a workload-owned SIGTERM handler is not displaced.
The documented boundary now matches the code: termination by signal is
covered; `os._exit` and SIGKILL are not, and cannot be without
incremental import journaling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the SIGTERM handler from the previous commit, which was worse
than the bug it fixed.
A Python signal handler only runs when the interpreter reaches a bytecode
boundary. A worker inside a long C call -- BLAS, zlib, pickle -- latches
the signal and never dies, and both `Pool._terminate_pool` and
`util._exit_function` join workers with no timeout. So the handler could
hang the workload indefinitely:
worker in zlib.compress, on terminate()
without roar exit -15, joined in 0.02s
with the handler still alive after 8s, needed SIGKILL
A missing shard is a thin record. A hung pipeline is a stopped campaign.
Instead the fork child writes its shard eagerly, right after forking, and
keeps the existing Finalize to rewrite the same per-PID path on an orderly
exit. A worker that is killed -- SIGTERM, SIGKILL -- or that calls
os._exit still contributes the state it inherited at fork, rather than
nothing. An orderly worker upgrades that with whatever it imported while
running. merge_inject_logs unions both, so the upgrade is free.
This also covers strictly more than the handler did: SIGKILL and os._exit
were previously uncoverable, and both now leave a valid shard.
The boundary, stated honestly and pinned by a test: imports a worker makes
*after* forking are lost if it is killed before exiting. Closing that
needs incremental journaling, not an exit hook. Coverage for a worker
killed while still bootstrapping is best-effort, so the pool test asserts
against workers that actually ran a task rather than an exact shard count.
Verified: hang gone; `with Pool(...)`, close()/join(), bare Process and
daemon children all report; 127 runtime tests pass; removing the install()
wiring, the eager write, or the Finalize each fails a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/bin/true exists on most Linux distributions but not on macOS, where the binary is only at /usr/bin/true. `roar run /bin/true` therefore exited 127 on both macOS runners and read as a roar failure rather than a missing test fixture. Resolve it via PATH so the test exercises what it means to -- a non-Python command -- on either platform. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/usr/bin/true is SIP-protected, so roar's preflight correctly refuses to preload-inject into it and exits 1: Tracer preflight failed for 'preload': macOS protected binary blocks preload injection That is the condition _MACOS_PROTECTED_BINARY already documents for the other launchers in this file, so mark this case the same way. The previous /bin/true exited 127 before ever reaching preflight, which masked it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix: fail closed on incomplete runtime dependency capture
Installing roar into the workload's own environment has two costs, and the hint names both rather than dwelling on either. Both sets of requirements must resolve together, so roar's pins can collide with the project's. And roar's dependencies are loaded into the traced process and recorded alongside the project's -- measured on an `import requests` workload, the freeze carries nine packages belonging to roar. They cannot be separated afterwards: roar's copy of a package and the workload's are the same file at the same path, so nothing -- path, name, or dist metadata -- can attribute them. Subtracting by name once stripped the workload's own tqdm and typing-extensions (P0-28), which is why `roar_footprint_paths` abstains and the freeze over-includes instead. #287 tried to fix the second cost by snapshotting sys.modules at the end of bootstrap and subtracting it. That is measurably inert -- roar's dependencies load lazily *after* the boundary -- while risking a real false negative, so it was closed in favour of saying this plainly. The comparison is against the interpreter the WORKLOAD would use, not roar's own. That distinction is the whole check: under `uv tool` or pipx, roar runs from its own venv and so always sits inside its own sys.prefix, so comparing roar against itself reports every correctly isolated install as shared -- nagging exactly the users who took the advice. Resolution mirrors a shell's: active virtualenv, else conda env, else the first python on PATH; unresolvable means stay quiet. Verified live in both layouts rather than by assumption: roar copied into a project venv warns; roar in its own venv with a project venv active stays silent; and with no venv active, a tool install still stays silent because the workload would run the system python. Printed once at `roar init`, not per run: it is a property of how roar was installed, and a per-run warning is noise people learn to skip. It rides the existing hint machinery, so `roar config set hints.enabled false` already silences it. Both uv and pipx are offered, with install routes, since not everyone has uv. Seven tests over the real layouts (pip-into-venv, tool-install with and without an active venv, system install, conda), each verified to fail against the naive roar-vs-its-own-prefix comparison. Detection cannot fail the command; a cosmetic hint is never worth breaking `roar init` for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat: say so at init when roar shares the workload's environment
fix: flush Python inject shards from fork workers
fix(release): preserve the Linux wheel baseline
feat(auth): support delegated GLaaS publishing
christophergeyer
marked this pull request as ready for review
August 19, 2026 19:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rc/0.4.4 → main (0.4.4 release integration)
Brings the full 0.4.4 reproduce-trustworthiness line to
main, and pulls in #262 which was merged directly tomain(so this PR is conflict-free andmainends up with everything).Version:
pyproject.tomlis0.4.4on this branch (bumped in #290);mainis0.4.3. There is exactly one version source —roar/__init__.pyderives__version__from installed dist metadata — so no second place to update.Reproduce / provenance
--step-timeout(default none) + SIGKILL the whole process group on timeout--export-requirements)-yoverrides)sys.pathdist-packagesimports, not justsite-packageswith Pool(...)terminates its workers)/procreports; a#!/usr/bin/env python3script now records as./train.shroar initIntegrations / auth / security
huggingface-hubruntime deprun.get_url(), and lerobot's wandb call shapesCI / release plumbing
Pulled in from main
tb/ray-fragment-reconstitution-repro) — wait for complete fragment lineage, merged into this branch somainis not rolled back.State at merge
mainis not ahead ofrc/0.4.4— zero commits onmainthat this branch lacks, so nothing is lost.rc/0.4.4is 81 commits ahead ofmain.3ec06bf), all 15 checks.🤖 Generated with Claude Code