roar reproduce fails for lineages with a 'roar get' source step (wrong roar in venv + dropped destination)

[christophergeyer]

Summary

roar reproduce --run of a lineage whose source step is roar get hf://datasets/... fails at that first step, cascading to the whole pipeline. Two compounding problems.

Problem 1 — reproduction venv installs the wrong roar

In the reproduce venv, .venv/bin/roar is a Python-2 package (PyPI name collision), not roar-cli. When reproduce replays the source step as roar run roar get ..., the inner roar get invokes the broken py2 tool:

File ".venv/bin/roar", line 328
    print 'Invalid Port: %s' % options.serve
SyntaxError: Missing parentheses in call to 'print'.

→ Step 1 exits 1. Root cause: the reproduction venv resolves roar to the wrong distribution; roar-cli's entrypoint isn't what ends up on PATH inside the venv.

Problem 2 — roar get destination dropped from the replayed command

The replayed step is roar get hf://datasets/yuanty/LIBERO-fastwam --force — the original destination arg (./data/libero_tarballs) is gone. Even with Problem 1 fixed, roar get would fetch to a default location while downstream steps read the recorded path, so a path-based handoff fails:

tar: ./data/libero_tarballs/*.tar.gz: Cannot open: No such file or directory

Impact

A pipeline that sources data via roar get (the recommended way to make data a tracked, reproducible input) cannot be reproduced end-to-end — steps after the get cascade-fail on missing data. This blocks the core reproducibility guarantee for any data-driven pipeline.

Suggested fixes

1. Install/resolve roar-cli (not a colliding roar) into the reproduction venv, or drive replayed roar get steps via the same roar entrypoint running the reproduction rather than a venv-local roar.
2. Preserve roar get arguments (including destination) in the replayed command, or restore the fetched artifact to its recorded path before downstream steps run.

Environment

roar 0.3.6 · reproduce --run --lineage · Hugging Face dataset source · Linux/conda

[christophergeyer]

Root cause pinned down — it's a distribution-name vs import-name mismatch in package capture.

roar show @<train-step> --all records the dependency as roar (the import module name), not roar-cli (the PyPI distribution name). roar-cli installs a top-level module named roar, but the runtime capture wrote down the module name and reproduce uses it as the pip install target.

Result in the reproduce venv:

• pip install roar resolves to an unrelated, squatted Python-2 package named roar on PyPI (print 'Invalid Port: %s' → SyntaxError).
• roar-cli is therefore never installed in the reproduce venv (pip show roar-cli → not found; pip list | grep roar → empty).
• So the replayed roar get ... step invokes .venv/bin/roar (the wrong py2 tool) → exit 1 → the whole pipeline cascade-fails.

Fix direction: map captured import modules back to their installing distribution before recording (e.g. importlib.metadata.packages_distributions() → roar ⇒ roar-cli), or special-case roar itself so the reproduction venv always installs roar-cli at the same version that drove the run.