You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(bench): LMEVAL_VARIANT env override for experiment isolation in Modal harness #1128
Add an optional LMEVAL_VARIANT env var (or --variant flag) to modal_fast50.py that appends a custom suffix to the volume variant key.
Why
The volume resume feature keys results by (strategy, temporal, mmr_lambda, question_id). This is sufficient for flag-driven experiments, but not for code-driven experiments: tweaking retrieval code (e.g. temporal window logic in temporal.py) without adding a new flag produces the same variant key, so a resumed run silently returns stale results computed by the old code — the exact failure mode the variant-keyed paths were introduced to prevent (see review finding on #1127).
Today the only workaround is manually clearing the Modal volume, which also destroys valid resume state for other variants.
Testing
LMEVAL_VARIANT= unset → behaviour identical to today (no suffix)
LMEVAL_VARIANT=exp1 → volume path includes suffix; resume scoped to it; baseline variants untouched
compare_fast50.py baseline exp1 still diffs correctly (outdir already namespaced)
Closing as deferred. The 500Q validation campaign this was filed for is complete (v0.16.0 released with validated numbers — see #1142 / #1145), and the single-dataset setup made experiment isolation a non-issue in practice. Will reopen if a new benchmark campaign with variant isolation needs comes up.
Activity
ajianaz commentedon Aug 29, 2026
Closing as deferred. The 500Q validation campaign this was filed for is complete (v0.16.0 released with validated numbers — see #1142 / #1145), and the single-dataset setup made experiment isolation a non-issue in practice. Will reopen if a new benchmark campaign with variant isolation needs comes up.