Skip to content

feat(bench): LMEVAL_VARIANT env override for experiment isolation in Modal harness #1128

Closed

Description

@ajianaz

What

Add an optional LMEVAL_VARIANT env var (or --variant flag) to modal_fast50.py that appends a custom suffix to the volume variant key.

Why

The volume resume feature keys results by (strategy, temporal, mmr_lambda, question_id). This is sufficient for flag-driven experiments, but not for code-driven experiments: tweaking retrieval code (e.g. temporal window logic in temporal.py) without adding a new flag produces the same variant key, so a resumed run silently returns stale results computed by the old code — the exact failure mode the variant-keyed paths were introduced to prevent (see review finding on #1127).

Today the only workaround is manually clearing the Modal volume, which also destroys valid resume state for other variants.

Testing

  • LMEVAL_VARIANT= unset → behaviour identical to today (no suffix)
  • LMEVAL_VARIANT=exp1 → volume path includes suffix; resume scoped to it; baseline variants untouched
  • compare_fast50.py baseline exp1 still diffs correctly (outdir already namespaced)

Activity

  1. ajianaz commented on Aug 29, 2026

    @ajianaz
    CollaboratorAuthor

    Closing as deferred. The 500Q validation campaign this was filed for is complete (v0.16.0 released with validated numbers — see #1142 / #1145), and the single-dataset setup made experiment isolation a non-issue in practice. Will reopen if a new benchmark campaign with variant isolation needs comes up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions