Windows: git clone fails because HELM test fixture directories contain ":" (invalid NTFS character)

[Mandark-droid]

Problem

git clone of this repository fails on Windows:

error: invalid path 'tests/data/helm/commonsense:dataset=hellaswag,method=multiple_choice_joint,model=eleutherai_pythia-1b-v0/per_instance_stats.json'
fatal: unable to checkout working tree

The HELM test fixtures use HELM's run-spec names as directory names, and those contain :, which is an illegal character in Windows filenames (NTFS reserves it for alternate data streams). Git for Windows refuses to create these paths (and with the default core.protectNTFS=true, refuses to even stage them in the index), so Windows contributors cannot clone or check out the repo at all.

Affected paths (everything under tests/data/helm/):

• tests/data/helm/commonsense:dataset=hellaswag,method=multiple_choice_joint,model=eleutherai_pythia-1b-v0/
• tests/data/helm/mmlu:subject=philosophy,method=multiple_choice_joint,model=openai_gpt2/
• tests/data/helm/narrative_qa:model=openai_gpt2/

Proposed fix

Rename the fixture directories to replace : with a Windows-safe character (e.g. _), and update the corresponding path strings in tests/test_helm_instance_level_adapter.py. The HELM adapter itself reads run_spec.json from inside each run directory rather than parsing the directory name, so nothing at runtime depends on the colon being present — real HELM output directories on users' machines are unaffected.

Optionally, a small CI check (or pre-commit hook) rejecting committed paths with characters invalid on Windows (: * ? " < > |) would prevent regressions.

I'm happy to submit a PR for this.

Workaround for Windows users meanwhile

git clone --no-checkout https://github.com/evaleval/every_eval_ever.git
cd every_eval_ever
git config core.protectNTFS false
git sparse-checkout set --no-cone '/*' '!/tests/data/helm/'
git checkout main

This checks out everything except the colon-named fixtures (the HELM adapter tests that need them can then only run on Linux/macOS, e.g. in CI).