Skip to content

Fix backend CI: upgrade datasets, use canonical squad id (closes #18) - #20

Merged
elkaix merged 1 commit into
mainfrom
fix/datasets-hub-compat
Jul 12, 2026
Merged

elkaix merged 1 commit into
mainfrom
fix/datasets-hub-compat

Conversation

@elkaix

@elkaix elkaix commented Jul 12, 2026

Copy link
Copy Markdown
Owner

The Backend tests CI check has been red since before #17 (merged via admin override) — the pre-existing #18 dependency incompatibility. This fixes it at the root.

Root cause

A three-way version conflict:

  • requirements.txt capped datasets>=2.16,<3. The datasets 2.x load path resolves a bare squad_v2 dataset id via huggingface_hub.
  • huggingface_hub floated to 1.x (transitively), which rejects bare ids — it requires namespace/name. So load_dataset("squad_v2") raises HfUriError on any machine without a warm cache (i.e. CI).
  • Pinning hub <1.0 is not an option: transformers 5.x requires huggingface_hub>=1.5.0.

Fix

  • requirements.txt: datasets>=2.16,<3 → datasets>=3 — datasets 5.x supports huggingface_hub 1.x, so the whole stack (datasets / hub / transformers / sentence-transformers) is coherent (pip check clean).
  • src/eval/datasets/squad_v2.py: load the canonical rajpurkar/squad_v2 id, which hub 1.x accepts. Same underlying data and row shape — the checked-in frozen artifact and the sampling tests are unaffected.

Verification

  • Full suite 304 passing on datasets 5.0.0 + huggingface_hub 1.23.0.
  • The 5 previously-failing squad tests (test_eval_datasets_squad.py) pass from an empty HF cache — the exact cold-start condition that fails in CI.

Closes #18. Unblocks the required Backend-tests check so future PRs merge without admin override.

CI's Backend tests check failed because `datasets` was capped `<3`, whose
dataset-loading path is incompatible with `huggingface_hub` 1.x — which
`transformers` 5.x requires (>=1.5.0). Resolving the bare `squad_v2` id under
hub 1.x raised HfUriError ("Repository id must be 'namespace/name'"). Pinning
hub down is impossible without breaking transformers.

Fix both halves of the conflict:
- requirements.txt: `datasets>=2.16,<3` -> `datasets>=3` (5.x supports hub 1.x).
- squad_v2 loader: load the canonical `rajpurkar/squad_v2` id, which hub 1.x
  accepts. Dataset content and row shape are unchanged, so the frozen artifact
  and the sampling tests still hold.

Verified: full suite 304 passing; the squad dataset tests pass from an empty
HF cache — the cold-start condition that fails in CI.

Closes #18.
@elkaix
elkaix merged commit 92d51ea into main Jul 12, 2026
2 checks passed
@elkaix
elkaix deleted the fix/datasets-hub-compat branch July 12, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: squad_v2 dataset loading broken on fresh environments (datasets/huggingface_hub incompatibility)

1 participant