Fix ILIAS/INQUIRE evaluation workflow: configurable paths, format mismatch, sample data generator, and dataset download scripts

[Copilot]

Users had no clear path to evaluate on ILIAS/INQUIRE: compute_embeds.py had hardcoded researcher-specific paths, eval_retrieval.py silently broke when loading distractor embeddings due to a key mismatch, there was no way to test the pipeline without downloading large datasets, and there were no download scripts or replication commands for the evaluation benchmarks.

compute_embeds.py — remove hardcoded paths

• --shard_dir and --out_dir are now required CLI arguments (were hardcoded to /u/ericx003/data/ilias/yfcc100m and ./yfcc_embeds)
• Added module docstring with output format spec and example commands for both ILIAS (YFCC100M) and INQUIRE (iNaturalist)

eval_retrieval.py — fix distractor loading and add baseline-only mode

compute_embeds.py writes {"keys": ..., "embeddings": ...} but the distractor loader called data['image_embeddings'], causing a KeyError at eval time.

Added load_distractor_embeddings_from_dir() that accepts both formats:

# compute_embeds.py output → "embeddings" key
# precompute_embeddings.py output → "image_embeddings" key

Explicit .float() cast on distractor tensors since compute_embeds.py defaults to fp16 storage.

Added --baseline_only flag that skips model loading entirely, so the full data pipeline can be smoke-tested without downloading any pretrained checkpoint. --checkpoint_path is now optional and only required when --baseline_only is not set.

create_sample_dataset.py — new synthetic data generator

Generates minimal WebDataset tar shards for smoke-testing the pipeline end-to-end without large downloads:

# Image-only shards for compute_embeds.py (mimics YFCC100M / iNaturalist)
python create_sample_dataset.py --mode distractors --out_dir ./sample/distractors --n_images 200

# Image+caption shards for precompute_embeddings.py (mimics ILIAS-core / INQUIRE queries)
python create_sample_dataset.py --mode pairs --out_dir ./sample/pairs --n_images 50 --captions_per_image 3

downloading/download_ilias.py and downloading/download_inquire.py — new dataset download scripts

Two new download scripts following the same snapshot_download pattern as the existing download_cc12m.py:

• download_ilias.py pulls ILIAS-core query shards from MVRL/ILIAS on HuggingFace and prints the next-step precompute_embeddings.py command. Includes notes on obtaining the separate YFCC15M distractor images.
• download_inquire.py pulls INQUIRE query annotations from olivialiu/INQUIRE on HuggingFace. Includes notes on obtaining the iNaturalist 2021 distractor images.

README.md — evaluation documentation

• Added a structured "Evaluation data" section under "Data Setup" with download commands for ILIAS and INQUIRE.
• Moved "Using Pretrained Models" before "Evaluation" and listed all four checkpoint names explicitly.
• Added full replication commands for ILIAS (using siglip2-large-patch16-512, the model confirmed to reproduce paper results) and INQUIRE.
• Smoke-test block now uses --baseline_only and requires no pretrained checkpoint to run end-to-end.

[jacobsn]

@copilot can you check to see if this PR addresses #1 (including factoring in the comments). Is there anything else we should do before merging this? Perhaps we need to have more detailed instructions for how to download the other datasets and specific command lines for running smoke tests and replication.

[Copilot]

@copilot can you check to see if this PR addresses #1 (including factoring in the comments). Is there anything else we should do before merging this? Perhaps we need to have more detailed instructions...

This PR does address issue #1 — the core workflow bugs (hardcoded paths in compute_embeds.py, key-format mismatch when loading distractor embeddings) are fixed, and the steps Eric described in the issue comments are now documented with concrete commands.

To close the remaining gaps (download instructions, specific model names, runnable replication), I've added in commit 9430b98:

• downloading/download_ilias.py and downloading/download_inquire.py — snapshot_download-based scripts (same pattern as download_cc12m.py) to pull the ILIAS-core and INQUIRE query shards from HuggingFace. Notes for the large distractor pools (YFCC15M / iNaturalist 2021) that must be obtained separately are included in each script's docstring.
• eval_retrieval.py --baseline_only — skips model loading so users can smoke-test the data pipeline end-to-end before downloading any checkpoint.
• README updated with:
  • Dataset download commands for both benchmarks
  • Full replication commands using the actual model names from download_ckpts.py (e.g. siglip2-large-patch16-512, the model Vladan confirmed reproduces ILIAS results)
  • A self-contained smoke-test block that runs with --baseline_only and requires no checkpoint

One thing worth confirming with @EricX003: the HuggingFace dataset repo IDs I've used (MVRL/ILIAS and olivialiu/INQUIRE) — please update the defaults in the download scripts if these differ from the actual published repos.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• ericx003.github.io
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
• huggingface.co
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[jacobsn]

@copilot resolve merge conflicts and make sure sure this PR "makes sense"

[Copilot]

@copilot resolve merge conflicts and make sure sure this PR "makes sense"

Resolved in commit 6ff6786. The only conflict was in eval_retrieval.py — compute_recall_at_k's function signature. Main had added query_image_ids/gallery_image_ids parameters for multi-caption recall, while our branch had removed them. The resolution keeps both: main's image_id-aware recall computation and our load_distractor_embeddings_from_dir + --baseline_only additions. The --baseline_only path now also correctly passes image_ids to evaluate_baseline for consistency.