Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models
Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos · WACV ’25
Paper · arXiv · Project page · DalleStreet on Hugging Face · Dollar Street subset on Hugging Face · Research overview and publications
Code and published assets for reproducing the Crossroads project.
DALLE Street dataset: https://huggingface.co/datasets/iamshnoo/dallestreet
This project is reproducible from:
- the GitHub repository
- the GitHub release assets under
data-assets-2026-03-30 - your own GPU access and API credentials
The repo alone is not self-contained. The release archives provide the data trees and intermediate outputs that the scripts expect.
Use Python 3.10 or 3.11.
Create the environment:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip setuptools wheel
pip install \
pandas numpy pillow requests tqdm openai datasets \
matplotlib seaborn scikit-learn transformers accelerate \
huggingface-hub opencv-python diffusers torch torchvision \
torchmetrics supervisionIf you want to run edit.py, also install GroundingDINO:
git clone https://github.com/IDEA-Research/GroundingDINO.git
pip install -e ./GroundingDINOCopy the example environment file:
cp env.example.sh env.sh
source env.sh
mkdir -p "$CROSSROADS_DOLLARSTREET_CACHE" "$CROSSROADS_MODEL_CACHE" "$CROSSROADS_INSTRUCTBLIP_CACHE" "$CROSSROADS_EDITS_CACHE"Create secrets.json and point CROSSROADS_SECRETS_FILE to it if it is not in the repo root:
{
"AZURE_OPENAI_ENDPOINT": "...",
"AZURE_OPENAI_API_KEY": "...",
"GPT4V_OPENAI_ENDPOINT": "...",
"GPT4V_OPENAI_API_KEY": "...",
"OPENAI_API_KEY": "..."
}Path variables used by the repo:
CROSSROADS_DATA_ROOT: extracted project-data root; defaults to the repo rootCROSSROADS_SECRETS_FILE: path tosecrets.jsonCROSSROADS_DOLLARSTREET_CACHE: Hugging Face cache for Dollar StreetCROSSROADS_MODEL_CACHE: cache root for LLaVA and related modelsCROSSROADS_INSTRUCTBLIP_CACHE: cache root for InstructBLIPCROSSROADS_EDITS_CACHE: cache root for editing and similarity models
Download the published release assets:
- Release:
data-assets-2026-03-30 - URL:
https://github.com/iamshnoo/crossroads/releases/tag/data-assets-2026-03-30
Reconstruct the split archives:
cat backup.tar.gz.part* > backup.tar.gz
cat dataset_dollarstreet.tar.gz.part* > dataset_dollarstreet.tar.gzVerify them:
sha256sum backup.tar.gz
sha256sum dataset_dollarstreet.tar.gzExpected SHA-256 values:
backup.tar.gz:94ba8c7e4c5413c5db31e6b8fa88c648d6bc0fe1e82ab2642398dda719396cffdataset_dollarstreet.tar.gz:cff780f4418bd40ae60aa7ed3a7fa3619d25de30e2178c94f8e63b9b1c92a97f
Extract the archives. If you extract them outside the repo root, set:
export CROSSROADS_DATA_ROOT="/path/to/extracted/data/root"The release archives supply the folders below. These are the important ones for reproduction.
results/
- raw model outputs and generated artifacts
- examples used directly by scripts:
results/dalle_images.csvresults/gpt/azure/*.csvresults/llava/*.csvresults/dalle_eval/.../*.csvresults/marvl/.../*.csvresults/dalle_objects/...results/human-study/study1/*.csvresults/human-study/study2/*.csv
corrected/
- normalized and aggregated outputs used for benchmarking and plotting
- this is the main comparison surface for reproducing the paper numbers
marvl/
- MARVL metadata and file mappings
- most importantly:
marvl/marvl_images.csvplus the referenced MARVL image paths
These folders should exist under CROSSROADS_DATA_ROOT. If you extract the archives into the repo root, no extra path editing is required.
Run the project in this order.
Primary script:
gen_dalle_street.py
Command:
./.venv/bin/python gen_dalle_street.py --category carThis produces images under results/dalle_natural/<category>/<country>/. The script currently loops over all categories internally.
Required file:
results/dalle_images.csv
This manifest is consumed by the DALLE Street evaluation and artifact-extraction scripts.
Dollar Street:
classify_dollar_street_gpt.pyclassify_dollar_street_llava.py
for split in car cups_mugs_glasses family_snapshots front_door home kitchen plate_of_food social_drink wall_decoration wardrobe; do
./.venv/bin/python classify_dollar_street_gpt.py --split "$split"
./.venv/bin/python classify_dollar_street_llava.py --split "$split"
doneDALLE Street:
classify_dalle_street_gpt.pyclassify_dalle_street_llava.py
for kind in natural vivid; do
for concept in car cups_mugs_glasses family_snapshots front_door home kitchen plate_of_food social_drink wall_decoration wardrobe; do
./.venv/bin/python classify_dalle_street_gpt.py --type "$kind" --concept "$concept"
./.venv/bin/python classify_dalle_street_llava.py --type "$kind" --concept "$concept"
done
doneMARVL:
classify_marvl_gpt.pyclassify_marvl_llava.py
for lang in id sw ta tr zh; do
./.venv/bin/python classify_marvl_gpt.py --country "$lang"
./.venv/bin/python classify_marvl_llava.py --country "$lang"
doneclassify_evals.pymarvl_evals.py
./.venv/bin/python classify_evals.py
./.venv/bin/python marvl_evals.pyoverall_acc.pyplot.pyplot_marvl.pydata_stats.pyhuman_evals.pyrgb_calculate.pypeople_counter.py
./.venv/bin/python overall_acc.py
./.venv/bin/python plot.py
./.venv/bin/python plot_marvl.py
./.venv/bin/python data_stats.py
./.venv/bin/python human_evals.py
./.venv/bin/python rgb_calculate.py
./.venv/bin/python people_counter.pyRun these in order:
dalle_obj_detect.pydalle_obj_process.pydalle_obj_det_process.pydalle_obj_counts.pycountry_obj_dict_remove_adj.pycountry_obj_cooccur.pycoocurrence_statistics.py
Standalone helpers:
e2e-caption.pyedit.py
Recovered helper scripts:
evals/artifacts.pyevals/cap-edit-captioning.pyevals/cap-edit-caption-editing.pyevals/cap-edit-image-editing-preprocess.pyevals/cap-edit-pnp-config-maker.pyevals/cap-edit-image-editing-pnp.pyevals/cap-edit-eval.pyevals/cultureadapt.pyevals/eval_img_similarity.pyevals/consolidate-eval-results.py
These are the main reference files already present in the published release data. After rerunning the pipeline, compare your regenerated outputs against these files.
corrected/dollar_street_gpt.csvcorrected/dollar_street_llava.csvcorrected/dalle_street_gpt.csvcorrected/dalle_street_llava.csvcorrected/marvl_gpt.csvcorrected/marvl_llava.csv
corrected/accuracy_by_income_quartiles_dollar_street_gpt.csvcorrected/accuracy_by_income_quartiles_dollar_street_llava.csvcorrected/country_wise_accuracy_by_subregion_dollar_street_gpt.csvcorrected/country_wise_accuracy_by_subregion_dollar_street_llava.csvcorrected/country_wise_accuracy_by_subregion_dalle_street_gpt.csvcorrected/country_wise_accuracy_by_subregion_dalle_street_llava.csvcorrected/country_wise_accuracy_dollar_street_gpt.csvcorrected/country_wise_accuracy_dollar_street_llava.csvcorrected/country_wise_accuracy_dalle_street_gpt.csvcorrected/country_wise_accuracy_dalle_street_llava.csvcorrected/marvl_accuracy_gpt.csvcorrected/marvl_accuracy_llava.csv
corrected/objects_proc.csvcorrected/objects_proc_filtered.csvcorrected/country_obj_dict_adj.jsoncorrected/country_obj_dict_adj_unfiltered.jsoncorrected/country_obj_dict_no_adj.jsoncorrected/country_obj_dict_no_adj_unfiltered.jsoncorrected/country_obj_adj_tfidf.csvcorrected/country_obj_adj_tfidf.jsoncorrected/country_obj_no_adj_tfidf.csvcorrected/country_obj_no_adj_tfidf.json
For paper reproduction, the most important comparison surface is still corrected/.
There is not a single final_metrics.csv in this repo. The final paper metrics are distributed across the benchmark CSVs below.
corrected/dollar_street_llava.csvcorrected/dollar_street_gpt.csvcorrected/dalle_street_gpt.csvcorrected/dalle_street_llava.csvcorrected/marvl_gpt.csvcorrected/marvl_llava.csv
Running:
./.venv/bin/python overall_acc.pyprints the final headline accuracies from those six files.
corrected/accuracy_by_income_quartiles_dollar_street_gpt.csvcorrected/accuracy_by_income_quartiles_dollar_street_llava.csvcorrected/country_wise_accuracy_by_subregion_dollar_street_gpt.csvcorrected/country_wise_accuracy_by_subregion_dollar_street_llava.csvcorrected/country_wise_accuracy_by_subregion_dalle_street_gpt.csvcorrected/country_wise_accuracy_by_subregion_dalle_street_llava.csvcorrected/country_wise_accuracy_dollar_street_gpt.csvcorrected/country_wise_accuracy_dollar_street_llava.csvcorrected/country_wise_accuracy_dalle_street_gpt.csvcorrected/country_wise_accuracy_dalle_street_llava.csvcorrected/marvl_accuracy_gpt.csvcorrected/marvl_accuracy_llava.csv
results/human-study/artifacts.csvresults/consolidated_metrics.csvresults/cap_edit/metrics/*.csvresults/cultureadapt/metrics/*.csv
If you can regenerate these files and they match the published release artifacts closely, you have reproduced the paper’s reported outputs.
@InProceedings{Mukherjee_2025_WACV,
author = {Mukherjee, Anjishnu and Zhu, Ziwei and Anastasopoulos, Antonios},
title = {Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models},
booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)},
month = {February},
year = {2025},
pages = {1755-1764}
}