Anjishnu Mukherjee, Ziwei Zhu, Antonios Anastasopoulos · EMNLP ’26 (Main)
Paper · arXiv · Project page · LocalNewsQA on Hugging Face · Research overview and publications
This repository contains the reusable software and public artifacts for the metadata localization submission. The codebase supports the full component surface used by the project:
- building metadata-conditioned training corpora from geographically grounded news data;
- running MAPLE pretraining recipes through the project Nanotron fork;
- generating, validating, auditing, and releasing LocalNewsQA benchmark data;
- supervised fine-tuning and LoRA merge workflows;
- evaluating pretrained and instruction-tuned models on local knowledge, cultural projection, and external benchmark surfaces;
- reproducing the paper tables, plots, and summary analyses from tracked result artifacts.
The top-level layout separates the reusable package, benchmark data, training recipes, evaluation scripts, and paper artifacts so each component can be used directly.
Clone the repository and install the reusable culture-map package:
git clone https://github.com/iamshnoo/metadata_localization.git
cd metadata_localization
python -m pip install -e culture_map
culture-map --helpRender the released cultural-values map and overlay model projections:
culture-map download-paper-assets --data-dir data/paper_osf
culture-map plot-map \
--data-dir data/paper_osf \
--with-paper-models \
--output outputs/culture_map.pngBuild the tracked QA benchmark export:
python qa_data/build_hf_dataset.py --helpInspect the reproducibility pipeline:
python tools/repo.py components
python tools/repo.py localnewsqa-pipeline
python tools/repo.py sft
python tools/repo.py evals| Need | Start here | What it provides |
|---|---|---|
| Installable software package | culture_map/ |
CLI and Python modules for WVS-style cultural projection, plotting, and provider/model evaluation. |
| LocalNewsQA gold benchmark | qa_data/localnewsqa_core/final_gold_20260516/, qa_data/hf_dataset_localnewsqa_gold_20260516/ |
Final 18,700 row explicit/ambiguous gold benchmark as JSONL and parquet. |
| Dataset generation and audit | qa_data/localnewsqa_core/, docs/DATASET_AND_AUDIT.md |
LocalNewsQA candidate generation, human validation, web evidence, automated audit, repair, and release scripts. |
| Legacy QA benchmark | qa_data/ |
Legacy qa_metacul JSON, JSONL, Hugging Face dataset export, and builder. |
| Nanotron pretraining | src/step3_pretraining/, docs/PRETRAINING.md |
MAPLE Nanotron training recipes, checkpoint conversion, and pretrained-model evaluation. |
| SFT training and merge | src/step4_sft/4b_sft/, docs/SFT_AND_EVALUATION.md |
Supervised fine-tuning, LoRA merge, and cluster launch manifests for metadata variants. |
| Evaluation stack | src/step3_pretraining/3b_pretrain_eval/, src/step4_sft/4c_sft_eval/, slurm_live/, docs/SFT_AND_EVALUATION.md |
LocalNewsQA, external benchmarks, WVS/culture-map, open-ended probes, significance, and plots. |
| Reproducibility pipeline | src/ |
Step-structured scripts from data processing through model evaluation and paper plots. |
| Live experiment scripts | src_live/, slurm_live/ |
Flat mirrors of active scripts and cluster launchers used for paper runs. |
| Paper artifacts | results/ |
Tracked summary tables, plots, benchmark outputs, and compressed large result files. |
| Python component index | tools/repo.py |
Small no-dependency helper for discovering component entrypoints. |
| Workspace sync utility | sync_from_workspace.py |
Copies curated code/results from the experiment workspace into this repo layout. |
metadata_localization/
├── culture_map/ # installable package: cultural map projection/evaluation
├── docs/ # component guides and artifact manifests
├── qa_data/ # QA benchmark data, LocalNewsQA Core, and builders
├── results/ # tracked summary results, plots, and compressed artifacts
├── slurm_live/ # cluster job launchers used by the experiments
├── src/ # organized pipeline by paper stage
├── src_live/ # live flat script mirror from the experiment workspace
├── tools/ # Python helper entrypoints for repository discovery
├── sync_from_workspace.py
└── README.md
The src/ tree is the canonical organized view:
src/
├── step0_dataset/ # raw NOW processing and corpus statistics
├── step1_lda_analysis/ # topic modeling and document theme scaffolding
├── step2_process_data/ # metadata, splits, and HF/MECO dataset creation
├── step3_pretraining/ # checkpoint conversion and pretraining evaluation
├── step4_sft/ # QA generation, SFT, and SFT evaluation
└── step5_plots/ # paper figures, significance, and analysis scripts
Use python tools/repo.py components to list the main reusable entrypoints.
culture_map is an installable package we use for WVS-style cultural
projection. Install it with pip install -e culture_map; it exposes the
culture-map command and separate modules for asset loading, scoring,
projection, plotting, provider runners, and local checkpoint evaluation.
Useful commands:
culture-map download-paper-assets
culture-map derive-projection
culture-map plot-map --with-paper-models
culture-map run-openai-part1 --model <model-name>
culture-map run-anthropic-part1 --model <model-name>
culture-map run-gemini-part1 --model <model-name>
culture-map run-together-part1 --model <model-name>
culture-map run-local-country-eval --variant <variant-name>See culture_map/README.md and docs/COMPONENTS.md for the package API,
expected inputs, and common workflows.
The other reusable components are script-level pipelines:
qa_data/localnewsqa_core/for LocalNewsQA generation, validation, automated audit, and final release construction.src/step3_pretraining/for MAPLE pretraining recipes usinghttps://github.com/iamshnoo/nanotron, checkpoint conversion, and pretrained evaluation.src/step4_sft/4b_sft/for SFT training and LoRA merge.src/step4_sft/4c_sft_eval/andsrc/step5_plots/for benchmark evaluation, significance testing, and paper plots.
The repository includes code and summary artifacts. Large raw corpora, licensed inputs, model checkpoints, and cluster logs are intentionally excluded from Git. The pipeline expects those external assets to be present in the workspace paths configured by the individual scripts.
Recommended reading order:
docs/COMPONENTS.mdfor module boundaries and reusable entrypoints.docs/PRETRAINING.mdfor the Nanotron training, conversion, and evaluation workflow.docs/DATASET_AND_AUDIT.mdfor LocalNewsQA generation and audit.docs/SFT_AND_EVALUATION.mdfor SFT and benchmark evaluation.docs/REPRODUCIBILITY.mdfor the end-to-end pipeline and external inputs.docs/RESULTS.mdfor the tracked result directories and paper artifacts.qa_data/README.mdfor the QA benchmark surface.culture_map/README.mdfor cultural projection workflows.
The repository tracks source code, small benchmark artifacts, summary result tables, plots, and compressed result files needed for review. It does not track:
- raw NOW corpus data;
- generated document metadata and full training corpora;
- model checkpoints and training logs;
- API keys, local environment files, or cluster-local state;
- raw result files larger than GitHub's 100 MB object limit.
@inproceedings{mukherjee-etal-2026-maple,
title = "{MAPLE}: Metadata Conditioned {LLM} Pretraining for Locale-Aware Question Answering",
author = "Mukherjee, Anjishnu and
Zhu, Ziwei and
Anastasopoulos, Antonios",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Junyi Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.2053/",
doi = "10.18653/2026.emnlp-main.2053",
pages = "46746--46772",
ISBN = "979-8-89176-459-0"
}