diff --git a/README.md b/README.md index 482c18a..e456668 100644 --- a/README.md +++ b/README.md @@ -232,7 +232,7 @@ ORBIT includes a compact benchmark suite spanning text-only and multimodal routi | ------------------ | ---------- | ------: | ------------------------------------------------------------ | | **RouterBench** | Text | 11 | Small text-only pool for controlled budgeted routing evaluation. | | **RouterEval** | Text | Dynamic | Large text-only pool determined by the common models in the downloaded release. | -| **MMR-Bench** | Multimodal | 10 | Multimodal routing with visual inputs. | +| **MMR-Bench** | Multimodal | 9 complete | V1 default keeps complete candidates; explicit coverage and split protocols. | | **MixInstruct** | Text | 12 | Text-only instruction-style queries across mixed sources. | | **LLMRouterBench** | Text | 12 | Performance-cost routing over ten tasks with per-query model costs. | | **xRouteBench** | Text / text-view | 18 | Official train/test outcomes with recorded token usage and frozen pricing. | @@ -244,6 +244,13 @@ ORBIT includes a compact benchmark suite spanning text-only and multimodal routi > Steps 1 and 2. ORBIT reads named JSONL/Parquet/CSV exports, not positional or > pickle matrices. See [data preparation, image handling and evaluation limitations](docs/vl_routerbench.md). +MMR-Bench V1 parses mixed boolean/numeric labels, retains original monetary +costs, and explicitly selects fully observed candidates without imputing missing +cells or dropping queries. The default in-domain split remains `legacy`; +`configs/examples/mmrbench_joint_grouped.json` opts into joint image/text +grouping for a shared leakage-resistant split. See [coverage, response-field +compatibility and protocol limitations](docs/mmrbench_protocol.md). + > Note: RouterEval provides 12 dataset files with nested model pools. To ensure a consistent evaluation protocol, ORBIT deterministically sorts the common models from Group A's six files and aligns performance and cost columns to that canonical order. The bundled metadata currently contains 239 model descriptions; description-based routers validate coverage by model name and report any missing entries instead of silently misaligning them. > Note: The LLMRouterBench loader uses the Performance-Cost subset and ORBIT's standard 20/80 in-domain split. It aligns 12 models over 12,446 queries, preserves released costs for auditing, repairs usable zero-cost records from token usage or model-task medians, and globally scales effective costs to `[0, 1]`. Failed zero-usage calls retain zero performance and zero cost. diff --git a/configs/benchmarks/mmrbench.json b/configs/benchmarks/mmrbench.json index b720dcf..cd6956a 100644 --- a/configs/benchmarks/mmrbench.json +++ b/configs/benchmarks/mmrbench.json @@ -8,6 +8,8 @@ "dataset": { "name": "MMRBench", "dataset_dir": "./data/mmrbench", + "candidate_policy": "complete-models", + "protocol": "legacy", "split": { "mode": "in-domain", "ratios": { diff --git a/configs/examples/mmrbench_joint_grouped.json b/configs/examples/mmrbench_joint_grouped.json new file mode 100644 index 0000000..bf6b599 --- /dev/null +++ b/configs/examples/mmrbench_joint_grouped.json @@ -0,0 +1,12 @@ +{ + "dataset": { + "name": "MMRBench", + "dataset_dir": "./data/mmrbench", + "candidate_policy": "complete-models", + "protocol": "joint-grouped", + "split": { + "mode": "in-domain", + "ratios": {"train": 0.2, "test": 0.8} + } + } +} diff --git a/docs/mmrbench_protocol.md b/docs/mmrbench_protocol.md new file mode 100644 index 0000000..c8668ae --- /dev/null +++ b/docs/mmrbench_protocol.md @@ -0,0 +1,72 @@ +# MMR-Bench V1 data integrity and evaluation protocols + +The production `MMRBench` loader reads the released CSV and image archive. +It converts literal `True`/`False` and numeric quality labels into numeric +values in [0,1], and preserves original monetary costs, including valid zeros. +Malformed/out-of-range labels and negative/infinite costs fail explicitly. +Missing observations are never converted to incorrect answers or zero costs. + +## Candidate coverage + +The historical default candidate order contains ten models. In the audited V1 +release, GPT-5-nano has no score/cost for MathVision's 3,040 queries. The shipped +benchmark configuration now explicitly selects +`dataset.candidate_policy="complete-models"`: retain only candidates with +finite quality and cost on every query, in their original order. For that +release this means nine models and all 10,370 queries across seven datasets, +including MathVision and OCRBench. No query is dropped to complete the pool. +Unknown extra CSV candidates are not automatically added to the declared pool. + +Excluded model names and coverage counts are logged and available through +`loader.excluded_models` and `loader.model_coverage`. Candidate selection +depends only on observation availability, never performance magnitude. +`dataset.candidate_policy="error"` rejects incomplete candidates instead; +programmatic configurations without a policy also default to this strict mode. +The loader fails if fewer than two complete candidates remain. + +The corrected nine-model pool is not the historical ten-model evaluation. +Report the candidate policy and actual returned model list with results. + +## Response compatibility and images + +Each actual released answer is exposed as `model_i_response` for the shared +router/EA-RAM contract. The historical `model_i_prediction` field is retained +as an identical alias. Missing response strings are not fabricated; EA-RAM's +existing response-availability checks still apply. + +Images may be in `DATASET_DIR/TASK/ID.ext` (the archive's layout) or in +`DATASET_DIR/images/TASK/ID.ext`. SEEDBench2_Plus maps to SEEDBenchv2Plus. +Missing or ambiguous images fail with the query ID, rather than yielding a +missing path. Joint grouping additionally decodes every image and rejects +unreadable files; it never substitutes a zero image. + +## Optional joint image/text grouping + +The default `dataset.protocol="legacy"` retains the previous seeded row-random +**in-domain split**, with a warning that repeated inputs can cross train/test. +It does not bypass label/cost validation or restore incomplete observations. + +For a common split that also protects text-view baselines, opt in explicitly: + +```bash +python main.py --dataset MMRBench --method knn \ + --config configs/examples/mmrbench_joint_grouped.json --device cuda:0 +``` + +Set the example's dataset directory to your local release. `--config` replaces +top-level objects, so the example includes the complete `dataset` settings. + +`dataset.protocol="joint-grouped"` forms connected components: queries with +identical decoded RGB images **or** identical whitespace-normalized text must +remain together, including transitive links. Different image encodings with +the same decoded pixels are grouped. Original prompts are not normalized for +encoder input, and quality, costs and answers are never used for grouping. +The configured seed and train fraction drive `GroupShuffleSplit` on these +components. The query fraction is approximate and the split is not per-task +stratified; a whole task can fall in test. Report this as a distinct protocol, +not the untouched historical split or a guaranteed within-task evaluation. + +This grouping changes only MMR-Bench V1 in-domain splitting. Out-of-domain task +splitting, other benchmarks, method algorithms and metric definitions are +unchanged. It prevents input overlap across the experiment split, not exposure +during external pretrained-model training. diff --git a/tests/test_mmrbench_loader.py b/tests/test_mmrbench_loader.py new file mode 100644 index 0000000..cb56d9e --- /dev/null +++ b/tests/test_mmrbench_loader.py @@ -0,0 +1,239 @@ +"""Production MMR-Bench loader regressions; no experiment/cache dependencies.""" + +import json +from pathlib import Path +import tempfile +import unittest +from unittest.mock import patch + +import numpy as np +import pandas as pd +from PIL import Image + +from main import load_config +from methods.EARAMRouter import EARAMRouter +from utils.data import BaseDatasetLoader, MMRBenchLoader + + +MODELS = ["Qwen2.5-VL-3B-Instruct", "Qwen2.5-VL-72B-Instruct", "InternVL3-78B", "Gemma3-4B", + "Qwen2.5-VL-7B-Instruct", "Claude3-7V_Sonnet", "gpt-5-nano-2025-08-07", "GeminiPro2-5", + "gpt-5-2025-08-07", "GeminiFlash2-5"] + + +class MMRBenchLoaderTest(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name) + (self.root / "MMStar").mkdir() + for index in range(20): + Image.new("RGB", (8, 8), (index, 5 * index, 40)).save(self.root / f"MMStar/{index}.png") + self.raw = pd.DataFrame({"dataset_idx": [f"MMStar_{i}" for i in range(20)], + "question": [f" query {i}\n" for i in range(20)], "answer": ["A"] * 20}) + labels = ["True", "False", "1", "0", True, False, 1, 0, .5, .25] * 2 + for index, model in enumerate(MODELS): + self.raw[f"{model}_correct"] = labels + self.raw[f"{model}_cost"] = np.arange(20) * (index + 1) * .0001 + self.raw[f"{model}_prediction"] = [f"actual answer {index} {i}" for i in range(20)] + self.args = {"seed": 42, "dataset": {"name": "MMRBench", "candidate_policy": "complete-models", + "protocol": "legacy", "split": {"mode": "in-domain", "ratios": {"train": .2, "test": .8}}}} + + def load(self): + self.raw.to_csv(self.root / "MMR-Bench.csv", index=False) + loader = MMRBenchLoader(str(self.root), self.args) + models, frame = loader.process(str(self.root)) + return loader, models, frame + + def test_mixed_labels_become_numeric_and_original_costs_stay_exact(self): + loader, models, frame = self.load() + self.assertEqual(models, MODELS) + expected = [1, 0, 1, 0, 1, 0, 1, 0, .5, .25] * 2 + for index, model in enumerate(models): + np.testing.assert_array_equal(frame[f"model_{index}_performance"], expected) + np.testing.assert_allclose(frame[f"model_{index}_cost"], self.raw[f"{model}_cost"], rtol=0, atol=1e-16) + pd.testing.assert_series_equal(frame[f"model_{index}_response"], frame[f"model_{index}_prediction"], check_names=False) + self.assertEqual(loader.excluded_models, []) + self.assertEqual(loader.model_coverage, dict.fromkeys(MODELS, 20)) + self.assertEqual(frame.model_0_cost.iloc[0], 0) + + def test_complete_models_policy_retains_every_query_and_canonical_order(self): + self.raw.loc[3:7, [f"{MODELS[6]}_correct", f"{MODELS[6]}_cost"]] = np.nan + with self.assertLogs(level="WARNING") as logs: + loader, models, frame = self.load() + self.assertEqual(models, MODELS[:6] + MODELS[7:]) + self.assertEqual(loader.excluded_models, [MODELS[6]]) + self.assertEqual(loader.model_coverage[MODELS[6]], 15) + self.assertEqual(len(frame), 20) + self.assertIn("no queries dropped", " ".join(logs.output)) + np.testing.assert_array_equal(frame.model_6_cost.to_numpy(), pd.read_csv(self.root / "MMR-Bench.csv")[f"{MODELS[7]}_cost"].to_numpy()) + self.assertTrue(np.isfinite(frame.filter(regex=r"_performance$|_cost$").to_numpy()).all()) + self.assertNotIn("model_9_performance", frame) + + def test_strict_policy_and_missing_policy_fail_on_incomplete_cells(self): + self.raw.loc[0, f"{MODELS[6]}_cost"] = np.nan + for policy in ("error", None): + with self.subTest(policy=policy): + if policy is None: + self.args["dataset"].pop("candidate_policy", None) + else: + self.args["dataset"]["candidate_policy"] = policy + with self.assertRaisesRegex(ValueError, "incomplete candidate observations"): + self.load() + + def test_missing_cost_or_score_is_never_imputed_as_zero(self): + for suffix in ("cost", "correct"): + with self.subTest(suffix=suffix): + column = f"{MODELS[0]}_{suffix}" + old = self.raw[column].copy() + self.raw.loc[0, column] = np.nan + with self.assertLogs(level="WARNING"): + _, models, frame = self.load() + self.assertNotIn(MODELS[0], models) + self.assertEqual(len(frame), 20) + self.raw[column] = old + + def test_malformed_labels_raise_instead_of_selecting_candidates_by_quality(self): + for invalid in ("false-ish", "infinity", "-0.1", "1.1"): + with self.subTest(value=invalid): + self.raw.loc[0, f"{MODELS[0]}_correct"] = invalid + with self.assertRaisesRegex(ValueError, "correctness"): + self.load() + + def test_invalid_costs_raise_instead_of_silently_dropping_model(self): + for invalid in (-1., float("inf"), "invalid"): + with self.subTest(value=invalid): + self.raw[f"{MODELS[0]}_cost"] = self.raw[f"{MODELS[0]}_cost"].astype(object) + self.raw.loc[0, f"{MODELS[0]}_cost"] = invalid + with self.assertRaisesRegex(ValueError, "cost"): + self.load() + + def test_aliases_allow_ea_ram_to_fit_the_actual_response_evaluator(self): + _, models, frame = self.load() + class TextEncoder: + def run_embed(self, texts=None, images=None): + import torch + return torch.tensor([[len(str(text)), 1.] for text in texts], dtype=torch.float32) + router = EARAMRouter.__new__(EARAMRouter) + router.args = {"training": {"epochs": 1, "batch_size": 32, "hidden_dim": 8, "lr": .001}} + router.train_df, router.model_list = frame, models + router.embedder = TextEncoder() + router.seed = 42 + router.hidden_dim, router.batch_size, router.epochs = 8, 32, 1 + router.learning_rate, router.weight_decay = .001, 0. + import torch + router.device = torch.device("cpu") + router.expost_evaluator = None + router._fit_expost_evaluator() + self.assertIsNotNone(router.expost_evaluator) + + def test_nested_images_layout_and_seedbench_alias(self): + nested = self.root / "images/SEEDBenchv2Plus" + nested.mkdir(parents=True) + self.raw.loc[0, "dataset_idx"] = "SEEDBench2_Plus_0" + (self.root / "MMStar/0.png").rename(nested / "0.png") + _, _, frame = self.load() + self.assertEqual(frame.eval_name.iloc[0], "SEEDBenchv2Plus") + self.assertEqual(Path(frame.image_path.iloc[0]), nested / "0.png") + + def test_missing_and_ambiguous_images_fail_explicitly(self): + (self.root / "MMStar/0.png").rename(self.root / "unused.png") + with self.assertRaisesRegex(FileNotFoundError, "image missing"): + self.load() + (self.root / "unused.png").rename(self.root / "MMStar/0.png") + Image.new("RGB", (8, 8)).save(self.root / "MMStar/0.jpg") + with self.assertRaisesRegex(ValueError, "ambiguous image"): + self.load() + + def test_joint_groups_are_transitive_across_text_and_decoded_pixels(self): + # Different PNG encodings/paths, identical pixels; then a text-only link. + with Image.open(self.root / "MMStar/0.png") as image: + image.save(self.root / "MMStar/1.png", compress_level=0) + self.raw.loc[1, "question"] = "shared question" + self.raw.loc[2, "question"] = " shared\n question\t" + _, _, frame = self.load() + groups = MMRBenchLoader._joint_input_groups(frame) + self.assertEqual(groups[0], groups[1]) + self.assertEqual(groups[1], groups[2]) + self.assertNotEqual(groups[2], groups[3]) + + def test_joint_split_is_deterministic_disjoint_and_retains_original_inputs(self): + self.args["dataset"]["protocol"] = "joint-grouped" + self.raw.loc[1, "question"] = self.raw.loc[0, "question"] + with Image.open(self.root / "MMStar/2.png") as image: + image.save(self.root / "MMStar/3.png") + loader, _, frame = self.load() + train, test = loader.split(frame) + again_train, again_test = loader.split(frame) + pd.testing.assert_frame_equal(train, again_train) + pd.testing.assert_frame_equal(test, again_test) + def texts(data): + return set(data.prompt.map(lambda value: " ".join(str(value).split()))) + self.assertFalse(texts(train) & texts(test)) + groups = loader._joint_input_groups(frame) + membership = dict(zip(frame.sample_id, groups)) + self.assertFalse({membership[sid] for sid in train.sample_id} & {membership[sid] for sid in test.sample_id}) + combined = pd.concat([train, test]).set_index("sample_id").loc[frame.sample_id].reset_index() + pd.testing.assert_frame_equal(combined, frame) + + def test_legacy_split_exactly_matches_previous_seeded_row_random_protocol(self): + loader, _, frame = self.load() + with self.assertLogs(level="WARNING"): + train, test = loader.split(frame) + expected_train, expected_test = BaseDatasetLoader.split_in(loader, frame) + pd.testing.assert_frame_equal(train, expected_train) + pd.testing.assert_frame_equal(test, expected_test) + + def test_unknown_policies_and_missing_columns_fail_clearly(self): + self.args["dataset"]["candidate_policy"] = "unknown" + with self.assertRaisesRegex(ValueError, "candidate_policy"): + self.load() + self.args["dataset"]["candidate_policy"] = "complete-models" + loader, _, frame = self.load() + self.args["dataset"]["protocol"] = "unknown" + with self.assertRaisesRegex(ValueError, "dataset.protocol"): + loader.split(frame) + self.raw = self.raw.drop(columns=[f"{MODELS[0]}_correct"]) + with self.assertRaisesRegex(ValueError, "missing source columns"): + self.load() + + def test_joint_grouping_rejects_empty_queries_and_corrupt_images(self): + loader, _, frame = self.load() + for value in (None, " \t\n"): + bad = frame.copy() + bad.loc[0, "prompt"] = value + with self.assertRaisesRegex(ValueError, "query text"): + loader._joint_input_groups(bad) + # A real non-image local file must not become a zero image in grouping. + bad = frame.copy() + bad.loc[0, "image_path"] = str(self.root / "MMR-Bench.csv") + with self.assertRaises(OSError): + loader._joint_input_groups(bad) + + def test_shipped_configs_are_explicit_and_joint_grouping_is_opt_in(self): + args = load_config("MMRBench", "knn") + self.assertEqual(args["dataset"]["candidate_policy"], "complete-models") + self.assertEqual(args["dataset"]["protocol"], "legacy") + example = json.loads(Path("configs/examples/mmrbench_joint_grouped.json").read_text()) + self.assertEqual(example["dataset"]["protocol"], "joint-grouped") + + def test_zero_quality_is_retained_and_insufficient_complete_pool_is_rejected(self): + self.raw[f"{MODELS[0]}_correct"] = 0 + _, models, _ = self.load() + self.assertIn(MODELS[0], models) + for model in MODELS[1:]: + self.raw[f"{model}_cost"] = np.nan + with self.assertRaisesRegex(ValueError, "at least two complete"): + self.load() + + def test_invalid_query_identity_fails_before_image_lookup(self): + for value in (None, "MMStar_1", "no-separator", "../MMStar_0", ".._0"): + with self.subTest(value=value): + old = self.raw.loc[0, "dataset_idx"] + self.raw.loc[0, "dataset_idx"] = value + with self.assertRaises(ValueError): + self.load() + self.raw.loc[0, "dataset_idx"] = old + + +if __name__ == "__main__": + unittest.main() diff --git a/utils/data.py b/utils/data.py index 55ca6d1..dce4189 100644 --- a/utils/data.py +++ b/utils/data.py @@ -556,6 +556,86 @@ class MMRBenchLoader(BaseDatasetLoader): hf_id = "gh0stHunter/MMR-Bench" default_target_path = "./data/MMRBench" + @staticmethod + def _quality_values(values: pd.Series, model: str) -> np.ndarray: + """Parse released bool/string/numeric labels without filling missing cells.""" + normalized = values.astype("string").str.strip().str.lower().replace({"true": "1", "false": "0"}) + try: + parsed = pd.to_numeric(normalized, errors="raise").to_numpy(dtype=np.float64, na_value=np.nan) + except (ValueError, TypeError) as exc: + raise ValueError(f"MMRBench {model}: invalid correctness labels") from exc + if np.isinf(parsed).any() or ((parsed < 0) | (parsed > 1)).any(): + raise ValueError(f"MMRBench {model}: correctness labels must be in [0, 1]") + return parsed + + def _image_location(self, sample_id: str) -> tuple[str, str]: + task, separator, identifier = str(sample_id).rpartition("_") + if (not separator or not task or not identifier or task in {".", ".."} or identifier in {".", ".."} + or any(c in str(sample_id) for c in ("/", "\\"))): + raise ValueError(f"MMRBench invalid dataset_idx: {sample_id!r}") + task = "SEEDBenchv2Plus" if task == "SEEDBench2_Plus" else task + roots = (Path(self.target_path), Path(self.target_path) / "images") + candidates = [root / task / f"{identifier}{ext}" for root in roots for ext in (".jpg", ".jpeg", ".png")] + paths = list(dict.fromkeys(path.resolve() for path in candidates if path.is_file())) + if not paths: + raise FileNotFoundError(f"MMRBench image missing for {sample_id!r}; searched {candidates}") + if len(paths) != 1: + raise ValueError(f"MMRBench ambiguous image files for {sample_id!r}: {paths}") + return task, paths[0].resolve().as_posix() + + @staticmethod + def _joint_input_groups(frame: pd.DataFrame) -> np.ndarray: + """Connected components of identical decoded RGB images OR query text.""" + import hashlib + from PIL import Image + + if frame["prompt"].isna().any(): + raise ValueError("MMRBench joint grouping requires non-missing query text") + texts = frame["prompt"].astype(str).map(lambda text: " ".join(text.split())) + if (texts == "").any(): + raise ValueError("MMRBench joint grouping requires nonempty query text") + image_keys, cache = [], {} + for image_path in frame["image_path"]: + path = str(Path(image_path).resolve()) + if path not in cache: + with Image.open(path) as image: + rgb = image.convert("RGB") + cache[path] = hashlib.sha256(str(rgb.size).encode() + rgb.tobytes()).hexdigest() + image_keys.append(cache[path]) + parents = list(range(len(frame))) + def find(index): + while parents[index] != index: + parents[index] = parents[parents[index]] + index = parents[index] + return index + for keys in (image_keys, texts): + first = {} + for index, key in enumerate(keys): + if key in first: + a, b = find(index), find(first[key]) + parents[max(a, b)] = min(a, b) + else: + first[key] = index + return np.asarray([find(index) for index in range(len(frame))]) + + def split_in(self, dataset: Any) -> Tuple[pd.DataFrame, pd.DataFrame]: + protocol = self.args.get("dataset", {}).get("protocol", "legacy") + if protocol == "legacy": + logging.warning("[MMRBench] protocol=legacy: row-random splitting does not isolate repeated image/text inputs") + return super().split_in(dataset) + if protocol != "joint-grouped": + raise ValueError("MMRBench dataset.protocol must be 'legacy' or 'joint-grouped'") + from sklearn.model_selection import GroupShuffleSplit + + frame = dataset.copy() if isinstance(dataset, pd.DataFrame) else pd.DataFrame(dataset) + groups = self._joint_input_groups(frame) + splitter = GroupShuffleSplit(n_splits=1, train_size=self.args["dataset"]["split"]["ratios"]["train"], + random_state=self.args["seed"]) + train_ids, test_ids = next(splitter.split(frame, groups=groups)) + logging.info("[MMRBench] protocol=joint-grouped seed=%s groups=%d train=%d test=%d; not per-task stratified", + self.args["seed"], len(np.unique(groups)), len(train_ids), len(test_ids)) + return frame.iloc[train_ids].reset_index(drop=True), frame.iloc[test_ids].reset_index(drop=True) + def download(self) -> str: """ Download `images.tar.gz` and `MMR-Bench.csv` to `self.target_path`, @@ -630,59 +710,52 @@ def process(self, dataset: Any) -> Any: 9: "GeminiFlash2-5" } - model_list = [model_dict[i] for i in range(len(model_dict))] - logging.info(f"[utils.data.py] Model List: {model_list}") - - data_dir = self.target_path - - csv_files = os.path.join(data_dir, "MMR-Bench.csv") - - df = pd.read_csv(csv_files) - - def get_image_path(dataset_idx): - match = re.search(r'^(.*)_([^_]+)$', dataset_idx) - subdir = match.group(1) - qid = match.group(2) - if subdir == "SEEDBench2_Plus": - subdir = "SEEDBenchv2Plus" - - for ext in ['.jpg', '.jpeg', '.png']: - img_path = f"{subdir}/{qid}{ext}" - full_path = f"{data_dir}/{img_path}" - if os.path.exists(full_path): - return subdir, full_path.replace('\\', '/') - - eval_name, image_path = zip(*df['dataset_idx'].apply(get_image_path)) - df['image_path'] = image_path - df['eval_name'] = eval_name - - logging.info(f"[utils.data.py] Combined DataFrame shape: {df.shape}") - - new_columns = ['sample_id', 'prompt', 'answer', 'eval_name', 'image_path'] - for model_id,model_name in model_dict.items(): - new_columns.append(f'model_{model_id}_performance') - new_columns.append(f'model_{model_id}_cost') - new_columns.append(f'model_{model_id}_prediction') - - new_df = pd.DataFrame(columns=new_columns) - column_mapping = { - 'sample_id': 'dataset_idx', - 'prompt': 'question', - 'answer': 'answer', - 'eval_name': 'eval_name', - 'image_path': 'image_path' - } - for new_col, old_col in column_mapping.items(): - new_df[new_col] = df[old_col] - for model_id, model_name in model_dict.items(): - perf_col = f'{model_name}_correct' - cost_col = f'{model_name}_cost' - pred_col = f'{model_name}_prediction' - new_df[f'model_{model_id}_performance'] = df[perf_col] - new_df[f'model_{model_id}_cost'] = df[cost_col] - new_df[f'model_{model_id}_prediction'] = df[pred_col] - logging.info(f"[utils.data.py] Final DataFrame shape: {new_df.shape}") - return model_list, new_df + candidates = list(model_dict.values()) + policy = self.args.get("dataset", {}).get("candidate_policy", "error") + if policy not in {"error", "complete-models"}: + raise ValueError("MMRBench dataset.candidate_policy must be 'error' or 'complete-models'") + df = pd.read_csv(Path(self.target_path) / "MMR-Bench.csv") + required = {"dataset_idx", "question", "answer"} + required.update(f"{model}_{suffix}" for model in candidates for suffix in ("correct", "cost", "prediction")) + missing = required - set(df.columns) + if missing: + raise ValueError(f"MMRBench missing source columns: {sorted(missing)}") + if df.empty or df.dataset_idx.isna().any() or df.dataset_idx.duplicated().any(): + raise ValueError("MMRBench requires nonempty data with unique non-missing query IDs") + quality, costs, self.model_coverage = {}, {}, {} + for model in candidates: + quality[model] = self._quality_values(df[f"{model}_correct"], model) + try: + costs[model] = pd.to_numeric(df[f"{model}_cost"], errors="raise").to_numpy(dtype=np.float64, na_value=np.nan) + except (ValueError, TypeError) as exc: + raise ValueError(f"MMRBench {model}: invalid monetary costs") from exc + if np.isinf(costs[model]).any() or (costs[model] < 0).any(): + raise ValueError(f"MMRBench {model}: costs must be finite and nonnegative when present") + self.model_coverage[model] = int((np.isfinite(quality[model]) & np.isfinite(costs[model])).sum()) + self.excluded_models = [model for model in candidates if self.model_coverage[model] != len(df)] + if self.excluded_models and policy == "error": + counts = {model: self.model_coverage[model] for model in self.excluded_models} + raise ValueError(f"MMRBench incomplete candidate observations (of {len(df)}): {counts}; " + "select candidate_policy='complete-models' explicitly to retain complete candidates without dropping queries") + if self.excluded_models: + logging.warning("[MMRBench] candidate_policy=complete-models excludes incomplete candidates %s; " + "coverage=%s of %d; no queries dropped or missing cells imputed", + self.excluded_models, {m: self.model_coverage[m] for m in self.excluded_models}, len(df)) + model_list = [model for model in candidates if model not in self.excluded_models] + if len(model_list) < 2: + raise ValueError("MMRBench requires at least two complete candidate models") + locations = [self._image_location(value) for value in df.dataset_idx] + result = pd.DataFrame({"sample_id": df.dataset_idx, "prompt": df.question, "answer": df.answer, + "eval_name": [item[0] for item in locations], "image_path": [item[1] for item in locations]}) + for index, model in enumerate(model_list): + result[f"model_{index}_performance"] = quality[model] + result[f"model_{index}_cost"] = costs[model] + # Keep the historical field and expose the BaseRouter/EA-RAM + # response contract as an identical alias, never a fabricated answer. + result[f"model_{index}_prediction"] = df[f"{model}_prediction"] + result[f"model_{index}_response"] = df[f"{model}_prediction"] + logging.info("[MMRBench] candidate_policy=%s models=%s rows=%d", policy, model_list, len(result)) + return model_list, result def split_out(self, dataset: Any) -> Any: ds = dataset