Issues

• Tau subsampling can drop complete rows or divide by zero

  google-research/mt-metrics-eval#31 · Filter by author sylvesterkaczmarek opened on Oct 4, 2026

• PairwiseConfidenceError(filter_nones=True) reshapes a None-filtered flat vector into per-system rows, misaligning segments when missing gold scores are not column-aligned

  google-research/mt-metrics-eval#30 · Filter by author shaurya416 opened on Sep 30, 2026

• Sample ignores seed zero and uses global randomness for stratified allocation

  google-research/mt-metrics-eval#28 · Filter by author sylvesterkaczmarek opened on Sep 28, 2026

• HTTP Error 403: Forbidden when running mtme --download

  google-research/mt-metrics-eval#27 · Filter by author marialcasimiro opened on Mar 20, 2026
  4

• Initial Unit Tests Fail

  google-research/mt-metrics-eval#26 · Filter by author orfeas-menis opened on Dec 4, 2025
  1

• Automatically download data if not present

  google-research/mt-metrics-eval#25 · Filter by author zouharvi opened on Apr 7, 2025

• WMT24 ESA spans are missing

  google-research/mt-metrics-eval#22 · Filter by author zouharvi opened on Feb 18, 2025

• No module named mt_metrics_eval.score_mqm

  google-research/mt-metrics-eval#17 · Filter by author Mohamed-E-Fayed opened on Jul 7, 2024

• Missing de-en sentence-level data

  google-research/mt-metrics-eval#15 · Filter by author LeonardoEmili opened on Apr 9, 2024

• Random chance performance for Pairwise Accuracy in "Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration"

  google-research/mt-metrics-eval#14 · Filter by author linzhiqiu opened on Jan 21, 2024