fix(eval): chunk + paginate rolling_mean GetMetricData (>500 MetricDataQueries cap — Sat-SF EvalRollingMean ERROR)

[cipher813]

Root cause

evals/rolling_mean.py issued a single cw.get_metric_data call with one MetricDataQuery per (rubric × dimension) combo. AWS GetMetricData hard-caps MetricDataQueries at 500 per call. As the rubric/dimension matrix grew, len(queries) exceeded 500, so the call raised:

ValidationError: The collection MetricDataQueries must not have a size greater than 500.

This made the Saturday-SF EvalRollingMean state return {"status":"ERROR"}, contributing to the terminal PipelineFailure on the 2026-05-17 weekend run.

Fix

Introduce a small private helper:

def _get_metric_data_all(cw, queries: list[dict], start: datetime, end: datetime) -> list[dict]

• Chunks queries into ≤500-query batches via the named constant _GET_METRIC_DATA_MAX_QUERIES = 500.
• For each chunk, follows NextToken pagination to exhaustion (response.get("NextToken") → re-call with NextToken=...).
• Flat-merges all MetricDataResults across chunks/pages into one list.

The flat merge is correct because every query Id (m{idx} over the whole combos list) is globally unique, so the downstream by_id = {r["Id"]: r for r in ...} mapping is unchanged. The ≤500 case is behaviourally identical (single chunk, paginated only if AWS returns a NextToken).

AWS 500 cap citation

GetMetricData MetricDataQueries collection size is hard-limited to 500 per request — see the AWS GetMetricData API reference.

Test additions

+7 tests in tests/test_eval_rolling_mean.py (new TestGetMetricDataAll class), mirroring the existing MagicMock-CloudWatch style:

• cap constant is 500
• >500 queries split into multiple calls, each ≤500, all results merged in order
• NextToken pagination followed within a chunk
• ≤500 single-call path unchanged
• end-to-end >500-combo compute_and_emit_4w_mean run maps every result back to its combo via the unique m{idx} Id scheme

Full research suite: 1343 passed.

🤖 Generated with Claude Code