[kenkyl]

Problem

A prospect running msstats.py against a large production project got:

429 Maximum response size of 200000000 bytes reached. Consider querying less data by increasing the step or interval, using more filters and aggregations, or limiting the time duration.

The commands/calls query fetches 7 days of 60s rates for every instance × node × command in one request. On larger projects this exceeds Cloud Monitoring's 200 MB limit, which applies to the whole query, so paging doesn't help. After the error:

• msstats.py crashed in create_workbooks (the error path returned a tuple instead of a dict).
• memorystore.py swallowed the error and wrote a CSV with no throughput or command-category data, with no warning.

Changes

• Automatic query splitting: new msstats.list_time_series helper, also used by memorystore.py. On the "Maximum response size" 429 it halves the time window on a --step boundary and merges the results, oldest first. Projects that work today still make a single request. Other errors, including quota 429s, are re-raised unchanged.
• msstats.py
  • Error paths return {} instead of (project_id, {}), so a failed query no longer crashes the script.
  • A failed memory or max-memory query no longer reuses the commands query results.
• memorystore.py: every failed query is printed as an error instead of silently ignored. The CSV is still written exactly as before, and the script exits 1 if any query failed.
• README: the 429 note now says splitting is automatic, so --step and --duration are optional.

Behavior changes

• memorystore.py exits 1 (was 0) when any Monitoring query fails. The CSV contents are unchanged. Nothing in the repo reads the exit code (batch_run_memorystore.sh ignores it).
• msstats.py now reports the error and writes an empty workbook instead of crashing with a traceback. Its exit code is unchanged.

Testing

• pytest test_msstats.py: 36 passed (9 new).
• black --check *.py: clean.
• Each new test was checked by reverting its fix and confirming the test fails.
• Not yet run against a live GCP project large enough to trigger the 429.

Out of scope (follow-ups)

• get_project_from_service_account_and_authenticate returns before setting GOOGLE_APPLICATION_CREDENTIALS, so service-account JSON files are never actually used.
• The README batch section lists ./batch_run_msstats.sh twice (the second should be batch_run_memorystore.sh).

🤖 Generated with Claude Code

[kenkyl]

Fix for large deployments that hit Cloud Monitoring's 200MB response limit. When a query hits the limit, the script now automatically splits its time range in half, and keeps splitting until each piece fits, then merges the results. Smaller deployments are unaffected and still make a single request.

Also fixes error handling in both scripts: msstats.py no longer crashes after a failed query, and memorystore.py now reports failed queries instead of silently writing a CSV with missing data.

Potential impact: memorystore.py now exits with code 1 if any query fails (it previously always exited 0). The CSV output is unchanged. Nothing in the repo checks the exit code, but flag it if you have scripts that do.

Changes are in msstats.py, memorystore.py, README.md, and test_msstats.py (9 new tests). Tested with mocks only so far; not yet verified against a live project large enough to hit the limit.

[ajGingrich]

P2 follow-up: the XLSX export in msstats.py still calls client.list_time_series directly for memory usage and capacity, so those queries don't get the new automatic splitting. If either response is too large, the export can be missing memory columns. An error fetching a later page can also still crash the run because iteration happens outside the try block. Routing both queries through the new helper would cover these cases.

I reproduced both cases with simulated responses. This is a pre-existing gap, so I don't see it as a blocker for the commands-query fix in this PR; the follow-up is tracked in #27.

Validation: all 36 existing tests passed, along with additional simulated checks for split CSV/XLSX exports across Redis, Valkey, and Redis Cluster. Live queries against the configured GCP project succeeded but returned no Memorystore data in the one-hour window, so recovery from a real oversized response is still unverified.