[kurok]

Closes #237

The bug

_collect_one caught TimeoutError / ConnectionError / OSError, but _deserialize_stats raises ValueError, AttributeError or KeyError on any non-conforming JSON — a status-code key that is not a number, a list where an object belongs, a missing "stats".

So one buggy worker turned the entire cluster run into a traceback, taking every other node's results with it. And since the control plane is unauthenticated unless --worker-secret is set, the payload need not even come from a worker anyone trusts.

Reproduced: _deserialize_stats({"status_codes": {"abc": 1}}) → ValueError: invalid literal for int() with base 10: 'abc', propagating straight out of run_master.

The fix

The deserialize is wrapped. A malformed result is logged against the worker that sent it and discarded; every other node's results are still merged and reported.

_deserialize_stats now validates the shapes it reads rather than trusting them, and every rejection names the field. That matters because the message is what the master logs, per worker:

status_codes key 'abc' is not a status code: invalid literal for int() with base 10: 'abc'

reads better in an operator's log than the bare invalid literal did. latencies must be an array, got str, rps_timeline entry must be an array, got int, and so on. True is rejected for a numeric field too — isinstance(True, int) is True in Python, and a bool in total_requests is a bug worth surfacing.

Tests

• tests/test_audit_distributed.py::TestMasterSurvivesBadWorker::test_master_survives_malformed_result_from_one_worker — two real sockets, one valid result of 250 requests and one {"status_codes": {"abc": 1}}. Asserts merged.total_requests == 250 and that the bad worker was reported. Pre-fix: ValueError: invalid literal for int() with base 10: 'abc' out of run_master, no result at all.
• tests/test_distributed.py::TestMalformedStats — seven malformed shapes as subtests, each asserting the message names its field, plus a round-trip test so the hardening did not break the valid path. All seven subtests fail pre-fix.

Checks

ruff format/check clean, mypy clean, 1913 passed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!