Repository navigation
fix(distributed): split -n and --rate across the workers - #289
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Only feeder rows were sharded. num_requests, rate and rate_ramp went to every node unchanged, so --master --expect-workers 3 -n 1000 ran 3,000 requests and --rate 100 drove 300 rps -- while the README said "the master splits the workload evenly across workers". A capacity or SLO gate applied three times the load it was asked for, and nothing said so. num_requests is now divided with the remainder going to the low-numbered nodes, so the shards sum to exactly -n (1000/3 -> 334/333/333). rate and rate_ramp are divided by the node count. connections stays per node -- it sizes that node's own pool, not a share of the work -- and so does duration, since every node runs the whole window. The split is logged when the config goes out. tests/test_pywrkr.py::test_master_worker_multiple asserted `total_requests >= 20` for `-n 10` across two workers, which encoded the bug. It now asserts the cluster total is 10. Closes #238 Signed-off-by: Yurii Riabikov <22548029+kurok@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kurok
force-pushed
the
fix/distributed-shard-load
branch
from
September 19, 2026 12:38
5828b73 to
39bfeec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #238
The bug
Only feeder rows were sharded.
num_requests,rateandrate_rampwent to every node unchanged, so:--master --expect-workers 3 -n 1000ran 3,000 requests--rate 100drove 300 rpswhile the README said "The master splits the workload evenly across workers". A capacity or SLO gate applied three times the load it was asked for, and nothing in the output said so.
The fix
num_requestsis divided with the remainder going to the low-numbered nodes, so the shards sum to exactly-n(1000 across 3 →334/333/333, and no node does more than one request more than any other).rateandrate_rampare divided by the node count.connectionsis deliberately not sharded — it sizes that node's own pool, not a share of the cluster's work — and neither isduration, since every node runs the whole window.The split is logged when the config goes out:
Together with #236 this closes the loop on distributed load shape: the profile now reaches the workers, and the base rate it scales is now the node's share rather than the cluster total.
A test that asserted the bug
tests/test_pywrkr.py::test_master_worker_multipleassertedtotal_requests >= 20for-n 10across two workers, with the comment "Each worker sends at least num_requests, so merged should have >= 2x". That is the defect, written down as the expectation. It now asserts the cluster total is 10, with a note saying what it used to claim and why.Tests
tests/test_audit_distributed.py::TestMasterShardsLoad::test_master_shards_num_requests_and_rate_across_workers— two real sockets read their config off the wire. Assertsnum_requestssums to 10, eachrateis 3.0, andconnectionsis untouched. Pre-fix:AssertionError: 20 != 10.tests/test_distributed.py::TestLoadSharding— six cases on the arithmetic: shards sum to the total for 2/3/4/7 workers and stay within one of each other, rates sum back to the target,connections/durationare per node, more workers than requests gives the surplus nodes0, a single worker is returned untouched, and a duration-only run has nothing to shard.Checks
ruff format/check clean, mypy clean,
1920 passed.