Skip to content

fix(distributed): split -n and --rate across the workers - #289

Merged
kurok merged 1 commit into
mainfrom
fix/distributed-shard-load
Sep 19, 2026
Merged

kurok merged 1 commit into
mainfrom
fix/distributed-shard-load

Conversation

@kurok

@kurok kurok commented Sep 19, 2026 •

Copy link
Copy Markdown
Owner

Closes #238

The bug

Only feeder rows were sharded. num_requests, rate and rate_ramp went to every node unchanged, so:

  • --master --expect-workers 3 -n 1000 ran 3,000 requests
  • --rate 100 drove 300 rps

while the README said "The master splits the workload evenly across workers". A capacity or SLO gate applied three times the load it was asked for, and nothing in the output said so.

The fix

num_requests is divided with the remainder going to the low-numbered nodes, so the shards sum to exactly -n (1000 across 3 → 334/333/333, and no node does more than one request more than any other). rate and rate_ramp are divided by the node count.

connections is deliberately not sharded — it sizes that node's own pool, not a share of the cluster's work — and neither is duration, since every node runs the whole window.

The split is logged when the config goes out:

Master: splitting the load across 3 workers: -n 1,000 -> 334/333/333, --rate 100 -> 33.3333 rps each

Together with #236 this closes the loop on distributed load shape: the profile now reaches the workers, and the base rate it scales is now the node's share rather than the cluster total.

A test that asserted the bug

tests/test_pywrkr.py::test_master_worker_multiple asserted total_requests >= 20 for -n 10 across two workers, with the comment "Each worker sends at least num_requests, so merged should have >= 2x". That is the defect, written down as the expectation. It now asserts the cluster total is 10, with a note saying what it used to claim and why.

Tests

  • tests/test_audit_distributed.py::TestMasterShardsLoad::test_master_shards_num_requests_and_rate_across_workers — two real sockets read their config off the wire. Asserts num_requests sums to 10, each rate is 3.0, and connections is untouched. Pre-fix: AssertionError: 20 != 10.
  • tests/test_distributed.py::TestLoadSharding — six cases on the arithmetic: shards sum to the total for 2/3/4/7 workers and stay within one of each other, rates sum back to the target, connections/duration are per node, more workers than requests gives the surplus nodes 0, a single worker is returned untouched, and a duration-only run has nothing to shard.

Checks

ruff format/check clean, mypy clean, 1920 passed.

@codecov

codecov Bot commented Sep 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.29630% with 1 line in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/pywrkr/distributed.py 96.29% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Only feeder rows were sharded. num_requests, rate and rate_ramp went to
every node unchanged, so --master --expect-workers 3 -n 1000 ran 3,000
requests and --rate 100 drove 300 rps -- while the README said "the
master splits the workload evenly across workers". A capacity or SLO
gate applied three times the load it was asked for, and nothing said so.

num_requests is now divided with the remainder going to the low-numbered
nodes, so the shards sum to exactly -n (1000/3 -> 334/333/333). rate and
rate_ramp are divided by the node count. connections stays per node --
it sizes that node's own pool, not a share of the work -- and so does
duration, since every node runs the whole window. The split is logged
when the config goes out.

tests/test_pywrkr.py::test_master_worker_multiple asserted
`total_requests >= 20` for `-n 10` across two workers, which encoded the
bug. It now asserts the cluster total is 10.

Closes #238

Signed-off-by: Yurii Riabikov <22548029+kurok@users.noreply.github.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kurok
kurok force-pushed the fix/distributed-shard-load branch from 5828b73 to 39bfeec Compare September 19, 2026 12:38
@kurok
kurok merged commit 63d9cf4 into main Sep 19, 2026
18 checks passed
@kurok
kurok deleted the fix/distributed-shard-load branch September 19, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(distributed): -n and --rate are sent whole to every worker, contradicting the README

1 participant