Skip to content

test(distributed): a worker that has not reported yet is not reporting - #293

Merged
kurok merged 1 commit into
mainfrom
test/streaming-worker-count-race
Sep 19, 2026
Merged

kurok merged 1 commit into
mainfrom
test/streaming-worker-count-race

Conversation

@kurok

@kurok kurok commented Sep 19, 2026 •

Copy link
Copy Markdown
Owner

Fourth CI flake this session. It turned PR #291 red on macos-latest 3.13.

Root cause

test_the_master_streams_the_cluster_during_the_run asserted:

self.assertTrue(all(e["workers"] == "2" for e in intervals), master_exports)

The first interval snapshot can legitimately land before a worker has sent its first progress message. macos-latest 3.13 produced exactly that:

workers '1' (total 60), then '2' (126), '2' (126), '2' (290), final '2' (418)

That is a correct cluster view during ramp-up — a worker that has not reported yet is not reporting — and the assertion called it a failure.

The fix

Assert what is actually true: the view converges.

counts = [int(e["workers"]) for e in intervals]
self.assertLessEqual(max(counts), 2, master_exports)   # never invents a third
self.assertEqual(counts[-1], 2, master_exports)        # both counted by the end
self.assertEqual(int(master_exports[-1]["workers"]), 2, master_exports)  # final is complete

The point of the original assertion — that the master merges both workers rather than one — is preserved by the last two checks. Ramp-up is not a defect.

Unrelated to #291's change: the workers_reported tags added there go on the end-of-run report config, while this count comes from the progress aggregator. #291 only shifted the timing that exposed it.

Checks

ruff format/check clean, 1920 passed. Ran the test twice locally.

test_the_master_streams_the_cluster_during_the_run asserted that every
interval snapshot counted both workers. The first snapshot can
legitimately land before a worker has sent its first progress message,
and macos-latest 3.13 produced exactly that -- workers 1, then 2, 2, 2 --
so a correct cluster view failed the build.

The claim worth making is that the view converges: at most two workers
are ever counted, the last interval has both, and so does the final
snapshot. Ramp-up is not a defect.

Signed-off-by: Yurii Riabikov <22548029+kurok@users.noreply.github.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@kurok
kurok merged commit dcbda0a into main Sep 19, 2026
17 checks passed
@kurok
kurok deleted the test/streaming-worker-count-race branch September 19, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant