Repository navigation
fix(cli): --worker reports whether the run actually happened - #292
Merged
Merged
Conversation
run_worker_node returned None on every failure path -- auth timeout, wrong message type, config timeout, HTTP/2 refusal, master closed -- and the CLI did `asyncio.run(...); sys.exit(0)`. systemd, a Kubernetes Job or a Jenkins agent could not tell a completed benchmark from a five-minute wait for a master that never appeared. ConnectionRefusedError escaped as a raw traceback, and Ctrl-C printed a KeyboardInterrupt traceback. run_worker_node returns 0 only when a benchmark ran and its results reached the master; every give-up path returns 1. The connect is wrapped so an unreachable master is a logged error rather than a traceback, and the CLI exits 130 on Ctrl-C. Closes #240 Signed-off-by: Yurii Riabikov <22548029+kurok@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #240
The bug
run_worker_nodereturnedNoneon every failure path — auth timeout, wrong message type, config timeout, HTTP/2 refusal, master closed — and the CLI did:So systemd, a Kubernetes Job or a Jenkins agent could not tell a completed benchmark from a five-minute wait for a master that never appeared. Both were a success.
Two smaller edges came with it:
ConnectionRefusedErrorescaped as a raw traceback, and Ctrl-C printed aKeyboardInterrupttraceback at whoever pressed it.The fix
run_worker_nodereturnsint:0only when a benchmark ran and its results reached the master; every give-up path returns1. All eleven barereturns becamereturn 1, and the contract is in the docstring.The connect is wrapped, so an unreachable master is a logged error naming the address rather than a traceback. The CLI exits
130on Ctrl-C — the shell's convention for SIGINT.The return type changes from
Nonetoint, which is backwards-compatible for the existing callers that ignore it (pywrkr.run_worker_nodeis public API).Tests
tests/test_distributed.py::TestWorkerExitCode, all four verified against the unpatched source:test_worker_cli_exits_nonzero_when_master_sends_wrong_message— the issue's named case. Pre-fixAssertionError: None != 1.test_worker_exits_nonzero_when_the_master_is_not_listening— pre-fix the test dies onConnectionRefusedError: [Errno 61], which is the traceback the issue describes.test_worker_exits_nonzero_when_the_master_hangs_up.test_worker_exits_zero_after_a_completed_run— the success path still says success, and the master really did receive aresultmessage. This one matters: returning1everywhere would also pass the first three.The success test waits on an
asyncio.Eventset by the server handler rather than asserting straight afterrun_worker_nodereturns — the handler is its own task, and my first version raced it into anIndexError.Checks
ruff format/check clean, mypy clean,
1917 passed.