Repository navigation
🐛 [v6] contributor relay: send a finished verdict before exiting on SIGTERM - #17235
Conversation
A relay stopped a second or two after its agent's HIVE_VERDICT lost the task_complete. The finish is synchronous (execSync label calls, and quitLiveCLI sleeps about 3 s), so the signal handler only ran after the frame was handed to the socket, then called process.exit(0) with the write still queued. In a container the entrypoint (PID 1) also exited right after signalling the relay, which SIGKILLs it. The hub booked the run as "abandoned: connection lost" and offered the issue again. - task_complete and task_failed are sent with a write callback and tracked until it fires. - SIGTERM/SIGINT sets shuttingDown, then waits for those writes, bounded by HIVE_RELAY_SHUTDOWN_FLUSH_MS (default 5000 ms, below the 10 s stop timeout), then runs cleanup() and exits. With nothing pending it exits at once, as before. A second signal exits at once. - While shutting down: no ready, and a task_assign is handed back. - contributor-agent.sh waits for the relay before stopping tmux and exiting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
There was a problem hiding this comment.
Read the diff against the v6 tree (relay finishCurrentTask, cleanup, stopAgentForTaskExit/quitLiveCLI, and the entrypoint's signal path). This looks correct to me.
What I verified:
bin/contributor-relay.js:8513-8537—task_completeis sent afterstopAgentForTaskExit()and beforecurrentTask = null, all synchronously. So when the signal handler finally runs,currentTaskis already null andcleanup()(:10364) skips thestopAgentForTaskExitbranch — the flush wait is not followed by another ~3 s ofexecSyncsleeps. The 5 s deadline arithmetic in the comment holds for this path.sendHeldFrameadds the entry beforews.send, anddoneis idempotent viaSet.delete, so a synchronous or double callback cannot double-settle. Atask_faileddecline emitted during the wait is itself a held frame andwaitForHeldFramesre-enters for it — the test at "gives up on an unwritten task_complete at the deadline, and takes no work meanwhile" covers exactly that.requestWork(:2377) and thereadychoke point insendTo(:2526) both honourshuttingDown, so areadythat the just-finished task queued cannot leave during the wait.bin/contributor-agent.sh:842—wait "$RELAY_PID"inside the trap is on a direct background child (:695-696), and the main loop already re-waits after a trapped signal, so the trap'swaitreturns when the relay really exits. tmux is killed after it, as the comment says.
One pre-existing note, not introduced here: the synchronous finish can take longer than the "about 3 s" the design comment assumes when the pane does not return to a shell — quitLiveCLI (:5305) can spend 1 + 2 + up to 5 (waitForPaneShell) + 0.5 + up to 5 s before the handler runs, which already exceeds the runtime's 10 s. This PR does not make that worse, and the flush wait is still strictly better than the old immediate process.exit; just worth knowing the deadline only protects the common case.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. Review findings from Correctness review (read at head 6135bc4). The flush mechanism itself checks out: The shutdown decline books a real failure against the contributor and the issue ( The late -const SHUTDOWN_HELD_FRAMES = new Set(['task_complete', 'task_failed']);
+const SHUTDOWN_HELD_FRAMES = new Set(['task_complete', 'task_failed', 'task_declined']);
...
if (shuttingDown) {
console.log(`Declining ${taskKey(msg)} from ${hub.url} — the relay is shutting down`);
- sendTo(hub, { type: 'task_failed', seq: nextSeq(), task_id: msg.task_id, reason: 'contributor relay is shutting down', failure_kind: 'environment' });
+ if (hubSupportsQuotaPreflight(hub)) {
+ // Local condition, not a task failure: a pre-credential decline releases
+ // the lease without a failure cooldown or reclaim cooldown (#6833).
+ sendTo(hub, { type: 'task_declined', seq: nextSeq(), task_id: msg.task_id, task_gen: msg.task_gen, reason: 'local_capacity_guard', message: 'contributor relay is shutting down' });
+ } else {
+ sendTo(hub, { type: 'task_failed', seq: nextSeq(), task_id: msg.task_id, task_gen: msg.task_gen, reason: 'contributor relay is shutting down', failure_kind: 'environment' });
+ }
break;
}The window is narrow (the healthy-socket wait is milliseconds), so this does not block the fix. Not run locally: the 5 new tests; I read them and they exercise the stated paths. Evidence bundle `#172356135bc4` sha256:a336c04d0e002633eeef0d740a0a975856d566a3128eecdaeda47b2c5f9d9bd5 — download from the dashboard or `hivectl review evidence #17235`. |
Summary
A contributor relay that is stopped a second or two after its agent prints a verdict now reports that task to the hub before it exits. Before this, the
task_completewas lost: the hub booked the run as abandoned and offered the issue again. The relay now waits up to 5 seconds for a finished task's frame to be written. It takes no new work while it waits. The container entrypoint now waits for the relay before it ends the container.What went wrong
Seen live on 2026-11-08 on a Danathar hive. The relay logged
Task ct-Danathar/arch-bootc-627-1794141490 completed — signal=verdict (HIVE_VERDICT: no_work_needed)and thenDetected no_work_needed verdict .... About a second later adocker stopprintedShutting down...and the process was gone. The hub never gottask_complete. It recorded "abandoned: connection lost with the task still held", and the issue was worked again 13 minutes later. The same thing happened toct-Danathar/atomic-image-builder-834-1794141933.Two things combined:
sendinfinishCurrentTaskis synchronous. The label calls useexecSync, and stopping the agent (quitLiveCLI) sleeps for at least 3 seconds. A signal that lands in that window is handled only after the send. The old handler then calledprocess.exit(0)at once, andprocess.exitdoes not wait for a socket write that is still queued.bin/contributor-agent.shis the container's PID 1. Itscleanup()signalled the relay, killed tmux and exited straight away. When PID 1 exits, everything else in the container is SIGKILLed, so the relay never got to run its handler at all.What changes
task_completeandtask_failedframes are now sent with thewswrite callback and tracked until it fires. These are the frames that report a finished task.cleanup()and exits. If nothing is pending, it exits at once, as before.HIVE_RELAY_SHUTDOWN_FLUSH_MS, which defaults to 5000 ms. Docker and podman SIGKILL 10 seconds after the stop signal. The handler only runs once the synchronous finish has returned, which has already used about 3 of those seconds, so 5 seconds keeps the whole shutdown inside 10. A healthy socket finishes the write in milliseconds; the cap only matters when the socket is stuck.ready(checked insendToand inrequestWork), and hands back anytask_assignas atask_failedwithfailure_kind: environment.contributor-agent.shnow waits for the relay before it kills tmux and exits. tmux is killed last because the relay's own shutdown still stops the agent in it.send(), and the wait tracks the write on whichever hub it went to.What "pending" means: a
task_completeortask_failedthat has been handed to the socket but whose write callback has not fired yet. Nothing can be half-way throughfinishCurrentTaskwhen the handler runs, because the relay has no async work in that path, so the handler can only see the finish either not started or already sent. There is no double send: the handler never builds or sends a frame itself. It only waits for the write that is already under way.One limit: the
readythatfinishCurrentTasksends in the same synchronous step goes out before the handler can run. Any assignment that answers it during the wait is handed back.Out of scope: how the hub books abandonment, and the runtime's stop timeout.
Files
bin/contributor-relay.js: tracks the frames, adds the shutdown handler, and adds thereadyandtask_assignguards.bin/contributor-agent.sh:cleanup()waits for the relay.bin/contributor-relay.test.js: five new tests.src/docs/contributor-relay.md,src/docs/env-vars.md: document the shutdown wait and the new variable.changelog.d/fixed-relay-sigterm-verdict.mdTests
fba3c4609): the child logsQUEUED task_completeand then exits after 4 ms, with noWRITTEN task_complete. With this change it logsWRITTEN task_completeand then exits after about 300 ms.readygoes out and a latetask_assignis handed back.On this host,
node bin/contributor-relay.test.jshad 66 failures on the unmodified tip that depend on the host. There are 66 after this change, and they are the same tests. The 5 new tests pass. The docs citation and link checks pass, and so doesgo test ./pkg/config/(env-var doc parity).🤖 Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1