Skip to content

🐛 [v6] contributor relay: send a finished verdict before exiting on SIGTERM - #17235

Merged
hivecommons-hive[bot] merged 2 commits into
hivecommons:v6from
Danathar:fix/v6-relay-sigterm-verdict
Nov 8, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
hivecommons:v6from
Danathar:fix/v6-relay-sigterm-verdict

Conversation

@Danathar

@Danathar Danathar commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Summary

A contributor relay that is stopped a second or two after its agent prints a verdict now reports that task to the hub before it exits. Before this, the task_complete was lost: the hub booked the run as abandoned and offered the issue again. The relay now waits up to 5 seconds for a finished task's frame to be written. It takes no new work while it waits. The container entrypoint now waits for the relay before it ends the container.

What went wrong

Seen live on 2026-11-08 on a Danathar hive. The relay logged Task ct-Danathar/arch-bootc-627-1794141490 completed — signal=verdict (HIVE_VERDICT: no_work_needed) and then Detected no_work_needed verdict .... About a second later a docker stop printed Shutting down... and the process was gone. The hub never got task_complete. It recorded "abandoned: connection lost with the task still held", and the issue was worked again 13 minutes later. The same thing happened to ct-Danathar/atomic-image-builder-834-1794141933.

Two things combined:

  1. The relay exited with the frame still queued. Everything from the "completed" log line to the send in finishCurrentTask is synchronous. The label calls use execSync, and stopping the agent (quitLiveCLI) sleeps for at least 3 seconds. A signal that lands in that window is handled only after the send. The old handler then called process.exit(0) at once, and process.exit does not wait for a socket write that is still queued.
  2. The container ended before the relay could finish. bin/contributor-agent.sh is the container's PID 1. Its cleanup() signalled the relay, killed tmux and exited straight away. When PID 1 exits, everything else in the container is SIGKILLed, so the relay never got to run its handler at all.

What changes

  • task_complete and task_failed frames are now sent with the ws write callback and tracked until it fires. These are the frames that report a finished task.
  • On SIGTERM or SIGINT the relay first sets a shutting-down flag. If one of those frames is still being written, it waits for the write, then runs cleanup() and exits. If nothing is pending, it exits at once, as before.
  • The wait is capped by HIVE_RELAY_SHUTDOWN_FLUSH_MS, which defaults to 5000 ms. Docker and podman SIGKILL 10 seconds after the stop signal. The handler only runs once the synchronous finish has returned, which has already used about 3 of those seconds, so 5 seconds keeps the whole shutdown inside 10. A healthy socket finishes the write in milliseconds; the cap only matters when the socket is stuck.
  • While shutting down, the relay sends no ready (checked in sendTo and in requestWork), and hands back any task_assign as a task_failed with failure_kind: environment.
  • A second signal during the wait exits at once.
  • contributor-agent.sh now waits for the relay before it kills tmux and exits. tmux is killed last because the relay's own shutdown still stops the agent in it.
  • Multi-hub relays: the frame goes to the task's own hub through the existing send(), and the wait tracks the write on whichever hub it went to.

What "pending" means: a task_complete or task_failed that has been handed to the socket but whose write callback has not fired yet. Nothing can be half-way through finishCurrentTask when the handler runs, because the relay has no async work in that path, so the handler can only see the finish either not started or already sent. There is no double send: the handler never builds or sends a frame itself. It only waits for the write that is already under way.

One limit: the ready that finishCurrentTask sends in the same synchronous step goes out before the handler can run. Any assignment that answers it during the wait is handed back.

Out of scope: how the hub books abandonment, and the runtime's stop timeout.

Files

  • bin/contributor-relay.js: tracks the frames, adds the shutdown handler, and adds the ready and task_assign guards.
  • bin/contributor-agent.sh: cleanup() waits for the relay.
  • bin/contributor-relay.test.js: five new tests.
  • src/docs/contributor-relay.md, src/docs/env-vars.md: document the shutdown wait and the new variable.
  • changelog.d/fixed-relay-sigterm-verdict.md

Tests

  • SIGTERM during a verdict finish waits for task_complete to be written before exiting. This runs a short-lived child relay with a fake socket that completes writes 300 ms later. It raises a real SIGTERM, then finishes the task in the same synchronous turn, which is the live order. It fails on the v6 tip (fba3c4609): the child logs QUEUED task_complete and then exits after 4 ms, with no WRITTEN task_complete. With this change it logs WRITTEN task_complete and then exits after about 300 ms.
  • SIGTERM with no finished task to report still exits at once. A queued progress frame does not hold the exit. Passes before and after.
  • The shutdown deadline, in-process: a write that never completes gives up at the deadline. During the wait, no ready goes out and a late task_assign is handed back.
  • A second signal during the wait exits at once.
  • The default deadline is below 10 s.

On this host, node bin/contributor-relay.test.js had 66 failures on the unmodified tip that depend on the host. There are 66 after this change, and they are the same tests. The 5 new tests pass. The docs citation and link checks pass, and so does go test ./pkg/config/ (env-var doc parity).

🤖 Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

A relay stopped a second or two after its agent's HIVE_VERDICT lost the
task_complete. The finish is synchronous (execSync label calls, and
quitLiveCLI sleeps about 3 s), so the signal handler only ran after the
frame was handed to the socket, then called process.exit(0) with the write
still queued. In a container the entrypoint (PID 1) also exited right after
signalling the relay, which SIGKILLs it. The hub booked the run as
"abandoned: connection lost" and offered the issue again.

- task_complete and task_failed are sent with a write callback and tracked
  until it fires.
- SIGTERM/SIGINT sets shuttingDown, then waits for those writes, bounded by
  HIVE_RELAY_SHUTDOWN_FLUSH_MS (default 5000 ms, below the 10 s stop
  timeout), then runs cleanup() and exits. With nothing pending it exits at
  once, as before. A second signal exits at once.
- While shutting down: no ready, and a task_assign is handed back.
- contributor-agent.sh waits for the relay before stopping tmux and exiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17025 🌱 Forward-merge v5 into v6 (1 commits) same files src/docs/env-vars.md

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the v6 tree (relay finishCurrentTask, cleanup, stopAgentForTaskExit/quitLiveCLI, and the entrypoint's signal path). This looks correct to me.

What I verified:

  • bin/contributor-relay.js:8513-8537 — task_complete is sent after stopAgentForTaskExit() and before currentTask = null, all synchronously. So when the signal handler finally runs, currentTask is already null and cleanup() (:10364) skips the stopAgentForTaskExit branch — the flush wait is not followed by another ~3 s of execSync sleeps. The 5 s deadline arithmetic in the comment holds for this path.
  • sendHeldFrame adds the entry before ws.send, and done is idempotent via Set.delete, so a synchronous or double callback cannot double-settle. A task_failed decline emitted during the wait is itself a held frame and waitForHeldFrames re-enters for it — the test at "gives up on an unwritten task_complete at the deadline, and takes no work meanwhile" covers exactly that.
  • requestWork (:2377) and the ready choke point in sendTo (:2526) both honour shuttingDown, so a ready that the just-finished task queued cannot leave during the wait.
  • bin/contributor-agent.sh:842 — wait "$RELAY_PID" inside the trap is on a direct background child (:695-696), and the main loop already re-waits after a trapped signal, so the trap's wait returns when the relay really exits. tmux is killed after it, as the comment says.

One pre-existing note, not introduced here: the synchronous finish can take longer than the "about 3 s" the design comment assumes when the pane does not return to a shell — quitLiveCLI (:5305) can spend 1 + 2 + up to 5 (waitForPaneShell) + 0.5 + up to 5 s before the handler runs, which already exceeds the runtime's 10 s. This PR does not make that worse, and the flush wait is still strictly better than the old immediate process.exit; just worth knowing the deadline only protects the common case.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness review (read at head 6135bc4). The flush mechanism itself checks out: sendHeldFrame registers before ws.send (bin/contributor-relay.js:2466-2481), waitForHeldFrames re-loops for a decline sent during the wait (:2494-2498), cleanup() is cheap on this path because finishCurrentTask has already nulled currentTask (:8536, :10391), and contributor-agent.sh:842 now actually gives the relay that time. One advisory:

The shutdown decline books a real failure against the contributor and the issue (bin/contributor-relay.js:9887-9892, in-scope, low)

The late task_assign is handed back as task_failed / failure_kind: environment. On the hub, task_failed is a failure regardless of kind: recordTaskFailureForTask (per-issue cooldown + quarantine counter) and bookReclaimCooldown (the long per-contributor exclusion from #13093) both run before the kind is even read (src/pkg/dashboard/contribute_ws.go:4310-4321), and an assign→failed sub-minute also counts toward the fast-failure streak (contribute_failure_streak.go:22-24). A docker stop for a routine restart that races an assignment therefore keeps the restarted contributor off that issue and nudges it toward a claim pause — for something that is not a task failure. The hub already has a no-penalty frame for exactly this moment: task_declined at pre-credential time releases the lease "without recording failure cooldown or trust impact" (contribute_ws.go:3460-3505), and the relay already uses it for the quota guard with a task_failed fallback for hubs without quota_preflight_v1 (:9958-9972). Suggest the same shape here; task_declined would also need to join SHUTDOWN_HELD_FRAMES so the wait tracks it:

-const SHUTDOWN_HELD_FRAMES = new Set(['task_complete', 'task_failed']);
+const SHUTDOWN_HELD_FRAMES = new Set(['task_complete', 'task_failed', 'task_declined']);
...
       if (shuttingDown) {
         console.log(`Declining ${taskKey(msg)} from ${hub.url} — the relay is shutting down`);
-        sendTo(hub, { type: 'task_failed', seq: nextSeq(), task_id: msg.task_id, reason: 'contributor relay is shutting down', failure_kind: 'environment' });
+        if (hubSupportsQuotaPreflight(hub)) {
+          // Local condition, not a task failure: a pre-credential decline releases
+          // the lease without a failure cooldown or reclaim cooldown (#6833).
+          sendTo(hub, { type: 'task_declined', seq: nextSeq(), task_id: msg.task_id, task_gen: msg.task_gen, reason: 'local_capacity_guard', message: 'contributor relay is shutting down' });
+        } else {
+          sendTo(hub, { type: 'task_failed', seq: nextSeq(), task_id: msg.task_id, task_gen: msg.task_gen, reason: 'contributor relay is shutting down', failure_kind: 'environment' });
+        }
         break;
       }

The window is narrow (the healthy-socket wait is milliseconds), so this does not block the fix. Not run locally: the 5 new tests; I read them and they exercise the stated paths.

Evidence bundle `#172356135bc4` sha256:a336c04d0e002633eeef0d740a0a975856d566a3128eecdaeda47b2c5f9d9bd5 — download from the dashboard or `hivectl review evidence #17235`.

@hivecommons-hive
hivecommons-hive Bot merged commit 38f29f8 into hivecommons:v6 Nov 8, 2026
68 of 69 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at 6135bc4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant