fix(opencode): reap fatal provider process groups

.github/workflows/fatal-provider-process-group-quality-ci.yml

@@ -0,0 +1,63 @@
+name: Fatal Provider Process Group Quality CI
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - ".github/workflows/fatal-provider-process-group-quality-ci.yml"
+      - "scripts/ci/run_opencode_review_model_pool.sh"
+      - "tests/test_opencode_model_pool_runner.py"
+      - "docs/doctoring/opencode-process-group-termination.md"
+      - "CHANGELOG.md"
+
+permissions:
+  contents: read
+
+concurrency:
+  group: fatal-provider-process-group-quality-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  exact-head-contract:
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: ubuntu-24.04
+    timeout-minutes: 10
+    steps:
+      - name: Checkout exact source revision
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ github.event.pull_request.head.sha || github.sha }}
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.14"
+
+      - name: Install exact hash-verified test runner dependencies
+        env:
+          PIP_DISABLE_PIP_VERSION_CHECK: "1"
+          PIP_NO_INPUT: "1"
+        shell: bash --noprofile --norc -e -o pipefail {0}
+        run: |
+          cat >"${RUNNER_TEMP}/fatal-provider-quality-requirements.txt" <<'EOF'
+          iniconfig==2.1.0 --hash=sha256:9deba5723312380e77435581c6bf4935c94cbfab9b1ed33ef8d238ea168eb760
+          packaging==26.2 --hash=sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e
+          pluggy==1.6.0 --hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746
+          pygments==2.20.0 --hash=sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176
+          pytest==9.1.1 --hash=sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c
+          EOF
+          python -m pip install \
+            --only-binary=:all: \
+            --require-hashes \
+            -r "${RUNNER_TEMP}/fatal-provider-quality-requirements.txt"
+
+      - name: Verify fail-first and exact-head contracts
+        shell: bash --noprofile --norc -e -o pipefail {0}
+        run: |
+          test "$(git rev-parse HEAD)" = "${{ github.event.pull_request.head.sha || github.sha }}"
+          python -m pytest -q tests/test_opencode_model_pool_runner.py \
+            -k 'fatal_provider_error_kills_hung_opencode_run_early or delisted_openrouter_model_error_kills_hung_run_early'
+          python -m compileall -q tests/test_opencode_model_pool_runner.py
+          bash -n scripts/ci/run_opencode_review_model_pool.sh
+          git diff --exit-code

CHANGELOG.md

@@ -215,6 +215,7 @@ Semantic Versioning where the repository publishes a release.
 
 ### Fixed
 
+- Launch each OpenCode model attempt with `setsid` and signal the negative process-group id on a fatal provider event so descendant `opencode` processes cannot keep coverage pipes open after the launcher exits.
 - Web verification now checks services through local readiness addresses only.
   Start the backend and frontend on this computer and use their local health
   URLs when running the check.

docs/doctoring/opencode-process-group-termination.md

@@ -0,0 +1,62 @@
+# OpenCode fatal-provider process-group termination
+
+## Incident
+
+The exact-head coverage-evidence job for `.github` pull request #799 reached the repository test suite but did not complete inside its bounded measurement step. A focused reproduction identified `test_fatal_provider_error_kills_hung_opencode_run_early`: the launcher detected a fatal provider event and terminated the `timeout` wrapper, while a descendant fake `opencode` process could remain alive with inherited output pipes. The parent Python process then waited for end-of-file even though the launcher had returned.
+
+## Decision
+
+Each bounded `opencode run` starts in a new session with `setsid` when that
+tool is present (Linux CI / util-linux). On a structured fatal-provider
+event, the launcher sends `SIGTERM` to the negative process-group
+identifier, waits for bounded group disappearance, and then sends
+`SIGKILL` to the same group if necessary. Darwin local runners without
+`setsid` keep PID-directed `TERM`/`KILL` so the repository suite still
+executes. The ordinary timeout contract remains
+`timeout --kill-after=30s`; only the early-fatal cleanup boundary
+changes.
+
+The launcher explicitly disables Bash job control before it starts an
+attempt. That preserves the POSIX identity relied on by the negative-PGID
+signal: the background `setsid` process executes directly, and its PID becomes
+the new session and process-group id. The implementation captures that id at
+launch time instead of probing `PATH` again during cleanup.
+
+The group signal is deliberately scoped to the session created for one model attempt. It does not target the workflow shell, unrelated model attempts, or the runner process. The production Ubuntu image already installs `util-linux`, which supplies `setsid`.
+
+CWE-400 describes uncontrolled resource consumption when a child outlives the
+intended bound (MITRE, 2026). NIST SP 800-53 Rev. 5 SI-4 requires monitoring
+that detects and contains anomalous process behavior rather than treating a
+returned parent as a complete cleanup (Joint Task Force, 2020). Killing only
+the `timeout` wrapper therefore leaves a descendant that can stall coverage
+evidence; the negative process-group identifier is the contained unit.
+
+## Verification
+
+The existing behavioral regressions in
+`tests/test_opencode_model_pool_runner.py` use fake providers that emit fatal
+structured events and sleep for 120 seconds. Before the change, the tests
+exceeded their subprocess boundary because a descendant retained the capture
+pipes. With process-group termination, both the context/quota and delisted-model
+cases complete in under 25 seconds. The focused workflow executes those real
+process tests on Ubuntu, and shell syntax plus the repository-wide evidence
+command remain required before merge.
+
+## Rollback
+
+Rollback requires an independently reviewed change and a replacement mechanism that proves every descendant of a fatal model attempt is reaped without terminating unrelated runner work. Restoring PID-only termination is not acceptable because it reintroduces the pipe-retention failure mode.
+
+## APA 7th references
+
+IEEE & The Open Group. (2024). *The Open Group base specifications issue 8: System interfaces, `kill()`*. https://pubs.opengroup.org/onlinepubs/9799919799/functions/kill.html
+
+Joint Task Force. (2020). *Security and privacy controls for information systems
+and organizations* (NIST SP 800-53 Rev. 5). National Institute of Standards and
+Technology. https://doi.org/10.6028/NIST.SP.800-53r5
+
+MITRE. (2026). *CWE-400: Uncontrolled resource consumption*.
+https://cwe.mitre.org/data/definitions/400.html
+
+Free Software Foundation. (n.d.). *GNU Coreutils manual: `timeout`: Run a command with a time limit*. Retrieved August 7, 2026, from https://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html
+
+Linux man-pages project. (2026, February 8). *setsid(2) — Linux manual page* (Linux man-pages 6.18). https://man7.org/linux/man-pages/man2/setsid.2.html

scripts/ci/run_opencode_review_model_pool.sh

@@ -1,5 +1,8 @@
 #!/usr/bin/env bash
 set -euo pipefail
+# This script owns each background attempt's PID. Keep Bash job control off so
+# `setsid` execs directly and that PID is also the new process-group id.
+set +m
 
 : "${GITHUB_OUTPUT:=/dev/null}"
 
@@ -456,7 +459,7 @@ run_one_model_attempt() {
 	local opencode_json_file="$7"
 	local opencode_export_file="$8"
 	local run_timeout_seconds export_timeout_seconds opencode_status session_id opencode_stderr_file
-	local opencode_pid fatal_poll_seconds
+	local opencode_pid opencode_process_group fatal_poll_seconds
 
 	run_timeout_seconds="${OPENCODE_RUN_TIMEOUT_SECONDS:-3600}"
 	export_timeout_seconds="${OPENCODE_EXPORT_TIMEOUT_SECONDS:-120}"
@@ -465,17 +468,38 @@ run_one_model_attempt() {
 
 	rm -f "$opencode_json_file" "$opencode_stderr_file" "$opencode_export_file" "$candidate_output_file"
 	set +e
-	timeout --kill-after=30s "${run_timeout_seconds}s" \
-		env -u GH_TOKEN -u GITHUB_TOKEN -u OPENCODE_APP_TOKEN \
-		-u ACTIONS_ID_TOKEN_REQUEST_TOKEN -u ACTIONS_ID_TOKEN_REQUEST_URL \
-		opencode run "$(cat "$prompt_file")" \
-		--pure \
-		--agent "$agent" \
-		--model "$model_candidate" \
-		--format json \
-		--title "PR #${PR_NUMBER} OpenCode bounded review ${model_candidate} attempt ${attempt}/${attempts}" \
-		>"$opencode_json_file" 2>"$opencode_stderr_file" &
-	opencode_pid=$!
+	# Linux CI has setsid (util-linux). Start the timeout wrapper in its own
+	# session so a fatal-provider abort can terminate the complete provider
+	# process group. Killing only the timeout wrapper leaves descendants
+	# holding stdout/stderr pipes open. Darwin local runners lack setsid;
+	# they keep PID-directed TERM/KILL so the repository suite still runs.
+	if command -v setsid >/dev/null 2>&1; then
+		setsid timeout --kill-after=30s "${run_timeout_seconds}s" \
+			env -u GH_TOKEN -u GITHUB_TOKEN -u OPENCODE_APP_TOKEN \
+			-u ACTIONS_ID_TOKEN_REQUEST_TOKEN -u ACTIONS_ID_TOKEN_REQUEST_URL \
+			opencode run "$(cat "$prompt_file")" \
+			--pure \
+			--agent "$agent" \
+			--model "$model_candidate" \
+			--format json \
+			--title "PR #${PR_NUMBER} OpenCode bounded review ${model_candidate} attempt ${attempt}/${attempts}" \
+			>"$opencode_json_file" 2>"$opencode_stderr_file" &
+		opencode_pid=$!
+		opencode_process_group="$opencode_pid"
+	else
+		timeout --kill-after=30s "${run_timeout_seconds}s" \
+			env -u GH_TOKEN -u GITHUB_TOKEN -u OPENCODE_APP_TOKEN \
+			-u ACTIONS_ID_TOKEN_REQUEST_TOKEN -u ACTIONS_ID_TOKEN_REQUEST_URL \
+			opencode run "$(cat "$prompt_file")" \
+			--pure \
+			--agent "$agent" \
+			--model "$model_candidate" \
+			--format json \
+			--title "PR #${PR_NUMBER} OpenCode bounded review ${model_candidate} attempt ${attempt}/${attempts}" \
+			>"$opencode_json_file" 2>"$opencode_stderr_file" &
+		opencode_pid=$!
+		opencode_process_group=""
+	fi
 	# Some providers (github-models ContextOverflowError) log a fatal error and
 	# then hang instead of exiting, burning the whole run timeout. Watch the JSON
 	# log while opencode runs and kill the process early so the pool falls
@@ -484,12 +508,21 @@ run_one_model_attempt() {
 		if has_fatal_provider_error_event "$opencode_json_file"; then
 			printf 'OpenCode %s attempt %s/%s logged a fatal provider error while still running; killing the hung process instead of waiting out the %ss run timeout.\n' \
 				"$model_candidate" "$attempt" "$attempts" "$run_timeout_seconds"
-			kill "$opencode_pid" 2>/dev/null
-			for _ in $(seq 1 30); do
-				kill -0 "$opencode_pid" 2>/dev/null || break
-				sleep 1
-			done
-			kill -9 "$opencode_pid" 2>/dev/null
+			if [ -n "$opencode_process_group" ]; then
+				kill -TERM -- "-$opencode_process_group" 2>/dev/null || true
+				for _ in $(seq 1 30); do
+					kill -0 -- "-$opencode_process_group" 2>/dev/null || break
+					sleep 1
+				done
+				kill -KILL -- "-$opencode_process_group" 2>/dev/null || true
+			else
+				kill -TERM "$opencode_pid" 2>/dev/null || true
+				for _ in $(seq 1 30); do
+					kill -0 "$opencode_pid" 2>/dev/null || break
+					sleep 1
+				done
+				kill -KILL "$opencode_pid" 2>/dev/null || true
+			fi
 			break
 		fi
 		sleep "$fatal_poll_seconds"