guard(queue): restore the kill switch + atomic lock + worktree/timeout failure containment (deep-scan A1-A3,A7)

[matt82198]

Four VERIFIED P1s in the autonomous merge-queue advancer -- the thing that merges to main unattended every 5 minutes. From the Opus deep-scan of 2026-08-03 (findings A1-A3, A7). Every fix is TDD: the test was written first and observed RED against the current code.

A1 -- the kill switch was dead (highest)

tools/halt.py writes .HALT to the state dir it resolves (AESOP_STATE_ROOT env > config state_root > ./state). daemons/run-merge-queue.sh and daemons/run-watchdog.sh both read a hard-coded $AESOP_ROOT/state/.HALT -- while run-merge-queue.sh's own header documents AESOP_STATE_ROOT as a separate variable. Set them to different paths and halt.py set writes a sentinel no daemon ever reads: halt.py --status prints HALTED, the queue keeps merging. The human abort silently does nothing.

RED evidence (tests/test-daemon-halt-sentinel.sh, before the fix):

=== Test 2: merge-queue pass halts on the sentinel halt.py actually wrote ===
[mock-cycle] ran
MERGE-QUEUE: PASSED
FAIL: merge-queue advanced while halted -- the kill switch is dead

Fix: both scripts now check EVERY location the writer may have used -- $AESOP_STATE_ROOT/.HALT first (halt.py's own precedence), $AESOP_ROOT/state/.HALT second -- deduped so a single halt logs exactly once. The watchdog diverged the same way and got the same one-line-class fix.

A2 -- double-advancer lock race

acquire_lock reclaimed a stale lock with shutil.rmtree(ignore_errors=True) followed by os.mkdir. Two steps: pass B could claim in the window, and pass A's rmtree then deleted B's fresh lock and A claimed it too. Two advancers running the same queue concurrently is the observed #727/#728 double-batch shape.

RED evidence: AssertionError: 2 != 1 : exactly one pass may hold the advancer lock, got [True, True]

Fix: os.mkdir is the lock primitive (atomic; FileExistsError means you did not get it). A stale lock is reclaimed only by a single atomic os.rename onto a private name -- whoever wins that rename is the only process that can go on to claim -- and the result is then VERIFIED against the (pid, timestamp) fingerprint of the lock judged stale. A live lock taken by mistake is renamed straight back. The interleave test forces the race deterministically rather than racing threads.

A3 -- stranded worktree

A gh pr create failure returned from build_batch without restoring main, leaving the SHARED tree on integrate/q-*. Every later pass then died unsafe_worktree -- and record_exception's dedupe wrote that row once and went quiet. The queue stops merging and says nothing.

Fix: batch construction is try/finally from the moment the integration branch is created. Main is restored on every exit path including a raise, and the branch delete moved into the finally after that restore, because git branch -D refuses the branch you are standing on. Covered for pr-create failure, push failure, success, and an exception thrown mid-construction.

A7 -- no timeout handler

The shared transport runs subprocess with timeout=60 (gh) / 120 (git) and nothing in merge_queue caught TimeoutExpired. A hang escaped as a traceback out of a scheduled task: no exception row, possibly a stranded tree, and the only record is a stack trace nobody reads.

Fix: contained in the precondition phase, in the pass body, and as a main() backstop. Each writes a subprocess_timeout ledger row, restores the tree to main, and exits non-zero cleanly. Cleanup paths use a git_safe wrapper so a finally can never raise a second time over the real failure.

Not touched (deliberately)

classify_check, required_checks_green, merge_and_verify -- the audit proved the green-check logic sound and nothing here weakens it. No --admin, --auto, force-push, --no-verify, or stash anywhere.

Verification

• tests/test_merge_queue.py: 128 -> 140 (12 new, all RED first), full suite green
• tests/test-daemon-halt-sentinel.sh: new, 5 tests, proves both daemons find the sentinel halt.py actually writes (and still honour the legacy location)
• Full shell suite: 15/15 passed; pre-push-policy.sh --test 18/18
• Zero-sleep AST proof + forbidden-operations lint: still green
• tools/verify_test_suite_count.py --check: OK (shell count regenerated 13 -> 14)
• tools/secret_scan.py --staged: exit 0

tools/CLAUDE.md is touched only because the doc-sync gate blocks the push otherwise -- a surgical append to the existing merge_queue.py bullet, no restructuring, to minimise conflict with PR #751.

🤖 Generated with Claude Code