feat(m2): WorktreeMutex + crucible cleanup CLI

[suzuke]

Summary

Stacked on #7 (M2 PR 13 smolagents). Final M2 PR. Adds the worktree-level concurrency primitive (WorktreeMutex) that future parallel BFTS workers will use to claim a worktree without stepping on each other. For v1.0 single-process mode this is a near-free no-op; the value is the invariant ("one attempt per worktree at any time", per spec §4) plus crash recovery via crucible cleanup.

Also closes the gap noted in spec §11 INV-4: TrialLedger's per-write fcntl.flock is fine for one process, but cross-process worktree contention had no story until now.

What's new

src/crucible/locking.py (new, ~330 LOC)

Design contract (reviewer round 1 PRIMARY):

• The kernel flock(LOCK_EX | LOCK_NB) IS the authority. The JSON sentinel file is metadata only — useful for diagnostics ("who holds it?") and cleanup classification, but NEVER used to bypass a failed acquisition.
• Lock files live OUTSIDE any workspace at {tempdir}/crucible-locks/{sha256(resolved_workspace_path)[:32]}.lock, unreachable by smolagents _path_acl (rejects absolute paths) and Claude Code's workspace-scoped Read/Glob/Grep hooks.
• WorktreeMutex.__init__ validates the resolved lock path is NOT inside the workspace tree — WorktreeLockConfigError with actionable message if CRUCIBLE_LOCK_DIR is misconfigured into the workspace.

API:

• WorktreeMutex(workspace, *, timeout=30.0) context manager with acquire() / release() / held property.
• WorktreeLocked raised on contention timeout, carries owner metadata for diagnostics.
• LockOwner records pid / host / process_create_time (best-effort via psutil) for PID-reuse defence.
• is_owner_alive() / classify_worktree() / safe_cleanup_lock() for the cleanup CLI — metadata-driven classification but actual deletion gated on non-blocking flock acquisition.

crucible cleanup CLI

Inspect / refresh stale worktree lock metadata for the current project:

crucible cleanup                # dry-run; report status
crucible cleanup --refresh      # rewrite stale sentinel under flock
crucible cleanup --refresh --yes  # skip confirmation

Reviewer round 2 F1 / F2 safety contract:

• Default is read-only inspection.
• --refresh does NOT unlink the lock file. The acquired flock has already overwritten the sentinel with current owner metadata; release leaves the file with up-to-date content. Unlinking would re-introduce inode-split (process B holds old inode, cleanup unlinks path, C creates new file → both have "valid" flocks on different inodes).
• Worktree directories are NEVER deleted by this command. Worktree pruning is M3 territory once the parallel-worker layout exists.

Orchestrator integration

_run_loop_serial() wraps the entire loop in a single mutex acquire (5s timeout, fail-loud on contention). Single-process mode pays effectively zero overhead; future parallel workers each pin their own worktree.

Windows: explicit branch logs a clear warning (WorktreeMutex unsupported on Windows in v1.0 — running without cross-process lock enforcement. Concurrent crucible runs against the same workspace will race; use one process at a time.) and proceeds. POSIX path is unchanged.

Reviewer trail

Round	Verdict	Key finding
1 (design)	NEEDS_TWEAK	PRIMARY: stale-lock auto-steal would split-brain. flock is inode-based; unlinking a path while A holds the old inode lets B lock a new inode at the same path → both think they own.
2 (impl)	REJECTED	F1: cleanup unlinked lock file (split-brain reintroduced). F2: CLI claimed "delete worktrees" but only removed lock pathname. F3: lock at <workspace>/logs/locks/... was agent-readable via Claude Code Read hooks. F4: orchestrator silently swallowed Windows RuntimeError.
3 (fix)	PARTIAL	F1 / F2 / F4 verified. F3 had a config hole: CRUCIBLE_LOCK_DIR accepted verbatim, so an env value pointing into the workspace re-exposed the lock.
4 (fix)	VERIFIED	F3 closed: WorktreeMutex.__init__ containment-checks resolved lock path against workspace, raises WorktreeLockConfigError on violation. Reviewer ran 4-case repro: absolute-into-workspace, nested-into-workspace, legit-external (passes), relative-via-cwd-into-workspace — all behave correctly.

Stats

• 3 commits on feat/m2-ledger-lock (b7a6fe2 initial + a6be393 round-2 fix + 6f36a1d round-3 F3 follow-up)
• 5 files changed, +1,223 / -111 LOC
• 24 lock tests in test_locking.py (was 17 round-2; +7 across rounds)
• Full suite: 2,632 passed / 4 skipped, 0 regressions vs PR 13 baseline (2,608)

Test plan

Acquire / release / sentinel mechanics

• Acquire / release / re-acquire from same process
• Context manager releases on normal + exception exit
• Double acquire on same instance → RuntimeError
• Sentinel written under flock, persists across release, overwritten cleanly on next acquire

CRITICAL split-brain prevention (reviewer R1 primary)

• test_no_split_brain_stale_sentinel_with_live_holder: subprocess holds flock + writes "PID 99999999 / 2020-01-01" sentinel; main process MUST get WorktreeLocked, MUST NOT unlink, sentinel content asserted unchanged after attempt

Subprocess contention

• B times out while A holds; error message contains owner pid and timeout

Cleanup safety (reviewer R2 F1+F2)

• test_cleanup_does_not_unlink_lock_file: pre-write stale sentinel → safe_cleanup_lock acquires & releases → file STILL exists, sentinel pid is current runner not 99999999
• Busy worktree skipped (cleanup must not break running worker)
• Unheld stale sentinel → cleanup acquires successfully

Lock-outside-workspace invariant (reviewer R2 F3 + R3 F3 follow-up)

• test_lock_path_is_outside_workspace: default placement is NOT a descendant of workspace
• test_env_override_inside_workspace_rejected: CRUCIBLE_LOCK_DIR=<ws>/locks → WorktreeLockConfigError
• test_env_override_inside_workspace_subdir_rejected: nested workspace path rejected
• test_env_override_outside_workspace_ok: legitimate external override works
• test_env_override_relative_pointing_into_workspace_rejected: relative env value resolved against cwd-inside-workspace → rejected
• test_default_tempdir_placement_passes_containment: no-override path is outside workspace

Windows (reviewer R2 F4)

• WorktreeMutex.acquire() raises RuntimeError on Windows (mirrors TrialLedger v1.0 stance)
• Orchestrator handles via explicit branch + WARNING log; no silent swallowing

Cleanup classification

• no-lock-file / live / orphan (malformed sentinel) / stale (dead PID, same host)
• is_owner_alive: dead pid / self / cross-host conservative

Known limitations / non-blockers

• Worktree-directory pruning: deferred to M3 once parallel-worker layout exists. v1 cleanup is metadata-only.
• Multi-host coordination: is_owner_alive returns conservative-true on cross-host PIDs. v1 assumes single-host operation; distributed locking via Redis/etcd is out of scope.
• Lock fairness / queueing: first-come is fine at this scale; no queue.
• PID reuse on systems without psutil: when psutil is unavailable, is_owner_alive falls back to PID-only matching — small window for false positives, acceptable for cleanup classification (acquire path doesn't trust this anyway).

Closing M2

This is the LAST M2 PR. With #4 (doom-loop pruning), #5 (compare mode), #6 (HMAC seal), #7 (smolagents backend), and now #8, the M2 spec deliverables are complete:

Module	M2 deliverable	Status
SearchStrategy	+ doom-loop pruning	#4 ✅
Reporter	+ comparison	#5 ✅
Seal	+ HMAC upgrade	#6 ✅
AgentBackend	+ smolagents adapter	#7 ✅
StateStore	+ concurrency lock	#8 (this PR) ✅

🤖 Generated with Claude Code