feat(state_store): stale-primary failover + fencing generations (multibox Inc 5)

[matt82198]

Multibox Increment 5 — stale-instance detection, stale-primary failover, and fencing generations. Stacked on feat/multibox-inc4b-durability.

Answers the one question the claim log alone cannot: which peer is driving, and how a peer that was partitioned away is stopped from resuming when it comes back.

1. Tier-S heartbeats (HeartbeatDir)

Tier L keeps appending one instance_heartbeat event per beat. Tier S cannot — 5 boxes at 10s is ~43k records/day in a log whose every read is a full directory listing. So a Tier-S beat is one file per (instance_id, epoch), instances/<instance_id>.<epoch>.hb, atomically REPLACED (os.replace), never appended, keeping the directory bounded by fleet size forever. Same Inc 4b durability order (temp → fsync bytes → replace → _fsync_dir), .hb.tmp invisible to readers, instance_id sanitized in the filename with the true value read from the body (: is illegal on Windows).

Unreadable heartbeat evidence fails toward ALIVE (mtime fallback; unreadable dir → []) — deliberately the opposite direction from the claim log's fail-closed, because wrongly declaring a live peer dead would let its claims be reclaimed underneath it.

instance_projection.detect_stale_instances(store, threshold, source=None, now=None) gains the transport-aware source (record list, callable, or anything with read_heartbeats()). Threshold semantics, the 300s default and the result shape are unchanged on both transports — pinned by a test that asserts the boundary on both paths — so the reclamation path stays transport-blind. Stale-claim reclamation itself is not changed: it stays fold_fs_claims TTL behaviour. Inc 5 only makes staleness observable, via multibox_staleness_summary() — the data surface Inc 7's MCP fleet_multibox_summary will read. mcp/ is not touched.

2. Primary election

The primary is simply the holder of the reserved resource orchestrator_lock, taken through the same claim protocol as any file — same settle window, same (lamport, epoch_ms, instance_id, uuid) sort key, same TTL-expiry-at-fold. No election algorithm, no consensus, no quorum: the shared log is the arbiter. A live holder is never pre-empted; losing a concurrent takeover is not an error (the log is re-folded and the actual winner returned, so N challengers converge with no retry loop).

fold_primary(records, now, max_skew) -> PrimaryState is a pure function over a list of dicts and delegates liveness to fold_fs_claims rather than reimplementing it, so lowest-key-wins, TTL-at-fold, tombstones, heartbeat extension, skew-only-lengthens and corrupt-fails-closed all apply to the lock unchanged, for free.

3. Fencing generations

TTL takeover alone is unsafe: a merely partitioned primary returns after its successor took over and both drive. So every takeover bumps a monotonic generation, and assert_fenced / fenced_write / fenced_backend_write reject any coordination write whose generation is below the fold's current one. epoch (Inc 3) fences a restarted process against its own prior incarnation; generation fences a partitioned primary against its successor — different failure modes, both needed.

PrimaryState.generation is the max over all lock records including expired and tombstoned ones — a dead record is still proof a generation was issued, and forgetting it would let the fence go backwards.

The generation is carried inside the claim record as a second claimed path orchestrator_lock/gen/<NNNNNNNNNNNN>. That is load-bearing rather than a smuggled field: because the token is itself a claimed path, two instances racing for generation N collide on the token as well as the lock, so a generation can never be occupied twice.

backend_records() refuses a backend that exposes no listing (UnsupportedBackendError) instead of reading it as an empty log — which would report "no primary" and hand the lock to a second driver.

Falsifiability: the fence is proven load-bearing by mutation, not assertion

Replacing the comparison in assert_fenced with a bare return turns 5 tests red:

FAIL: TestFencing.test_stale_generation_is_rejected
FAIL: TestFencing.test_returning_old_primary_is_fenced_after_the_fold_shows_n_plus_1
FAIL: TestFencing.test_fence_is_load_bearing_the_same_write_lands_unguarded
FAIL: TestFencing.test_fenced_write_is_checked_before_any_side_effect
FAIL: TestElectPrimaryOnFsClaimLog.test_returning_old_primary_cannot_resume

test_fence_is_load_bearing_the_same_write_lands_unguarded shows why: the same write, same caller, same payload, is ACCEPTED when not routed through the guard, and produces no effect when it is. So the guard — and nothing else — is what stands between an old primary and split-brain.

Test counts

Suite	Count	Status
tests/test_failover.py (new)	93	green
tests/test_fs_claim_log.py (Inc 4a)	67	green, unchanged
tests/test_fs_claim_log_durability.py (Inc 4b)	42	green, unchanged
tests/test_claim_backend.py (Inc 2 contract)	22	green, unchanged
tests/test_instance_manager.py	43	green, unchanged
tests/test_multi_dispatch_claim.py	10	green, unchanged
tests/test_lease_claims.py	20	green (1 skip), unchanged

The 93 break down as 7 generation-token, 21 pure fold_primary (sole instance elected; lapse frees the lock while the fence remembers it; exactly one successor; 3-way simultaneous takeover → one winner and order-independent; generation never decreases; tombstone/heartbeat/skew/corrupt/pre-fencing-record/non-lock-claim; input not mutated), 12 fencing, 17 tmpdir integration with injected clock and settle=0, 14 HeartbeatDir (50 beats → still 1 file — the whole reason Tier S exists), 12 transport-parity, 10 summary/dataclass.

Full gate battery

npm run test:py 4469 OK (20 skips) · npm run test:node 315 pass, 0 fail · npm run test:sh 14/14 suites PASS · secret_scan --staged exit 0 · verify_test_coverage --check · claudemd_lint · claudemd_contract · claudemd_sync_gate --check · claudemd_drift · stateapi_lint · watcher_linter --check · spec_contract_validator --check · subprocess_guard --check · agent_prompt_hygiene · portability_check · self_stats --check · verify_test_suite_count --check (233 → 234) · metrics_gate — all exit 0.

Hermetic: tempdirs only, injectable clock, no real sleeps, no network, encoding='utf-8' throughout, ASCII source.

Scope

Touched: state_store/failover.py (new), state_store/instance_projection.py (transport-aware stale source only), tests/test_failover.py (new), state_store/CLAUDE.md, tests/CLAUDE.md (regenerated count). Not touched: fs_claim_log.py, identity.py, claim_backend.py, paths.py, lease_claims.py, tools/**, ui/**, mcp/**, .github/**.

Flag: reachable only under multibox.enabled (wired in Inc 7). Tier L is untouched.

🤖 Generated with Claude Code

[matt82198]

Known gap: stacked PRs get no CI. This PR targets feat/multibox-inc4b-durability, not main, and the workflows are wired to main-targeted PRs — so no checks will report here. Verification was run locally on this exact HEAD: npm run test:py 4469 OK (20 skips), npm run test:node 315 pass / 0 fail, npm run test:sh 14/14 suites PASS, plus the full CI gate battery (verify_test_coverage, claudemd_lint/contract/sync_gate/drift, stateapi_lint, watcher_linter, spec_contract_validator, subprocess_guard, agent_prompt_hygiene, portability_check, self_stats, verify_test_suite_count, metrics_gate, secret_scan) all exit 0. Real CI arrives when the stack lands on main — the base PRs (Inc 4a/4b) must merge first.