feat(state_store): FsClaimLog durability + skew bounds + GC (multibox Inc 4b)

[matt82198]

Stacked on #697 (Inc 4a). Base is feat/multibox-inc4a-fsclaimlog, not main.

Closes the three things Inc 4a deliberately left open. All production changes are in
state_store/fs_claim_log.py; the pure fold from 4a is untouched and its 67 tests stay green.

Durability

_append no longer writes the final name directly. It writes a temp name in the same directory
(os.replace is only atomic within one filesystem), flush() + os.fsync() to make the BYTES
durable, os.replace to publish the name atomically, then _fsync_dir() to make the directory
ENTRY durable. The order is the contract: a peer must never see a record name whose bytes are not
yet durable, and no reader may see a partially written .json.

The temp suffix is deliberately not .json, so _read_records skips it. A crash mid-write
leaves a file that was never published — no grant was ever made from it, so ignoring it
reproduces the exact pre-write state. This is not a weakening of 4a's fail-closed rule: a
truncated .json was published and still blocks every path (regression test included).

A real defect the tests caught

_fsync_dir uses os.open(dir, O_RDONLY) + os.fsync on POSIX. Windows refuses to open a
directory as a file, so that raises and falls through to a ctypes FlushFileBuffers fallback.
Asserting that the fallback succeeds against a real directory — rather than mocking it — found
a live bug: FlushFileBuffers requires GENERIC_WRITE. With GENERIC_READ the handle opens
happily and the flush then fails ERROR_ACCESS_DENIED (5), i.e. the helper degrades to a silent
no-op — precisely the failure mode a "best-effort" path hides best. Also fixed: undeclared
argtypes truncate the returned HANDLE to a C int on 64-bit.

Clock skew (writer side; 4a shipped the fold side)

Ordering stays lamport-keyed and skew-immune; only TTL uses the wall clock. Two guards:

1. _writer_epoch_ms() clamps our stamp so it can never regress. A backwards NTP step would
   otherwise publish a record that looks older than one we already wrote, shortening our own
   lease. Clamping upward only LENGTHENS — the safe direction.
2. ClockSkewError (new) fails claim() closed, before anything is written, when a peer
   record is stamped past now + max_skew_seconds.

Rationale for (2): the fold's + max_skew only makes "skew lengthens, never shortens" true
inside the bound. A peer behind by more than the bound would have its live lease expired early
by our fold — a double-grant. That direction is undetectable from a record alone (an old-looking
stamp is indistinguishable from an old record), but the symmetric ahead direction is directly
observable, so we fail closed on the observable half and leave measurement of both halves to Inc 0
preflight / Inc 7 startup gating.

Skew matrix (re-derived from the shipped fold)

peer skew	held at true deadline	early expiry
ahead +5 (within bound)	yes	no
ahead +10 (at bound)	yes	no
behind -5 (within bound)	yes	no
behind -10 (exactly at bound)	yes	no
behind -15 (past bound)	no	yes

For every |skew| <= max_skew the computed deadline is >= the true deadline, so no early expiry
ever. The past-bound cell is asserted as the falsifiable case — that is what makes max_skew a
load-bearing measured precondition rather than decoration (same spirit as Inc 6 assertion 2).

GC

compact(retain_seconds=0.0) -> int deletes only what it can prove dead past
ttl + max_skew + settle + retain; anything unprovable is kept forever.

• Group-atomic. All records of a lease (request + heartbeats + tombstone) go together or not at
  all. Deleting a tombstone while its request survives would resurrect a released claim;
  deleting a request while its heartbeats survive would silently shorten a live lease.
• Never live. A lease still winning a path in the current fold is untouched (belt-and-braces on
  top of the deadline test); the + max_skew term keeps that true for a skewed peer.
• Never unprovable. No group containing a ttl-less legacy record is ever collectable.
• Tombstoned leases are held to the same full bound (a tombstone frees the path at fold time,
  not at GC time), so no deletion depends on how fresh a peer's listing is.
• Corrupt records have no readable deadline, so mtime is the only evidence: collected only past
  mtime + default_ttl + max_skew + settle + retain — verified to sit a full settle window after
  the fold stops treating them as a live unknown-holder claim.
• Idempotent, and safe to run concurrently (a file another compactor already removed is not an
  error; an unreadable directory collects nothing).

Tests

42 new in tests/test_fs_claim_log_durability.py; 4a's 67 unchanged and green.

• 12 durability cases asserting the call order over a recording os shim —
  fsync -> replace -> open -> fsync -> close. The proof is the ordering, since on tmpfs no
  assertion about a real fsync's effect is possible. Plus the refuse-dir-open platform branch,
  same-directory temp name, temp invisible to the fold, truncated .json still blocks, and the
  real-host dir-sync success assertion (the one that caught the GENERIC_WRITE bug).
• 6 skew-matrix cells + 7 writer-guard cases (including: a refusal writes no record at all).
• 17 GC cases: never-delete-live under a skewed clock and under a heartbeat-extended lease,
  settle margin, ttl-less legacy, group atomicity, corrupt-by-mtime on both sides of the bound,
  idempotence, concurrent-removal race, compact never changes the answer of the fold, and a
  seeded property sweep of 200 randomized claim/renew/release/advance histories across the
  settle x max_skew x ttl x retain space asserting that same invariant.

Hermetic throughout: tempdirs only, injected clock, no real sleeps, no network, no cwd pollution.

Gates

• tests.test_fs_claim_log 67 OK (4a, unchanged) - tests.test_fs_claim_log_durability 42 OK
• tests.test_claim_backend 22 OK (Inc 2 contract suite still runs against FsClaimLog unmodified)
• Full Python suite 4372 tests OK (20 skipped) - npm run test:node 315/315 - npm run test:sh exit 0
• claudemd_lint / claudemd_sync_gate / claudemd_contract / verify_test_suite_count /
  metrics_gate / ci_gate_runability / dispatch_lint / pre-push-policy.sh --test all clean
• secret_scan.py --staged exit 0

Docs

First commit prunes state_store/CLAUDE.md 148 -> 88 lines by densifying verbose bullet
sections into prose — no information removed, every contract fact, event type, API name and
measured number retained (word-level diff verified). Inc 4b's section brings it to 96/150,
leaving 54 lines of headroom for Inc 5 and Inc 7.

Do not merge before #697.

🤖 Generated with Claude Code

[matt82198]

Heads-up for whoever picks this up: CI will not run on this PR while it is stacked. .github/workflows/ci.yml triggers only on pull_request: branches: [main], and this PR targets feat/multibox-inc4a-fsclaimlog. gh pr checks 722 reports "no checks reported" — that is absence of CI, not a green.

Checks will fire once #697 merges and GitHub auto-retargets this base to main. Until then the proof is the local battery recorded in the PR body (full Python suite 4372 OK / 20 skipped, node 315/315, shell exit 0, plus the gate tools). Do not read the empty check list as a pass.

[matt82198]

Full Python suite re-run on the exact committed tree (all three commits, including the seeded GC property sweep):

Ran 4375 tests in 713.963s
OK (skipped=20)

That supersedes the 4372 figure in the PR body, which was captured one commit earlier.