Skip to content

fix(state_store): wire canonical paths into leases, de-vacuum the guard test, make epoch fencing real (deep-scan B1-B3) - #754

Open
matt82198 wants to merge 2 commits into
mainfrom
fix/multibox-canonical-epoch
Open

fix(state_store): wire canonical paths into leases, de-vacuum the guard test, make epoch fencing real (deep-scan B1-B3)#754
matt82198 wants to merge 2 commits into
mainfrom
fix/multibox-canonical-epoch

Conversation

@matt82198

Copy link
Copy Markdown
Owner

Three verified P1s in the multibox coordination layer (Opus deep-scan 2026-08-03), all with red-before / green-after evidence.

B1 — split-brain: the canonicalizer was wired in, but under a host-dependent policy

LeaseStore derived claim keys via canonical_claim_path(path, case_policy="platform"). "platform" case-folds when os.name == 'nt' and preserves case otherwise, so two instances sharing one coordination database derived different keys for the same file. _check_conflicts is an exact-match lookup, so it missed and both instances were granted the claim.

Red, before the fix:

B1 pre-fix _normalize_path nt    = 'tools/runner.py'
B1 pre-fix _normalize_path posix = 'tools/Runner.py'
IDENTICAL? False
SPLIT-BRAIN: BOTH instances hold tools/Runner.py -- no LeaseConflict raised

Green, after:

nt= 'tools/runner.py'  posix= 'tools/runner.py'  IDENTICAL? True
BLOCKED: LeaseConflict Path conflict with win: ['tools/runner.py']

paths.py logic is untouched — this PR wires it in correctly.

Case-policy decision: default "insensitive", config-driven

Resolution order: explicit case_policy= arg → config["multibox"]["case_policy"]$AESOP_CLAIM_CASE_POLICYDEFAULT_CASE_POLICY ("insensitive").

Two reasons for "insensitive" over "platform":

  1. It is the only policy identical on every host, which is the property multibox requires. "sensitive" is also host-independent, but it under-collides.
  2. It errs in the safe direction. A claim keyspace may safely collide two genuinely distinct files — the loser simply waits for the lease. It may never fail to collide two names for the same file, because then both instances write it. Over-collision costs throughput; under-collision costs correctness.

Local single-box case-sensitive semantics are preserved as an explicit opt-in: LeaseStore(db, case_policy="sensitive") or multibox.case_policy. An unrecognized policy raises ValueError (fail-closed) rather than silently selecting a different keyspace — a silent fallback would be the same bug class as B1 itself. When neither an explicit policy nor a config dict is supplied, resolution is deferred to call time so environment changes apply without a restart.

config is now threaded through LocalLeaseBackend / get_backend so multibox.case_policy actually reaches the keyspace.

B2 — the guard test was vacuous and masked B1

TestLeaseClaimsHeterogeneityGuard claimed to verify "canonical_claim_path (via _normalize_path) produces identical results regardless of platform", but it never called _normalize_path (zero references) and passed case_policy="insensitive", a value production never used. It asserted a property of an argument it supplied itself.

Proof it was vacuous — mutating _normalize_path to the identity function, the old guard still passes:

test_heterogeneity_guard_47c967b_case_multiplatform ... ok
test_heterogeneity_guard_47c967b_separator_multiplatform ... ok
OLD GUARD SURVIVED IDENTITY MUTATION: True

The rewritten guard exercises the real production entry point (_normalize_path, and LeaseStore.claim / get_holder end-to-end). Same mutation against the new guard:

Ran 6 tests
FAILED (failures=3)
NEW guard survived identity mutation: False

test_path_case_sensitivity_linux_style asserted the host-dependent keyspace as a requirement; it is rewritten to assert the host-independent default, with the case-sensitive path moved to an explicit opt-in test.

B3 — epoch fencing was inert; implemented the increment (vocabulary kept)

_init_identity_file wrote epoch=1 once and nothing in the repo ever incremented it, so get_identity_with_epoch returned 1 forever and could not distinguish a pre- from a post-crash instance — while identity.py:8 documented it as "a monotonic boot counter; restart increments it". Per the task decision, the increment is implemented rather than the vocabulary removed, since Inc 5 (#735) consumes epoch as its fencing token.

Acquisition now increments and durably persists the epoch via atomic read-modify-write (temp file + fsync + os.replace), before returning it, so a crash immediately after cannot hand out the same epoch twice.

Preserved: the fail-closed-on-corruption hardening from 7e0c9f86; fresh box still starts at 1; the value is still cached per process (repeated calls in one process return one epoch).

Added fail-closed paths:

  • EpochPersistError when the bumped epoch cannot be written. It subclasses IdentityCorruptionError, so existing except IdentityCorruptionError handlers keep working. Returning the un-bumped epoch would hand a restarting instance the same fencing token its pre-crash self may still be using.
  • A non-integer or < 1 epoch is corrupt, not merely odd: it cannot be incremented monotonically.

Red, before (identity suite): 7 failures, 2 errors — including test_incremented_epoch_is_persisted_to_disk (1 != 2) and the two release_own_stale tests.

release_own_stale decision: explicit NotImplementedError

Chose the raise over a real implementation. Real prior-epoch reclamation requires epoch-tagged claims coordinated with the lease backend — that is Inc 5 (#735), stacked and unmerged; implementing it here would collide with shipped-but-unmerged work.

The unconditional return True was worse than an absent function: a caller trusting it believed reclamation had happened and would proceed to write files its pre-crash self still holds a live lease on. Prior claims are reclaimed by TTL expiry until Inc 5 lands — which is what the stub was silently providing anyway, minus the false assurance. Callers wanting best-effort behavior can catch NotImplementedError and wait out the TTL.

Note on a corrected test

test_id_stability_across_two_processes asserted "Epoch should be identical across processes on first read" — the bug stated as a requirement. If two live processes share a fencing token, the token fences nothing. It now asserts epoch_2 == epoch_1 + 1 across two real OS processes (the strongest B3 proof in the suite); stable_id is still asserted identical.

test_read_only_valid_id_file_succeeds is replaced by test_existing_id_file_is_loaded_and_bumped: its premise ("no write is needed on the read path") is exactly what B3 identified as the bug. The write-failure branch is now covered deterministically by mocking the atomic writer, because chmod-based coverage is unreliable (a no-op for root on Linux CI).

Verification

Suite Result
tests.test_lease_claims 32 pass
tests.test_state_store_identity 20 pass
tests.test_claim_backend 22 pass
tests.test_state_store_concurrency 12 pass
tests.test_state_store_paths 22 pass
tests.test_multi_dispatch_claim 10 pass
full unittest discover -s tests exit 0
npm run test:node 321 pass, 0 fail

Gates: secret_scan --staged CLEAN (exit 0), verify_test_suite_count OK, claudemd_sync_gate OK, verify_test_coverage OK, ci_gate_runability OK.

BEGIN IMMEDIATE atomicity is untouched (the audit proved it sound); paths.py's own logic is untouched.

Stacked PRs

This lands on main beneath the stacked multibox PRs (#697 / #722 / #735 / #738 / #739); they may need a trivial rebase. #735 (Inc 5) additionally consumes release_own_stale, which now raises instead of returning True — that call site needs a decision (catch and fall back to TTL, or implement reclamation as part of Inc 5).

🤖 Generated with Claude Code

…rd test, make epoch fencing real

Deep-scan findings B1/B2/B3 in the multibox coordination layer.

B1 (split-brain): LeaseStore derived claim keys with case_policy="platform",
which case-folds when os.name == 'nt' and preserves case otherwise. Two
instances sharing one coordination db therefore derived DIFFERENT keys for the
same file ('tools/Runner.py' -> 'tools/runner.py' on Windows, 'tools/Runner.py'
on Linux), so _check_conflicts (exact match) missed and BOTH were granted the
claim. Case policy is now configuration, not a property of the host OS, resolved
explicit-arg > multibox.case_policy > $AESOP_CLAIM_CASE_POLICY > "insensitive".
An unrecognized policy raises ValueError (fail-closed).

B2 (vacuous guard): the heterogeneity guard called canonical_claim_path directly
with case_policy="insensitive" -- a value production never passed -- and never
touched _normalize_path, so it stayed green while B1 was live and survived
mutating _normalize_path to the identity function. Rewritten to exercise the real
production entry point, plus an end-to-end LeaseStore split-brain reproducer.

B3 (inert epoch): nothing in the repo ever incremented the persisted epoch, so
get_identity_with_epoch returned 1 forever and could not distinguish a pre- from
a post-crash instance, while the module documented it as a monotonic boot
counter. Acquisition now increments and durably persists the epoch (temp file +
fsync + os.replace), fail-closed on persist failure (EpochPersistError, a
subclass of IdentityCorruptionError) and on a non-integer epoch. Existing
corrupt-file fail-closed hardening and fresh-box epoch=1 are preserved.
release_own_stale() now raises NotImplementedError instead of returning an
unconditional True while doing nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@matt82198 matt82198 added the merge-queue Queued for the merge-queue advancer daemon label Aug 3, 2026
@matt82198 matt82198 added queue-rejected Evicted from the merge queue (red or culprit) and removed merge-queue Queued for the merge-queue advancer daemon labels Aug 3, 2026
@matt82198

Copy link
Copy Markdown
Owner Author

Evicted from the merge queue: required check(s) absent from rollup: windows

matt82198 added a commit that referenced this pull request Aug 3, 2026
…754 adaptation)

PR #754 makes identity.release_own_stale() raise NotImplementedError instead of
returning an unconditional True while doing nothing. Inc 5 needs no call-site
change -- it has no call site: its reclamation mechanism already IS the fallback,
TTL expiry at fold time (ttl + max_skew), as failover.py documents.

Pins that so it cannot silently regress when #754 lands:
- 5 behavioural cells run the whole election/takeover/fencing/summary path under
  every shape the stub can present (raises, False, the old misleading True, symbol
  absent), including the falsifiable half: before the TTL a restarted incarnation
  gets NOTHING, which is the proof no proactive reclamation is running.
- A falsified source scan asserts no state_store module calls release_own_stale
  without handling NotImplementedError, so a future call site must carry the TTL
  fallback or go red.

Verified against PR #754 actual identity.py (swapped in, not committed): all
101 failover tests green. Nothing claims reclamation happened.

Real epoch-tagged reclamation filed as a backlog item.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@matt82198 matt82198 added the merge-queue Queued for the merge-queue advancer daemon label Aug 3, 2026
@matt82198 matt82198 removed the queue-rejected Evicted from the merge queue (red or culprit) label Aug 4, 2026
@matt82198 matt82198 added queue-rejected Evicted from the merge queue (red or culprit) and removed merge-queue Queued for the merge-queue advancer daemon labels Aug 4, 2026
@matt82198

Copy link
Copy Markdown
Owner Author

Evicted from the merge queue: ci (0) is not green

Failing run: https://github.com/matt82198/aesop/actions/runs/30876463260/job/91888856592

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

queue-rejected Evicted from the merge queue (red or culprit)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant