fix(state_store): wire canonical paths into leases, de-vacuum the guard test, make epoch fencing real (deep-scan B1-B3) - #754
Open
matt82198 wants to merge 2 commits into
Open
Conversation
…rd test, make epoch fencing real
Deep-scan findings B1/B2/B3 in the multibox coordination layer.
B1 (split-brain): LeaseStore derived claim keys with case_policy="platform",
which case-folds when os.name == 'nt' and preserves case otherwise. Two
instances sharing one coordination db therefore derived DIFFERENT keys for the
same file ('tools/Runner.py' -> 'tools/runner.py' on Windows, 'tools/Runner.py'
on Linux), so _check_conflicts (exact match) missed and BOTH were granted the
claim. Case policy is now configuration, not a property of the host OS, resolved
explicit-arg > multibox.case_policy > $AESOP_CLAIM_CASE_POLICY > "insensitive".
An unrecognized policy raises ValueError (fail-closed).
B2 (vacuous guard): the heterogeneity guard called canonical_claim_path directly
with case_policy="insensitive" -- a value production never passed -- and never
touched _normalize_path, so it stayed green while B1 was live and survived
mutating _normalize_path to the identity function. Rewritten to exercise the real
production entry point, plus an end-to-end LeaseStore split-brain reproducer.
B3 (inert epoch): nothing in the repo ever incremented the persisted epoch, so
get_identity_with_epoch returned 1 forever and could not distinguish a pre- from
a post-crash instance, while the module documented it as a monotonic boot
counter. Acquisition now increments and durably persists the epoch (temp file +
fsync + os.replace), fail-closed on persist failure (EpochPersistError, a
subclass of IdentityCorruptionError) and on a non-integer epoch. Existing
corrupt-file fail-closed hardening and fresh-box epoch=1 are preserved.
release_own_stale() now raises NotImplementedError instead of returning an
unconditional True while doing nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Evicted from the merge queue: required check(s) absent from rollup: windows |
matt82198
added a commit
that referenced
this pull request
Aug 3, 2026
…754 adaptation) PR #754 makes identity.release_own_stale() raise NotImplementedError instead of returning an unconditional True while doing nothing. Inc 5 needs no call-site change -- it has no call site: its reclamation mechanism already IS the fallback, TTL expiry at fold time (ttl + max_skew), as failover.py documents. Pins that so it cannot silently regress when #754 lands: - 5 behavioural cells run the whole election/takeover/fencing/summary path under every shape the stub can present (raises, False, the old misleading True, symbol absent), including the falsifiable half: before the TTL a restarted incarnation gets NOTHING, which is the proof no proactive reclamation is running. - A falsified source scan asserts no state_store module calls release_own_stale without handling NotImplementedError, so a future call site must carry the TTL fallback or go red. Verified against PR #754 actual identity.py (swapped in, not committed): all 101 failover tests green. Nothing claims reclamation happened. Real epoch-tagged reclamation filed as a backlog item. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Owner
Author
|
Evicted from the merge queue: ci (0) is not green Failing run: https://github.com/matt82198/aesop/actions/runs/30876463260/job/91888856592 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three verified P1s in the multibox coordination layer (Opus deep-scan 2026-08-03), all with red-before / green-after evidence.
B1 — split-brain: the canonicalizer was wired in, but under a host-dependent policy
LeaseStorederived claim keys viacanonical_claim_path(path, case_policy="platform")."platform"case-folds whenos.name == 'nt'and preserves case otherwise, so two instances sharing one coordination database derived different keys for the same file._check_conflictsis an exact-match lookup, so it missed and both instances were granted the claim.Red, before the fix:
Green, after:
paths.pylogic is untouched — this PR wires it in correctly.Case-policy decision: default
"insensitive", config-drivenResolution order: explicit
case_policy=arg →config["multibox"]["case_policy"]→$AESOP_CLAIM_CASE_POLICY→DEFAULT_CASE_POLICY("insensitive").Two reasons for
"insensitive"over"platform":"sensitive"is also host-independent, but it under-collides.Local single-box case-sensitive semantics are preserved as an explicit opt-in:
LeaseStore(db, case_policy="sensitive")ormultibox.case_policy. An unrecognized policy raisesValueError(fail-closed) rather than silently selecting a different keyspace — a silent fallback would be the same bug class as B1 itself. When neither an explicit policy nor a config dict is supplied, resolution is deferred to call time so environment changes apply without a restart.configis now threaded throughLocalLeaseBackend/get_backendsomultibox.case_policyactually reaches the keyspace.B2 — the guard test was vacuous and masked B1
TestLeaseClaimsHeterogeneityGuardclaimed to verify "canonical_claim_path (via_normalize_path) produces identical results regardless of platform", but it never called_normalize_path(zero references) and passedcase_policy="insensitive", a value production never used. It asserted a property of an argument it supplied itself.Proof it was vacuous — mutating
_normalize_pathto the identity function, the old guard still passes:The rewritten guard exercises the real production entry point (
_normalize_path, andLeaseStore.claim/get_holderend-to-end). Same mutation against the new guard:test_path_case_sensitivity_linux_styleasserted the host-dependent keyspace as a requirement; it is rewritten to assert the host-independent default, with the case-sensitive path moved to an explicit opt-in test.B3 — epoch fencing was inert; implemented the increment (vocabulary kept)
_init_identity_filewroteepoch=1once and nothing in the repo ever incremented it, soget_identity_with_epochreturned1forever and could not distinguish a pre- from a post-crash instance — whileidentity.py:8documented it as "a monotonic boot counter; restart increments it". Per the task decision, the increment is implemented rather than the vocabulary removed, since Inc 5 (#735) consumes epoch as its fencing token.Acquisition now increments and durably persists the epoch via atomic read-modify-write (temp file +
fsync+os.replace), before returning it, so a crash immediately after cannot hand out the same epoch twice.Preserved: the fail-closed-on-corruption hardening from
7e0c9f86; fresh box still starts at1; the value is still cached per process (repeated calls in one process return one epoch).Added fail-closed paths:
EpochPersistErrorwhen the bumped epoch cannot be written. It subclassesIdentityCorruptionError, so existingexcept IdentityCorruptionErrorhandlers keep working. Returning the un-bumped epoch would hand a restarting instance the same fencing token its pre-crash self may still be using.< 1epoch is corrupt, not merely odd: it cannot be incremented monotonically.Red, before (identity suite): 7 failures, 2 errors — including
test_incremented_epoch_is_persisted_to_disk(1 != 2) and the tworelease_own_staletests.release_own_staledecision: explicitNotImplementedErrorChose the raise over a real implementation. Real prior-epoch reclamation requires epoch-tagged claims coordinated with the lease backend — that is Inc 5 (#735), stacked and unmerged; implementing it here would collide with shipped-but-unmerged work.
The unconditional
return Truewas worse than an absent function: a caller trusting it believed reclamation had happened and would proceed to write files its pre-crash self still holds a live lease on. Prior claims are reclaimed by TTL expiry until Inc 5 lands — which is what the stub was silently providing anyway, minus the false assurance. Callers wanting best-effort behavior can catchNotImplementedErrorand wait out the TTL.Note on a corrected test
test_id_stability_across_two_processesasserted "Epoch should be identical across processes on first read" — the bug stated as a requirement. If two live processes share a fencing token, the token fences nothing. It now assertsepoch_2 == epoch_1 + 1across two real OS processes (the strongest B3 proof in the suite);stable_idis still asserted identical.test_read_only_valid_id_file_succeedsis replaced bytest_existing_id_file_is_loaded_and_bumped: its premise ("no write is needed on the read path") is exactly what B3 identified as the bug. The write-failure branch is now covered deterministically by mocking the atomic writer, becausechmod-based coverage is unreliable (a no-op for root on Linux CI).Verification
tests.test_lease_claimstests.test_state_store_identitytests.test_claim_backendtests.test_state_store_concurrencytests.test_state_store_pathstests.test_multi_dispatch_claimunittest discover -s testsnpm run test:nodeGates:
secret_scan --stagedCLEAN (exit 0),verify_test_suite_countOK,claudemd_sync_gateOK,verify_test_coverageOK,ci_gate_runabilityOK.BEGIN IMMEDIATEatomicity is untouched (the audit proved it sound);paths.py's own logic is untouched.Stacked PRs
This lands on
mainbeneath the stacked multibox PRs (#697 / #722 / #735 / #738 / #739); they may need a trivial rebase. #735 (Inc 5) additionally consumesrelease_own_stale, which now raises instead of returningTrue— that call site needs a decision (catch and fall back to TTL, or implement reclamation as part of Inc 5).🤖 Generated with Claude Code