diff --git a/DEBATES.md b/DEBATES.md index 5c882e4..385eb95 100644 --- a/DEBATES.md +++ b/DEBATES.md @@ -1145,3 +1145,8 @@ and real-world prediction accuracy were not tested. ## Phase 007E — Durable audit: integrity versus producer authority (2026-11-01) Thesis: canonical SHA-256 evidence supports deterministic, portable post-restart integrity checks. Antithesis: a writable data file and its unkeyed digest may be forged together; asserted authorization references are not cryptographic permissions. Resolution: implement narrowly scoped single-writer checkpoint export/import bound to re-executed World evidence, optionally check an externally anchored digest, and retain authentication/concurrency as explicit stops. No new ledger or distributed coordinator. + + +## Phase 007F — Commit acknowledgements and single-writer recovery versus physical guarantees + +Option A retained a caller-managed checkpoint but allowed accidental successful acknowledgements across ambiguous rename/fsync windows. Selected, without accepting an ADR: Option B, a narrow opt-in coordinator that validates a complete Phase 007E audit, publishes via the existing single-writer checkpoint writer, re-reads the evidence, and issues a checksum-bound receipt. Option C, consensus/database/broker infrastructure, remains unapproved. A failed directory fsync cannot authorize SUCCESS even if new content is visible. A stored checksum cannot authenticate its producer; without an externally trusted retained receipt it cannot reveal rollback to an older self-consistent image. Falsifiers and measured counterexamples: spec/financial-commit-recovery-007f.md and evals/financial-commit-recovery-007f.md. diff --git a/DECISIONS.md b/DECISIONS.md index 0c4b85a..63ab68f 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -551,3 +551,8 @@ On 2026-11-01, a bounded, explicitly opt-in Python `PopulationExecutor` adapter ## Phase 007E — Proposed, not Accepted ADR (2026-11-01) Bounded implementation record only: opt-in canonical financial audit v1 with strict trace-and-policy restoration and atomic single-writer checkpoint. No change to existing accepted DEC IDs or new Accepted ADR. Producer authentication, actual authorization and shared-state serializability remain undecided pending operator permission and separate threat-model evidence. See `spec/durable-financial-provenance-v1.md`. + + +## Phase 007F — Implementation checkpoint; no accepted architecture decision + +2026-11-01: Opt-in single-writer commit coordinator, versioned deterministic receipt and recovery validation added within the already approved domain boundary. A revalidated complete audit is required before publication; a changed checkpoint requires the previous trusted receipt; a successful acknowledgement requires publication plus revalidation; uncertain syscall outcomes raise an explicit recovery-required exception. Exactly-once network delivery, multi-writer serializability, producer authentication, crash-proof file hardware, and arbitrary valid rollback prevention are not accepted or demonstrated. Existing DEC-009 and Phase 007E contracts remain unchanged. See spec/financial-commit-recovery-007f.md. No new Accepted ADR. diff --git a/OUT_OF_THE_BOX.md b/OUT_OF_THE_BOX.md index 36a439b..bd809ea 100644 --- a/OUT_OF_THE_BOX.md +++ b/OUT_OF_THE_BOX.md @@ -543,3 +543,8 @@ Verified source-to-sale World witnesses and explicit author-asserted external tr ## Phase 007E research frontier (not implemented) Potential future externally anchored digest catalogs, HMAC with operator-owned key lifecycle, append-only authenticated logs and stateful concurrency arbitration require independent operators/requirements. Hash chaining without an external trust root cannot turn mutable records into authenticated evidence. No such capabilities are claimed or authorized in this phase. + + +## Phase 007F frontier — authenticated rollback fences and writer arbitration (unimplemented) + +Potential future research would specify operator-controlled immutable anchors, signature trust roots, durable fencing tokens, and a concrete multi-writer arbitration threat model. None is authorized by the present checksum-bound single-writer commit coordinator. A production durability promise additionally requires filesystem/hardware-specific power-loss experiments; an unanchored hash chain cannot independently prohibit older valid checkpoint substitution. diff --git a/evals/financial-commit-recovery-007f-measurements.json b/evals/financial-commit-recovery-007f-measurements.json new file mode 100644 index 0000000..134d02d --- /dev/null +++ b/evals/financial-commit-recovery-007f-measurements.json @@ -0,0 +1,151 @@ +{ + "python": "3.12.13 (main, Jul 18 2026, 16:55:18) [Clang 22.1.3 ]", + "os": "macOS-27.0.1-arm64-arm-64bit", + "source_sha256": "c64f1caaf28330152c4bc06c44f6a98b8041a8009f5dff7a8f76bed403b3bf46", + "commit_source_sha256": "ac711634db15b31f57dbdb7b59126adc3f79d6bd0e1ffa06f51b9f46c516c69e", + "provenance_version": "world-event-sale-v1", + "audit_schema": "funnelscript-financial-audit-v1", + "receipt_schema": "funnelscript-financial-commit-receipt-v1", + "memory_kind": "parent tracemalloc allocations; excludes child process", + "failure_model": "single writer, local atomic rename, external trusted receipt", + "runs": [ + { + "population": 10, + "segments": 2, + "seed": 11, + "population_seconds": 0.003071, + "financial_seconds": 0.001581, + "serialization_seconds": 0.0016, + "initial_publication_seconds": 0.008605, + "commit_publication_seconds": 0.013834, + "recovery_seconds_child": 0.002075, + "replay_seconds_child": 0.000923, + "total_seconds": 0.275531, + "tracemalloc_peak_bytes_parent": 147496, + "sales": 3, + "actions": 6, + "entries": 9, + "audit_sha256": "01d52cd9f847a78ed08a0b04304fb60d25b7611f224ae6676dbad4b12557458a", + "receipt_id": "5e2924e1450987ed3348a62105d99400dc217f86558d027a337ea27e74d998de", + "cash": "105.00", + "receipt_verified": true, + "replay_agrees": true, + "injected_rename_interruption_detected": true + }, + { + "population": 10, + "segments": 2, + "seed": 29, + "population_seconds": 0.002826, + "financial_seconds": 0.001381, + "serialization_seconds": 0.001554, + "initial_publication_seconds": 0.008415, + "commit_publication_seconds": 0.013396, + "recovery_seconds_child": 0.002123, + "replay_seconds_child": 0.000936, + "total_seconds": 0.28703, + "tracemalloc_peak_bytes_parent": 155219, + "sales": 3, + "actions": 6, + "entries": 9, + "audit_sha256": "fa261c007f4611fede17a99ec5864a4b57c42514cf7c0b0edd0795a9ca2f9664", + "receipt_id": "565bda19c2930c0f504e814a5474836fc7f505c3edb1446f095e00dd9d5f57a0", + "cash": "105.00", + "receipt_verified": true, + "replay_agrees": true, + "injected_rename_interruption_detected": true + }, + { + "population": 100, + "segments": 2, + "seed": 11, + "population_seconds": 0.011599, + "financial_seconds": 0.009224, + "serialization_seconds": 0.010509, + "initial_publication_seconds": 0.052181, + "commit_publication_seconds": 0.084247, + "recovery_seconds_child": 0.008696, + "replay_seconds_child": 0.005262, + "total_seconds": 0.578981, + "tracemalloc_peak_bytes_parent": 842966, + "sales": 30, + "actions": 33, + "entries": 63, + "audit_sha256": "8bdaac8d169e813054303bd1a462105f4ea2b3a696d4105b9f1719d898a87e7c", + "receipt_id": "1c1ffffca0df9efac3f0aa5c0f99da86a441aa7e560426de8a350a5d4ef3d92a", + "cash": "1050.00", + "receipt_verified": true, + "replay_agrees": true, + "injected_rename_interruption_detected": true + }, + { + "population": 100, + "segments": 2, + "seed": 29, + "population_seconds": 0.010714, + "financial_seconds": 0.009333, + "serialization_seconds": 0.010341, + "initial_publication_seconds": 0.054047, + "commit_publication_seconds": 0.080507, + "recovery_seconds_child": 0.008669, + "replay_seconds_child": 0.005264, + "total_seconds": 0.568981, + "tracemalloc_peak_bytes_parent": 839996, + "sales": 30, + "actions": 33, + "entries": 63, + "audit_sha256": "fd1ba43b2d53c986f3609dde46120bf7d48affa465d593b13df2e27f2793613c", + "receipt_id": "d84ee18a1f77002e977da686ac39debaab4faa8226a8e5c2ffc2456140996b43", + "cash": "1050.00", + "receipt_verified": true, + "replay_agrees": true, + "injected_rename_interruption_detected": true + }, + { + "population": 1000, + "segments": 2, + "seed": 11, + "population_seconds": 0.084326, + "financial_seconds": 0.091919, + "serialization_seconds": 0.102311, + "initial_publication_seconds": 0.536631, + "commit_publication_seconds": 0.875634, + "recovery_seconds_child": 0.06497, + "replay_seconds_child": 0.047724, + "total_seconds": 3.756048, + "tracemalloc_peak_bytes_parent": 7579168, + "sales": 300, + "actions": 303, + "entries": 603, + "audit_sha256": "8dc0984352f736df426e12a250682c1e5c4f2f41f1344fb9316604444bca3f50", + "receipt_id": "e2505329c54bfbae5179759c0b80e8fa3db3aae9b82ec8de225650936f8967c3", + "cash": "10500.00", + "receipt_verified": true, + "replay_agrees": true, + "injected_rename_interruption_detected": true + }, + { + "population": 1000, + "segments": 2, + "seed": 29, + "population_seconds": 0.087176, + "financial_seconds": 0.085504, + "serialization_seconds": 0.093552, + "initial_publication_seconds": 0.471817, + "commit_publication_seconds": 0.88301, + "recovery_seconds_child": 0.066213, + "replay_seconds_child": 0.05268, + "total_seconds": 3.66868, + "tracemalloc_peak_bytes_parent": 7601305, + "sales": 300, + "actions": 303, + "entries": 603, + "audit_sha256": "67a701966bab42d3490b8a0b47efe9f6081c475140a2aded1019ec606569d885", + "receipt_id": "c014601b07b914ddd84761cdca29a0120d6e79dcd4dff1216c90d1a384776f7c", + "cash": "10500.00", + "receipt_verified": true, + "replay_agrees": true, + "injected_rename_interruption_detected": true + } + ] +} diff --git a/evals/financial-commit-recovery-007f.md b/evals/financial-commit-recovery-007f.md new file mode 100644 index 0000000..79b1afe --- /dev/null +++ b/evals/financial-commit-recovery-007f.md @@ -0,0 +1,52 @@ +# Phase 007F — Verified Local Experiments and Failure Model + +Research hypothesis contract: [financial-commit-recovery-007f.md](../spec/financial-commit-recovery-007f.md). Machine-readable evidence: [financial-commit-recovery-007f-measurements.json](financial-commit-recovery-007f-measurements.json). Runner: application `scripts/phase007f_commit_experiment.py`. Independent measurements validator: `tests/test_financial_commit_recovery_reference.py`. + +## Experimental observations (local; not field durability proof) + +Python 3.12.13, macOS 27.0.1 arm64. Each of six configurations executes World transitions with 2 segments, reconciles Phase 007D actions against an independently specified integer-minor-unit oracle, creates an initial checkpoint, injects a pre-rename failure while attempting a second financial state, recovers the unchanged initial state, commits the second state, verifies its SHA-256-bound deterministic receipt, then restores and replays all actions in a new Python process. All six experiments completed with receipt equality and replay accounting agreement. A separately covered deterministic directory fsync failure yields `CommitOutcomeUncertain` rather than success. These tests simulate failures at Python syscall boundaries; they do not simulate actual power loss or corrupted hardware. + +| Agents | Seeds | Sales each | Final cash (SAR) | Commit publication (seconds, observed) | Child recovery (seconds, observed) | +| --- | --- | --- | --- | --- | --- | +| 10 | 11, 29 | 3 | 105.00 | 0.013834, 0.013396 | 0.002075, 0.002123 | +| 100 | 11, 29 | 30 | 1,050.00 | 0.084247, 0.080507 | 0.008696, 0.008669 | +| 1,000 | 11, 29 | 300 | 10,500.00 | 0.875634, 0.883010 | 0.064970, 0.066213 | + +Source and commit implementation SHA-256 fingerprints, OS and interpreter versions, raw timings, parent traced peak memory, evidence hashes, transaction counts and commit identities are present in the machine-readable file. The parent process memory excludes child allocations. Single-run wall time is not a scaling estimator or performance guarantee. + +## Fault injection and evidence coverage + +| Fault / claim | Test evidence | Expected response | +| --- | --- | --- | +| Commit and deterministic receipt | `test_commit_receipt_public_contract` | Canonical receipt matches recovered audit and exact ledger | +| Second commit and retry | `test_second_commit_idempotent_settlement_writeoff_and_independent_oracle` | No double posting | +| Changed checkpoint without trusted previous anchor | `test_changed_state_without_trusted_previous_anchor_fails_closed` | Reject | +| Valid but older checkpoint substitution | `test_outdated_anchor_and_valid_old_checkpoint_substitution_are_detected` | Detect only with externally retained trusted receipt | +| Changed amount/kind/actor/authorization/trial/currency | `test_conflicting_candidate_never_replaces_checkpoint` | Reject | +| Authorization reuse; truncating committed transaction history | `test_reused_authorization_and_removed_old_action_fail_before_publish` | Reject | +| Corrupt checkpoint, forged receipt, wrong policy/World | `test_receipt_mutations_wrong_trial_world_policy_and_damage_fail_closed` | Reject | +| Temporary creation/file sync/rename/parent directory sync failure | `test_publication_failure_never_returns_success_ack` | Uncertain, never acknowledge success | +| Lost acknowledgement and subsequent retry | `test_ambiguous_post_rename_commit_resolved_by_restart_recovery` | Inspect persisted evidence; retry same candidate without reposting | +| Python hash seed variation and restart | `test_cross_process_receipt_determinism_and_recovery` and six process experiments | Same committed identity | +| Missing source event; forged event; over-settlement/writeoff; delay; horizon | Existing `test_durable_financial_audit.py`, `test_financial_consequence_provenance.py` | Reject invalid reference | +| Agent enrollment reordering and unrelated-agent insertion | Existing Phase 007A/007C population conformance tests | Keyed per-agent invariance within declared independence | +| Shared-budget coupling; multi-writer race | Counterexamples remain architectural STOP | No claimed serializability | +| Actual sudden power loss / untrusted storage | Not tested | No claim of physical persistence or authentication | + +## Genuine TDD receipts and review findings + +- RED 1: `uv run pytest -q tests/test_financial_commit_recovery.py::test_commit_receipt_public_contract` returned exit 1: `AttributeError: module 'funnelscript.stdlib.marketing' has no attribute 'commit_financial_checkpoint'` (missing public behavior). +- GREEN 1: rerunning the identical command after the minimal initial coordinator implementation returned exit 0, `1 passed`. +- RED 2 (adversarial security review): `uv run pytest -q tests/test_financial_commit_recovery.py::test_idempotent_retry_must_not_ack_when_durability_sync_fails` returned exit 1, `Failed: DID NOT RAISE CommitOutcomeUncertain`. An existing image was incorrectly acknowledged on retry without re-synchronizing. +- GREEN 2: after adding a file-and-parent-directory `fsync` gate plus post-sync revalidation, the Phase 007F suite returned exit 0, `19 passed`. A transitory code mistake (calling the policy object as a function) produced two additional failing tests and was corrected; the tests were not weakened. +- Full final local application `uv run pytest -q` and `uv run pytest -v`: `541 passed, 58 subtests`. `uv build`, `compileall`, Bun `test` and `build`, installed-wheel smoke and `git diff --check` succeeded. +- Research `uv run --no-project --with pytest pytest -q/-v`: `55 passed, 149 subtests`. These commands are semantically equivalent to `uv run pytest` for this no-project research repository. +- Manual non-Codex evidence-based review: isolated opt-in module, no new dependencies, no modifications to compiler or World execution, exact `MarketingLedger` remains sole money engine, and full audit/policy/World revalidation happens before replacing bytes. A2M Code Review Graph could not inspect the isolated worktree because it lies outside its approved workspace; no independent graph-review result is claimed. + +Open boundaries: untrusted paths, compromised OS/filesystem, physical power-loss durability and multi-writer arbitration are outside verified scope; no fabricated independent security approval, signature or true rollback-proof authority is claimed. + +## Security and recovery scope + +The public coordinator is opt-in and single-writer. The file path and original World/policy are assumed trusted. The candidate must pass `export_audit` revalidation and the checkpoint is re-read and revalidated before success. The versioned receipt includes trial identity, complete checkpoint checksum, ledger identity set, fixed-minor-unit snapshot fingerprint and domain-separated derived identifier. Receipt equality is only checksum-bound consistency under known inputs. Directory `fsync` completion and supported filesystem semantics are prerequisites for the documented acknowledgement path; external power-fail atomicity and stable-media durability were not independently established. A process dying after `replace` but before successful directory sync produces an uncertain commit, irrespective of the candidate's present namespace visibility. An externally retained receipt can flag older valid content substitution, but a receipt stored next to the checkpoint on equally untrusted media is not an anti-rollback anchor. + +The protocol is not a database, does not serialize competing processes, does not authenticate the claimed external financial actor, does not prove real-world economic causation, and does not provide network/physical exactly-once delivery. No accepted ADR was created. No keys, signing infrastructure, distributed locks or extra accounting engine were added. diff --git a/spec/financial-commit-recovery-007f.md b/spec/financial-commit-recovery-007f.md new file mode 100644 index 0000000..8ecb2ec --- /dev/null +++ b/spec/financial-commit-recovery-007f.md @@ -0,0 +1,48 @@ +# Phase 007F — Crash-Consistent Single-Writer Financial Commit Contract (Proposed) + +Status: research hypothesis and implementation evaluation; not an accepted ADR. + +## Research anchors and falsifiable hypotheses + +Grounding: write-ahead/logical transaction recovery distinctions (Gray and Reuter, *Transaction Processing*, 1992); POSIX rename atomic namespace replacement and fsync boundaries; deterministic replay and content-addressed evidence. This contract does not claim stable-media correctness on every filesystem, producer identity authentication, Byzantine recovery, or concurrent-writer serializability. + +- H-FCOMMIT1: a candidate must round-trip through Phase 007E World/policy/audit/ledger validation before publication. Falsifier: a rejected candidate changes the checkpoint. +- H-FCOMMIT2: receipt identity must be a deterministic digest of the audit, trial, operation identities and ledger snapshot. Falsifier: receipt verifies for different evidence. +- H-FCOMMIT3: a success receipt must follow successful publication plus re-read/validation. A failure after rename and before directory fsync is uncertain. Falsifier: an injected post-rename failure returns SUCCESS. +- H-FCOMMIT4: identical redelivery cannot change state, whereas conflicting identifiers fail. Falsifier: duplicated replay changes cash or receivables. +- H-FCOMMIT5: on restart the persisted checkpoint, not the attempted write, determines observed state. Falsifier: a nonexistent posting is recovered. +- H-FCOMMIT6: cash = initial + settlements - ad spend - other costs; receivables = sales - settlements - writeoffs; profit = sales - ad spend - other costs - writeoffs, exactly in minor units. Falsifier: restored snapshot differs from the independent reference oracle. +- H-FCOMMIT7: byte-identical inputs yield identical receipts across Python processes and hash seeds. Falsifier: a cross-process mismatch. +- H-FCOMMIT8: single trusted writer, trusted input World, externally asserted authorizations and a trusted external receipt anchor are preconditions; local digests alone cannot prove source identity, prevent valid older checkpoint substitution or arbitrate concurrent writes. Falsifier: documentation or tests claim more than this. + +## Architectural alternatives + +A: caller-managed lifecycle — smallest change, but cannot standardize ambiguous acknowledgements. +B: opt-in domain financial commit coordinator using the existing canonical audit, exact ledger and atomic writer. Selected for direct testability and limited production surface. +C: generic distributed transactional infrastructure — rejected: no requirement for consensus, broker, multi-writer or a second ledger. + +## State machine and intended contracts + +ABSENT → VALIDATED → PUBLICATION_IN_PROGRESS → (COMMITTED_VERIFIED | PUBLICATION_UNCERTAIN | NOT_PUBLISHED). +COMMITTED_VERIFIED → RECOVERED_VERIFIED → (UNCHANGED_IDEMPOTENT | VALIDATED_NEXT). +Recovered validation uses original World and policy. Caller-supplied previous receipt is an optional trusted compare-before-publish anchor for subsequent state updates; a missing anchor on an existing different checkpoint must fail closed. Same-state receipt after restart is idempotent. Retained older valid content is not detectable without externally retained trusted receipt. + +An OSError during a write is not proof of rollback. On an interrupted publication, inspection of persisted evidence is mandatory; if a failed directory fsync leaves a candidate image, return no success receipt, signal uncertainty, and require a later recovery/revalidation. No interface may claim crash durability beyond filesystem and OS guarantees. + +## Receipt scope + +Deterministic, versioned receipt fields: trial, audit schema, SHA-256 digest of complete canonical checkpoint bytes, exact financial transaction identifiers, exact minor-unit financial-state fingerprint, and derived domain-separated commit identity. No process clock, random salt, untrusted authorization proof or secret key. A valid receipt proves equality with the revalidated local financial evidence under declared trusted inputs, not authenticated authority. + +## Independent reference oracle + +SAR 100 initial cash; SAR 20 ad spend; SAR 5 other cost; SAR 60 recognized sales; three SAR 10 delayed settlements. Day 1 cash 75, receivables 60, revenue 60, receipts 0. Day 3 cash 105, receivables 30, revenue 60, receipts 30, profit 35. With a SAR 5 writeoff: cash 105, receivables 25, profit 30. This oracle is expressed independently of the production ledger implementation. + +## Failure boundaries + +- Before namespace replacement: old valid image or no image should survive, absent a malicious or concurrently writing actor. +- After namespace replacement but before successful parent fsync: crash persistence is uncertain; surviving content must be inspected, not guessed. +- After success and lost acknowledgement: recovering and comparing receipt allows identical retry without additional posting. +- Wrong trial/currency/World/policy or corrupt/truncated image: fail closed. +- Older valid image and competing writers: require an external trusted anchor and separately enforced single-writer ownership respectively; never describe either protection as intrinsic authentication/serializability. + +No accepted architecture decision or multi-writer infrastructure is authorized by this research proposal. diff --git a/tests/test_financial_commit_recovery_reference.py b/tests/test_financial_commit_recovery_reference.py new file mode 100644 index 0000000..58a87f7 --- /dev/null +++ b/tests/test_financial_commit_recovery_reference.py @@ -0,0 +1,47 @@ +"""Phase 007F independent numerical and evidence-contract validation. + +No import of the production ledger or checkpoint implementation. +""" + +from decimal import Decimal +from hashlib import sha256 +import json +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +DATA = ROOT / "evals/financial-commit-recovery-007f-measurements.json" + + +def test_six_measured_commits_reconcile_against_independent_minor_unit_oracle(): + report = json.loads(DATA.read_text()) + assert report["receipt_schema"] == "funnelscript-financial-commit-receipt-v1" + assert report["audit_schema"] == "funnelscript-financial-audit-v1" + assert len(report["source_sha256"]) == len(report["commit_source_sha256"]) == 64 + assert {(r["population"], r["seed"]) for r in report["runs"]} == { + (n, seed) for n in (10, 100, 1000) for seed in (11, 29)} + for row in report["runs"]: + n = row["population"] + sales = (n // 10) * 3 + assert row["sales"] == sales + assert row["actions"] == sales + 3 + assert row["entries"] == sales * 2 + 3 + cash_minor = (n * 1000) + (sales * 1000) - (n * 200) - (n * 50) + assert Decimal(row["cash"]) == Decimal(cash_minor) / 100 + assert (row["receipt_verified"], row["replay_agrees"], + row["injected_rename_interruption_detected"]) == (True, True, True) + assert len(row["audit_sha256"]) == len(row["receipt_id"]) == 64 + for field in ( + "population_seconds", "financial_seconds", "serialization_seconds", + "initial_publication_seconds", "commit_publication_seconds", + "recovery_seconds_child", "replay_seconds_child", "total_seconds", + ): + assert type(row[field]) in (float, int) and row[field] >= 0 + assert row["tracemalloc_peak_bytes_parent"] > 0 + + +def test_report_file_provenance_matches_documented_runner_source(): + report = json.loads(DATA.read_text()) + assert report["memory_kind"] == "parent tracemalloc allocations; excludes child process" + assert "single writer" in report["failure_model"] + assert report["python"] and report["os"] + assert len({(r["population"], r["seed"]) for r in report["runs"]}) == 6