Merge pull request #37 from engineeredsoftware/v30/gate-5-testnet-ledger-projection-hardening

V30 Gate 5: Testnet Ledger Projection Hardening

Author: geraldarthurdavis

.github/workflows/bitcode-gate-quality.yml

@@ -83,6 +83,7 @@ jobs:
             node scripts/check-v30-gate2-protocol-package-api-boundaries.mjs --skip-branch-check
             node scripts/check-v30-gate3-bitcoin-taproot-psbt-fee-rigor.mjs --skip-branch-check
             node scripts/check-v30-gate4-btd-assetpack-mint-read-receipts.mjs --skip-branch-check
+            node scripts/check-v30-gate5-testnet-ledger-projection-hardening.mjs --skip-branch-check
           else
             echo "Unexpected BITCODE_SPEC.txt pointer: $POINTER" >&2
             exit 1

BITCODE_SPEC_V30.md

@@ -353,7 +353,9 @@ The canonical report is produced from:
 
 - ledger observed facts: fee receipts, ledger anchors, journal entries, BTD ranges, ownership events, read licenses, and finality state;
 - database projected facts: the Supabase readback rows that claim those ledger facts are durable in the application projection;
+- object-storage artifact facts: source-safe preview, pipeline evidence, telemetry, delivery manifest, and ledger projection artifact roots that prove artifacts are durable without exposing protected source;
 - private metaphysical canonical facts: protected source metadata, need/fit context, access policy documents, encrypted storage pointers, disputes, and telemetry context represented only by roots;
+- staging-testnet readback facts: secret-free Supabase project reference, REST/DB hosts, out-of-band admin-credential state, table readback counts, and synchronized-or-blocked state;
 - settlement conservation checks: BTC debit/credit and fee/payment roots that must conserve before unlock;
 - delivery evidence: post-settlement pull-request visibility and recovery posture.
 
@@ -363,24 +365,33 @@ Drift is classified before repair:
 - `ledger_root_mismatch`: the database projection points at a different root than the ledger observation;
 - `ledger_finality_mismatch`: the projected finality differs from the observed finality;
 - `database_orphan_projection`: the database projects a fact that has no matching ledger observation;
+- `missing_object_storage_artifact`: the ledger or projection expects an artifact root that is not durable yet;
+- `object_storage_root_mismatch`: the database projection and object-storage artifact root disagree;
+- `staging_testnet_readback_blocked`: staging-testnet Supabase readback cannot prove the expected rows without retry;
 - `settlement_conservation_drift`: BTC debit/credit or fee/payment accounting does not conserve.
 
 Repair actions are canonical and auditable:
 
 - `retry_database_readback`;
+- `retry_object_storage_write`;
+- `retry_staging_testnet_readback`;
 - `project_ledger_fact`;
 - `update_finality_state`;
 - `quarantine_database_projection`;
+- `quarantine_object_storage_artifact`;
 - `pause_settlement_unlock`;
 - `recover_delivery`.
 
 The reconciliation state is one of aligned, retryable, repairable, approval required, or blocked.
 Confirmed ledger facts that are missing from the database require operator-approved projection repair.
 Reorged or failed finality, database-only orphan projection, or settlement conservation drift blocks unlock and delivery.
+Missing durable object-storage artifacts are retryable unlock blockers.
+Object-storage root mismatches require quarantine until the artifact and projection agree.
 Delivery recovery is allowed only when settlement is otherwise aligned and the full AssetPack delivery target is not visible.
 
-Terminal must show drift classes, blockers, repair actions, proof roots, observed facts, projected facts, canonical facts, journal entries, and repair receipts before raw payloads.
+Terminal must show drift classes, blockers, repair actions, proof roots, observed facts, projected facts, object-storage facts, canonical facts, journal entries, and repair receipts before raw payloads.
 The Vercel Sandbox harness must store the reconciliation report as settlement evidence when it claims ledger readback.
+Supabase staging-testnet readback receipts must never store service-role JWTs, `sb_secret__` keys, OpenAI keys, database passwords, or any raw secret in tracked code or persisted proof payloads; only host/project identifiers, credential presence state, and proof roots are admissible.
 The API may persist schema-compatible repair receipts while richer repair actions and proof roots remain report evidence until the registry schema is formally expanded.
 
 ## V30 canonical subsystem surfaces

BITCODE_SPEC_V30_DELTA.md

@@ -178,6 +178,39 @@ Closure acceptance:
 - Supabase staging-testnet readback can prove synchronized or blocked state without secrets in tracked files;
 - reconciliation tests cover drift, quarantine, retry, and unlock blocking.
 
+Gate 5 implementation centers on `packages/btd/src/reconciliation.ts`.
+`LedgerDatabaseReconciliationReport` now carries distinct ledger-observed
+facts, database-projected facts, object-storage artifact facts, private
+metaphysical facts, settlement-conservation checks, and optional
+`SupabaseStagingTestnetProjectionReadback` evidence. Object-storage facts name
+artifact kind, source visibility, durable root, protected-source/encryption
+posture, and manifest root. Supabase readback receipts name only safe project
+and host identifiers, table counts, synchronized-or-blocked state, credential
+presence, and proof roots; secret-looking values are rejected.
+
+The deterministic repair surface now includes
+`missing_object_storage_artifact`, `object_storage_root_mismatch`, and
+`staging_testnet_readback_blocked` drift classes plus
+`retry_object_storage_write`, `retry_staging_testnet_readback`, and
+`quarantine_object_storage_artifact` repair actions. Missing durable artifacts
+block unlock as retryable repairs, object-storage root mismatches require
+quarantine/approval, and staging-testnet readback failures remain blocked until
+the table readback proves synchronized.
+
+The Sandbox harness emits object-storage artifact roots and secret-free
+staging-testnet readback receipts into settlement reconciliation evidence.
+Terminal reconciliation now renders object-storage artifacts as a fourth fact
+group beside ledger, database, and metaphysical facts and includes report proof
+roots and repair actions in the existing journal repair cockpit.
+
+Gate 5 evidence is covered by
+`packages/btd/__tests__/reconciliation.test.ts`,
+`packages/api/src/routes/__tests__/btd-crypto.test.ts`,
+`packages/pipeline-hosts/src/__tests__/asset-pack-harness.test.ts`,
+`uapi/tests/terminalJournalReconciliation.test.ts`,
+`uapi/tests/terminalTransactionDetailSnapshot.test.ts`, and
+`scripts/check-v30-gate5-testnet-ledger-projection-hardening.mjs`.
+
 ### Gate 6: Source-To-Shares Proof Cleanup
 
 Gate 6 cleans contribution measurement and settlement conservation.

BITCODE_SPEC_V30_NOTES.md

@@ -137,6 +137,36 @@ coerce receipt payloads under camelCase or database-style snake_case keys, and
 the transaction read model counts receipt payloads in closure and journal
 availability.
 
+## Gate 5 testnet ledger projection notes
+
+Gate 5 extends the V29 reconciliation cockpit from ledger/database/private-root
+separation into a four-fact projection model:
+
+- ledger observed facts remain fee, anchor, journal, range, ownership, license,
+  and finality observations;
+- database projected facts remain Supabase rows and projected roots/finality;
+- object-storage artifact facts now carry source-safe preview, evidence,
+  telemetry, delivery, and ledger-projection roots separately from database
+  rows;
+- private metaphysical facts remain root-only protected source and policy
+  context.
+
+Object storage may prove a durable artifact exists, but it may not leak
+protected source before settlement. Unencrypted protected source artifacts are
+rejected. Encrypted protected source artifacts are admissible only as roots and
+encrypted posture, not visible source.
+
+Supabase staging-testnet readback is an explicit receipt. It records project
+ref, REST host, optional DB host, admin credential presence state, table readback
+counts, synchronized-or-blocked state, and proof root. It rejects secret-shaped
+values so tracked files, telemetry, and persisted proof payloads cannot carry
+service-role JWTs, `sb_secret__` keys, OpenAI keys, or database passwords.
+
+Gate 5 keeps the existing registry table shape for repair rows. The API
+persists schema-compatible repair receipts; richer drift classes, repair
+actions, readback receipts, and proof roots remain in the report payload until a
+future schema expansion admits more columns.
+
 4. **Gate 4: BTD AssetPack Mint And Read Receipts**
    - Make BTD mint, read, and rights-transfer receipts typed, proof-rooted, stored, streamed, and source-safe.
 

BITCODE_SPEC_V30_PARITY_MATRIX.md

@@ -58,7 +58,7 @@ No `_legacy/` source is active source truth.
 | Protocol package API boundaries | Gate 2 | `packages/btd/src/api-boundaries.ts`, `packages/api/src/routes/btd-crypto.ts`, package READMEs/tests | drafted | Shared Protocol/BTD objects have package-owned builders, parsers, validators, JSON-safe serializers, and tests. |
 | Bitcoin Taproot PSBT fee rigor | Gate 3 | `packages/btd/src/btc-fee-operation.ts`, `packages/btd/src/bitcoin-fees.ts`, BTD/API tests, gate checker | drafted | BTC fee and signer states are typed, testnet/mainnet-safe, no-custody, Taproot/PSBT aware, and proof-rooted. |
 | BTD AssetPack mint/read receipts | Gate 4 | `packages/btd/src/receipts.ts`, `packages/btd/src/api-boundaries.ts`, asset-pack harness evidence, Terminal detail snapshot/read model tests, gate checker | drafted | Mint, read, and rights-transfer receipts bind BTD range, preview, paid unlock, delivery, and ledger projection. |
-| Testnet ledger projection hardening | Gate 5 | BTD reconciliation, Supabase readback, object-storage evidence, repair tests | pending | Ledger/database/object-storage projections are synchronized or blocked with deterministic repair posture. |
+| Testnet ledger projection hardening | Gate 5 | `packages/btd/src/reconciliation.ts`, API route tests, asset-pack harness evidence, Terminal journal reconciliation UI/tests, gate checker | drafted | Ledger/database/object-storage/private facts are distinct; staging-testnet readback is secret-free; drift, quarantine, retry, and unlock blocking are tested. |
 | Source-to-shares proof cleanup | Gate 6 | BTD/source-to-shares proof primitives, settlement conservation tests | pending | Measurement contribution, fee allocation, zero-cell/refit tail, and conservation invariants are testable. |
 | Bridge-readiness research boundaries | Gate 7 | Protocol/BTD research notes, policy posture tests, docs | pending | Bridge paths are documented as research until admitted by explicit future proof and policy. |
 | Protocol telemetry/proof hooks | Gate 8 | telemetry schema, proof hooks, generated-artifact inventory, tests | pending | Receipts, fee states, projections, and source-to-shares facts emit source-safe telemetry and proof hooks. |
@@ -150,6 +150,15 @@ No `_legacy/` source is active source truth.
 | Tests cover package and Terminal receipt posture | `packages/btd/__tests__/api-boundaries.test.ts`, `uapi/tests/terminalTransactionDetailSnapshot.test.ts`, and `uapi/tests/terminalTransactionReadModel.test.ts` | drafted |
 | Gate checker protects the receipt surface | `scripts/check-v30-gate4-btd-assetpack-mint-read-receipts.mjs`, `pnpm run check:v30-gate4`, gate-quality workflow | drafted |
 
+## Gate 5 Parity
+
+| Requirement | Source evidence | Current V30 judgment |
+| --- | --- | --- |
+| Ledger projection report separates four fact classes | `packages/btd/src/reconciliation.ts`, `uapi/app/terminal/terminal-journal-reconciliation.ts`, `uapi/app/terminal/TerminalTransactionJournalReconciliationCard.tsx` | drafted |
+| Object-storage and staging-testnet repair classes are deterministic | `packages/btd/__tests__/reconciliation.test.ts`, `packages/api/src/routes/__tests__/btd-crypto.test.ts` | drafted |
+| Harness and Terminal consume the same report evidence | `packages/pipeline-hosts/src/asset-pack-harness.ts`, `packages/pipeline-hosts/src/__tests__/asset-pack-harness.test.ts`, `uapi/tests/terminalJournalReconciliation.test.ts`, `uapi/tests/terminalTransactionDetailSnapshot.test.ts` | drafted |
+| Gate checker protects projection hardening | `scripts/check-v30-gate5-testnet-ledger-projection-hardening.mjs`, `pnpm run check:v30-gate5`, gate-quality workflow | drafted |
+
 ## Gate 4 accepted boundaries
 
 - Gate 4 does not require production-mainnet BTC settlement.
@@ -158,6 +167,14 @@ No `_legacy/` source is active source truth.
 - Gate 4 does not harden all ledger/database/object-storage projection repair; Gate 5 owns that.
 - Gate 4 does not finish source-to-shares contribution cleanup; Gate 6 owns that.
 
+## Gate 5 accepted boundaries
+
+- Gate 5 does not expand the physical reconciliation repair registry schema.
+- Gate 5 does not store or print Supabase service-role JWTs, `sb_secret__` keys, OpenAI keys, database passwords, or Vercel tokens.
+- Gate 5 does not make protected AssetPack source visible before paid unlock and delivery admission.
+- Gate 5 does not implement source-to-shares contribution accounting; Gate 6 owns that.
+- Gate 5 does not promote V30 or change the active canon pointer.
+
 ## completion condition
 
 Gate 1 is complete when the V30 draft family validates, `check:v30-gate1` passes, workflow posture is V30-aware, README and roadmap reflect V30 initiation, V31-V37 scopes are current enough to guide future gates, diff hygiene passes, and the gate branch is committed and pushed for review into `version/v30`.

package.json

@@ -58,6 +58,7 @@
     "check:v30-gate2": "node scripts/check-v30-gate2-protocol-package-api-boundaries.mjs",
     "check:v30-gate3": "node scripts/check-v30-gate3-bitcoin-taproot-psbt-fee-rigor.mjs",
     "check:v30-gate4": "node scripts/check-v30-gate4-btd-assetpack-mint-read-receipts.mjs",
+    "check:v30-gate5": "node scripts/check-v30-gate5-testnet-ledger-projection-hardening.mjs",
     "check:spec-quality": "node scripts/run-bitcode-spec-quality.mjs --mode basic",
     "check:spec-quality:title": "node scripts/run-bitcode-spec-quality.mjs --mode strict-from-title",
     "check:spec-quality:v24": "node scripts/run-bitcode-spec-quality.mjs --mode strict-version --version V24",

packages/api/src/routes/__tests__/btd-crypto.test.ts

@@ -1338,6 +1338,24 @@ describe('BTD crypto API builders', () => {
               projectedLedgerRoot: 'confirmed-root',
               projectedFinalityState: 'broadcast',
             },
+            {
+              factId: 'artifact-api-1',
+              projectedLedgerRoot: 'artifact-ledger-root',
+              projectedFinalityState: 'prepared',
+              projectedObjectStorageRoot: 'artifact-database-root',
+            },
+          ],
+          objectStorageArtifacts: [
+            {
+              factId: 'artifact-api-1',
+              artifactId: 'artifact-api-1',
+              artifactKind: 'pipeline_evidence',
+              storageRoot: 'artifact-storage-root',
+              sourceVisibility: 'proof_public',
+              durable: true,
+              containsProtectedSource: false,
+              encrypted: false,
+            },
           ],
           metaphysicalFacts: [
             {
@@ -1348,6 +1366,27 @@ describe('BTD crypto API builders', () => {
               private: true,
             },
           ],
+          stagingTestnetReadback: {
+            kind: 'btd.supabase_projection_readback',
+            readbackId: 'staging-readback-api-1',
+            lane: 'staging-testnet',
+            supabaseProjectRef: 'tkpyosihuouusyaxtbau',
+            restHost: 'tkpyosihuouusyaxtbau.supabase.co',
+            adminCredentialState: 'provided_out_of_band',
+            secretValuesStored: false,
+            tableReadbacks: [
+              {
+                table: 'btd_asset_pack_ranges',
+                expectedCount: 1,
+                observedCount: 1,
+                synchronized: true,
+              },
+            ],
+            state: 'synchronized',
+            blockingReasons: [],
+            proofRoot: 'btd-proof-root:supabase-projection-readback:api',
+            issuedAt,
+          },
           settlementConservationChecks: [
             {
               checkId: 'settlement-conservation-api-1',
@@ -1369,6 +1408,9 @@ describe('BTD crypto API builders', () => {
     expect(body.report.blocking).toBe(true);
     expect(body.report.state).toBe('blocked');
     expect(body.report.driftKindCounts.settlement_conservation_drift).toBe(1);
+    expect(body.report.driftKindCounts.object_storage_root_mismatch).toBe(1);
+    expect(body.report.objectStorageArtifacts[0].storageRoot).toBe('artifact-storage-root');
+    expect(body.report.stagingTestnetReadback.secretValuesStored).toBe(false);
     expect(body.report.metaphysicalFacts[0].canonicalRoot).toBe('private-source-root');
     expect(body.committed).toBe(true);
     expect(insertReconciliationRepair).toHaveBeenCalledWith(

packages/btd/README.md

@@ -15,6 +15,10 @@ This package owns:
   Reader and Depositor identities, source-safe preview roots, paid unlock,
   delivery admission, and ledger projection roots without leaking protected
   source before settlement
+- ledger/database/object-storage projection reconciliation, including
+  deterministic repair classes, source-safe object artifact roots,
+  secret-free Supabase staging-testnet readback receipts, quarantine/retry
+  actions, and settlement-unlock blocking posture
 - BTC fee operation posture, including quote lifecycle, signer recovery,
   no-server-custody PSBT handoff, Taproot/script posture, broadcast/finality
   observation, replacement/reorg repair, and testnet/mainnet network policy
@@ -44,6 +48,8 @@ import {
   buildBtdReadReceiptBoundarySettlement,
   buildBtdRightsTransferReceipt,
   buildBtdRegistrySnapshot,
+  buildSupabaseStagingTestnetProjectionReadback,
+  reconcileLedgerDatabaseProjection,
   toBtdJsonSafe,
   calculateLlmBtcFeeEstimate,
   buildLicensedReadRevenueRoute,