Skip to content

Commit 9d2edfa

Browse files
wip v28
1 parent 0257b4d commit 9d2edfa

25 files changed

Lines changed: 6169 additions & 1334 deletions

.env.example

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ NEXT_PUBLIC_SOFT_LAUNCH=false
3636
NEXT_PUBLIC_SUPABASE_ANON_KEY=<Supabase anon key>
3737
NEXT_PUBLIC_SUPABASE_URL=<Supabase project URL>
3838
SUPABASE_ANON_KEY=<Supabase anon key>
39+
SUPABASE_DB_URL=<Supabase Postgres connection string for data-health checks>
3940
SUPABASE_JWT_SECRET=<Supabase JWT secret>
4041
SUPABASE_PUBLISHABLE_KEY=<Supabase publishable key>
4142
SUPABASE_SECRET_KEY=<Supabase secret key>

BITCODE_SPEC_V28_NOTES.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@ The V28 natural path is:
6464

6565
This recalibration means the QA document, parity matrix, and implementation work should now classify every finding by the lowest-level source-of-truth it affects: wallet identity, GitHub/source scope, Fit-finding, AssetPack synthesis, ledgerized journal, PostgreSQL projection, pipeline runtime, telemetry/alerting, or legacy residue.
6666

67+
Supabase/PostgreSQL data-health checks are part of the V28 acceptance apparatus, not a separate operator convenience. The ORM package now owns reusable schema, identity, Terminal, ledger, and operational check manifests plus live runners for daily staging/production validation, CI E2E validation, local QA reports, and generated schema-type coverage. These checks must preserve the core boundary: Supabase rows are ledger-derived or Bitcode-canonical projections, while ledger anchors, signed wallet proofs, Terminal journal receipts, and canonical receipts remain the source of cryptographic truth.
68+
6769
### Deterministic Model Boundary
6870

6971
V28 must remove the idea that a user can choose ledgerized Bitcode synthesis models.

BITCODE_SPEC_V28_PARITY_MATRIX.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ It links manual screenshots, console logs, server logs, Supabase SQL checks, and
113113
| --- | --- | --- | --- |
114114
| BTC/D identity and wallet-backed session | `uapi/app/api/wallet/_shared.ts`, `uapi/app/api/wallet/authenticate/route.ts`, `uapi/app/api/wallet/oauth/authorization-code/route.ts`, `uapi/app/api/wallet/oauth/token/route.ts`, `uapi/app/api/wallet/oauth/userinfo/route.ts`, `uapi/lib/bitcoin-wallet-client.ts`, `uapi/lib/bitcode-wallet-local.ts`, `packages/btd/src/wallet.ts`, `packages/btd/src/bitcoin-provider.ts`, `supabase/migrations/003_user_connections_provider_scope.sql`, `uapi/tests/bitcoinWalletClient.test.ts` | partial | Live staging-testnet wallet connect with Xverse or Leather, signed challenge proof, top-chrome identity hydration, Wallet pane provider/address/balance readout, Supabase auth/session projection, sign-out clearing, and SQL evidence in `auth.users`, `user_connections`, and `user_profiles`. Leather parity additionally requires documented `getAddresses`, `open`, `signMessage`, hex `signPsbt`, and `sendTransfer` request shapes to stay covered by tests before Terminal BTC fee usage depends on them. |
115115
| GitHub App and repository source scope | `uapi/app/api/vcs/_shared.ts`, `uapi/app/api/vcs/[provider]/callback/route.ts`, `uapi/app/api/vcs/[provider]/setup/route.ts`, `uapi/app/api/vcs/[provider]/repositories/route.ts`, `packages/api/src/vcs/github-service.ts`, Auxillaries Externals pane, `supabase/migrations/001_v26_production.sql` `user_connections` and `vcs_repositories` | partial | GitHub App install/callback returns to Externals, installation/user state is stored under provider-scoped connection data, repositories appear for Give/Need source scope, and SQL evidence confirms `user_connections.provider='github'` plus `vcs_repositories` rows. |
116-
| PSQL/Supabase data reality | `supabase/migrations/001_v26_production.sql`, `002_v27_btd_crypto_registry.sql`, `003_user_connections_provider_scope.sql`, `internal-docs/BITCODE_EXCHANGE_DATABASE.md` | partial | Staging database has all required auth/profile/connection/repository/BTD registry/journal/telemetry tables, RLS posture does not hide required user-scoped reads, and QA SQL queries match Terminal/Wallet/Externals UI state. |
116+
| PSQL/Supabase data reality | `supabase/migrations/001_v26_production.sql`, `002_v27_btd_crypto_registry.sql`, `003_user_connections_provider_scope.sql`, `packages/orm/src/data-health/*`, `packages/orm/scripts/run-data-health.ts`, `packages/orm/scripts/check-schema-types.ts`, `supabase/DATA_HEALTH.md`, `supabase/queries/*` | partial | Staging database has all required auth/profile/connection/repository/BTD registry/journal/telemetry tables, RLS posture does not hide required user-scoped reads, generated schema types cover canonical public tables, automated data-health checks pass, and QA SQL queries match Terminal/Wallet/Externals UI state. |
117117
| Fit-finding | `uapi/app/terminal/TerminalGiveNeedWorkbench.tsx`, `TerminalNeedScenarioPanel.tsx`, `terminal-give-need-workbench.ts`, `terminal-need-scenarios.ts`, `packages/protocol/src/canonical/*`, `protocol-demonstration/src/*` reference witness | partial | Terminal simplest Need produces or reads Fit candidates, review decision, qualities, rejection/block reasons, source roots, proof/dedupe roots, and visible readiness state without relying on demonstration runtime imports. |
118118
| AssetPack synthesis and deterministic pipeline configuration | `internal-docs/ASSETPACK_EXECUTION.md`, `packages/protocol/src/canonical/run-artifacts.js`, `packages/btd/src/semantic-volume.ts`, `measuremint.ts`, `range.ts`, `receipts.ts`, `replay.ts`, `uapi/app/api/btd/mint-draft/route.ts`, Terminal AssetPack components | partial with blocker | Staging Need/Fit path creates or reads AssetPack evidence, semantic volume, measuremint/range or zero-cell receipt, and access-policy hash using protocol-specified model/pipeline configuration. Any broad user model-selection UI that can affect ledgerized synthesis must be removed or scoped to non-ledgerized conversation UX. |
119119
| Ledgerized journal and ledger anchors | `packages/btd/src/terminal-journal.ts`, `ledger-anchor.ts`, `reconciliation.ts`, `uapi/app/api/btd/terminal-journal/route.ts`, `asset-pack-ledger-anchor/route.ts`, `ledger-database-reconciliation/route.ts`, `supabase/migrations/002_v27_btd_crypto_registry.sql` | partial | Terminal shows journal rows for Need/Fit/AssetPack/BTC fee/anchor/reconciliation events, ledger anchor or blocked-readiness state is explicit, and SQL evidence in `btd_terminal_journal_entries`, `btd_asset_pack_ledger_anchors`, and reconciliation repair tables matches the UI. |

BITCODE_V28_QA.md

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -810,7 +810,10 @@ Automated checks passed:
810810
- `pnpm -C uapi exec tsc --noEmit --pretty false`: pass.
811811
- `pnpm -C packages/protocol test`: 2 passed.
812812
- `pnpm -C packages/protocol run typecheck`: pass.
813-
- `pnpm -C packages/orm test`: 15 passed.
813+
- `pnpm -C packages/orm test`: 19 passed, with live data-health E2E tests skipped unless enabled by env flags.
814+
- `pnpm -C packages/orm run test:data-health`: data-health contract tests passed; live DB and schema-type E2E tests are gated by env flags.
815+
- `pnpm -C packages/orm run schema-types:check`: generated Supabase and ORM database types cover canonical public tables.
816+
- `pnpm -C packages/orm run build`: pass after generated type refresh.
814817
- `pnpm -C uapi run build` with the staging-testnet env block: Next.js production build passed.
815818
- `git diff --check`: pass.
816819
@@ -825,11 +828,21 @@ External Supabase readiness check:
825828
The staging Supabase migration history now contains the required V28 tables from local migrations `001`, `002`, and `003`, followed by dashboard-origin RLS migration `20260510223914`.
826829
`supabase db push --dry-run` reports the remote database is up to date.
827830
831+
Data-health tooling added for repeatable Supabase/PostgreSQL validation:
832+
833+
- `packages/orm/src/data-health/checks.ts` defines schema, identity, Terminal, ledger, and operational checks.
834+
- `packages/orm/scripts/run-data-health.ts` runs live checks with `SUPABASE_DB_URL`, `DATABASE_URL`, or equivalent Postgres connection env.
835+
- `packages/orm/scripts/check-schema-types.ts` checks generated Supabase/ORM type coverage against canonical public tables.
836+
- `supabase/DATA_HEALTH.md` documents daily, CI, and QA usage.
837+
- Saved Supabase SQL query files are available under `supabase/queries/` with reusable names for projection overview, wallet/GitHub readiness, and BTD ledger reconciliation.
838+
828839
Required next validation before live onboarding QA:
829840
830-
1. Re-run saved Supabase queries `v28_qa_01_*` through `v28_qa_05*`.
831-
2. Rebuild/redeploy if Vercel env changed after the build.
832-
3. Re-run Pass 1A from a fresh browser profile.
841+
1. Run `SUPABASE_DB_URL='<postgres-connection-string>' pnpm db:data-health:daily` against staging-testnet.
842+
2. Run saved Supabase queries from `supabase/queries/` and paste result excerpts into this QA log.
843+
3. Run `pnpm db:schema-types:check` after generated Supabase types are refreshed from the staging database.
844+
4. Rebuild/redeploy if Vercel env changed after the build.
845+
5. Re-run Pass 1A from a fresh browser profile.
833846
834847
Deployment judgment:
835848

package.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,12 @@
6464
"db:dump": "bash scripts/supabase.sh db:dump -f supabase/migrations/000_squashed.sql",
6565
"db:gen:types": "bash scripts/supabase.sh gen:types",
6666
"db:codegen": "bash scripts/supabase.sh codegen:db",
67+
"db:data-health": "pnpm -C packages/orm run data-health",
68+
"db:data-health:ci": "pnpm -C packages/orm run data-health:ci",
69+
"db:data-health:daily": "pnpm -C packages/orm run data-health:daily",
70+
"db:data-health:qa": "pnpm -C packages/orm run data-health:qa",
71+
"db:schema-types:check": "pnpm -C packages/orm run schema-types:check",
72+
"db:schema-types:refresh": "pnpm -C packages/orm run schema-types:refresh",
6773
"db:squash": "bash scripts/db-squash.sh",
6874
"render:prompts": "pnpm -C packages/pipelines/asset-pack exec ts-node --transpile-only scripts/render-prompts.ts",
6975
"render:prompts:all": "pnpm -C packages/pipelines/asset-pack exec ts-node --transpile-only scripts/render-prompts.ts --scope=all",

packages/orm/README.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ Type-safe database access layer for the Bitcode platform. Provides 1:1 mapping w
77
- **Client Management**: Separate admin and user clients
88
- **Model System**: Type-safe model classes for all entities
99
- **AssetPack Evidence Vectors**: pgvector-backed storage helpers for AssetPack evidence rows
10-
- **Profile Contracts**: V26 readiness and profile helpers for Bitcode account state
10+
- **Profile Contracts**: Readiness and profile helpers for Bitcode account state
11+
- **Data Health**: Supabase/PostgreSQL projection checks for schema, identity, Terminal, ledger, telemetry, and reconciliation parity
1112

1213
## Client Types
1314

@@ -53,7 +54,8 @@ const relatedVectors = await client.assetPackVectors.listByAssetPackEvidenceId(a
5354
- **pgvector Integration**: Native PostgreSQL vector storage for admitted AssetPack evidence
5455
- **Profile Readiness**: Canonical user/profile readiness checks used by app and API routes
5556
- **Schema Contracts**: Generated database types and typed model helpers kept in lockstep with Supabase migrations
57+
- **Data Health Checks**: `pnpm -C packages/orm run data-health -- --suite daily` validates live Supabase projection health when a Postgres connection string is supplied through `SUPABASE_DB_URL`, `DATABASE_URL`, or compatible env.
5658

5759
## Architecture
5860

59-
Provides database as a typed service philosophy. 1:1 mapping with Supabase schema ensures consistency. Vector search, profile contracts, and generated types support Bitcode application, Exchange, Terminal, and proof-facing storage without admitting experimental non-Bitcode query corridors.
61+
Provides database as a typed service philosophy. 1:1 mapping with Supabase schema ensures consistency. Vector search, profile contracts, generated types, and data-health checks support Bitcode Terminal, wallet, GitHub, AssetPack, and proof-facing storage without admitting experimental non-Bitcode query corridors.

packages/orm/jest.config.cjs

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,13 @@ module.exports = {
77
testMatch: ['**/__tests__/**/*.test.(ts|tsx)'],
88
roots: ['<rootDir>'],
99
moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx', 'json', 'node'],
10-
globals: {
11-
'ts-jest': {
12-
tsconfig: '<rootDir>/../../tsconfig.json',
13-
diagnostics: false,
14-
},
10+
transform: {
11+
'^.+\\.(ts|tsx|js|jsx)$': [
12+
'ts-jest',
13+
{
14+
tsconfig: '<rootDir>/../../tsconfig.json',
15+
diagnostics: false,
16+
},
17+
],
1518
},
1619
};
17-

packages/orm/package.json

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,17 +7,27 @@
77
"scripts": {
88
"build": "tsc",
99
"codegen:db": "ts-node --transpile-only scripts/generate-db-types.ts",
10+
"data-health": "ts-node --transpile-only scripts/run-data-health.ts",
11+
"data-health:ci": "ts-node --transpile-only scripts/run-data-health.ts --suite ci --fail-on critical",
12+
"data-health:daily": "ts-node --transpile-only scripts/run-data-health.ts --suite daily --fail-on critical",
13+
"data-health:qa": "ts-node --transpile-only scripts/run-data-health.ts --suite qa --fail-on warning",
14+
"schema-types:check": "ts-node --transpile-only scripts/check-schema-types.ts",
15+
"schema-types:refresh": "ts-node --transpile-only scripts/refresh-schema-types.ts",
1016
"clean": "rm -rf dist",
1117
"test": "jest --passWithNoTests",
12-
"generate-types": "supabase gen types typescript --local > src/types/database.ts"
18+
"test:data-health": "jest --runTestsByPath src/__tests__/data-health.contract.test.ts src/__tests__/data-health.e2e.test.ts src/__tests__/schema-types.e2e.test.ts --runInBand",
19+
"test:data-health:live": "BITCODE_RUN_DB_HEALTH_E2E=true BITCODE_RUN_SCHEMA_TYPES_E2E=true jest --runTestsByPath src/__tests__/data-health.contract.test.ts src/__tests__/data-health.e2e.test.ts src/__tests__/schema-types.e2e.test.ts --runInBand",
20+
"generate-types": "mkdir -p ../../.tmp/data-health && supabase gen types typescript --local > ../../.tmp/data-health/database.local.ts && pnpm run schema-types:refresh -- --input ../../.tmp/data-health/database.local.ts"
1321
},
1422
"dependencies": {
1523
"@supabase/supabase-js": "^2.0.0",
24+
"pg": "^8.16.3",
1625
"zod": "^3.0.0"
1726
},
1827
"devDependencies": {
1928
"@types/jest": "^29.5.12",
2029
"@types/node": "^20.0.0",
30+
"@types/pg": "^8.15.6",
2131
"jest": "^29.7.0",
2232
"ts-jest": "^29.1.1",
2333
"ts-node": "10.9.2",
Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
import { readdirSync, readFileSync } from 'fs';
2+
import path from 'path';
3+
4+
import {
5+
CANONICAL_PUBLIC_TABLES,
6+
getRequiredGeneratedTypeTableNames,
7+
} from '../src/data-health/checks';
8+
9+
interface SchemaTypeReport {
10+
ok: boolean;
11+
generatedTypesPath: string;
12+
ormTypesPath: string;
13+
migrationDirectory: string;
14+
missingFromGeneratedTypes: string[];
15+
missingFromOrmTypes: string[];
16+
missingFromMigrations: string[];
17+
}
18+
19+
function main() {
20+
const repoRoot = path.resolve(__dirname, '../../..');
21+
const generatedTypesPath = path.join(repoRoot, 'packages/orm/src/types/database.generated.ts');
22+
const ormTypesPath = path.join(repoRoot, 'packages/orm/src/types/database.ts');
23+
const migrationDirectory = path.join(repoRoot, 'supabase/migrations');
24+
const generatedTypes = readFileSync(generatedTypesPath, 'utf8');
25+
const ormTypes = readFileSync(ormTypesPath, 'utf8');
26+
const migrationSource = readMigrationSource(migrationDirectory);
27+
const requiredTables = getRequiredGeneratedTypeTableNames();
28+
29+
const report: SchemaTypeReport = {
30+
ok: true,
31+
generatedTypesPath,
32+
ormTypesPath,
33+
migrationDirectory,
34+
missingFromGeneratedTypes: missingTableNames(generatedTypes, requiredTables),
35+
missingFromOrmTypes: missingTableNames(ormTypes, requiredTables),
36+
missingFromMigrations: missingTablesFromMigrations(migrationSource, CANONICAL_PUBLIC_TABLES),
37+
};
38+
39+
report.ok =
40+
report.missingFromGeneratedTypes.length === 0 &&
41+
report.missingFromOrmTypes.length === 0 &&
42+
report.missingFromMigrations.length === 0;
43+
44+
if (process.argv.includes('--json')) {
45+
console.log(JSON.stringify(report, null, 2));
46+
} else {
47+
printReport(report);
48+
}
49+
50+
process.exitCode = report.ok ? 0 : 1;
51+
}
52+
53+
function readMigrationSource(migrationDirectory: string): string {
54+
return readdirSync(migrationDirectory)
55+
.filter((fileName) => fileName.endsWith('.sql'))
56+
.sort()
57+
.map((fileName) => readFileSync(path.join(migrationDirectory, fileName), 'utf8'))
58+
.join('\n');
59+
}
60+
61+
function missingTableNames(source: string, tableNames: readonly string[]): string[] {
62+
return tableNames.filter((tableName) => !new RegExp(`\\b${escapeRegExp(tableName)}\\s*:`).test(source));
63+
}
64+
65+
function missingTablesFromMigrations(source: string, tableNames: readonly string[]): string[] {
66+
return tableNames.filter((tableName) => {
67+
const quoted = new RegExp(
68+
`CREATE\\s+TABLE\\s+IF\\s+NOT\\s+EXISTS\\s+"?public"?\\."?${escapeRegExp(tableName)}"?\\s*\\(`,
69+
'i',
70+
);
71+
const unquoted = new RegExp(
72+
`CREATE\\s+TABLE\\s+IF\\s+NOT\\s+EXISTS\\s+public\\.${escapeRegExp(tableName)}\\s*\\(`,
73+
'i',
74+
);
75+
return !quoted.test(source) && !unquoted.test(source);
76+
});
77+
}
78+
79+
function escapeRegExp(value: string): string {
80+
return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
81+
}
82+
83+
function printReport(report: SchemaTypeReport) {
84+
console.log('[Bitcode schema types] generated types:', report.generatedTypesPath);
85+
console.log('[Bitcode schema types] ORM types:', report.ormTypesPath);
86+
console.log('[Bitcode schema types] migrations:', report.migrationDirectory);
87+
88+
printList('Missing from generated Supabase types', report.missingFromGeneratedTypes);
89+
printList('Missing from ORM exported database types', report.missingFromOrmTypes);
90+
printList('Missing from migrations', report.missingFromMigrations);
91+
92+
if (report.ok) {
93+
console.log('[Bitcode schema types] PASS');
94+
} else {
95+
console.log('[Bitcode schema types] FAIL');
96+
console.log('Refresh generated types after applying migrations, then rerun this check.');
97+
}
98+
}
99+
100+
function printList(label: string, values: readonly string[]) {
101+
if (values.length === 0) {
102+
console.log(`${label}: none`);
103+
return;
104+
}
105+
106+
console.log(`${label}:`);
107+
for (const value of values) {
108+
console.log(` - ${value}`);
109+
}
110+
}
111+
112+
main();

0 commit comments

Comments
 (0)