wip v28

Author: geraldarthurdavis

BITCODE_SPEC_V28_NOTES.md

@@ -66,6 +66,8 @@ This recalibration means the QA document, parity matrix, and implementation work
 
 Supabase/PostgreSQL data-health checks are part of the V28 acceptance apparatus, not a separate operator convenience. The ORM package now owns reusable schema, identity, Terminal, ledger, and operational check manifests plus live runners for daily staging/production validation, CI E2E validation, local QA reports, and generated schema-type coverage. These checks must preserve the core boundary: Supabase rows are ledger-derived or Bitcode-canonical projections, while ledger anchors, signed wallet proofs, Terminal journal receipts, and canonical receipts remain the source of cryptographic truth.
 
+May 14 staging-testnet verification added the first repository-owned RLS repairs after the dashboard-origin RLS bootstrap. `20260514173000_enable_pipeline_runs_rls.sql` enables RLS on `public.pipeline_runs`, and `20260514175000_enable_pipeline_runtime_rls.sql` enables RLS on server-managed pipeline runtime tables reported by Supabase Security Advisor. These preserve existing service-role pipeline posture and move runtime internals to deny-by-default for browser roles unless a narrower policy is later specified. The live daily data-health suite now passes 24/24 checks against the staging Supabase project, live ORM E2E tests can load `.env.local` directly for local QA parity with the CLI runner, and the advisory-style RLS probe returns no public tables with policies/browser grants while RLS is disabled.
+
 ### Deterministic Model Boundary
 
 V28 must remove the idea that a user can choose ledgerized Bitcode synthesis models.

BITCODE_SPEC_V28_PARITY_MATRIX.md

@@ -113,7 +113,7 @@ It links manual screenshots, console logs, server logs, Supabase SQL checks, and
 | --- | --- | --- | --- |
 | BTC/D identity and wallet-backed session | `uapi/app/api/wallet/_shared.ts`, `uapi/app/api/wallet/authenticate/route.ts`, `uapi/app/api/wallet/oauth/authorization-code/route.ts`, `uapi/app/api/wallet/oauth/token/route.ts`, `uapi/app/api/wallet/oauth/userinfo/route.ts`, `uapi/lib/bitcoin-wallet-client.ts`, `uapi/lib/bitcode-wallet-local.ts`, `packages/btd/src/wallet.ts`, `packages/btd/src/bitcoin-provider.ts`, `supabase/migrations/003_user_connections_provider_scope.sql`, `uapi/tests/bitcoinWalletClient.test.ts` | partial | Live staging-testnet wallet connect with Xverse or Leather, signed challenge proof, top-chrome identity hydration, Wallet pane provider/address/balance readout, Supabase auth/session projection, sign-out clearing, and SQL evidence in `auth.users`, `user_connections`, and `user_profiles`. Leather parity additionally requires documented `getAddresses`, `open`, `signMessage`, hex `signPsbt`, and `sendTransfer` request shapes to stay covered by tests before Terminal BTC fee usage depends on them. |
 | GitHub App and repository source scope | `uapi/app/api/vcs/_shared.ts`, `uapi/app/api/vcs/[provider]/callback/route.ts`, `uapi/app/api/vcs/[provider]/setup/route.ts`, `uapi/app/api/vcs/[provider]/repositories/route.ts`, `packages/api/src/vcs/github-service.ts`, Auxillaries Externals pane, `supabase/migrations/001_v26_production.sql` `user_connections` and `vcs_repositories` | partial | GitHub App install/callback returns to Externals, installation/user state is stored under provider-scoped connection data, repositories appear for Give/Need source scope, and SQL evidence confirms `user_connections.provider='github'` plus `vcs_repositories` rows. |
-| PSQL/Supabase data reality | `supabase/migrations/001_v26_production.sql`, `002_v27_btd_crypto_registry.sql`, `003_user_connections_provider_scope.sql`, `packages/orm/src/data-health/*`, `packages/orm/scripts/run-data-health.ts`, `packages/orm/scripts/check-schema-types.ts`, `supabase/DATA_HEALTH.md`, `supabase/queries/*` | partial | Staging database has all required auth/profile/connection/repository/BTD registry/journal/telemetry tables, RLS posture does not hide required user-scoped reads, generated schema types cover canonical public tables, automated data-health checks pass, and QA SQL queries match Terminal/Wallet/Externals UI state. |
+| PSQL/Supabase data reality | `supabase/migrations/001_v26_production.sql`, `002_v27_btd_crypto_registry.sql`, `003_user_connections_provider_scope.sql`, `20260514173000_enable_pipeline_runs_rls.sql`, `20260514175000_enable_pipeline_runtime_rls.sql`, `packages/orm/src/data-health/*`, `packages/orm/scripts/run-data-health.ts`, `packages/orm/scripts/check-schema-types.ts`, `supabase/DATA_HEALTH.md`, `supabase/queries/*` | baseline pass; first-run QA evidence still pending | Staging database has all required auth/profile/connection/repository/BTD registry/journal/telemetry tables, RLS posture does not hide required user-scoped reads, generated schema types cover canonical public tables, automated data-health checks pass, and QA SQL queries match Terminal/Wallet/Externals UI state. Current baseline: `pnpm db:data-health:daily` passes 24/24 against staging, live data-health/schema E2E passes with `.env.local` loading, and direct advisory-style SQL returns zero public tables with policies/browser grants while RLS is disabled. |
 | Fit-finding | `uapi/app/terminal/TerminalGiveNeedWorkbench.tsx`, `TerminalNeedScenarioPanel.tsx`, `terminal-give-need-workbench.ts`, `terminal-need-scenarios.ts`, `packages/protocol/src/canonical/*`, `protocol-demonstration/src/*` reference witness | partial | Terminal simplest Need produces or reads Fit candidates, review decision, qualities, rejection/block reasons, source roots, proof/dedupe roots, and visible readiness state without relying on demonstration runtime imports. |
 | AssetPack synthesis and deterministic pipeline configuration | `internal-docs/ASSETPACK_EXECUTION.md`, `packages/protocol/src/canonical/run-artifacts.js`, `packages/btd/src/semantic-volume.ts`, `measuremint.ts`, `range.ts`, `receipts.ts`, `replay.ts`, `uapi/app/api/btd/mint-draft/route.ts`, Terminal AssetPack components | partial with blocker | Staging Need/Fit path creates or reads AssetPack evidence, semantic volume, measuremint/range or zero-cell receipt, and access-policy hash using protocol-specified model/pipeline configuration. Any broad user model-selection UI that can affect ledgerized synthesis must be removed or scoped to non-ledgerized conversation UX. |
 | Ledgerized journal and ledger anchors | `packages/btd/src/terminal-journal.ts`, `ledger-anchor.ts`, `reconciliation.ts`, `uapi/app/api/btd/terminal-journal/route.ts`, `asset-pack-ledger-anchor/route.ts`, `ledger-database-reconciliation/route.ts`, `supabase/migrations/002_v27_btd_crypto_registry.sql` | partial | Terminal shows journal rows for Need/Fit/AssetPack/BTC fee/anchor/reconciliation events, ledger anchor or blocked-readiness state is explicit, and SQL evidence in `btd_terminal_journal_entries`, `btd_asset_pack_ledger_anchors`, and reconciliation repair tables matches the UI. |

BITCODE_V28_QA.md

@@ -812,10 +812,12 @@ Automated checks passed:
 - `pnpm -C packages/protocol run typecheck`: pass.
 - `pnpm -C packages/orm test`: 19 passed, with live data-health E2E tests skipped unless enabled by env flags.
 - `pnpm -C packages/orm run test:data-health`: data-health contract tests passed; live DB and schema-type E2E tests are gated by env flags.
+- `BITCODE_RUN_DB_HEALTH_E2E=true BITCODE_RUN_SCHEMA_TYPES_E2E=true pnpm -C packages/orm run test:data-health:live`: 6 passed against the staging-testnet Supabase project.
 - `pnpm -C packages/orm run schema-types:check`: generated Supabase and ORM database types cover canonical public tables.
 - `pnpm -C packages/orm run build`: pass after generated type refresh.
 - `pnpm -C uapi run build` with the staging-testnet env block: Next.js production build passed.
 - `git diff --check`: pass.
+- Supabase Security Advisor RLS class probe: direct SQL returned zero public tables with policies or browser-role grants while RLS is disabled.
 
 Build warnings and interpretation:
 
@@ -825,8 +827,9 @@ Build warnings and interpretation:
 
 External Supabase readiness check:
 
-The staging Supabase migration history now contains the required V28 tables from local migrations `001`, `002`, and `003`, followed by dashboard-origin RLS migration `20260510223914`.
-`supabase db push --dry-run` reports the remote database is up to date.
+The staging Supabase migration history now contains the required V28 tables from local migrations `001`, `002`, and `003`, followed by dashboard-origin RLS migration `20260510223914` and repository RLS migrations `20260514173000_enable_pipeline_runs_rls.sql` and `20260514175000_enable_pipeline_runtime_rls.sql`.
+`scripts/supabase.sh migration list` reports local and remote migration history aligned.
+The added repository migrations enable RLS on `public.pipeline_runs`, `public.run_jobs`, `public.stream_logs`, and server-managed `public.deliverable_pipeline_*` runtime internals, closing the live data-health finding and the Supabase Security Advisor criticals shown during QA.
 
 Data-health tooling added for repeatable Supabase/PostgreSQL validation:
 
@@ -838,19 +841,17 @@ Data-health tooling added for repeatable Supabase/PostgreSQL validation:
 
 Required next validation before live onboarding QA:
 
-1. Run `SUPABASE_DB_URL='<postgres-connection-string>' pnpm db:data-health:daily` against staging-testnet.
-2. Run saved Supabase queries from `supabase/queries/` and paste result excerpts into this QA log.
-3. Run `pnpm db:schema-types:check` after generated Supabase types are refreshed from the staging database.
-4. Rebuild/redeploy if Vercel env changed after the build.
-5. Re-run Pass 1A from a fresh browser profile.
+1. Run saved Supabase queries from `supabase/queries/` and paste result excerpts into this QA log during first-run onboarding.
+2. Rebuild/redeploy if Vercel env changed after the build.
+3. Re-run Pass 1A from a fresh browser profile on `bitcode.exchange`.
 
 Deployment judgment:
 
 - Code/build readiness: pass.
 - Wallet/OAuth/GitHub source readiness in code: pass.
 - Formal protocol and ORM migration source readiness: pass.
-- Live staging data-plane readiness: migrations applied; pending saved-query verification and first-run onboarding QA.
-- Manual first-run onboarding may proceed after deployment env is confirmed and saved-query verification is captured.
+- Live staging data-plane readiness: pass baseline. `pnpm db:data-health:daily` checked 24 and passed 24 against `tkpyosihuouusyaxtbau`; the advisor-style RLS probe returned zero rows; saved-query verification still belongs in the first-run onboarding QA evidence.
+- Manual first-run onboarding may proceed after deployment env is confirmed; capture saved-query evidence during the pass.
 
 ## Issue Template
 

packages/orm/scripts/run-data-health.ts

@@ -8,6 +8,7 @@ import {
 } from '../src/data-health/checks';
 import {
   createDataHealthClient,
+  loadDataHealthEnvFiles,
   readDataHealthConnectionInfo,
   redactConnectionString,
   resolveDataHealthConnectionString,
@@ -34,6 +35,7 @@ const DEFAULT_OPTIONS: CliOptions = {
 };
 
 async function main() {
+  loadDataHealthEnvFiles();
   const options = parseArgs(process.argv.slice(2));
 
   if (options.list) {

packages/orm/src/__tests__/data-health.e2e.test.ts

@@ -1,5 +1,6 @@
 import {
   createDataHealthClient,
+  loadDataHealthEnvFiles,
   resolveDataHealthConnectionString,
   runDataHealthChecks,
 } from '../data-health/runner';
@@ -9,6 +10,7 @@ const maybeDescribe =
 
 maybeDescribe('Supabase data-health live E2E', () => {
   it('runs the CI health suite against the configured database', async () => {
+    loadDataHealthEnvFiles();
     const connectionString = resolveDataHealthConnectionString();
     expect(connectionString).toBeTruthy();
 

packages/orm/src/data-health/checks.ts

@@ -34,8 +34,19 @@ export const CANONICAL_APPLICATION_TABLES = [
   'vcs_repositories',
   'notifications',
   'pipeline_runs',
+  'run_jobs',
+  'stream_logs',
   'deliverables',
   'deliverable_vectors',
+  'deliverable_pipeline_runs',
+  'deliverable_pipeline_events',
+  'deliverable_pipeline_generated_assets',
+  'deliverable_pipeline_phase_delegations',
+  'deliverable_pipeline_agent_steps',
+  'deliverable_pipeline_generations',
+  'deliverable_pipeline_otf_instructions',
+  'deliverable_pipeline_substeps',
+  'deliverable_pipeline_tool_executions',
   'error_logs',
   'events',
 ] as const;
@@ -607,7 +618,7 @@ export const DATA_HEALTH_CHECKS: DataHealthCheckDefinition[] = [
           count(*)::bigint AS range_count
         FROM public.btd_asset_pack_ranges
       ),
-      overlaps AS (
+      range_overlaps AS (
         SELECT a.asset_pack_id AS left_asset_pack_id, b.asset_pack_id AS right_asset_pack_id
         FROM public.btd_asset_pack_ranges a
         JOIN public.btd_asset_pack_ranges b ON a.id < b.id
@@ -626,10 +637,10 @@ export const DATA_HEALTH_CHECKS: DataHealthCheckDefinition[] = [
       SELECT
         coalesce((SELECT total_minted FROM state), -1) = coalesce((SELECT summed_token_count FROM range_stats), 0)
         AND coalesce((SELECT next_token_id FROM state), -1) = coalesce((SELECT max_range_end FROM range_stats), 0)
-        AND (SELECT count(*) FROM overlaps) = 0
+        AND (SELECT count(*) FROM range_overlaps) = 0
         AND (SELECT count(*) FROM invalid_ranges) = 0 AS ok,
         (
-          (SELECT count(*) FROM overlaps)
+          (SELECT count(*) FROM range_overlaps)
           + (SELECT count(*) FROM invalid_ranges)
           + CASE
               WHEN coalesce((SELECT total_minted FROM state), -1) <> coalesce((SELECT summed_token_count FROM range_stats), 0)
@@ -641,7 +652,7 @@ export const DATA_HEALTH_CHECKS: DataHealthCheckDefinition[] = [
           'state_total_minted', (SELECT total_minted FROM state),
           'state_next_token_id', (SELECT next_token_id FROM state),
           'range_stats', (SELECT to_jsonb(range_stats) FROM range_stats),
-          'overlaps', coalesce((SELECT jsonb_agg(to_jsonb(overlaps)) FROM overlaps), '[]'::jsonb),
+          'overlaps', coalesce((SELECT jsonb_agg(to_jsonb(range_overlaps)) FROM range_overlaps), '[]'::jsonb),
           'invalid_ranges', coalesce((SELECT jsonb_agg(asset_pack_id ORDER BY asset_pack_id) FROM invalid_ranges), '[]'::jsonb)
         ) AS details;
     `,

packages/orm/src/data-health/runner.ts

@@ -1,3 +1,5 @@
+import { existsSync, readFileSync } from 'fs';
+import path from 'path';
 import { Client, type QueryResultRow } from 'pg';
 
 import {
@@ -82,14 +84,27 @@ export function resolveDataHealthConnectionString(env: NodeJS.ProcessEnv = proce
   );
 }
 
+export function loadDataHealthEnvFiles(cwd: string = process.cwd()): void {
+  const repoRoot = path.resolve(__dirname, '../../../..');
+  const candidates = [
+    path.join(repoRoot, '.env.local'),
+    path.join(repoRoot, 'uapi/.env.local'),
+    path.join(cwd, '.env.local'),
+  ];
+
+  for (const filePath of candidates) {
+    loadEnvFile(filePath);
+  }
+}
+
 export function redactConnectionString(connectionString: string): string {
   try {
     const url = new URL(connectionString);
     if (url.password) {
-      url.password = '[redacted]';
+      url.password = 'redacted';
     }
     if (url.username) {
-      url.username = url.username ? '[redacted]' : '';
+      url.username = url.username ? 'redacted' : '';
     }
     return url.toString();
   } catch {
@@ -98,9 +113,10 @@ export function redactConnectionString(connectionString: string): string {
 }
 
 export async function createDataHealthClient(connectionString: string): Promise<Client> {
+  const normalizedConnectionString = normalizePgConnectionString(connectionString);
   const client = new Client({
-    connectionString,
-    ssl: shouldUseSsl(connectionString) ? { rejectUnauthorized: false } : undefined,
+    connectionString: normalizedConnectionString,
+    ssl: shouldUseSsl(normalizedConnectionString) ? { rejectUnauthorized: false } : undefined,
   });
   await client.connect();
   return client;
@@ -233,6 +249,44 @@ function shouldUseSsl(connectionString: string): boolean {
   }
 }
 
+function normalizePgConnectionString(connectionString: string): string {
+  try {
+    const url = new URL(connectionString);
+    url.searchParams.delete('sslmode');
+    return url.toString();
+  } catch {
+    return connectionString;
+  }
+}
+
+function loadEnvFile(filePath: string): void {
+  if (!existsSync(filePath)) return;
+
+  for (const line of readFileSync(filePath, 'utf8').split(/\r?\n/)) {
+    const trimmed = line.trim();
+    if (!trimmed || trimmed.startsWith('#')) continue;
+
+    const match = trimmed.match(/^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)=(.*)$/);
+    if (!match) continue;
+
+    const key = match[1];
+    if (process.env[key] !== undefined) continue;
+
+    process.env[key] = stripEnvQuotes(match[2]);
+  }
+}
+
+function stripEnvQuotes(value: string): string {
+  const trimmed = value.trim();
+  if (
+    (trimmed.startsWith('"') && trimmed.endsWith('"')) ||
+    (trimmed.startsWith("'") && trimmed.endsWith("'"))
+  ) {
+    return trimmed.slice(1, -1);
+  }
+  return trimmed;
+}
+
 async function inspectAvailableRelations(client: Queryable): Promise<Set<string>> {
   const result = await client.query<{
     relation_name: string;

supabase/migrations/20260514173000_enable_pipeline_runs_rls.sql

@@ -0,0 +1 @@
+ALTER TABLE "public"."pipeline_runs" ENABLE ROW LEVEL SECURITY;

supabase/migrations/20260514175000_enable_pipeline_runtime_rls.sql

@@ -0,0 +1,7 @@
+ALTER TABLE IF EXISTS "public"."deliverable_pipeline_agent_steps" ENABLE ROW LEVEL SECURITY;
+ALTER TABLE IF EXISTS "public"."deliverable_pipeline_generations" ENABLE ROW LEVEL SECURITY;
+ALTER TABLE IF EXISTS "public"."deliverable_pipeline_otf_instructions" ENABLE ROW LEVEL SECURITY;
+ALTER TABLE IF EXISTS "public"."deliverable_pipeline_substeps" ENABLE ROW LEVEL SECURITY;
+ALTER TABLE IF EXISTS "public"."deliverable_pipeline_tool_executions" ENABLE ROW LEVEL SECURITY;
+ALTER TABLE IF EXISTS "public"."run_jobs" ENABLE ROW LEVEL SECURITY;
+ALTER TABLE IF EXISTS "public"."stream_logs" ENABLE ROW LEVEL SECURITY;