Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ updates:
# its blocker below clears, then let CI judge the upgrade on its own.
# eslint 10 — eslint-config-next 16.2.12 calls the pre-10 scope-manager
# API (scopeManager.addGlobals); `eslint .` crashes outright.
# jsdom 30 — requires Node >=22.22.2; CI runs Node 20.
# jest-dom 7 — requires Node >=22 and a new @testing-library/dom peer.
# (jsdom 30 and jest-dom 7 were ignored while copy-gate ran Node 20;
# every workflow now reads .nvmrc (Node 22), so CI judges those majors.)
# typescript 7 — typescript-eslint 8.x (via eslint-config-next) peers
# typescript <6.1.0; `npm run lint` crashes in
# typescript-estree ("reading 'Cjs'"). Backed out in
Expand All @@ -50,10 +50,6 @@ updates:
# package.json's range and `npm ci` fails (#3138, #3157).
- dependency-name: "eslint"
update-types: ["version-update:semver-major"]
- dependency-name: "jsdom"
update-types: ["version-update:semver-major"]
- dependency-name: "@testing-library/jest-dom"
update-types: ["version-update:semver-major"]
- dependency-name: "typescript"
update-types: ["version-update:semver-major"]
- dependency-name: "@vitest/coverage-v8"
Expand Down
22 changes: 22 additions & 0 deletions .github/workflows/backend-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ on:
- 'frontend/src/lib/public-openapi.ts'
- 'frontend/package-lock.json'
- 'alembic/**'
# The two non-Alembic schema channels, guarded by
# scripts/check_schema_channels.py in alembic-revision-check.
- 'supabase/migrations/**'
- 'sql/prod-apply/**'
- 'server/**'
- 'services/**'
- 'shared/**'
Expand All @@ -43,6 +47,10 @@ on:
- 'frontend/src/lib/public-openapi.ts'
- 'frontend/package-lock.json'
- 'alembic/**'
# The two non-Alembic schema channels, guarded by
# scripts/check_schema_channels.py in alembic-revision-check.
- 'supabase/migrations/**'
- 'sql/prod-apply/**'
- 'server/**'
- 'services/**'
- 'shared/**'
Expand Down Expand Up @@ -355,6 +363,10 @@ jobs:
--extend-exclude __pycache__,.venv,venv,build,dist,services \
--statistics

# Every top-level entry in scripts/ has a row in scripts/README.md (WI-08).
- name: Guard the scripts/ registry
run: python scripts/check_scripts_registered.py

# ── RLS fallback-UUID guard (hard gate) ──────────────────
# Scans migrations for the fail-open RLS pattern that #1091 fixed.
# Any new migration that reintroduces `COALESCE(..., <fallback-UUID>)`
Expand Down Expand Up @@ -425,6 +437,16 @@ jobs:
- name: Guard against new orphan migrations
run: python scripts/check_orphan_migrations.py

# ── Schema channel guard (hard gate) ──────────────────
# supabase/migrations/ and sql/prod-apply/ may hold rendered copies of
# Alembic revisions and nothing else. A file there must start with
# ``-- alembic-revision: <id>`` and ``-- rendered-from: ...`` headers
# naming an existing revision. Pre-decision files are grandfathered
# by name inside the script; the list only shrinks. See
# docs/decisions/2026-10-30-schema-change-channel.md.
- name: Guard the non-Alembic schema channels
run: python scripts/check_schema_channels.py

# ── Dep version drift guard (hard gate) ─────────────────
# Scans services/*/requirements*.txt to ensure no per-service file
# lowers the floor for shared packages below pyproject.toml. Prevents
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/copy-gate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ jobs:
persist-credentials: false
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v6
with:
node-version: '20'
# One Node version for every workflow: .nvmrc (WI-13, 2026-10-30).
node-version-file: .nvmrc
- run: node --test scripts/copy-lint/check-copy.test.mjs
- name: Read base copy baseline
env:
Expand Down
6 changes: 3 additions & 3 deletions ASYNC_PROCESSES.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ trigger was removed in migration v059.
|-----------|---------|--------------------|----------------------|
| `nlp_extraction` | `_handle_nlp_extraction` → `services.nlp.app.extractor.extract_entities` | 300s | topic `documents.ingested` |
| `review_item` | `_handle_review_item` → admin HallucinationTracker | 30s | topic `nlp.needs_review` |
| `graph_update` | **No handler registered** — a timeout (120s) is configured, but `TASK_HANDLERS` registers only `nlp_extraction` and `review_item`; an enqueued `graph_update` row sits unclaimed | 120s | topic `graph.update` |

(Extraction-path hardening for the pg pipeline, including the `review_item` handler, is
in flight on a sibling branch — treat the handler details above as the pre-fix baseline.)
`graph_update` was declared as a task type with a 120 s timeout but nothing enqueued it and
no handler existed; it was removed on 2026-10-30 (tech-debt WI-15). High-confidence
extractions are logged as `fsma_high_confidence_no_graph_sink` until a graph sink exists.

Mechanics:

Expand Down
94 changes: 8 additions & 86 deletions docs/MIGRATION_CONSOLIDATION.md
Original file line number Diff line number Diff line change
@@ -1,88 +1,10 @@
# Migration Consolidation Plan

## Current State (3 parallel systems)

| System | Location | Count | Format |
|--------|----------|-------|--------|
| Alembic | `alembic/versions/` | 10 | Python, revision chain |
| Raw SQL (main) | `migrations/` | 28 | Flyway V-numbered |
| Raw SQL (admin) | `services/admin/migrations/` | 46 | Flyway V-numbered |

**Total: 84 migration files**, many with overlapping operations.

## Why Alembic Wins

- Alembic's baseline (`20260324_baseline_consolidate_v002_through_v042.py`) already consolidates the first 41 Flyway migrations
- Alembic has revision-chain integrity (down_revision links)
- Alembic supports both online (live DB) and offline (SQL script) modes
- Python migrations allow conditional logic (IF NOT EXISTS, data backfill)
- All new migrations since March 2026 are already Alembic

## What's Already Covered by Alembic

V002-V042 (baseline), V043-V050 (control plane + task queue), V051-V055 (RLS + tenant tables + indexes).

## What Needs Porting (raw SQL not yet in Alembic)

### High Priority — unique schema changes not in any Alembic migration

| File | Operation | Priority |
|------|-----------|----------|
| `V049__transformation_links_adjacency.sql` | Lot-to-lot ingredient traceability | HIGH |
| `V053__add_growing_cte_type.sql` | Add 'growing' to event_type CHECK | HIGH |
| `V057__ftl_cottage_cheese_exemption.sql` | FTL update | MEDIUM |
| `SQL_V048__rls_sysadmin_defense_in_depth.sql` | RLS hardening | HIGH |
| `SQL_V050__rls_reference_tables.sql` | RLS on reference tables | HIGH |
| `V054__rls_variable_standardization.sql` | Fix RLS variable names | CRITICAL |
| `V056__rls_obligations_and_ftl.sql` | Close last RLS gaps | HIGH |

### Already duplicated in Alembic (can skip)

| File | Covered by |
|------|-----------|
| `V043__canonical_traceability_events.sql` | Alembic 20260325 |
| `V044__versioned_rules_engine.sql` | Alembic 20260325 |
| `V045__exception_remediation_queue.sql` | Alembic 20260325 |
| `V046__request_response_workflow.sql` | Alembic 20260325 |
| `V047__identity_resolution.sql` | Alembic 20260325 |
| `V052__ensure_tenant_tables.sql` | Alembic 20260411 V052 |
| `V052__cte_events_composite_idempotency_key.sql` | Alembic 20260411 V052 |

### Admin migrations — disposition

- **V1-V35**: Core admin schema. These run against `ADMIN_DATABASE_URL` (separate DB). Keep as-is until admin DB is merged into main DB during monolith consolidation.
- **V36-V40**: Funnel events, password reset, tool leads, supplier portal, PCOS removal. Same — admin-DB-scoped.

## Execution Plan

### Phase 1: Port remaining raw SQL to Alembic (1 day)
Create 3 new Alembic migrations:
1. `20260415_rls_hardening_v056.py` — combines SQL_V048, SQL_V050, V054, V056
2. `20260415_transformation_links_v057.py` — ports V049
3. `20260415_schema_updates_v058.py` — ports V053, V057

### Phase 2: Validate against real database (half day)
```bash
# Against a fresh database
alembic upgrade head

# Against an existing database (stamp first)
alembic stamp head # if migrations were already applied via Flyway
```

### Phase 3: Remove raw SQL directory (after validation)
- Delete `migrations/` directory
- Update `Dockerfile` (already done — only copies `alembic/`)
- Update `scripts/run-migrations.sh` to use `alembic upgrade head` only

### Phase 4: Admin DB merge (deferred to monolith consolidation)
- Port admin V1-V40 into main Alembic chain
- Requires merging `ADMIN_DATABASE_URL` into `DATABASE_URL`
- Depends on monolith consolidation completing first

## Status

- [x] Phase 1: Port remaining raw SQL to Alembic (3 migrations created, April 15 2026)
- [x] Phase 2: Validate against real database (Supabase, alembic upgrade head passes)
- [x] Phase 3: Remove raw SQL directory (`migrations/` deleted)
- [ ] Phase 4: Admin DB merge (deferred to monolith consolidation)
Superseded on 2026-10-30. This file described the March 2026 state (10
Alembic revisions, a Flyway-style `migrations/` directory that no longer
exists) and was not kept current.

The current decision and inventory of every schema channel is
`docs/decisions/2026-10-30-schema-change-channel.md`. The procedure for
reconciling the production version row is
`docs/runbooks/ALEMBIC_VERSION_RECONCILE.md`.
Loading
Loading