From 89796a399b25a6967bc142eaefb0ba21d24781a6 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 5 Oct 2026 17:37:24 +0200 Subject: [PATCH 1/2] TML-3474: shape the migration-statements project spec and design notes Signed-off-by: willbot Signed-off-by: Will Madden Co-Authored-By: Claude Fable 5.1 --- projects/migration-statements/README.md | 12 ++ projects/migration-statements/design-notes.md | 148 ++++++++++++++++++ projects/migration-statements/plan.md | 16 ++ projects/migration-statements/spec.md | 139 ++++++++++++++++ 4 files changed, 315 insertions(+) create mode 100644 projects/migration-statements/README.md create mode 100644 projects/migration-statements/design-notes.md create mode 100644 projects/migration-statements/plan.md create mode 100644 projects/migration-statements/spec.md diff --git a/projects/migration-statements/README.md b/projects/migration-statements/README.md new file mode 100644 index 000000000000..2d3f274c31e3 --- /dev/null +++ b/projects/migration-statements/README.md @@ -0,0 +1,12 @@ +# Migration statements + +Project folder for [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), second attempt. The user states migration intent as statements on the `migration plan` and `db update` command line, in contract vocabulary, and the planner refuses data loss that no statement covers. + +- [`spec.md`](./spec.md) — project spec: purpose, the statement vocabulary, non-goals, cross-cutting requirements, project DoD. +- [`design-notes.md`](./design-notes.md) — the design discussion of 2026-10-05: the verb and noun matrix, the scenarios that stress it, prior art, alternatives rejected, and what changed from the shelved first attempt. +- [`plan.md`](./plan.md) — slice plan. +- `slices//` — per-slice spec, plan and reviews. + +The first attempt, which stated intent in the contract source with `@@hint(was:)`, is shelved in prisma/orm#30557 and prisma/orm#30570 under `projects/intent-hints/` on those branches. Its planner substrate is reused here. + +Everything in this folder is transient. At close-out the durable parts move to `docs/` and an ADR, and the folder is deleted. diff --git a/projects/migration-statements/design-notes.md b/projects/migration-statements/design-notes.md new file mode 100644 index 000000000000..1e2916d61daa --- /dev/null +++ b/projects/migration-statements/design-notes.md @@ -0,0 +1,148 @@ +# Design notes — Migration statements + +Record of the design discussion on 2026-10-05 between the operator and the orchestrator, after the contract-source attempt was shelved. Each section carries the reasoning and the alternatives rejected. + +## Principles + +- The planner never guesses. It acts on a destructive-looking change only when the user has stated what it is. +- The user states intent in the vocabulary of their contract. Storage names, SQL and literal values never appear on the command line. +- A statement is sugar over the hand-written route. Whatever it produces, a user could have written in `migration.ts`, and that route stays. +- `migration plan` and `db update` share one planner. A statement means the same thing to both, and `db update` is a migration planned in memory and applied at once. + +## Why not the contract source + +The first attempt put `@@hint(was: "Profile")` on the model. It was shelved because stating intent in the source felt unintuitive, and because intent in the source has to survive until every environment has consumed it, which produced the rules about spent and ignored hints and the unhashed contract section that snapshots had to strip. A statement on the command line applies to one planning run. For `migration plan` the migration file is the record; for `db update` the database is. Nothing has to be left behind or cleaned up. + +The prior art is honest about the trade-off. Atlas and Terraform both started with the command line, Atlas with an interactive prompt and Terraform with `state mv`, and both moved to a declaration in the source, because a command-line statement is not in version control and a prompt cannot be answered by a script or an agent. We accept the first objection knowingly: the migration file is in version control, and `db update` is dev-only. We avoid the second by having no prompt. + +## Prior art + +| Tool | How a rename is stated | Non-interactive answer | +| --- | --- | --- | +| Prisma Migrate 2 to 7 | Not detected; edit the generated SQL with `--create-only` | the edited migration file | +| ActiveRecord | No diffing; `rename_column` in a hand-written migration; generator shorthand writes `add_column` calls from command arguments | the migration file | +| Alembic, EF Core, pgroll | hand-written or hand-edited migration | the migration file | +| Django `makemigrations` | detects candidate pairs and asks "Did you rename X to Y?" | `--noinput` treats it as drop and add; no way to pre-answer | +| Drizzle Kit | detects candidate pairs and asks | none shipped; an open request proposes `--preflight` to export questions and `--answers` to answer them; internally the planner takes a `HintsHandler` whose only source is the prompt | +| Atlas | asked at plan time until 2024; now `renamed_from` on the object in the HCL or SQL source | the source declaration; the prompt was abandoned because an agent cannot answer it | +| Terraform | `terraform state mv`, then the `moved` block in the source from 1.1 | the source declaration | +| PlanetScale, sqldef, Skeema | no rename; expand, dual-write, backfill, contract | nothing to answer | + +The ActiveRecord generator is the closest precedent to what we build: command arguments that produce explicit operations in a migration file, which is then the record. Nobody offers a rename argument on a plan command. + +## The vocabulary + +Four verbs and four nouns, all in the application domain. + +| | namespace | model | field | value | +| --- | --- | --- | --- | --- | +| rename | `--rename billing:finance` | `--rename Profile:User`, `--rename auth.User:public.User` | `--rename User.name:User.fullName`, `--rename Address.street:Address.streetName` | `--rename Status.ARCHIVED:Status.RETIRED` | +| delete | `--delete billing` | `--delete Legacy` | `--delete User.name` | `--delete Status.ARCHIVED` | +| convert | | `--convert Bug` (discriminator value changed) | `--convert User.age` | | +| backfill | | | `--backfill User.phone` | | + +Rename is an identity change, delete is a removal, convert is an in-place change to a definition that existing data must follow, backfill fills existing rows for a field that has become required. A model rename whose namespace part changes is a move; there is no separate move verb because the coordinate already says it. `--delete` is consent, not a hint: the contract already lacks the thing, the planner already plans the drop, and the statement allows it. + +Coordinates are the names the user wrote in the contract source. The namespace is omitted when the contract has one. Model and value object names share one namespace for resolution, and an ambiguous reading is an error that says which readings were tried. The separator is a colon because `->` redirects in a shell and names cannot contain a colon. + +### What each statement replaces + +| Statement | Diff shows | Postgres and SQLite | Mongo | +| --- | --- | --- | --- | +| rename namespace | every model removed under one, added under the other | alter schema rename | not applicable, one namespace | +| rename model | model removed and added | rename table, retarget foreign keys, rename constraints and indexes to destination names | rename collection | +| rename model across namespaces | removed in one, added in another | alter table set schema | not applicable | +| rename variant, single-table | model removed and added, storage unchanged | nothing unless the discriminator value changed | nothing | +| rename variant, multi-table | same | rename the variant's table | not applicable | +| rename value object | removed and added under value objects | nothing, reported as applied | nothing | +| rename field | field removed and added | rename column, rename constraints and indexes on it | update every document | +| rename value object field | removed and added under the value object | rewrite the JSON in every column of that type, arrays included | rewrite subdocuments | +| rename relation field | relation removed and added | nothing, reported as applied | nothing | +| rename enum value | value removed and added | update rows holding the old value; native Postgres enums rename the value | update documents | +| delete model | model removed | drop table | drop collection | +| delete variant, single-table | model removed | delete rows with that discriminator value, drop its columns; the refusal says it deletes rows | delete documents | +| delete field | field removed | drop column | unset in every document | +| delete value object field | removed | nothing or a cleanup rewrite; consent still required, the application loses the data | same | +| delete enum value | value removed | set the field to null where nullable, otherwise refuse and point at rename or convert; never deletes rows | same | +| convert field | type changed | `alterColumnType` with `using: placeholder(...)`; a data transform scaffold only where a separate pass is needed, such as nulls before not null | data transform scaffold, then the validator change | +| convert variant | discriminator value changed | update rows holding the old value, no placeholder, both values are known | update documents | +| backfill field | field added non-nullable, or nullable to required | the scaffolded backfill transform; without the statement the temporary-default recipe | data transform scaffold | + +Not offered, deliberately: split or merge tables, move a field between models, change a primary key, move a variant between single-table and multi-table storage, make an existing model a variant. These are data migrations and belong in a hand-written file. + +### Why no value or expression on the command line + +Two proposals were rejected in turn. A conversion expression in the family's language with a token for the current value, `--convert User.age:"nullif(trim($value),'')::integer"`: users cannot be expected to type SQL on a command line. A fill value as a JSON literal, `--fill 0`: JSON has no types, which is the gap the data-type and cast design closes for `@default` and would reopen here. A fill value in the `@default` grammar was considered and dropped for the same reason the expression was: the command line carries intent only. The conversion and the value are written into the migration file where the placeholder is, which is the existing editing flow. + +### Why no prompt and no detection + +A prompt only helps in a terminal and only when a candidate pair exists; the operator distrusts automatic detection and called the prompt an ugly guardrail. Django, Drizzle and the old Atlas all carry open complaints about it. The refusal message does the prompt's job without guessing: it names the operation and prints the statements. + +## Resolution and the origin contract + +Both sides of a rename resolve against a contract, so the user writes model and field names and never storage names. Resolution happens in the framework, against the origin contract and the destination contract; the family receives resolved entities. The new side is always available, the destination is the emitted contract. The old side needs the origin contract. `migration plan` has it from its origin ref. `db update` today passes the planner `fromContract: null` and must instead resolve the marker's hash in the local snapshot store. + +When the origin contract cannot be found, every rename and convert statement fails with one error. `--delete` still works because consent resolves against the diff's own operations. The user then consents and, in a migration project, edits the file. The marker's `contract_json` column is not used: the docs call it optional and diagnostic, and the planner is offline. + +The contract-level comparison also covers changes the schema differ cannot see: value object fields, discriminator values, and on Mongo any field the validator does not carry. Those changes are visible in a diff of the two contracts and get the same refusal and statements as a dropped column. Without the origin contract they are invisible, which is one more reason the error is loud. + +## Scenarios that stress the model + +1. **Composition in order.** `--rename Profile:User --rename User.name:User.fullName`: the second uses the new name. Statements apply to the working copy in the order given. +2. **Swap.** `--rename A:B --rename B:A` needs a temporary name. Refused in this project with a clear message. +3. **Rename plus convert on one field.** `--rename User.age:User.years --convert User.years` works because convert resolves after the rename. +4. **Old name missing.** Error before planning, naming the contract searched. +5. **New name missing.** Error, naming the destination contract. +6. **Both names exist in the origin.** Error: cannot rename onto an existing entity. +7. **The rename changed nothing in storage.** The user renamed the model and kept `@@map("profiles")`. The statement resolves, no operation is emitted, and the plan reports it as applied with nothing to do. +8. **Repeated run.** `db update --rename Profile:User` twice. The second run fails because `Profile` is not in the origin. Chosen deliberately: a statement is intent for this run, and a script that always passes it is wrong. +9. **No origin contract.** Renames and converts fail, deletes work. Baselines and `--from @empty` have no origin models and nothing to rename. +10. **Rollback.** `migration plan --to ^` reverses the diff; statements are written in the direction of this plan. +11. **Re-running the migration file** reproduces the operations from the file. Mongo and value object rewrites are data transform operations the facade offers. +12. **Control policy forbids altering the model.** The statement is an error, not ignored. Statements express intent; intent that makes no sense is rejected. +13. **Extension contract spaces.** A name that resolves only in an extension's space is an error. +14. **Ambiguous coordinates.** `A.B.C` could be namespace, model, field or model, field, value. Resolution usually settles it; when two readings resolve, error. The first slice's spec fixes the grammar. +15. **Renaming the default namespace.** `public` on Postgres is target-owned. Needs a check in the slice that adds namespace renames. +16. **Interactive consent.** In a terminal the refusal becomes the per-operation consent question, and an answer equals the statement. No prompt for renames. + +## The type-change walkthrough and its evaluation + +Postgres, a migration project, `age String` to `age Int`. + +1. Edit the contract source, `contract emit`. +2. `migration plan --name age-to-int` refuses; nothing is written; the message prints `--convert User.age` and `--delete User.age`. +3. `migration plan --name age-to-int --convert User.age` writes the migration with `alterColumnType({ ..., using: placeholder('typechange-users-age:using') })`, `ops.json` as `[]`, and the existing pending-placeholders warning. +4. The user replaces the placeholder with the expression their data needs. +5. `node migrations//migration.ts` re-emits. +6. `db migrate`. + +The round trip in step 2 is the expected outcome: the user says which of the two they meant, and `db update` behaves the same way. The cost is paid by making the refusal copy-and-paste. Two substrate defects found on the way: the existing scaffold puts placeholders in a separate data transform and leaves the alter bare, so a user who fills the two slots gets a migration that fails at apply, because Postgres will not cast text to integer without `using`; and the alter's rendered options expose internal field names. Both are fixed in the convert slice. `placeholder()` is typed `never` and the CLI catches its error by code, so it can sit in any slot of any operation; ADR 200's statement that only data transforms have holes described the scaffolder at the time, not a constraint. + +`--convert` promises more than it delivers today, since it scaffolds a slot rather than converting. It keeps the name because it names the intent, and the planner can do more of the work later without the statement changing. `--placeholder` was considered and rejected: it names the mechanism and merges two intents. + +## What changed from the shelved design + +| Shelved | Now | +| --- | --- | +| `@@hint(was:)` in the contract source, lowered to an unhashed contract section stripped from snapshots | statements on the command line; no contract change | +| a spent hint is silently ignored; a non-applicable hint is ignored | an unusable statement is an error | +| the old side is a storage name resolved against the origin schema | both sides are contract names resolved against the two contracts | +| `deleted: true` tombstone blocks and a reserved `deprecated` | `--delete` per operation; no lifecycle vocabulary | +| blanket `--confirm` kept as the fallback | `--confirm` replaced by `--delete` | +| field hints, refusal, tombstones as later slices | rename, delete, convert, backfill as verbs over four nouns | + +Kept unchanged: destructive means data loss; the working schema shared by the planner and the facade; companion constraint names from the destination contract; foreign-key pairing by own and referenced columns; operations should carry dependency information and `migration.ts` should become an ordered graph walk, with no composite operations. + +## Open questions + +- Grammar details for coordinates and the separator, settled in the first slice's spec. +- Whether `--delete` on a type-changed field keeps the word or gets its own. +- Whether the Mongo slice needs a collection rename operation added to the Mongo migration facade first, or ships it inside the slice. + +## References + +- [`spec.md`](./spec.md) +- [Atlas declarative renames](https://atlasgo.io/changelog/declarative-schema-renames), [Atlas v0.22 rename detector](https://atlasgo.io/blog/2024/05/01/atlas-v-0-22) +- [Django ticket 24735](https://code.djangoproject.com/ticket/24735), [Drizzle issue 5307](https://github.com/drizzle-team/drizzle-orm/issues/5307), [Drizzle issue 6053](https://github.com/drizzle-team/drizzle-orm/issues/6053) +- [Prisma customizing migrations](https://prisma.io/docs/orm/prisma-migrate/workflows/customizing-migrations), [PlanetScale renames](https://planetscale.com/docs/learn/handling-table-and-column-renames) +- prisma/orm#30557 and prisma/orm#30570, the shelved attempt diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md new file mode 100644 index 000000000000..840f42494d0f --- /dev/null +++ b/projects/migration-statements/plan.md @@ -0,0 +1,16 @@ +# Project plan — Migration statements + +_Drafted after the spec is accepted. Replace this placeholder._ + +**Spec:** [`spec.md`](./spec.md) + +## Slices + +_To be decomposed._ + +## Close-out (required) + +- [ ] Verify every project DoD item in [`spec.md`](./spec.md) +- [ ] Migrate long-lived docs into `docs/` and write the ADR +- [ ] Strip repo-wide references to `projects/migration-statements/**` +- [ ] Delete `projects/migration-statements/` diff --git a/projects/migration-statements/spec.md b/projects/migration-statements/spec.md new file mode 100644 index 000000000000..c626579152b5 --- /dev/null +++ b/projects/migration-statements/spec.md @@ -0,0 +1,139 @@ +# Project spec — Destructive changes need stated intent + +**Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9) ([TML-3474](https://linear.app/prisma-company/issue/TML-3474)) · **Branch:** `tml-3474-migration-statements` + +## Purpose + +The migration planner diffs two states and cannot tell intent from the diff. A renamed model looks exactly like a dropped table plus a created one, and a removed model looks the same as one the user forgot. Today the planner plans the destruction and relies on a blanket consent prompt, or on a hand-written migration, to keep rows safe. Projects that use `db update` and keep no migration history cannot hand-write anything. + +This project makes two things true. The planner refuses any operation that loses data unless the user has said what they mean about that operation. And the user says it on the command line, in the vocabulary of their contract, as a statement the planner acts on: a rename, a deletion, a conversion or a backfill. Nothing is inferred, nothing is asked, and nothing is written into the contract source. + +This is the second attempt. The first stated intent in the contract source with `@@hint(was:)` and was shelved on 2026-10-05 as unintuitive. Its planner substrate is reused; its contract section is not. + +## At a glance + +A user renames the `Profile` model to `User`, renames its `name` field, and removes the `Legacy` model for good. The contract source already says all of that. The command says what it means: + +```text +$ prisma migration plan --name tidy-users --rename Profile:User --rename User.name:User.fullName --delete Legacy +✔ Planned 5 operation(s) + Rename table "Profile" to "User" widening + Rename constraint "Profile_pkey" to "User_pkey" widening + Rename column "User"."name" to "fullName" widening + Rename constraint "User_name_key" to "User_fullName_key" widening + Drop table "Legacy" destructive +``` + +`migration.ts` carries `renameTable`, `renameColumn` and `dropTable` calls, the same file a user could write by hand. `prisma db update` with the same statements does the same against a live database. Without `--delete Legacy`, both commands refuse: + +```text +✖ The plan would lose data. State what you mean about each operation: + Table "Legacy" would be dropped and its rows lost. + To rename it: --rename Legacy: + To delete it: --delete Legacy +``` + +A type change gets the same treatment with its own statements: + +```text +$ prisma migration plan --name age-to-int --convert User.age +⚠ migration.ts has a placeholder: replace it with the conversion, then run the file to re-emit. +``` + +## Decisions settled in discussion (2026-10-05) + +The reasoning and the alternatives rejected are in [`design-notes.md`](./design-notes.md). + +1. **Data loss is refused by default, in both commands.** An operation is destructive only when it loses data: dropping a model or field, deleting a value rows still carry, or changing a field's type in a way the database cannot keep. Drops of indexes, constraints, checks, policies and defaults lose nothing and are widening. `db update` already refuses through its consent prompt; `migration plan` gains the same refusal for every plan, not only auto-baselines. +2. **Intent is stated on the command line, up front.** There is no interactive prompt that asks whether a drop is a rename, and no detection of candidate pairs. The refusal is where the user learns what to type, and it prints each statement ready to paste. +3. **Four verbs, four nouns.** The verbs are `rename`, `delete`, `convert` and `backfill`. The nouns are namespace, model, field and value, addressed by contract coordinates such as `User`, `User.name`, `Status.ARCHIVED`, `Bug` and `Address.street`, with the namespace prefixed when the contract has more than one. Rename and delete apply to all four nouns; a model rename across namespaces is a move. Convert applies to fields and to variants whose discriminator value changed. Backfill applies to fields. +4. **Statements are intent, not hints.** A statement that cannot be resolved against the origin and destination contracts, or cannot be applied, is an error and nothing is planned. There is no "spent" or "ignored" statement. A statement that resolves and applies but changes nothing in storage, such as renaming a value object or a relation field, is reported as applied with no operations. +5. **Everything is in the application domain.** The command line never carries a table, column, collection, SQL expression or literal value. The framework resolves coordinates against the two contracts; the family turns the resolved statement into storage operations. +6. **`--delete` is per-operation consent and replaces `--confirm`.** Each destructive operation needs its own statement. A namespace delete covers everything in the namespace and the refusal still lists each model. The interactive consent question stays and asks per operation; answering it is the same as typing the statement. +7. **Convert and backfill scaffold the existing placeholder migration.** `--convert` writes the type change with `placeholder()` in the slot that carries the conversion: the `using` expression on Postgres, a data transform on Mongo. `--backfill` writes the scaffolded backfill transform. The user fills the slot in `migration.ts` and re-emits, exactly as the placeholder flow works today. Neither is available on `db update`, which has no file to fill; without `--backfill` the planner's existing temporary-default recipe applies, which loses nothing. +8. **The origin contract is required for renames and conversions.** `migration plan` has it from its origin ref. `db update` resolves the marker's contract hash in the local snapshot store. When no origin contract can be found, every rename and convert statement fails with one error, `--delete` still works, and the user hand-edits the migration if they have one. +9. **Statements compose in order** over a working copy of the origin, so a later statement names things by their new names. A swap of two names is refused in this project. +10. **One statement surface for every family.** Postgres and SQLite rename tables and columns; Mongo renames collections and rewrites documents for field renames; a value object field rename rewrites the stored JSON on SQL and the subdocuments on Mongo. Same statement, each family's own operations. + +## Non-goals + +- No interactive prompt that asks whether a drop is a rename, and no rename detection from matching columns or any other heuristic. +- No intent in the contract source. `@@hint` and any attribute like it are not part of this project; the shelved attribute is not revived. +- No expressions, literals or values on the command line. A conversion or a backfill value is written into the migration file. +- No scaffolded migrations from `db update`. +- No split or merge of tables, no moving a field between models, no primary key changes, no moving a variant between single-table and multi-table storage, no making an existing model a variant. These stay hand-written data migrations. +- No `deprecated` lifecycle and no change to `db verify`. +- No reliance on the marker's `contract_json` column, which the docs call optional and diagnostic. The origin contract comes from the snapshot store or not at all. +- No reversal of the documented stance that `db update` is dev-only. + +## Place in the larger world + +- **Planner inputs.** Every planner takes a destination contract and an origin schema: derived from a contract snapshot for `migration plan`, introspected for `db update` and `db init`, and one differ runs over two schema IRs ([ADR 235](../../docs/architecture%20docs/adrs/ADR%20235%20-%20The%20schema%20differ%20walks%20two%20derived%20schema%20IRs.md)). `migration plan` also hands the SQL planner the origin contract as `fromContract` (`packages/1-framework/3-tooling/cli/src/control-api/operations/plan-resolution.ts`); the `db update` path passes `fromContract: null` (`packages/1-framework/3-tooling/migration/src/aggregate/strategies/plan-from-diff.ts`). Statements become the third input, and `db update` learns to supply the origin contract. +- **Where the origin contract comes from for `db update`.** The database marker stores the contract hash the database was last updated to ([ADR 021](../../docs/architecture%20docs/adrs/ADR%20021%20-%20Contract%20Marker%20Storage.md)), and `db update` advances the `db` ref to the same hash ([ADR 218](../../docs/architecture%20docs/adrs/ADR%20218%20-%20Refs%20with%20paired%20contract%20snapshots%20and%20universal%20graph-node%20invariant.md)). The content-addressed snapshot store ([ADR 240](../../docs/architecture%20docs/adrs/ADR%20240%20-%20Contract%20snapshots%20live%20in%20a%20content-addressed%20store.md)) holds the contract for that hash when the project has it. +- **Consent today.** `db update` pre-plans and refuses with `MIGRATION.DESTRUCTIVE_CHANGES` until the user types the database name or passes the engine's `--confirm ` token (`packages/1-framework/3-tooling/cli/src/orm/db/consent.ts`, `control-api/operations/db-update.ts`). `migration plan` applies the same refusal to auto-baselines only, with the project directory as the token (`control-api/operations/migration-plan.ts`). The CLI README's `db update` section still describes a `--yes` flag that no longer exists. +- **The placeholder flow.** `migration plan` already scaffolds data transforms with `placeholder()` slots for a required column on a non-empty table, a nullable-to-required change and an unsafe type change, writes `ops.json` as `[]` with `pendingPlaceholders`, and the user fills the slots and re-runs `migration.ts` to re-emit ([ADR 200](../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md), [ADR 196](../../docs/architecture%20docs/adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md), `packages/3-targets/3-targets/postgres/src/core/migrations/planner-strategies.ts`). Under `db update`, whose policy excludes the `data` class, those strategies stand down and direct DDL is emitted. The Postgres `alterColumnType` operation already has a `using` option the planner never fills. +- **Rename substrate.** `renameTable` exists on Postgres and SQLite from prisma/orm#30331. The shelved prisma/orm#30570 refactored it onto a working schema shared by the planner and the hand-written facade, with companion constraint names taken from the destination contract and foreign keys paired by their own and referenced columns, and reclassified non-data drops as widening. That code is the starting point; the contract `hints` section it also added is not. +- **Operation classes.** Additive, widening, destructive and data (`packages/2-sql/9-family/src/core/migrations/types.ts`). Every Postgres type change is destructive today except a short list of safe widenings; Mongo classifies by validator comparison (`packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts`). +- **Coordinates.** A contract entity is addressed by plane, namespace, kind and name ([ADR 221](../../docs/architecture%20docs/adrs/ADR%20221%20-%20Contract%20IR%20two%20planes%20with%20uniform%20entity%20coordinate%20and%20pack-contributed%20entity%20kinds.md), [ADR 224](../../docs/architecture%20docs/adrs/ADR%20224%20-%20Namespace%20concretions%20address%20entities%20by%20coordinate.md)); the contract source declares namespaces as `namespace { ... }` blocks; the query surface resolves a bare name through the sole namespace ([ADR 223](../../docs/architecture%20docs/adrs/ADR%20223%20-%20Target-owned%20default%20namespace.md)). Variants and discriminators are [ADR 173](../../docs/architecture%20docs/adrs/ADR%20173%20-%20Polymorphism%20via%20discriminator%20and%20variants.md); value objects are [ADR 178](../../docs/architecture%20docs/adrs/ADR%20178%20-%20Value%20objects%20in%20the%20contract.md), stored as one JSONB column on SQL and a subdocument on Mongo. +- **Family vocabulary.** `packages/1-framework` may not name a family or target (`no-family-vocabulary-in-framework`, enforced by the `lint:framework-vocabulary` ratchet). The statement parser and resolver live there and speak only contract vocabulary. +- **Documentation already promising something like this.** The Data Contract and Migration System subsystem docs and ADR 001 describe `@hint(was: "old_name")` in the contract source as the planner's source of intent. This project replaces that promise. + +## Cross-cutting requirements + +1. **Statements are the only source of intent.** The planner never infers a rename, never asks whether a drop is one, and never reads intent from the contract. A change with no statement is planned as the diff shows it, and refused if that loses data. +2. **Contract vocabulary only.** A statement names namespaces, models, fields and values as the contract source names them. The framework resolves every coordinate against the origin and destination contracts before any family code runs; the family receives resolved entities, never names to look up. +3. **One statement surface, both commands.** `migration plan` and `db update` accept the same statements with the same meaning, and the plan a statement produces is the same whichever command ran it. `db update` obtains its origin contract from the marker hash through the snapshot store. +4. **Refusal names the way out.** When either command refuses, the error lists every destructive operation with the rename and delete statements that would resolve it, spelled exactly as the user would type them. In a terminal the same list is the consent question, asked per operation. +5. **An unusable statement is an error.** A coordinate that does not resolve in the contract it must resolve in, a rename whose new name already exists in the origin, a statement the control policy forbids, or a rename and convert statement with no origin contract: each fails before anything is planned, naming what was searched and what was found. Running the same statements twice fails the second time. +6. **Statements compose in order.** Each statement is applied to the working copy of the origin before the next is resolved, so `--rename Profile:User --rename User.name:User.fullName` is valid and the second statement uses the new table name. A swap is refused with a message. +7. **Same output as the hand-written route.** A plan produced from statements writes a `migration.ts` whose calls a user could have written by hand, and re-running that file reproduces `ops.json`. Statements are never recorded anywhere; the migration file is the record. +8. **Companion objects follow a rename, never a rebuild.** Every constraint and index whose name derives from a renamed table or column is renamed on both SQL targets to the name the destination contract gives it, explicit or derived. +9. **Scaffolds reuse the placeholder flow.** `--convert` and `--backfill` produce the existing placeholder-bearing migration, with the placeholder in the operation that consumes the value. The pending state, the empty `ops.json` and the re-emit path are unchanged. +10. **Destructive means data loss.** Drops of indexes, unique and foreign-key constraints, checks, row-level-security policies, defaults and native enum types, and disabling row-level security, are widening on every target. A type change the family knows to be widening needs no statement. +11. **Framework stays family-agnostic.** The statement grammar, parser and resolver carry no family or target vocabulary, and each family documents what every verb means for its storage. + +## Transitional-shape constraints + +- The refusal in `migration plan` lands only after `--rename` exists for models and fields, so the first refusal a user meets has a rename-shaped way out for the common case. It ships with an upgrade fragment, since plans with drops change behaviour. +- `--delete` and the removal of `--confirm` land in the same slice, on both commands, with their upgrade fragment. No release has both. +- Each slice that adds a verb or a noun adds it for Postgres and SQLite in the same PR. Mongo follows in its own slice using the same framework surface, and the framework surface is designed for it from the first slice. +- The reclassification of non-data drops as widening lands with the first slice, salvaged from prisma/orm#30570, since a SQLite rename rebuilds indexes and would otherwise require consent. + +## Contract-impact + +None. `contract.json`, `contract.d.ts`, snapshots and migration manifests are unchanged. The `hints` section the shelved attempt added is not carried over. + +## Adapter-impact + +- **Postgres and SQLite:** planners take statements as an input; a column rename operation with prechecks and postchecks on each target; companion renames for column-derived names; the `using` placeholder on the Postgres type change; the SQLite type change scaffold; deletes under the existing drop operations; widening reclassification. +- **MongoDB:** a collection rename operation; field renames and value renames as data transforms; convert as a data transform scaffold; deletes under the existing drop and unset operations. + +## ADR pointer + +One ADR at close-out: destructive changes need stated intent, covering the refusal default and what counts as destructive, the statement vocabulary and its grammar, resolution against the two contracts, the per-operation consent model, the scaffold rule, and the origin-contract requirement. ADR 001, ADR 028 and the two subsystem docs that describe `@hint(was:)` are amended to describe statements instead. + +## Project DoD + +In addition to the team DoD floor in [`drive/calibration/dod.md`](../../drive/calibration/dod.md): + +- A journey on Postgres and on SQLite renames a model and a field and deletes another model in one `migration plan` with three statements, runs `migrate` on tables with rows, a unique constraint, a foreign key, a secondary index and a check, and afterwards the rows and objects are present under the new names, the deleted table is gone, a further plan is empty, and `db verify --schema-only` is clean. +- The same change through `db update` with the same statements on both targets produces the same database state and no consent prompt. Running the same `db update` again with the same statements fails on the first statement. +- The same change with `--delete` omitted makes both commands refuse, and the refusal text contains the exact statement that then succeeds. +- A type change through `migration plan --convert` writes a migration whose only placeholder is in the Postgres `using` slot; filling it and re-running the file produces an applicable migration that converts the rows. The same change without a statement is refused by both commands, and `--delete` on the field plans the direct alter. +- A new required field on a non-empty table through `migration plan --backfill` writes the scaffolded backfill transform; without the statement both commands plan the temporary-default recipe. +- On Mongo, the rename journey above with a collection and a document field, through both commands, and a value object field rename that rewrites subdocuments. +- A `db update` against a database whose marker hash has no local snapshot fails every rename and convert statement with one error naming the hash, and succeeds with `--delete` alone. +- `--confirm` is gone from both commands, the CLI README describes the statements and the consent model, and the upgrade fragments for the refusal and the consent change are validated by execution. +- The project's ADR and doc amendments are merged. + +## Open questions + +Settled in the first slice's spec, not here: the exact grammar of a coordinate when a namespace and a model share a name, the separator spelling, and whether `--delete` on a type-changed field keeps that word or gets its own. + +## References + +- [`design-notes.md`](./design-notes.md) — the discussion record, the verb and noun matrix, the scenarios, the prior art. +- [prisma/orm#30570](https://github.com/prisma/orm/pull/30570) — the shelved slice whose planner substrate this project reuses. +- [prisma/orm#30331](https://github.com/prisma/orm/pull/30331) — the rename-table operation. +- [ADR 200](../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md) — the placeholder flow convert and backfill reuse. +- [ADR 224](../../docs/architecture%20docs/adrs/ADR%20224%20-%20Namespace%20concretions%20address%20entities%20by%20coordinate.md) — entity coordinates. From 97820831c62ab435273c17277d3772f5b67c6b52 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 5 Oct 2026 17:41:08 +0200 Subject: [PATCH 2/2] TML-3474: plan the migration-statements project as four slices Signed-off-by: willbot Signed-off-by: Will Madden Co-Authored-By: Claude Fable 5.1 --- projects/migration-statements/plan.md | 59 +++++++++++++++++++++++++-- 1 file changed, 55 insertions(+), 4 deletions(-) diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index 840f42494d0f..afb25dce425c 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -1,16 +1,67 @@ # Project plan — Migration statements -_Drafted after the spec is accepted. Replace this placeholder._ +**Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474) · **Shaping PR:** prisma/orm#30604 · **Working branch:** `tml-3474-migration-statements` -**Spec:** [`spec.md`](./spec.md) +## Summary + +Four slices. The first three stack: the statement surface with renames, then the refusal and consent model, then the scaffolded verbs and the remaining nouns. The fourth, Mongo, builds on the second and runs in parallel with the third. Each slice is one PR, stacked on the working branch until the shaping PR merges and then retargeted to `main`. ## Slices -_To be decomposed._ +### Slice 1 — Statements on the command line, and renames of models and fields on Postgres and SQLite + +**Linear:** [TML-3475](https://linear.app/prisma-company/issue/TML-3475) · **Folder:** `slices/renames/` + +**Outcome.** Both commands accept `--rename old:new` for models and fields, including a model rename across namespaces. The framework parses each statement, resolves it against the origin and destination contracts, and applies it in order to a working copy of the origin. The SQL planners emit the table or column rename and every companion rename, with names from the destination contract. `db update` resolves its origin contract from the marker hash through the snapshot store and fails every rename when it cannot. An unusable statement is an error. Non-data drops are widening. The written migration is what a user could write by hand. + +**Builds on.** The planner substrate from prisma/orm#30570, without its contract section. + +**Hands to.** A statement type the framework owns, parsed and resolved, delivered to every family planner as resolved entities; the working-copy mechanism statements apply to; the origin-contract resolution for `db update`; a column rename operation on both SQL targets. + +### Slice 2 — Both commands refuse data loss by default, and `--delete` is the per-operation consent + +**Linear:** [TML-3476](https://linear.app/prisma-company/issue/TML-3476) · **Folder:** `slices/refusal/` + +**Outcome.** `migration plan` refuses any plan that loses data, with the error `db update` uses. The refusal lists each destructive operation with the statements that resolve it. `--delete` consents to one operation, for namespaces, models and fields, and replaces `--confirm` on both commands. The terminal consent question asks per operation. Upgrade fragments record both changes; the CLI README describes statements and consent. + +**Builds on.** Slice 1. + +**Hands to.** The refusal shape every later verb hooks its statements into; the per-operation consent model; `--confirm` gone. + +### Slice 3 — Convert and backfill scaffold the placeholder migration, and the remaining nouns + +**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/convert-backfill/` + +**Outcome.** `--convert` scaffolds the type change with the placeholder in the slot that carries the conversion, and `--backfill` the backfill transform; both refused on `db update`; the scaffolding stops being automatic. `--rename` on enum values and namespaces and `--convert` on a variant plan the row updates and the schema rename. `--delete` on an enum value nulls where nullable, else refuses. + +**Builds on.** Slice 2. + +**Hands to.** Project close-out for the SQL targets. + +### Slice 4 — The same statements on MongoDB + +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo/` + +**Outcome.** The Mongo planner takes the same resolved statements: collection rename, document rewrites for field and value object field renames, drops and unsets for deletes, a data transform scaffold for convert, under the slice 2 refusal and consent model. No family vocabulary enters the framework. + +**Builds on.** Slice 2. + +**Hands to.** Project close-out for Mongo. + +## Sequencing + +- **Stack:** slice 1 → slice 2 → slice 3. +- **Parallel:** slice 4 runs beside slice 3 once slice 2 has merged. + +## Dependencies + +- Nothing external. The shelved prisma/orm#30570 is a source to copy from, not a dependency; its branch stays as a draft. +- Users get the feature through the next published CLI release after slice 2 merges; slices 1 and 2 together are the minimum that changes behaviour a user sees. ## Close-out (required) - [ ] Verify every project DoD item in [`spec.md`](./spec.md) -- [ ] Migrate long-lived docs into `docs/` and write the ADR +- [ ] Write the ADR and amend ADR 001, ADR 028 and the Data Contract and Migration System subsystem docs +- [ ] Migrate long-lived docs into `docs/` - [ ] Strip repo-wide references to `projects/migration-statements/**` - [ ] Delete `projects/migration-statements/`