diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index 9d0718e7c125..e79746a3f208 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -65,3 +65,50 @@ Adding a required field to a populated collection, or making an optional field r `CLI.CONSENT_REQUIRED` lists one `user-choice` next action per flag form ("Run the command again with --delete Legacy", another for the rename form), not grouped by subject and not a complete runnable command, and its summary says "N subjects need a statement", which is project vocabulary. Grouping the forms per subject and printing a full command would need the engine to know the invocation; raise it on prisma/prisma-cli once the statement prompt has shipped. +## SQL variants stored in their own tables + +**Found:** slice 4a design (2026-10-09). Will: SQL will need to handle multi-table storage of variants in the future. + +On MongoDB a variant's field rename touches only that variant's documents. On SQL today a variant's field is a column in the shared table, and a rename applies to every row. When SQL supports variants stored in their own tables, a variant's field statements must act on that variant's table only. + + +## MongoDB `@map` accepts field names MongoDB cannot validate + +**Found:** slice 4a review of dispatch 3 (2026-10-09). For the MongoDB authoring owners, not this project. + +`@map("o.d")` or `@map("$x")` is accepted in a MongoDB schema. MongoDB reads such a name as a path or an operator, and an insert of `{ "o.d": ... }` already fails the validator Prisma writes. Slice 4a skips such fields when planning statements. The fix belongs in authoring: reject the name with a diagnostic. When it lands, remove the framework resolver's fallback that reads `User.o.d` as field `o.d` (added in slice 4a), and the planner's silent skip of such fields. + +## A repeated run reports outdated statements one at a time + +**Found:** slice 4a manual QA (2026-10-09), F3. Applies to every family. + +Running the same `db update` again after it applied fails on the first outdated `--rename` or `--delete`, so a user with three outdated flags needs three runs to learn about all of them. The resolver could report every statement that no longer resolves in one error. + +## A collection name MongoDB rejects reaches the runner + +**Found:** slice 4a manual QA (2026-10-09), F9. For the MongoDB authoring owners. + +`@@map("system.x")` is accepted in a MongoDB schema. A rename onto it is refused with a reason, but a plan that creates or drops such a collection fails at apply with `CLI.UNEXPECTED`. Authoring should reject collection names MongoDB rejects. + +## `db migrate` of a migration with an unfilled placeholder gives advice that does not mention the slot + +**Found:** slice 3a review (2026-10-12). Older than this project's slices. + +A migration written with a placeholder has `ops.json` as `[]`. Running `db migrate` before filling the slot applies nothing and then fails at the destination check with general advice. It should say that the migration still has an unfilled slot, name the slot, and say to fill it and run `node migration.ts`. + +## `db migrate` applies an `ops.json` older than its `migration.ts` + +**Found:** slice 4a review (2026-10-12). Older than this project. + +The Migration System doc lists emit-drift detection (re-emit `migration.ts` in memory and compare with `ops.json` before applying) as architecturally required and not yet implemented. Without it, a user who edits `migration.ts` and forgets to run it applies the old operations; in slice 4a's review that turned a wrong advice text into a dropped collection. Worth a ticket when this project closes, if none exists. + +## A `--convert` expression can leave values the field's type cannot hold + +**Found:** slice 3a review (2026-10-12). + +A SQLite `--convert` rebuild runs the user's expression without the storage-class check the direct rebuild gets, so `trim("n")` on a field changed to `Int` leaves `'abc'` as text, and the ORM returns `"abc"` for an `Int` field. Postgres refuses such a value itself. Consider running the same check after a converted copy. + +## `migration status --json` prints `{bin}` unreplaced + +**Found:** slice 3a review (2026-10-12). Outside this project. The summary text in the JSON output contains a raw `{bin}` placeholder. + diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index 0090e6f8d0d8..91c59007bc8c 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -1,82 +1,106 @@ # Project plan — Migration statements -**Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474) · **Shaping PR:** prisma/orm#30604 · **Working branch:** `tml-3474-migration-statements` +**Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474) -## Summary +## At a glance -Four slices. The first three stack: the statement surface with renames, then the refusal and consent model, then the scaffolded verbs and the remaining nouns. The fourth, Mongo, builds on the second and runs in parallel with the third. Each slice is one PR, stacked on the working branch until the shaping PR merges and then retargeted to `main`. +| Slice | What it delivers | Builds on | State | +| --- | --- | --- | --- | +| 1 | `--rename` for models and fields, Postgres and SQLite | — | Merged, prisma/orm#30638 | +| 2 | Data loss refused until answered; `--delete`, `--allow`; the interactive prompt | 1 | Merged, prisma/orm#30648 | +| 3a | `--convert` and `--backfill`, Postgres and SQLite | 2 | Spec: [`slices/convert-backfill/`](./slices/convert-backfill/spec.md) | +| 4a | Renames and deletes on MongoDB | 2 | Spec: [`slices/mongo-renames/`](./slices/mongo-renames/spec.md) | +| 4p | MongoDB contracts written in TypeScript get the same validator as PSL; removes 4a's temporary refusal | 4a | Next after 4a | +| 3b | The remaining SQL renames and deletes | 3a | Not started | +| 4b | `--convert`, `--backfill` and value object renames on MongoDB | 3b, 4a | Not started | -## Slices +```text +1 → 2 ─┬─ 3a ── 3b ─┐ + └─ 4a ───────┴─ 4b → close-out +``` -### Slice 1 — Statements on the command line, and renames of models and fields on Postgres and SQLite +- 3a and 4a run in parallel now. +- Each slice is one pull request against `main`. +- Slices 3 and 4 were split on 2026-10-08. Each was too large to review as one pull request. -**Linear:** [TML-3475](https://linear.app/prisma-company/issue/TML-3475) · **Folder:** `slices/renames/` +## Slices still to build -**Outcome.** Both commands accept `--rename old:new` for models and fields, with a model rename across namespaces resolved but refused until slice 3. The framework parses each statement, resolves it against the origin and destination contracts, and applies it in order to a working copy of the origin. The SQL planners emit the table or column rename and every companion rename, with names from the destination contract. `db update` resolves its origin contract from the marker hash through the snapshot store and fails every rename when it cannot. An unusable statement is an error. Non-data drops are widening. The written migration is what a user could write by hand. +### 3a — `--convert` and `--backfill` on Postgres and SQLite -**Builds on.** The planner substrate from prisma/orm#30570, without its contract section. +**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Spec:** [`slices/convert-backfill/spec.md`](./slices/convert-backfill/spec.md) -**Hands to.** A statement type the framework owns, parsed and resolved, delivered to every family planner as resolved entities; the working-copy mechanism statements apply to; the origin-contract resolution for `db update`; a column rename operation on both SQL targets. +- A lossy type change is answered with `--convert` (fill in the conversion) or `--delete`. +- `--backfill` writes a slot to fill existing rows of a field that became required. +- `migration plan` stops writing placeholders nobody asked for. +- A new required field gets a temporary value on both targets. SQLite lacks this today. +- The `SET NOT NULL` failure names the NULL rows (TML-3517). -### Slice 2 — Both commands refuse data loss by default, and `--delete` is the per-operation consent +**Hands to 3b and 4b:** the two flags and how their questions and placeholders work. -**Linear:** [TML-3476](https://linear.app/prisma-company/issue/TML-3476) · **Folder:** `slices/refusal/` +### 4a — Renames and deletes on MongoDB -**Outcome.** `migration plan` refuses any plan that loses data, with the error `db update` uses. The refusal lists each destructive operation with the statements that resolve it. `--delete` consents to one operation, for namespaces, models and fields, and replaces `--confirm` on both commands. The terminal consent question asks per operation. Upgrade fragments record both changes; the CLI README describes statements and consent. +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Spec:** [`slices/mongo-renames/spec.md`](./slices/mongo-renames/spec.md) -**Builds on.** Slice 1. Two things slice 1 leaves for it: the missing-origin check in the CLI statement resolver runs over the whole input and must move inside the per-statement loop so that `--delete` works without an origin contract; and the `@@map`-only rename gap in `deferred.md` must be decided before the refusal text is written. Decided 2026-10-07 on prisma/orm#30638: dropping a row-level-security policy and disabling row-level security stay `widening` (they lose no data), and slice 2 adds a separate consent question before `db update` applies an operation that widens who can read or write rows, answered per operation like the data-loss consent and refused in a non-interactive run unless consented. +- A model rename renames the collection. A field rename rewrites the documents. +- Removing a field is asked about. `--delete` removes it from every document. +- Both commands, including `db update`. +- The temporary setting that makes MongoDB refuse renames is deleted. -**Design settled 2026-10-07 (Will).** Consent is a statement the user answers with, not a token copied back: the CLI engine gains `ctx.prompt.statement`/`statements`, answered on the command line by verb flags (`--delete Legacy`, `--rename Legacy:Archive`, `--allow User`) and interactively by typing the statement when a human runs the command. The engine change is its own slice, built in prisma/prisma-cli first: [`slices/engine-statement-prompt/spec.md`](./slices/engine-statement-prompt/spec.md). Slice 2's spec is [`slices/refusal/spec.md`](./slices/refusal/spec.md). The `@@map`-only gap gets no statement (see `deferred.md`). Operations that lose no data (`SET NOT NULL`, safe type widenings, MongoDB validator and index changes) stop being destructive. +**Hands to 4b:** a MongoDB planner that takes statements. -**Hands to.** The refusal shape every later verb hooks its statements into; the per-operation consent model; `--confirm` no longer consents to data loss. +### 4p — TypeScript MongoDB contracts get the PSL validator -### Slice 3 — Convert and backfill scaffold the placeholder migration, and the remaining nouns +**Linear:** [TML-3553](https://linear.app/prisma-company/issue/TML-3553) · **Folder:** [`slices/mongo-ts-validator/`](./slices/mongo-ts-validator/investigation.md) -**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/convert-backfill/` +Decided by Will on 2026-10-12: contracts written in either form must behave the same, and this slice comes right after 4a, before 3b and 4b. Two pull requests: -**Outcome.** `--convert` scaffolds the type change with the placeholder in the slot that carries the conversion, and `--backfill` the backfill transform; both refused on `db update`; the scaffolding stops being automatic. `--rename` on enum values, namespaces and value object fields, a model move across namespaces (`alter table set schema` on Postgres, deferred from slice 1 on 2026-10-06 because no operation existed for it), and `--convert` on a variant, plan the row updates, the JSON rewrites and the schema rename. `--delete` on an enum value nulls where nullable, else refuses. +1. The validator derivation moves into `@internal/mongo-contract`. No output changes. +2. The TypeScript builder calls it. 4a's temporary refusal goes. An upgrade fragment covers the closed validator TypeScript users gain. -**Builds on.** Slice 2. +**Why now:** without a validator, a TypeScript contract's field names are not in its storage hash, so a MongoDB field rename on it gives a migration `db migrate` refuses. 4a refuses that case for now. -**Hands to.** Project close-out for the SQL targets. +### 3b — The remaining renames and deletes on Postgres and SQLite -### Slice 4 — The same statements on MongoDB +**Linear:** [TML-3537](https://linear.app/prisma-company/issue/TML-3537) · **Folder:** `slices/remaining-nouns/` -**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo/` +- `--rename` for enum values, namespaces and fields inside value objects. This slice builds the statement support for value objects, which 4b reuses. +- Moving a model to another namespace (`alter table set schema` on Postgres). +- `--convert` on a variant whose discriminator value changed. +- `--delete` on an enum value: sets it to NULL where the field is optional, otherwise refuses. +- `--delete` on a variant stored in its base's table: deletes its rows and drops its columns. The refusal says rows are deleted. +- `--delete ` answers every question for the models in it. -**Outcome.** The Mongo planner takes the same resolved statements: collection rename, document rewrites for field and value object field renames, drops and unsets for deletes, a data transform scaffold for convert, under the slice 2 refusal and consent model. No family vocabulary enters the framework. +**Risk to settle in its spec:** a value object can contain itself (ADR 178's `NavItem.children`). Renaming a field inside it means rewriting to any depth, which a fixed SQL JSON expression or MongoDB update cannot do. -**Builds on.** Slice 2. Slice 2 left a temporary member for it: `TargetMigrationsCapability.refusesRenameStatements`, set by MongoDB so a data-loss question offers no `rename` there. Slice 4 deletes the member, its one CLI read and its doc mention when the Mongo planner carries out renames. +**Hands to:** close-out for Postgres and SQLite. -**Hands to.** Project close-out for Mongo. +### 4b — `--convert`, `--backfill` and value object renames on MongoDB -## Stretch goal — interactive statements when a human runs the command +**Linear:** [TML-3538](https://linear.app/prisma-company/issue/TML-3538) · **Folder:** `slices/mongo-convert-backfill/` -Recorded 2026-10-06 at the operator's request. This is the eventual direction, and the slices build with it in mind so that it drops in on top of their mechanisms; it is not a slice of its own, and it is built once the mechanisms it needs exist (after slice 2 at the earliest). +- The same two flags as 3a, writing a data transform with a placeholder. +- Making a field required on a collection with documents is no longer silent. It is answered with `--backfill`, or the command says the existing documents need the field. +- Renaming a field inside a value object rewrites the embedded documents, including lists, dictionaries and unions. +- Removing a variant model is data loss: today its documents stay and fail every update. `--delete ` deletes the documents with that discriminator value, as 3b does for SQL rows. -When `migration plan` or `db update` detects a destructive operation and no statement covers it, the command today refuses and prints the statements that would resolve it. The stretch goal: when the command is run by a human in a terminal (stdin and stdout are a TTY, and no `--json` or similar non-interactive flag is set), the refusal becomes an interactive prompt that asks, per destructive operation, what the user means: rename it to a name they type, or delete it. Each answer is exactly the statement the refusal would have printed, applied in order, and the plan then proceeds as if the statements had been given on the command line. In a script, an agent, or CI (no TTY) the behaviour stays as it is: refuse and print the statements. +**Hands to:** close-out for MongoDB. -Rules that carry over unchanged: the prompt never guesses or proposes a rename candidate; it asks. The statements are not recorded anywhere; the migration file or the database is the record. The consent model of slice 2 (`--delete` per operation) is what the "delete" answer maps to. +### Alongside — a syntax for naming storage objects -What the slices do now so this drops in later (see the spec's cross-cutting requirement 12): +There is no statement for a model that keeps its name but changes its table (`@@map`). Will asked on 2026-10-07 for a critical discussion and a look at how established tools name tables on the command line before any syntax is chosen. `--rename table/users:app_users` was an off-the-cuff example, not a decision. -- The refusal is a structured value first and text second: a list of destructive operations, each with the entity it touches in domain coordinates and the statements that would resolve it, rendered to the error message by the command. The prompt consumes the same list. -- Statement text is parsed, resolved and applied through one function the command calls, which takes the statement strings and the two contracts and does not care whether the strings came from the command line or from an answer typed at a prompt. Statements can therefore be added after the command has started and the plan re-run. -- The per-operation consent question that `db update` already asks in a terminal is the seed of this prompt, so slice 2 keeps it as a per-operation question rather than folding it into a yes/no over the whole plan. +- The write-up goes to Will for a decision. +- It does not block close-out. +- Building the chosen syntax is not in this project unless Will adds it. -## Sequencing +## Delivered: the interactive prompt -- **Stack:** slice 1 → slice 2 → slice 3. -- **Parallel:** slice 4 runs beside slice 3 once slice 2 has merged. - -## Dependencies - -- Nothing external. The shelved prisma/orm#30570 is a source to copy from, not a dependency; its branch stays as a draft. -- Users get the feature through the next published CLI release after slice 2 merges; slices 1 and 2 together are the minimum that changes behaviour a user sees. +Recorded on 2026-10-06 as the stretch goal: when a human runs the command in a terminal, the refusal becomes a question per operation, and the answer is the same statement the flags take. Slice 2 built it on the CLI engine's statement prompt (prisma/prisma-cli#337). Scripts, agents and CI get the refusal that names the flags. ## Close-out (required) - [ ] Verify every project DoD item in [`spec.md`](./spec.md) +- [ ] Final retro - [ ] Write the ADR and amend ADR 001, ADR 028 and the Data Contract and Migration System subsystem docs - [ ] Migrate long-lived docs into `docs/` - [ ] Strip repo-wide references to `projects/migration-statements/**` diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md b/projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md new file mode 100644 index 000000000000..340b3935c712 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md @@ -0,0 +1,36 @@ +# Dispatch 1 — `--convert` and `--backfill` reach the planner, and questions know why data is lost + +**Slice:** [`../spec.md`](../spec.md) § The two flags, § How the pieces fit, § Implementation notes · **Plan:** [`../plan.md`](../plan.md) dispatch 1 · **Builds on:** slice 2 (merged on `main`) + +## Outcome + +Both flags are parsed, resolved against the two contracts, and handed to the SQL planners as destination coordinates. The planners do nothing new with them yet. Each data-loss entry says whether it is a `drop` or a `typeChange`, and the questions offer the verbs that fit. `db update` refuses both flags with a clear error. + +## What to build + +1. **Verbs per command.** `orm/statement-verbs.ts` declares verbs per command. `migration plan`: `rename`, `delete`, `convert`, `backfill`. `db update`: `rename`, `delete`, `allow`, plus `convert` and `backfill` declared only so they can be refused: any value fails with the new `MIGRATION.STATEMENT_NEEDS_MIGRATION_FILE`, whose message says `db update` has no migration file to fill and names `migration plan`. Error reference entry. +2. **Resolved statements gain a `kind`.** `ResolvedMigrationStatement` (`packages/1-framework/1-core/framework-components/src/control/migration-statements.ts`) becomes `kind: 'rename' | 'convert' | 'backfill'`. Every reader that assumes a rename switches on `kind` first: `describeMigrationStatement`, `migrationStatementJson`, `sameCoordinate` and `renamedModel` in `plan-questions.ts`, `originTable` and `originColumn` in the SQL family `operation-subjects.ts`, `StatementPlanner.plan` in `statement-planning.ts` (today it plans any non-model statement as a column rename), and the MongoDB planner (keeps refusing every statement it cannot carry out). +3. **Resolution rules** (in `resolve-statements.ts`): + - `convert Model.field`: names the field by its destination name (after `--rename User.age:User.years`, `--convert User.years`); the field exists in the destination and, through any earlier rename, in the origin; its type changes. Needs the origin contract, else `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`. + - `backfill Model.field`: the field is required in the destination and is either new on a model that exists in the origin or optional in the origin. + - Anything else is `MIGRATION.STATEMENT_UNRESOLVED` naming what was found. +4. **Handed to the planners.** `migration plan` takes `convert` and `backfill` from the run before planning, like `rename`. The SQL planners receive them as destination coordinates (namespace, table, column) on the strategy context (Postgres `planner-strategies.ts` around `StrategyContext`; SQLite equivalent). They do not go through `planStatements`, which applies renames to the working schema. No strategy reads them yet. +5. **The loss kind.** Each `dataLoss` entry gains `loss: 'drop' | 'typeChange'`, set by each target's operation-subjects function. `dataLossQuestion` stops inferring from whether the destination still has the field. A `typeChange` question offers `convert` and `delete` on `migration plan` and `delete` only on `db update`. A `drop` question offers `rename` and `delete`, as today. +6. **Re-plan with every statement.** `askPlanQuestions` re-plans with all planned statements, not only renames, so a typed `convert` re-plans. +7. **Reporting a delete on a type change.** A `delete` answering a `typeChange` question is described as `delete values of field "User.age" (type change)`, not `delete field`. + +## Not in this dispatch + +Any planner output change (dispatches 2–4). The temporary default. Docs beyond the error reference entry. Journeys. + +## Tests + +Unit tests in the CLI and framework-components packages: each resolution rule, both pass and fail; `convert` after a rename uses the new name; `convert` with no origin contract; the verbs offered per loss kind and per command; `db update --convert` fails with the new code before planning; a typed `convert` triggers a re-plan; the delete-on-type-change description; every `kind` switch (a `convert` statement is never planned as a column rename — write the test that would have caught it). + +## Halt conditions + +Stop and report if the engine cannot declare a verb on `db update` that the command refuses with its own error; or if `dataLoss` producers outside the SQL family (extension spaces applying recorded migrations, MongoDB) cannot set `loss` without a design decision. + +## Gate + +The plan's gate, plus `pnpm check:error-reference` and `pnpm lint:framework-vocabulary`. diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/02-postgres-convert.md b/projects/migration-statements/slices/convert-backfill/dispatches/02-postgres-convert.md new file mode 100644 index 000000000000..229db0040e60 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/02-postgres-convert.md @@ -0,0 +1,34 @@ +# Dispatch 2 — Postgres: `--convert` writes the conversion slot + +**Slice:** [`../spec.md`](../spec.md) § What changes for users, § What the planners write · **Plan:** [`../plan.md`](../plan.md) dispatch 2 · **Builds on:** dispatch 1 + +## Outcome + +On Postgres, a lossy type change plans no placeholder unless `--convert` names the field. With `--convert User.age`, `migration plan` writes one `alterColumnType` whose `using` is `placeholder(...)`, classed `data`. `alterColumnType` takes plain options. + +## What to build + +1. **Gate the type-change scaffold.** `typeChangeCallStrategy` (`packages/3-targets/3-targets/postgres/src/core/migrations/planner-strategies.ts`) runs only for a field that a `convert` statement names. Without one, the type change falls through to the direct alter: `widening` for a known-safe widening, `destructive` otherwise, which dispatch 1's questions already ask about. +2. **One slot, in the alter.** With `convert`, the strategy writes a single `alterColumnType` with `using: placeholder('.:using')` and no separate data transform. No separate NULL transform either: when the field also becomes required, `SET NOT NULL` follows the alter, and the conversion expression can map NULL. +3. **Class `data`.** `AlterColumnTypeClass` gains `data`. A converted alter is `data`, so it is not in `dataLoss`, and a typed `convert` at the prompt resolves the question on re-plan. +4. **Plain options.** The facade's `alterColumnType` (`postgres-migration.ts`) takes `{ table, column, type, using? }` (plus a namespace where the facade needs one; match the neighbouring facade calls). The rendered call shows that shape, with no `qualifiedTargetType`, `formatTypeExpected` or `rawTargetTypeForLabel`. The `using` slot renders as code, imports `placeholder`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result (`unfilledPlaceholderOperation`), so the CLI writes `ops.json` as `[]` with `pendingPlaceholders`. +5. **Upgrade instructions** for hand-written migrations that call `alterColumnType`, using the `record-upgrade-instructions` skill. + +## Not in this dispatch + +Backfill and the temporary default (dispatch 3). SQLite (dispatch 4). Docs beyond the upgrade instructions. + +## Tests + +- Planner: a lossy change with no statement plans the direct `destructive` alter; with `convert` plans one `data` alter with the slot; a safe widening ignores `convert` (dispatch 1's resolver already refuses a convert with nothing to convert — keep that). +- Rendering: the rendered `migration.ts` for a convert; filling the slot with `'"age"::integer'` and running the file produces an `ops.json` whose SQL is `ALTER TABLE ... TYPE integer USING "age"::integer`; that SQL runs on PGlite against rows holding numeric strings and converts them. +- CLI: `migration plan --convert User.age` reports `pendingPlaceholders`; a typed `convert` at the prompt re-plans and asks nothing more. +- Update `render-typescript.test.ts`, `postgres-issue-planner.test.ts` and any test that asserted the old automatic scaffold. + +## Halt conditions + +Stop and report if the `using` slot cannot render as code without changing the shared call-rendering helper for every call; or if any example migration under `examples/` calls `alterColumnType` with the old options in a way the upgrade instructions cannot describe. + +## Gate + +The plan's gate, plus `pnpm fixtures:check`, `pnpm check:upgrade-coverage --mode pr --prev $(git merge-base HEAD origin/main) --head HEAD`, and `test/integration/test/cli-journeys/data-transform-strategies.e2e.test.ts` (its type-change scenario changes). diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/03-postgres-backfill-and-temporary-default.md b/projects/migration-statements/slices/convert-backfill/dispatches/03-postgres-backfill-and-temporary-default.md new file mode 100644 index 000000000000..1e9439fccf7f --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/03-postgres-backfill-and-temporary-default.md @@ -0,0 +1,36 @@ +# Dispatch 3 — Postgres: `--backfill`, and the temporary default as a real call + +**Slice:** [`../spec.md`](../spec.md) § What changes for users, § What the planners write · **Plan:** [`../plan.md`](../plan.md) dispatch 3 · **Builds on:** dispatch 1 (dispatch 2 may land first; nothing here depends on it) + +## Outcome + +On Postgres, `migration plan` writes a backfill placeholder only for a field `--backfill` names. Without it, a new required field on an existing table gets the temporary default under `migration plan` as well as `db update`, and an optional field made required is a direct `SET NOT NULL`. The temporary default renders as a call that reproduces `ops.json`. + +## What to build + +1. **Gate the backfill scaffolds.** `notNullBackfillCallStrategy` and `nullableTighteningCallStrategy` (`planner-strategies.ts`) run only for a field a `backfill` statement names. + - New required field with `--backfill`: nullable add, `dataTransform` with placeholders, `SET NOT NULL` (today's recipe). + - Optional made required with `--backfill`: `dataTransform` with placeholders, `SET NOT NULL`. +2. **No flag.** + - New required field on an existing table: the temporary-default recipe (`notNullAddColumnCallStrategy`), now under `migration plan` too. It keeps today's refusal for a column in a new primary key, unique or foreign key, falling back to the empty-table direct add as today. + - Optional made required: a direct `SET NOT NULL`, `widening`. +3. **A real call.** `AddNotNullColumnWithTempDefaultCall` and `AddNotNullColumnDirectCall` render today as `rawSql({ id, label, operationClass })` with no SQL (`op-factory-call.ts`), so re-running `migration.ts` does not reproduce `ops.json`. Render each as a facade call (a new facade method on the Postgres migration class, or an existing one with the right options) that builds the same operation. Running the written `migration.ts` reproduces `ops.json` byte for byte. + +## Not in this dispatch + +SQLite (dispatch 4). The NOT NULL failure text (dispatch 5). Docs and upgrade fragment (dispatch 6). + +## Tests + +- Planner: each row of the spec's table for Postgres, with and without `--backfill`, under `migration plan` and `db update`. +- Rendering round trip: the temporary default and the direct add, written by `migration plan`, re-run, `ops.json` identical. +- On PGlite: the temporary-default migration applied to a table with rows leaves the column `NOT NULL` with no default and the fill value in old rows; the backfill scaffold, filled with an `UPDATE`, applies. +- Update `render-typescript.test.ts` (its pinned `rawSql` output changes), `nullability-backfill` and any test that asserted the automatic scaffold; `test/integration/test/cli-journeys/invariant-routing.e2e.test.ts` (around lines 130–160) needs `--backfill`. + +## Halt conditions + +Stop and report if the temporary default cannot be expressed as a facade call without exposing internal operation fields in `migration.ts`; or if a codec's temporary value cannot be rendered as code. + +## Gate + +The plan's gate, plus `pnpm fixtures:check` and the named journeys `data-transform-strategies`, `invariant-routing`, and any other file under `test/integration` or `test/e2e` that reaches `migration plan` or `db update` with a new required field (find them through the helpers in `test/integration/test/utils/journey-test-helpers.ts` as well as the command names). diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/04-sqlite.md b/projects/migration-statements/slices/convert-backfill/dispatches/04-sqlite.md new file mode 100644 index 000000000000..49799b641153 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/04-sqlite.md @@ -0,0 +1,31 @@ +# Dispatch 4 — SQLite: `--convert`, `--backfill` and the temporary value + +**Slice:** [`../spec.md`](../spec.md) § What the planners write, § Implementation notes · **Plan:** [`../plan.md`](../plan.md) dispatch 4 · **Builds on:** dispatches 2 and 3 + +## Outcome + +SQLite behaves like Postgres for every row of the spec's table. A table rebuild can copy a column through an expression, which carries the conversion slot and the temporary value. + +## What to build + +1. **Copy expressions.** `recreateTable` (`packages/3-targets/3-targets/sqlite/src/core/migrations/operations/tables.ts`) takes a per-column copy expression for the `INSERT INTO … SELECT` step. `RecreateTableCall` renders it as code, so it can hold `placeholder(...)`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result. +2. **`--convert`.** A lossy type change plans nothing without a statement (the destructive rebuild that dispatch 1's questions ask about). With `--convert`, the rebuild's copy expression for the column is `placeholder('.:using')`, and the rebuild is classed `data`. +3. **`--backfill`.** New required field: add the column nullable, a `dataTransform` with placeholders, then a tightening rebuild. Optional made required: `dataTransform`, then the tightening rebuild. `nullabilityTighteningBackfillStrategy` runs only for a field `--backfill` names. +4. **No flag.** Optional made required: the tightening rebuild, `widening`. New required field on an existing table: a rebuild whose copy expression for the new column is a fill value, under both commands. Refused for a column in a new primary key, unique or foreign key, as on Postgres. +5. **Fill values.** A SQLite fill-value map, consulted after the codec hook as on Postgres: integer `0`, real `0`, text `''`, blob `X''`, and whatever SQLite's boolean and date-time codecs store (find them in the SQLite target's codecs; do not guess). + +## Tests + +- Planner: each row of the spec's table for SQLite, with and without each flag, under `migration plan` and `db update`. +- Rendering round trip for the rebuild with a copy expression and with a placeholder. +- On SQLite with rows: the conversion slot filled with `CAST("age" AS INTEGER)` converts the values; the temporary value lands in old rows and the column is `NOT NULL`; the backfill scaffold, filled, applies. +- `delete-statements-migration.sqlite.e2e.test.ts` journey S3 now adds the required field to a table with rows (it used an empty table because of this gap). +- The deferred item "SQLite `db update` cannot add a required field to a table that has rows" is closed: remove it from `projects/migration-statements/deferred.md`. + +## Halt conditions + +Stop and report if a SQLite codec stores booleans or dates in a form that has no safe fill value; or if the copy expression cannot be added without changing the rebuild's postcheck SQL for plans that do not use it (existing `ops.json` must not change). + +## Gate + +The plan's gate, plus `pnpm fixtures:check` and every file under `test/integration` or `test/e2e` that runs `migration plan` or `db update` against SQLite with a type change or a new required field (directly or through `journey-test-helpers.ts`, F44). diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md b/projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md new file mode 100644 index 000000000000..fab087fafef0 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md @@ -0,0 +1,38 @@ +# Dispatch 5 — The NOT NULL failure names the NULL rows (TML-3517) + +**Slice:** [`../spec.md`](../spec.md) § How the pieces fit · **Plan:** [`../plan.md`](../plan.md) dispatch 5 · **Builds on:** dispatches 3 and 4 + +## Outcome + +When `SET NOT NULL` on Postgres, or a tightening rebuild on SQLite, meets NULL values, the command says how many rows hold NULL in which column, and how to fix it. It no longer talks about schema drift. + +Today, on Postgres: + +```text +✖ [MIGRATION.RUNNER_FAILED] Operation alterNullability.setNotNull.Item.note failed during precheck: ensure no NULL values in "note" + why: Migration runner failed +→ Inspect the reported conflict, reconcile schema drift if needed, then re-run `prisma db update` +``` + +After: + +```text +✖ [MIGRATION.RUNNER_FAILED] 3 rows of "Item" hold NULL in "note", so it cannot become required. +→ Set a value in those rows, or keep the field optional. +→ Or plan it with `prisma migration plan --backfill Item.note` and fill the backfill step. +``` + +## What to build + +1. **The runner reports the count.** When the precheck that refuses NULLs fails (Postgres `runner.ts` around lines 331–355), the runner counts the NULL rows and returns a `why` with the table, the column and the count. SQLite: the tightening rebuild's copy fails with `NOT NULL constraint failed`; turn that into the same report, with the count. +2. **A `fix` on the failure.** `MigrationRunnerFailure` gains an optional `fix`. The `db update`, `db init` and `migrate` error mappers (`db-update-failure.ts` around lines 23–27, and the matching spots for the other two) use it when present, instead of the fixed drift advice. +3. **A required column added to an empty table only** (a new unique, primary-key or foreign-key column, or a type with no temporary value such as a native enum) fails its empty-table precheck on a table with rows. Today `db migrate` says "Fix the issue and re-run" and `db update` says to reconcile schema drift. Give it its own advice naming the field: plan it with `migration plan --backfill ` and fill the step, make the field optional, or give it a default. On `db update`, point to `migration plan`. (Review finding D3-1 of dispatch 3; the reviewer confirmed the `--backfill` route applies for an enum column and a unique column.) +4. **The fix names a flag only where it applies.** `--backfill` is suggested for `db update` and `migration plan`, written in the field's contract name when the origin contract is known, and through `statementFlag` so it is shell-quoted. Under `migrate`, the advice says to fix the rows and re-run, since the migration is already written. + +## Tests + +On PGlite and SQLite, a table with NULLs in a field made required: each command prints the count and the fix; following the fix (set the values, re-run) applies; following the `--backfill` advice plans a backfill scaffold. The previous text no longer appears. + +## Gate + +The plan's gate, plus `pnpm check:error-reference` if a code or entry changes, and every journey that asserts the old drift advice. diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md b/projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md new file mode 100644 index 000000000000..43fd3bd9660a --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md @@ -0,0 +1,30 @@ +# Dispatch 6 — Journeys, docs and upgrade fragments + +**Slice:** [`../spec.md`](../spec.md) § Done when, § Scope · **Plan:** [`../plan.md`](../plan.md) dispatch 6 · **Builds on:** dispatches 1–5 + +## Outcome + +The slice's done conditions hold through journeys on Postgres and SQLite, and the docs, error reference, skill and upgrade fragment describe the new behaviour. + +## What to build + +1. **Journeys**, on tables with rows, on both targets: + - a type change: refused without a flag; `--convert`, fill the slot, run `migration.ts`, `migrate`, rows converted; `--delete` plans the direct change; + - a new required field: no flag gives the temporary value; `--backfill`, fill the step, `migrate`; + - an optional field made required with NULLs present: the failure names the rows; after setting them, it applies; + - `db update --convert` refused, and its printed `migration plan` command run as printed; + - afterwards a further plan is empty and `db verify --schema-only` is clean. +2. **Follow every printed advice to its end** (F46), on each target, including the `db update` refusal through `migrate` on a database created by `db init`. +3. **Docs.** + - Migration System doc § Statements: `--convert` and `--backfill`; the loss kinds; the questions per command; remove the sentence saying the Postgres planner writes no placeholder data transforms (§ Authoring intermediate state); § NOT NULL columns without defaults now applies to `migration plan` too. + - Migration System doc: what a `--delete` answer to a type change does (the database's cast on Postgres; the storage-class check after a SQLite rebuild), how it is reported, and the `--convert` advice when a cast fails. + - `docs/reference/error-reference.md`: every code added or changed. + - The CLI README sections for `migration plan` and `db update`. + - `skills/prisma-8/references/migrations.md`: type changes and new required fields. +4. **App upgrade fragment** (`record-upgrade-instructions` skill): lossy type changes now ask; placeholders only with `--convert` or `--backfill`; a new required field gets a temporary value under `migration plan`; `alterColumnType`'s options (if dispatch 2's instructions do not already cover it). + +The orchestrator writes the ADR 200 amendment; do not edit that file. + +## Gate + +The plan's gate, plus `pnpm check:upgrade-coverage --mode pr --prev $(git merge-base HEAD origin/main) --head HEAD`, `pnpm lint:docs`, the journey files, and every other file under `test/integration` or `test/e2e` that runs `migration plan` or `db update` on Postgres or SQLite, directly or through `journey-test-helpers.ts` (F44). diff --git a/projects/migration-statements/slices/convert-backfill/plan.md b/projects/migration-statements/slices/convert-backfill/plan.md new file mode 100644 index 000000000000..7d86a0aea322 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/plan.md @@ -0,0 +1,38 @@ +# Slice plan — 3a, `--convert` and `--backfill` + +**Spec:** [`spec.md`](./spec.md) · **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Branch:** `tml-3477-convert-backfill` + +One implementer and one reviewer, both on Opus, resumed across every dispatch. Briefs are in `dispatches/`. + +## Gate for every dispatch + +- `pnpm typecheck`; `pnpm --filter lint`; `pnpm --filter test`. +- `pnpm lint:deps` when imports change; `pnpm fixtures:check` when rendering or classes change; `pnpm check:error-reference` when error codes change; `pnpm lint:framework-vocabulary` when `packages/1-framework` changes; `pnpm lint:throws` and `pnpm lint:casts` (CI runs them, `lint:agent` skips them). +- Integration and e2e: only named files, run as `pnpm test `. Never `pnpm test:integration`, `test:e2e` or `test:all`. +- The reviewer runs code, not only reads it (`drive/calibration/failure-modes.md` F34): SQL the slice renders is compiled and run on PGlite and SQLite, and every printed advice is followed (F46). + +## Dispatches + +| # | Outcome | Builds on | +| --- | --- | --- | +| 1 | `--convert` and `--backfill` are parsed, resolved and handed to the planners; each data-loss entry says `drop` or `typeChange`; questions offer the right verbs; `db update` refuses both flags | slice 2 | +| 2 | Postgres: a lossy type change plans nothing without a statement; `--convert` writes `alterColumnType` with a `using` placeholder; `alterColumnType` takes plain options | 1 | +| 3 | Postgres: `--backfill` writes the backfill scaffold; without it a new required field gets the temporary default as a real call in `migration.ts`; optional-to-required is a direct `SET NOT NULL` | 1 | +| 4 | SQLite: the rebuild takes a per-column copy expression; `--convert`, `--backfill` and the temporary value work as on Postgres | 2, 3 | +| 5 | The NOT NULL failure names the NULL rows and suggests a fix, on Postgres and SQLite (TML-3517) | 3, 4 | +| 6 | Journeys for the slice's done conditions; docs; app upgrade fragment and facade upgrade instructions; every printed advice followed | 1–5 | + +**Hand-offs** +- 1 → 2, 3: planners receive convert and backfill as destination coordinates, and a converted change leaves `dataLoss`. +- 2 → 4: the converted alter is `data`; the `using` slot renders as code and blocks `toOp()` while unfilled. +- 3 → 4: the temporary default renders as a call that reproduces `ops.json`. +- 6 → slice DoD. + +## Coordination with 4a + +4a runs at the same time on `tml-3478-mongo-renames`. Both touch `plan-questions.ts` and the resolved statement type. Whichever merges second merges `main` and adapts. The MongoDB planner must keep refusing `convert` and `backfill` statements until 4b. + +## Open items + +- ADR 200 amendment: written by the orchestrator in dispatch 6's branch, not by the implementer. +- Both 3a and 4a add a framework type `MigrationDataLoss` (3a with an optional `label`, 4a with an optional `keepDataAdvice`). Whichever merges second combines them into one type. diff --git a/projects/migration-statements/slices/convert-backfill/spec.md b/projects/migration-statements/slices/convert-backfill/spec.md new file mode 100644 index 000000000000..7f320106fd87 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/spec.md @@ -0,0 +1,135 @@ +# Slice 3a — `--convert` and `--backfill` + +**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477), [TML-3517](https://linear.app/prisma-company/issue/TML-3517) · **Branch:** `tml-3477-convert-backfill` · **Builds on:** slice 2 (merged) · **Project:** [`../../spec.md`](../../spec.md) + +## In one example + +`User.age` changes from `String` to `Int` on Postgres. + +```text +$ prisma migration plan --name age-to-int +✖ [CLI.CONSENT_REQUIRED] 1 subject needs a statement, and the session is not interactive. + why: Change type of column "User"."age" would lose the data of field "User.age". +→ Pass --convert User.age +→ Pass --delete User.age + +$ prisma migration plan --name age-to-int --convert User.age +⚠ migration.ts has a placeholder: replace it with the conversion, then run the file to re-emit. +``` + +`migration.ts` then has one slot to fill: + +```ts +this.alterColumnType({ + table: 'User', + column: 'age', + type: 'int4', + using: placeholder('User.age:using'), +}), +``` + +The user writes `using: '"age"::integer'`, runs the file, and applies it. + +## What changes for users + +| Change in the schema | Today | After this slice | +| --- | --- | --- | +| Type change that loses data | Placeholder migration, written without asking | Refused. Answer `--convert` (fill in the conversion) or `--delete` (accept the loss) | +| Type change that keeps every value (`int4` → `int8`) | Planned, nothing asked | Same | +| Required field added to an existing table | Placeholder migration, written without asking | Filled with a temporary value (`''`, `0`, …), nothing asked. `--backfill` writes a slot to fill instead | +| Optional field made required | Placeholder migration, written without asking | `SET NOT NULL`, nothing asked. Fails if NULLs exist, and says how many. `--backfill` writes a slot to fill instead | +| Any of the above through `db update` | No placeholders | Same. `--convert` and `--backfill` are refused: `db update` has no file to fill | + +SQLite works the same way. Two SQLite gaps close on the way: +- `db update` can add a required field to a table with rows. Today it fails. +- A type change gets a slot for the conversion. Today SQLite just rebuilds the table. + +## Decisions to check + +1. **Temporary values without asking.** Without `--backfill`, `migration plan` writes fill values such as `''` or `0` into existing rows. The project spec already decided this. It is the biggest behaviour change in the slice. +2. **`alterColumnType` changes shape.** Its options become `{ table, column, type, using? }`. Today they expose internal names. Hand-written migrations that call it need the upgrade instructions. +3. **A type change answered with `delete` runs the database's cast** (decided by Will, 2026-10-12). Values the database can cast are kept; a value it cannot cast fails the migration, and the failure points to `--convert`. The statement is reported as `accept lossy type change of field "User.age"`, not as deleting the field or its values. + +## The two flags + +**`--convert `** +- Answers a type change that would lose data. +- `migration plan` only. +- Names the field by its new name. After `--rename User.age:User.years`, write `--convert User.years`. +- Needs the old contract, like `--rename`. + +**`--backfill `** +- Optional. Answers no question. +- `migration plan` only. +- The field must be required now, and either new on an existing table or optional before. + +**On `db update`,** both flags fail with `MIGRATION.STATEMENT_NEEDS_MIGRATION_FILE`, which points to `migration plan`. + +## What the planners write + +| Case | Postgres | SQLite | +| --- | --- | --- | +| `--convert` | `alterColumnType` with `using: placeholder(…)` | Table rebuild whose copy step has `placeholder(…)` for the column | +| `--backfill`, new field | Add nullable column, data transform with placeholders, `SET NOT NULL` | Add nullable column, data transform, tightening rebuild | +| `--backfill`, optional made required | Data transform with placeholders, `SET NOT NULL` | Data transform, tightening rebuild | +| New required field, no flag | Add with temporary default, drop the default | Rebuild that copies the temporary value into the new column | + +## How the pieces fit + +- **Questions know why data is lost.** Each data-loss entry says `drop` or `typeChange`. A `typeChange` question offers `convert` and `delete`; a `drop` question offers `rename` and `delete`. +- **A converted change stops being data loss.** It is classed `data`, because the user writes the conversion. So it is not asked about again. +- **A typed `convert` at the prompt re-plans,** the same way a typed rename does. +- **Placeholders appear only when a flag asks.** The four strategies that write them today run only for a field a flag names. +- **The temporary default becomes a real call in `migration.ts`.** Today it renders as an empty `rawSql(...)`, so re-running the file does not reproduce `ops.json`. +- **The NOT NULL failure names the NULLs (TML-3517).** It says how many rows hold NULL in which column, and suggests filling them, keeping the field optional, or `--backfill`. It no longer talks about schema drift. + +## Why one pull request + +Everything here serves one rule: a placeholder migration is written only when a flag asks for it. The reviewer reads that rule and its consequences. + +## Scope + +**In** +- Both flags on Postgres and SQLite; refused on `db update`. +- Ending automatic placeholders. +- The temporary default on both targets, as a real call. +- The `alterColumnType` reshape, with upgrade instructions. +- TML-3517. +- Docs: + - ADR 200: it says only a data transform can hold a placeholder. + - The Migration System doc: it says the Postgres planner never writes placeholders. + - Migration System § Statements, the error reference, the CLI README, `skills/prisma-8/references/migrations.md`. +- An app upgrade fragment for the behaviour changes. +- The project spec: "an existing table", not "a non-empty table". `migration plan` cannot see rows. + +**Out** +- `--convert` on a variant: slice 3b. +- MongoDB: slice 4b. +- A temporary value for a new unique, primary-key or foreign-key column. It stays refused, as on Postgres today. + +## Edge cases already known + +| Case | Handling | +| --- | --- | +| A temporary value in a unique column fails on the second row | Refused for new unique, primary-key and foreign-key columns | +| `--convert` after a rename in the same run | Uses the new name, which the planner already uses | +| A placeholder operation has no position until the file is re-run | The convert's reported positions are empty until then, as for every placeholder today | + +## Done when + +- The project's convert and backfill checks pass on Postgres and SQLite. +- Re-running every `migration.ts` that this slice's tests write reproduces its `ops.json`. + +## Implementation notes + +- `dataLoss` entries gain `loss: 'drop' | 'typeChange'`, set by each target's operation-subjects function. `dataLossQuestion` stops inferring it from whether the destination has the field. +- `ResolvedMigrationStatement` gains `kind: 'rename' | 'convert' | 'backfill'`. Every reader that assumes a rename switches on `kind` first, including the SQL `StatementPlanner`, which today plans any non-model statement as a column rename. +- `askPlanQuestions` re-plans with every planned statement, not only renames. +- Convert and backfill reach the strategies as destination coordinates through the strategy context. They do not go through `planStatements`, because they rename nothing. +- Strategies gated: Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`. +- `alterColumnType`'s after-check compares the column's `format_type` text, which `migration.ts` no longer carries. The facade derives it from `type` through the control stack's data types (any of a data type's texts, rendered as its `format_type` text; a type that names its own kind, such as an enum, from its quoted name), and the planner derives it the same way, so `ops.json` is unchanged by a re-run. A `type` the stack cannot read fails with a structured error. A test runs every Postgres data type through both and checks they agree. Decided 2026-10-09 by the orchestrator: comparing only the type OID would let a length-only change look already applied, so the runner would skip it. +- SQLite does not cast on a table rebuild: a value its new column cannot hold stays in its old storage class. So a direct rebuild of a retyped column (no statement, or `--delete`) checks, after the copy and inside the rebuild, that every value's storage class fits the new column, and fails naming the field with `--convert` advice otherwise. A `--convert` rebuild runs the user's expression unchecked. Decided 2026-10-12 by the orchestrator, applying Will's rule for `--delete` on type changes. +- `AlterColumnTypeClass` gains `data`. The `using` slot renders as code, imports `placeholder`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result. +- `recreateTable` gains a per-column copy expression, used by convert and by the SQLite temporary value. SQLite gets a fill-value map: integer `0`, real `0`, text `''`, blob `X''`, and what its boolean and date-time codecs store. +- `MigrationRunnerFailure` gains `fix`. The `db update`, `db init` and `migrate` error mappers prefer it. +- Grounding report (local, gitignored): `wip/grounding-3a.md`. diff --git a/projects/migration-statements/slices/engine-statement-prompt/spec.md b/projects/migration-statements/slices/engine-statement-prompt/spec.md index c2b105039d2d..32929d9ea75c 100644 --- a/projects/migration-statements/slices/engine-statement-prompt/spec.md +++ b/projects/migration-statements/slices/engine-statement-prompt/spec.md @@ -105,6 +105,6 @@ None. Will decided on 2026-10-07: consents carry a verb; the answer is a free-te ## References -- Project spec [`../../spec.md`](../../spec.md) decisions 2 and 6, requirement 12; [`../../plan.md`](../../plan.md) § Stretch goal. +- Project spec [`../../spec.md`](../../spec.md) decisions 2 and 6, requirement 12; [`../../plan.md`](../../plan.md) § Delivered: the interactive prompt. - prisma-cli: `packages/cli-engine/src/execution/prompts.ts`, `shared-flags.ts`, `clack-renderer.ts`, `src/context.ts`; `docs/product/cli-style-guide.md` § consent; ADR 0004 (engine version pinning). - `wip/slice-2-consent-options.md`: the option evaluation Will chose from. diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md b/projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md new file mode 100644 index 000000000000..d9a3de8aad32 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md @@ -0,0 +1,38 @@ +# Dispatch 1 — A `renameCollection` operation, end to end + +**Slice:** [`../spec.md`](../spec.md) § How it works, § Implementation notes · **Plan:** [`../plan.md`](../plan.md) dispatch 1 · **Builds on:** slice 2 (merged on `main`) + +## Outcome + +A hand-written MongoDB migration can call `renameCollection('profiles', 'users')`, and running it renames the collection with its documents, indexes and validator. The planner does not use it yet. You also report how variants are stored today. + +## What to build + +1. **The operation through the stack**, the way `createCollection` and `dropCollection` go: + - a `RenameCollectionCommand` and its visitor method in `packages/2-mongo-family/4-query/query-ast` (`ddl-commands.ts`, `ddl-visitors.ts`); + - its wire command in `packages/2-mongo-family/6-transport/mongo-lowering` and lowering in `packages/3-mongo-target/2-mongo-adapter`; + - a driver case in `packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts`; + - serializer and deserializer in the target's operation serializer; + - preview text in `packages/2-mongo-family/9-family/src/core/operation-preview.ts`; + - a `renameCollection` factory in `migration-factories.ts`, a `RenameCollectionCall` in `op-factory-call.ts` that renders `renameCollection('profiles', 'users')`, and the export from `src/exports/migration.ts`. +2. **Checks.** Before: the source collection exists and the target does not (`listCollections`). After: the target exists and the source does not. A re-run after success skips through the after-check. +3. **Class:** `widening`. +4. **Variant report (TML-2447).** Plan a contract with a base model and two variants that share one collection (`@@base`, `@@discriminator`) on current `main`, and report: which collections the planner creates, what the validator holds, and whether a variant's documents can be told apart by the discriminator value in the stored documents. Do not fix anything here; the orchestrator decides from the report. + +## Not in this dispatch + +Planning the rename (dispatch 3). Document rewrites (dispatch 2). Removing `renameStatements`. + +## Tests + +- Unit: factory, call rendering, serializer round trip, preview text. +- Against mongodb-memory-server (existing target-runner tests under `test/integration/test/mongo/target-runner/` show the setup): a collection with documents, a unique index and a closed validator is renamed, and all three are kept; re-running skips; renaming onto an existing collection fails at the before-check with nothing changed. +- `render-typescript` round trip: running the rendered `migration.ts` reproduces the operation in `ops.json`. + +## Halt conditions + +Stop and report if the driver's rename needs admin privileges that mongodb-memory-server does not give; or if a DDL command cannot carry a second collection name without changing the shared command shape. + +## Gate + +The plan's gate, plus the new target-runner test file. diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md b/projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md new file mode 100644 index 000000000000..f73944df2973 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md @@ -0,0 +1,36 @@ +# Dispatch 2 — `renameField` and `unsetField` rewrites, end to end + +**Slice:** [`../spec.md`](../spec.md) § How it works, § Decisions 3 and 5 · **Plan:** [`../plan.md`](../plan.md) dispatch 2 · **Builds on:** dispatch 1 + +## Outcome + +A hand-written MongoDB migration can call `renameField(...)` and `unsetField(...)`. Each runs one `updateMany` that skips validation, carries the class of its effect, and runs under `db update`'s classes. The planner does not use them yet. + +## What to build + +1. **Skipping validation.** The raw update command (`RawUpdateManyCommand`, `packages/2-mongo-family/4-query/query-ast/src/raw-commands.ts`) gains `bypassDocumentValidation`, through its wire form and the driver (`mongo-driver.ts` passes only `upsert` today). +2. **Two factories and calls.** + - `renameField(collection, from, to, { filter? })`: `$rename` on documents that hold `from` and match `filter`. Before: no matching document holds `to`. After: no matching document holds `from`. Class `widening`. + - `unsetField(collection, field, { filter? })`: `$unset` on documents that hold `field` and match `filter`. After: no matching document holds `field`. Class `destructive`. + - `filter` carries a variant's discriminator condition (`{ type: 'bug' }`); dispatch 3 supplies it. + - Calls render as `renameField('users', 'name', 'fullName')` and `unsetField(...)`, exported from the target's migration module, so a re-run of `migration.ts` reproduces `ops.json`. +3. **Telling rewrites from DDL by shape.** The runner (`mongo-runner.ts`) and the operation serializer recognise these by their shape (`run` against `execute`), not by `operationClass === 'data'`. `data` stays the class of user-written and scaffolded transforms. A `db update`-style policy without `data` accepts these two and still refuses a `data` transform. +4. **Storage name.** The family helper that names an operation's storage (`mongoStorageNameOf`, `packages/2-mongo-family/9-family/src/core/operation-storage-name.ts`) handles the new shape: `.`. + +5. **Preview text that mongosh accepts (review finding D1-1).** Every MongoDB preview line in `packages/2-mongo-family/9-family/src/core/operation-preview.ts` writes the collection as `db.getCollection("")`, with the name escaped as a string, so a name such as `my-coll` or one containing quotes is valid mongosh. This covers the existing lines (`drop`, `createIndex`, and the rest) and the new rename and rewrite lines. + +## Not in this dispatch + +Planning them (dispatch 3). Value object fields (slice 4b). + +## Tests + +Against mongodb-memory-server: a collection with a closed validator that requires `name` and a unique index on `name`. Drop the index, `renameField` to `fullName`, create the index on `fullName`: every document moved, the unique index builds, nothing failed validation. `renameField` when a document already holds `fullName` fails at the before-check with nothing changed. A re-run after success skips. `unsetField` removes the field from every matching document. A `filter` limits both to matching documents. Under a policy of `additive`, `widening`, `destructive`, both run and a `data` transform is refused. Unit: factories, rendering, serializer round trip, storage names. + +## Halt conditions + +Stop and report if skipping validation needs a server privilege mongodb-memory-server lacks; or if the serializer cannot tell the shapes apart without a new discriminating field in `ops.json` (that changes the file format and needs a decision). + +## Gate + +The plan's gate, plus the target-runner test files you add or change. diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md b/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md new file mode 100644 index 000000000000..12f33854b9ea --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md @@ -0,0 +1,36 @@ +# Dispatch 3 — The MongoDB planner carries out renames and deletes + +**Slice:** [`../spec.md`](../spec.md) § Decisions, § How it works · **Plan:** [`../plan.md`](../plan.md) dispatch 3 · **Builds on:** dispatches 1 and 2 + +## Outcome + +`migration plan` and `db update` on MongoDB rename collections and fields from `--rename`, and ask about a removed field, which `--delete` answers by removing it from every document. The temporary refusal (`renameStatements`, `keepDataByHand`) is gone. + +## What to build + +1. **Collection rename.** A model statement whose collection name changes renames the collection in the working copy of the origin schema before the diff (`mongo-planner.ts`, `planCalls`), and plans one `renameCollection`. Indexes and the validator then diff as usual. +2. **Field rename.** A field statement plans one `renameField`. For a variant's field, `filter` is `{ : }`, read from the base model (`discriminator.field`, `variants..value`; a variant's `fields` hold only its own fields). A statement whose stored names do not change plans nothing and reports applied with no operations. +3. **Field removal is data loss.** A field present in the origin model and absent from the destination, on a collection that is not dropped, plans one `unsetField` (with the variant filter for a variant's field) and reports a `field` subject in `dataLoss` (with `loss: 'drop'` once slice 3a's loss kinds are on `main`). Without an origin contract the subject is `{ kind: 'storage', name: '.' }`. The question offers `rename` and `delete`. +4. **Order.** Collection renames, collection creates, index drops, rewrites (renames and removals), index creates, validator changes, option changes, collection drops. Plans with no statements and no removed fields are unchanged; prove it with the existing planner tests passing untouched. +5. **Refusals** (`statementRejected`, reason named): renaming or removing `_id`; a field whose stored name contains `.` or starts with `$` (MongoDB reads it as a path or an operator; `@map` can produce one); renaming a model stored in a time-series collection or a view; a collection rename onto a name the working schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, starting with `system.`); a field stored on one side only; any statement when the control policy is not `managed`. +6. **`appliedStatements`** count only the rename or rewrite each statement accounts for. +7. **Delete the temporary refusal.** The `renameStatements` member in `control-migration-types.ts`; MongoDB's setting in `control-target.ts`; `keepDataByHand`, `keepTheData` and `NOT_IN_THIS_RELEASE` in `mongo-planner.ts`; `keepDataByHandFor` and its parameter in `plan-questions.ts`; the reads in `db-run.ts` and `migration-plan.ts`. Keep the unrelated `renameStatements()` helper in `statement-text.ts`. +8. **After slice 3a merges** (if it has by then): resolved statements have a `kind`. Carry out `kind: 'rename'`; refuse `convert` and `backfill` as 3a's MongoDB code does today. + +## Not in this dispatch + +Journeys, docs, ADR amendments and upgrade fragments (dispatch 4). + +## Tests + +- Planner unit tests for each item above, rewriting `mongo-planner.statements.test.ts` and `db-update-statements.test.ts` (both assert the old refusal), and the `keepDataByHand` cases in `mongo-planner.data-loss.test.ts`. +- CLI tests that used `keepDataByHand` or `renameStatements` (listed in `wip/grounding-4a.md` § 7 in the parent worktree). +- Against mongodb-memory-server, through the planner and runner: a model rename and a field rename with a unique index on the field and a closed validator; a field removal; a variant field rename where another variant stores a field with the same name, which keeps its value. + +## Halt conditions + +Stop and report if the working-schema rename cannot happen before the diff without changing the shared schema IR; or if removing a field from a variant cannot be told apart from removing it from the base in the diff. + +## Gate + +The plan's gate, plus `pnpm lint:framework-vocabulary`, `pnpm check:error-reference`, and the named files `test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts` and `mongo-db-update-consent.e2e.test.ts` (S4 of the first now succeeds instead of refusing). diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/04-journeys-docs-fragments.md b/projects/migration-statements/slices/mongo-renames/dispatches/04-journeys-docs-fragments.md new file mode 100644 index 000000000000..e51e046d2afe --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/dispatches/04-journeys-docs-fragments.md @@ -0,0 +1,33 @@ +# Dispatch 4 — Journeys, docs and upgrade fragments + +**Slice:** [`../spec.md`](../spec.md) § Done when, § Scope · **Plan:** [`../plan.md`](../plan.md) dispatch 4 · **Builds on:** dispatch 3 + +## Outcome + +The slice's done conditions hold through journeys on real MongoDB, and the docs, error reference, skill and upgrade fragments describe what MongoDB now does. + +## What to build + +1. **Journeys** (in `test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts` or a sibling file), on collections with documents, through `migration plan` then `migrate` and through `db update`: + - a model rename, a field rename with a unique index on the field, and a field removal with `--delete`, in one run; + - a variant's field rename in a collection where another variant stores a field with the same name; + - afterwards: documents under the new names; the removed field gone from every document; every document accepts an update; a further plan is empty; `db verify --schema-only` is clean; + - without `--delete`, both commands refuse, and the printed flags then succeed when run as printed; + - a second `db update` with the same statements fails on the first statement; + - `db update` with no snapshot: a removed field is named `.`, and `--delete .` works. +2. **Follow every printed advice to its end** (`drive/calibration/failure-modes.md` F46): each refusal, error and dry-run suggestion this slice adds or changes is run as printed in a journey or a probe, and the end state checked. +3. **Docs.** + - `docs/architecture docs/subsystems/7. Migration System.md` § Statements: MongoDB carries out renames and deletes; remove the `renameStatements` paragraph and the "MongoDB refuses every statement" limit; say how a field is named (contract key, the stored name on MongoDB) and that a variant's field rename touches only that variant's documents. + - `docs/architecture docs/subsystems/10. MongoDB Family.md`: the operation list gains `renameCollection`, `renameField`, `unsetField`; remove the stale "Known gaps" paragraph under § Polymorphic variants (TML-2447's gaps are fixed on `main`). + - `docs/reference/error-reference.md`: every code this slice adds or changes. + - The CLI README sections for `migration plan` and `db update`. + - `skills/prisma-8/references/migrations.md`: renames and deletes on MongoDB. +4. **Upgrade fragments** (`record-upgrade-instructions` skill): + - app: MongoDB renames work; removing a field now asks and `--delete` removes it from every document; + - extension: `TargetMigrationsCapability.renameStatements` and `keepDataByHand` are gone (shipped in 8.0.0-rc.17). + +The orchestrator writes the ADR 188 and ADR 264 amendments; do not edit those two files. + +## Gate + +The plan's gate, plus `pnpm check:upgrade-coverage --mode pr --prev $(git merge-base HEAD origin/main) --head HEAD`, `pnpm lint:docs`, the journey files, and every other file under `test/integration` or `test/e2e` that runs `migration plan` or `db update` against MongoDB, directly or through the helpers in `test/integration/test/utils/journey-test-helpers.ts` (F44). diff --git a/projects/migration-statements/slices/mongo-renames/plan.md b/projects/migration-statements/slices/mongo-renames/plan.md new file mode 100644 index 000000000000..113fae5dffcf --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/plan.md @@ -0,0 +1,36 @@ +# Slice plan — 4a, renames and deletes on MongoDB + +**Spec:** [`spec.md`](./spec.md) · **Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Branch:** `tml-3478-mongo-renames` + +One implementer and one reviewer, both on Opus, resumed across every dispatch. Briefs are in `dispatches/`. + +## Gate for every dispatch + +- `pnpm typecheck`; `pnpm --filter lint`; `pnpm --filter test`. +- `pnpm lint:deps` when imports change; `pnpm fixtures:check` when rendering or classes change; `pnpm check:error-reference` when error codes change; `pnpm lint:framework-vocabulary` when `packages/1-framework` changes; `pnpm lint:throws` and `pnpm lint:casts`. +- Integration and e2e: only named files, run as `pnpm test `. Never `pnpm test:integration`, `test:e2e` or `test:all`. +- The reviewer runs code against mongodb-memory-server, not only reads it (F34), and follows every printed advice (F46). + +## Dispatches + +| # | Outcome | Builds on | +| --- | --- | --- | +| 1 | A `renameCollection` operation exists end to end: factory, call, serializer, runner, preview text, with before and after checks. Also reports how variants are stored today (TML-2447) | slice 2 | +| 2 | `renameField` and `unsetField` rewrites exist end to end, skip validation, carry `widening` and `destructive`, and the runner and serializer tell them from DDL by shape | 1 | +| 3 | The planner carries out statements: collection rename on the working schema, field rename (filtered by discriminator for a variant), field removal as data loss, the operation order, the refusals. `renameStatements` and `keepDataByHand` are deleted | 2 | +| 4 | Journeys for the slice's done conditions through both commands; docs; app and extension upgrade fragments; every printed advice followed | 3 | + +**Hand-offs** +- 1 → 3: `renameCollection` can be planned and rendered. +- 2 → 3: `renameField` and `unsetField` can be planned and rendered, and run under `db update`'s classes. +- 3 → 4: both commands rename and delete on MongoDB. + +## Coordination with 3a + +3a runs at the same time on `tml-3477-convert-backfill` and adds a `kind` to resolved statements. Whichever merges second merges `main` and adapts. After 3a, the MongoDB planner handles `kind: 'rename'` and refuses `convert` and `backfill`. + +## Open items + +- ADR 188 and ADR 264 amendments: written by the orchestrator in dispatch 4's branch. +- Dispatch 1 found TML-2447's gaps fixed on `main`: variants share their base's collection, and the validator has a `oneOf` branch per variant pinned to its discriminator value. Variant field renames stay in 4a. Dispatch 4 removes the stale "Known gaps" paragraph from the MongoDB Family doc § Polymorphic variants. +- Both 3a and 4a add a framework type `MigrationDataLoss` (3a with an optional `label`, 4a with an optional `keepDataAdvice`). Whichever merges second combines them into one type. diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md new file mode 100644 index 000000000000..15a19ed225ef --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -0,0 +1,142 @@ +# Slice 4a — Renames and deletes on MongoDB + +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Branch:** `tml-3478-mongo-renames` · **Builds on:** slice 2 (merged) · **Project:** [`../../spec.md`](../../spec.md) + +## In one example + +A MongoDB user renames model `Profile` to `User`, renames field `name` to `fullName`, and removes field `nickname`. + +```text +$ prisma migration plan --name tidy --rename Profile:User --rename User.name:User.fullName --delete User.nickname +✔ Planned 6 operation(s) + Rename collection profiles to users widening + Drop index users.name_1 widening + Rename field name to fullName in users widening + Remove field nickname from users destructive + Create index users.fullName_1 additive + Update validator on users widening +``` + +`db update` with the same flags does the same against the live database. + +Without `--delete User.nickname`, both commands refuse, as they do on SQL: + +```text +✖ [CLI.CONSENT_REQUIRED] 1 subject needs a statement, and the session is not interactive. + why: Remove field "nickname" from collection "users" would lose the data of field "User.nickname". +→ Pass --rename 'User.nickname:' +→ Pass --delete User.nickname +``` + +## What changes for users + +| Change in the schema | Today | After this slice | +| --- | --- | --- | +| Model renamed, `--rename` given | Refused, with steps to run by hand in `mongosh` | Collection renamed; documents, indexes and validator kept | +| Field renamed, `--rename` given | Refused, with steps to run by hand | Every document's field renamed; indexes moved to the new name | +| Field removed | Validator updated, nothing asked. Every document that still holds the field then fails each update | Asked about. `--delete` removes the field from every document; `--rename` keeps its values | +| Model removed | Asked about; `--delete` drops the collection | Same, and `--rename` is offered too | + +## Decisions (settled with Will, 2026-10-09) + +1. **Removing a field is data loss.** It is asked about, and `--delete` removes it from every document. Today it is silent, and MongoDB's validator then rejects every update to a document that still holds the field. +2. **Fields are named by their contract key, as on SQL.** MongoDB's contract keys fields by their stored name, so `id @map("_id")` is `User._id` in a statement. Naming the schema field instead would need MongoDB's contract to keep a field-to-stored-name map, as SQL's does. That is outside this project. +3. **Rewrites are classed by what they do to data, as on SQL.** A field rename is `widening`, like a column rename. A field removal is `destructive`, like a column drop. The planner writes a `data` operation only as a placeholder scaffold. So `db update` runs these rewrites, as it runs SQL renames. ADR 188 changes one sentence: today it says every MongoDB data transform is `data`. +4. **Without the old contract, a removed field is named `.`,** for example `users.nickname`. SQL uses `..`, or `
.` on SQLite. +5. **A variant's field rename touches only that variant's documents.** The rewrite filters on the variant's discriminator value, so another variant in the same collection that stores a field with the same name keeps it. This differs from SQL single-table storage, where a rename renames the shared column for every row. +6. **Contracts without a validator get a temporary refusal** (Will, 2026-10-12). A MongoDB contract written in TypeScript has no validator today, so its field names are not in its storage hash, and a field rename or removal would give a migration whose start and end hashes are equal, which `db migrate` refuses. `migration plan` refuses such a change and says why; `db update` is unaffected. TML-3553, the next slice, gives TypeScript contracts the PSL validator and removes this refusal. + +## How it works + +**Collection rename** +- A new `renameCollection` operation. +- Before: the source exists and the target does not. After: the target exists. +- The rename happens before the diff, so MongoDB keeps the indexes and validator, and nothing is dropped and recreated. + +**Field rename** +- One `updateMany` with `$rename`, written as `renameField(...)` in `migration.ts`. +- Touches only documents that hold the old field. For a variant's field, only documents with that variant's discriminator value. +- Before: no document already holds the new name, which `$rename` would overwrite. After: no document holds the old name, so a re-run skips it. + +**Field removal** +- One `updateMany` with `$unset`, written as `unsetField(...)`. +- No unset when the whole collection is dropped. + +**Order in a plan** + +1. Collection renames +2. Collection creates +3. Index drops +4. Document rewrites (renames and removals) +5. Index creates +6. Validator changes +7. Option changes +8. Collection drops + +- Rewrites run after index drops, so a unique index on the old field doesn't fail once documents lose the field. +- They run before index creates, so a unique index on the new field sees the moved values. +- Rewrites skip validation. The old validator requires the old field, and the new one may require a field the same plan adds. +- Plans with no statements and no removed fields come out exactly as today. + +**Refused, with the reason named** +- Renaming a model stored in a time-series collection, or in a view. MongoDB cannot rename them. +- Renaming `_id`. +- Renaming a collection onto one that already exists. +- Renaming a field that is stored on only one side. +- Any statement when the model's control policy is not `managed`, as on SQL. + +**The temporary refusal goes.** MongoDB's `renameStatements: { refused: true, keepDataByHand }` setting, and all the code that reads it, is deleted. + +## Why one pull request + +It is one database learning two flags the SQL targets already support. It adds one new operation and one new rewrite. The collection rename touches six packages, but the change is the same in each and copies the existing collection operations. + +## Scope + +**In** +- Model rename, field rename (including a variant's field), field removal. +- Both commands. +- Deleting `renameStatements` and `keepDataByHand`. The unrelated `renameStatements()` helper in `statement-text.ts` stays. +- ADR amendments: + - ADR 188: planner-written rewrites carry the class of their effect. + - ADR 264: MongoDB now has a collection rename. +- Docs: Migration System § Statements, the error reference, the CLI README, `skills/prisma-8/references/migrations.md`. +- Upgrade fragments: + - App: renames work; removing a field now asks. + - Extension: the capability member is gone. It shipped in 8.0.0-rc.17. + +**Out** +- Renaming a field inside a value object. It needs statement support that slice 3b builds. Moved to 4b. +- `--convert`, `--backfill`, and the validator problem when a field becomes required: slice 4b. +- Batching rewrites on large collections. + +## Edge cases already known + +| Case | Handling | +| --- | --- | +| A unique index on the old field fails the rewrite once two documents lose it (slice 1 QA, F3) | Index drops run before the rewrite | +| A unique index on the new field fails if created before the rewrite | Index creates run after it | +| The old validator requires the old field and forbids the new one | The rewrite skips validation | +| Two variants in one collection store a field with the same name | The rewrite filters on the discriminator value | +| MongoDB migrations are not transactional | The after-check lets a re-run skip a finished rewrite | +| Skipping validation needs a privilege on hosted clusters | The failure names the privilege | +| TML-2447 (open): variants that share a collection get their own collection and an incomplete validator | The first dispatch checks how variants are stored today, before building on it | + +## Done when + +- On collections with documents, through `migration plan` then `migrate` and through `db update`: a model rename, a field rename with a unique index, and a field removal with `--delete`. Afterwards: + - the documents are under the new names; + - the removed field is gone from every document; + - every document accepts an update; + - a further plan is empty; + - `db verify --schema-only` is clean. +- `git grep -n "keepDataByHand"` finds nothing. + +## Implementation notes + +- `renameCollection` goes through the stack like the other collection operations: query AST command and visitor, wire command and lowering, driver, operation serializer, preview text, factory, call class. +- The raw update command gains `bypassDocumentValidation`, through its wire form and the driver. +- The runner and the operation serializer tell a rewrite from a DDL command by its shape (`run` against `execute`), not by `operationClass === 'data'`. +- Removed fields are reported as `field` subjects in `dataLoss`. +- Code to delete: the member in `control-migration-types.ts`; MongoDB's setting in `control-target.ts`; `keepDataByHand` and `keepTheData` in `mongo-planner.ts`; `keepDataByHandFor` and its parameter in `plan-questions.ts`; the reads in `db-run.ts` and `migration-plan.ts`; the paragraph in the Migration System doc. +- Grounding report (local, gitignored): `wip/grounding-4a.md`. diff --git a/projects/migration-statements/slices/mongo-ts-validator/investigation.md b/projects/migration-statements/slices/mongo-ts-validator/investigation.md new file mode 100644 index 000000000000..76f96f2cd4c8 --- /dev/null +++ b/projects/migration-statements/slices/mongo-ts-validator/investigation.md @@ -0,0 +1,100 @@ +# Grounding: `$jsonSchema` validators for TypeScript-authored MongoDB contracts + +Read-only investigation. Paths are repo-relative. No tests were run. + +## 1. The gap is real + +- PSL: the interpreter sets `coll['validator']` for every non-variant model's collection (`packages/2-mongo-family/2-authoring/contract-psl/src/interpreter.ts:1703-1733`), then passes it into `buildMongoStorage`, which computes the storage hash (`interpreter.ts:1737-1751`). +- TypeScript: `buildCollections` writes only `indexes`, `options` and `control` (`packages/2-mongo-family/2-authoring/contract-ts/src/contract-builder.ts:2395-2416`). `defineContract` hashes that body (`contract-builder.ts:2553-2606`). There is no validator anywhere in `contract-ts/src`. +- Committed emitted output confirms it: + - TS-authored `test/integration/test/mongo/fixtures/generated/contract.json`: `tasks` has only `kind`; `users` has `indexes`, `kind`, `options`. + - TS-authored `examples/bundle-size/src/mongo/generated/contract.json`: `notes` has only `kind`. + - PSL-authored `test/integration/test/mongo/temporal-presets/_fixture/generated/contract.json`: every collection has `kind` and `validator`. +- Side-by-side fixture `test/integration/test/authoring/side-by-side/mongo/` builds the same schema both ways. The test strips `validator` and `storageHash` before comparing (`test/integration/test/authoring/side-by-side-contracts.test.ts:273-292`, `306-327`). Everything else is equal, so the validator is the only storage difference. The stripping exists since commit `3d9d8bf1e9` (2026-04-10). +- The consequence for renames: a self-edge (`from === to`) migration is refused unless it runs a data transform (`packages/1-framework/3-tooling/migration/src/errors.ts:218-226`); `migration new` reports `MIGRATION.NO_CHANGES` (`packages/1-framework/3-tooling/cli/src/control-api/operations/migration-new.ts:189-190`). + +## 2. How PSL derives the validator + +- Code: `packages/2-mongo-family/2-authoring/contract-psl/src/derive-json-schema.ts` (196 lines). Two exports: `deriveJsonSchema` and `derivePolymorphicJsonSchema`. +- Inputs are all contract-level, not PSL-level: + - domain fields (`ContractField` from `@internal/contract/types`), including `nullable`, `many` (with `elementNullable`), scalar `codecId`, `valueSet`, and value-object references; + - `MongoTypeLookups` = codec lookup (`descriptorFor`) + data type lookup (`get`), defined in `packages/2-mongo-family/1-foundation/mongo-contract/src/mongo-data-type.ts:38-55`. `bsonTypesOfCodec` reads `bsonTypes` from the codec's data type; + - value objects (`ContractValueObject`), recursed into as nested closed objects; + - storage value sets (for the `enum` keyword); + - for polymorphism: base fields, discriminator field name, and per-variant `{ discriminatorValue, fields }`, producing `oneOf` branches. +- Behaviour: every object level is closed (`additionalProperties: false`); non-nullable fields go in `required`; an empty `bsonTypes` list (`Bson`) means "any value"; a codec with no Mongo data type returns `undefined` and the field is left out of `properties`. Level and action are fixed: `strict` / `error` (`derive-json-schema.ts:128-132`, `191-195`). +- Imports: only `@internal/contract/types` and `@internal/mongo-contract`. No PSL AST, no diagnostics. +- The per-collection loop (pick base models, build variant entries, choose plain or polymorphic) lives in the PSL interpreter (`interpreter.ts:1703-1733`). It reads only the resolved domain models (`base`, `discriminator`, `variants`, `fields`, `storage.collection`). It is also surface-neutral. +- Current callers: the PSL interpreter, its tests, and `packages/3-extensions/mongo/test/mongo.enum.e2e.test.ts:10` (through the `@internal/mongo-contract-psl` export). +- Where it can live: + - `@internal/mongo-contract` (`packages/2-mongo-family/1-foundation/mongo-contract`) is layer `foundation`, plane `shared` (`architecture.config.json:676-680`). It already owns `MongoValidator`, `bsonTypesOfCodec` and `buildMongoStorage`. Both authoring packages already depend on it. This is the right home. + - `9-family` is not possible: authoring (`shared` plane, lower layer) cannot import it. + - `contract-ts` is possible (`contract-psl` already depends on `@internal/mongo-contract-ts`), but it would make the TS builder the owner of a rule both surfaces share. + +## 3. Is the omission deliberate? + +- No ADR, doc or commit gives a reason. The original plan (`9d07bdb93f`, `projects/mongo-schema-migrations/plan.md`) said "Emitter: auto-derive `$jsonSchema` from model fields". The work landed as "PSL authoring + emitter" (`92d9841fef`, `9e1f160aeb`), and the TS builder was never wired. Closing the validators (`48f2f73a5e`, TML-2730) again touched only PSL. +- The gap is known and has been documented as behaviour since then: + - `skills/prisma-8/references/contract.md:313`: "A TypeScript Mongo contract has no collection validator and accepts these enums." + - Shipped upgrade instructions tell users to use a TS contract as a workaround: `upgrade-instructions/releases/8.0.0-rc.16-to-8.0.0-rc.17/sources/enum-members-hold-read-values/app/instructions.md:19` and `:78` ("author the enum ... in a TypeScript contract, which carries no collection validator"). Same wording in `skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md`. + - `upgrade-instructions/releases/8.0.0-rc.12-to-8.0.0-rc.13/sources/mongo-json-and-bson/app/instructions.md:39` and `:46` describe the TS case as having no validator. + - `test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts:84` switched to a PSL project because TS fixtures emit no validator (also `wip/beowulf-40-handbacks.txt:2117`). +- The Prisma 6 reader omits validators on purpose: `packages/2-mongo-family/2-authoring/contract-prisma6/README.md:9` ("Prisma 6 databases have none"). Its parity test strips them (`contract-prisma6/test/parity.test.ts:24-60`). +- No opt-out exists on either surface. Level and action are hard-coded. `validationLevel: 'off'` is not in the type (`mongo-contract/src/ir/mongo-validator.ts:3-4`). + +## 4. Blast radius + +TS-authored MongoDB contracts whose `contract.json` and storage hash would change: + +- Committed emitted fixtures (checked by `pnpm fixtures:check` / `emit:check`): + - `examples/bundle-size/src/mongo/generated/contract.{json,d.ts}` + - `test/integration/test/mongo/fixtures/generated/contract.{json,d.ts}` (emitted with `|| true` in `test/integration/package.json:7`) +- TS contracts emitted at test time (no committed JSON, but assertions may change): + - `test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-*.ts` (used by `cli-journeys/mongo-migration.e2e.test.ts`, `mongo-db-update-consent.e2e.test.ts`, `delete-statements-migration.mongo.e2e.test.ts`). `mongo-migration.e2e.test.ts:235-238` asserts every op is `additive`; the first plan will now also create each collection with its validator. + - `test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/control-policy/mongo/contract.ts` (used by `cli.control-policy.mongo.e2e.test.ts`). `tolerated`, `external` and `observed` collections gain an expected validator, so verify outcomes may change. + - `test/integration/test/fixtures/cli/cli-integration-test-app/fixtures/emit-command/contract.mongo.ts` (`cli.emit-command.test.ts`). + - `test/integration/test/contract-space-fixture-mongo/contract.ts`, `test/integration/test/mongo-runtime/query-builder.test.ts`, `test/integration/test/enum-members/enum-members.mongo-ts.test.ts`, `test/integration/test/authoring/mongo-pack-composition.test.ts`, `test/integration/test/ports/engines/queries/data_types/enum_type/enum_type.test.ts`, `packages/1-framework/3-tooling/cli/test/control-api/contract-space-seed-phase.mongo.test.ts`. +- Unit tests in `packages/2-mongo-family/2-authoring/contract-ts/test/`: + - `contract-builder.storage-hash-pin.test.ts` pins a hash; it changes. + - `contract-builder.dsl.test.ts:90-91, 307` assert collections equal `{ kind: 'mongo-collection' }`. + - Many tests use fake target packs with no codecs or data types (for example `storage-hash-pin.test.ts:13-20`). With today's derivation every field would be dropped and the validator would be a closed empty object. +- Side-by-side test: the validator strip can be removed, which is the parity check we want (`side-by-side-contracts.test.ts:273-327`). +- Prisma 6 reader: produces no validator by design and stays that way; its parity test keeps stripping. +- `contract infer`: Postgres only (`packages/3-targets/3-targets/postgres/src/core/psl-infer/`). No MongoDB infer exists. +- Examples: `mongo-demo`, `mongo-blog-leaderboard`, `retail-store` are PSL; they do not change. + +Three behaviours the TS builder would need that PSL already has: + +- **`_id` requirement.** PSL refuses a model without an `objectId` `_id` (`PSL_MONGO_ID_REQUIRED`, `interpreter.ts:1563-1580`). The TS builder has no such rule. A closed validator for a model without `_id` rejects every insert, because MongoDB adds `_id`. Several TS unit-test models have no `_id`. +- **Enums the validator cannot list.** PSL refuses enums whose codec stores a BSON type JSON cannot represent, or non-finite doubles (`packages/2-mongo-family/9-family/src/core/authoring-entity-types.ts:113, 151`). That check is in `9-family`, which the TS builder cannot import. Without it, a TS enum over `mongo/int64@1` would get a validator that refuses every write, which is exactly the bug rc.17 fixed for PSL. +- **Complete lookups.** The TS builder builds its codec lookup from the target only (`contract-builder.ts:2563`). PSL gets codec and data type lookups from the full control stack (`contract-psl/src/provider.ts:45-46`). The target pack carries both codecs and `dataTypes` (`packages/3-mongo-target/1-mongo-target/src/core/descriptor-meta-runtime.ts:22-27`); the adapter and family contribute none. Extension packs must be included, or an extension-typed field is dropped and then rejected by `additionalProperties: false`. + +## 5. What changes for an existing user of a TS contract + +- After upgrading and re-emitting, the storage hash changes. The runtime marker check warns once per runtime (`verifyMarker: 'onFirstUse'`). +- `db verify` fails: a validator the contract declares but the database lacks is a `fail` issue (`packages/2-mongo-family/9-family/src/core/schema-diff.ts:198-212`), subject to the collection's control policy. +- `migration plan` / `db update` plan one `collMod` per existing collection, id `validator..add`, class `widening` (`packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts:439-460`; ADR 195 note dated 2026-10-07). `db init` refuses when a collection holds data and needs a validator, and points at `db update` (`test/integration/test/cli-journeys/mongo-db-update-consent.e2e.test.ts:178-204`). +- Existing documents: MongoDB does not check existing documents when `collMod` sets a validator. With `validationLevel: 'strict'` and `validationAction: 'error'`, every later insert, and every update to any document, must produce a valid document. A stored document that does not match (a missing required field, a wrong BSON type, or any extra field, because the schema is closed) can no longer be updated unless the update makes it valid. Writes from other applications that add fields outside the contract start failing. This is the main user risk, and it is larger for TS users, who have run with an open schema until now. +- Precedent for upgrade instructions: + - `skills/upgrade/prisma-next-upgrade/upgrades/0.11-to-0.12/instructions.md`, change `re-emit-closed-mongo-contracts`: re-emit, then apply the validator migration with `db update`. + - ADR 258 (`docs/architecture docs/adrs/ADR 258 - List cardinality has independent container and element nullability.md:104`): re-emit, then apply a new validator migration; runtime deployment alone does not change stored validators. + - Current location for new instructions: `upgrade-instructions/pending//app/instructions.md` (see `upgrade-instructions/pending/`). + +## 6. Size + +One PR is possible but heavy. It would contain: moving the derivation and the per-collection loop to `@internal/mongo-contract`; moving the enum-listability check down from `9-family`; adding an `_id` rule to the TS builder; building full lookups in the TS builder; regenerating two committed fixtures; updating TS unit tests and several e2e assertions; removing the side-by-side strip; upgrade instructions; and correcting `skills/prisma-8/references/contract.md:313` plus the advice in shipped rc.17 instructions (a new pending note, since shipped notes do not change). The e2e changes (control policy, mongo migration journeys) are the least predictable part. + +A clean split is two PRs: + +1. Move `derive-json-schema.ts`, the per-collection loop and the enum-listability check to `@internal/mongo-contract`. PSL calls them. No output change, so no fixture change. +2. The TS builder calls them, adds the `_id` rule, builds lookups from target plus extensions, regenerates fixtures, updates tests, and adds upgrade instructions. + +## Design questions and recommendations + +1. **Where does the derivation live?** Recommend `@internal/mongo-contract`, as one function taking domain models, value objects, value sets and `MongoTypeLookups`, returning validators per collection. Both surfaces call it. +2. **Should the TS builder require an `objectId` `_id`?** Recommend yes, with the same rule as PSL, raised as a contract error. The alternative (always allow `_id` in the schema) would make the two surfaces differ again. +3. **What happens when a field's codec has no Mongo data type?** Today the field is silently dropped, then rejected by the closed schema. Recommend making it an error in the shared function. This also forces TS unit tests with fake packs to use the real target pack, or a pack with data types. +4. **Enums the validator cannot list.** Recommend moving the check to the shared package and applying it on both surfaces. Write a pending upgrade note that withdraws the rc.17 "use a TypeScript contract" workaround. +5. **Opt-out (`validator: false` or `validationLevel: 'moderate'`)?** Recommend no opt-out in this change. Neither surface has one. Adding one to TS only would recreate the asymmetry. If users need it, add it to both surfaces later. +6. **Prisma 6 reader?** Recommend leaving it without validators. Its README states the reason, and it describes an existing Prisma 6 database. +7. **One PR or two?** Recommend two: a no-output-change move first, then the TS behaviour change with fixtures and upgrade instructions. diff --git a/projects/migration-statements/spec.md b/projects/migration-statements/spec.md index 746c6e454693..6603c7b15683 100644 --- a/projects/migration-statements/spec.md +++ b/projects/migration-statements/spec.md @@ -45,7 +45,7 @@ $ prisma migration plan --name age-to-int --convert User.age The reasoning and the alternatives rejected are in [`design-notes.md`](./design-notes.md). 1. **Data loss is refused by default, in both commands.** An operation is destructive only when it loses data: dropping a model or field, deleting a value rows still carry, or changing a field's type in a way the database cannot keep. Drops of indexes, constraints, checks, policies and defaults lose nothing and are widening. `db update` already refuses through its consent prompt; `migration plan` gains the same refusal for every plan, not only auto-baselines. -2. **Intent is stated on the command line, up front.** There is no prompt that guesses whether a drop is a rename, and no detection of candidate pairs. The refusal is where the user learns what to type, and it prints each statement ready to paste. The eventual direction (see [`plan.md`](./plan.md) § Stretch goal and requirement 12 below) is that when a human runs the command in a terminal, the refusal becomes an interactive prompt that asks the user to state what each destructive operation is; the answers are the same statements. "No prompt" was the initial design stance, not a rule; the rule is that nothing is guessed. +2. **Intent is stated on the command line, up front.** There is no prompt that guesses whether a drop is a rename, and no detection of candidate pairs. The refusal is where the user learns what to type, and it prints each statement ready to paste. The eventual direction (see [`plan.md`](./plan.md) § Delivered: the interactive prompt and requirement 12 below) is that when a human runs the command in a terminal, the refusal becomes an interactive prompt that asks the user to state what each destructive operation is; the answers are the same statements. "No prompt" was the initial design stance, not a rule; the rule is that nothing is guessed. 3. **Four verbs, four nouns.** The verbs are `rename`, `delete`, `convert` and `backfill`. The nouns are namespace, model, field and value, addressed by contract coordinates such as `User`, `User.name`, `Status.ARCHIVED`, `Bug` and `Address.street`, with the namespace prefixed when the contract has more than one. Rename and delete apply to all four nouns; a model rename across namespaces is a move. Convert applies to fields and to variants whose discriminator value changed. Backfill applies to fields. 4. **Statements are intent, not hints.** A statement that cannot be resolved against the origin and destination contracts, or cannot be applied, is an error and nothing is planned. There is no "spent" or "ignored" statement. A statement that resolves and applies but changes nothing in storage, such as renaming a value object or a relation field, is reported as applied with no operations. 5. **Everything is in the application domain.** The command line never carries a table, column, collection, SQL expression or literal value. The framework resolves coordinates against the two contracts; the family turns the resolved statement into storage operations. @@ -71,7 +71,7 @@ The reasoning and the alternatives rejected are in [`design-notes.md`](./design- - **Planner inputs.** Every planner takes a destination contract and an origin schema: derived from a contract snapshot for `migration plan`, introspected for `db update` and `db init`, and one differ runs over two schema IRs ([ADR 235](../../docs/architecture%20docs/adrs/ADR%20235%20-%20The%20schema%20differ%20walks%20two%20derived%20schema%20IRs.md)). `migration plan` also hands the SQL planner the origin contract as `fromContract` (`packages/1-framework/3-tooling/cli/src/control-api/operations/plan-resolution.ts`); the `db update` path passes `fromContract: null` (`packages/1-framework/3-tooling/migration/src/aggregate/strategies/plan-from-diff.ts`). Statements become the third input, and `db update` learns to supply the origin contract. - **Where the origin contract comes from for `db update`.** The database marker stores the contract hash the database was last updated to ([ADR 021](../../docs/architecture%20docs/adrs/ADR%20021%20-%20Contract%20Marker%20Storage.md)), and `db update` advances the `db` ref to the same hash ([ADR 218](../../docs/architecture%20docs/adrs/ADR%20218%20-%20Refs%20with%20paired%20contract%20snapshots%20and%20universal%20graph-node%20invariant.md)). The content-addressed snapshot store ([ADR 240](../../docs/architecture%20docs/adrs/ADR%20240%20-%20Contract%20snapshots%20live%20in%20a%20content-addressed%20store.md)) holds the contract for that hash when the project has it. - **Consent today.** `db update` pre-plans and refuses with `MIGRATION.DESTRUCTIVE_CHANGES` until the user types the database name or passes the engine's `--confirm ` token (`packages/1-framework/3-tooling/cli/src/orm/db/consent.ts`, `control-api/operations/db-update.ts`). `migration plan` applies the same refusal to auto-baselines only, with the project directory as the token (`control-api/operations/migration-plan.ts`). The CLI README's `db update` section still describes a `--yes` flag that no longer exists. -- **The placeholder flow.** `migration plan` already scaffolds data transforms with `placeholder()` slots for a required column on a non-empty table, a nullable-to-required change and an unsafe type change, writes `ops.json` as `[]` with `pendingPlaceholders`, and the user fills the slots and re-runs `migration.ts` to re-emit ([ADR 200](../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md), [ADR 196](../../docs/architecture%20docs/adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md), `packages/3-targets/3-targets/postgres/src/core/migrations/planner-strategies.ts`). Under `db update`, whose policy excludes the `data` class, those strategies stand down and direct DDL is emitted. The Postgres `alterColumnType` operation already has a `using` option the planner never fills. +- **The placeholder flow.** `migration plan` already scaffolds data transforms with `placeholder()` slots on Postgres for a required column on an existing table, a nullable-to-required change and an unsafe type change (SQLite only for nullable-to-required), writes `ops.json` as `[]` with `pendingPlaceholders`, and the user fills the slots and re-runs `migration.ts` to re-emit ([ADR 200](../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md), [ADR 196](../../docs/architecture%20docs/adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md), `packages/3-targets/3-targets/postgres/src/core/migrations/planner-strategies.ts`). Under `db update`, whose policy excludes the `data` class, those strategies stand down and direct DDL is emitted. The Postgres `alterColumnType` operation already has a `using` option the planner never fills. - **Rename substrate.** `renameTable` exists on Postgres and SQLite from prisma/orm#30331. The shelved prisma/orm#30570 refactored it onto a working schema shared by the planner and the hand-written facade, with companion constraint names taken from the destination contract and foreign keys paired by their own and referenced columns, and reclassified non-data drops as widening. That code is the starting point; the contract `hints` section it also added is not. - **Operation classes.** Additive, widening, destructive and data (`packages/2-sql/9-family/src/core/migrations/types.ts`). Every Postgres type change is destructive today except a short list of safe widenings; Mongo classifies by validator comparison (`packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts`). - **Coordinates.** A contract entity is addressed by plane, namespace, kind and name ([ADR 221](../../docs/architecture%20docs/adrs/ADR%20221%20-%20Contract%20IR%20two%20planes%20with%20uniform%20entity%20coordinate%20and%20pack-contributed%20entity%20kinds.md), [ADR 224](../../docs/architecture%20docs/adrs/ADR%20224%20-%20Namespace%20concretions%20address%20entities%20by%20coordinate.md)); the contract source declares namespaces as `namespace { ... }` blocks; the query surface resolves a bare name through the sole namespace ([ADR 223](../../docs/architecture%20docs/adrs/ADR%20223%20-%20Target-owned%20default%20namespace.md)). Variants and discriminators are [ADR 173](../../docs/architecture%20docs/adrs/ADR%20173%20-%20Polymorphism%20via%20discriminator%20and%20variants.md); value objects are [ADR 178](../../docs/architecture%20docs/adrs/ADR%20178%20-%20Value%20objects%20in%20the%20contract.md), stored as one JSONB column on SQL and a subdocument on Mongo. @@ -91,7 +91,7 @@ The reasoning and the alternatives rejected are in [`design-notes.md`](./design- 9. **Scaffolds reuse the placeholder flow.** `--convert` and `--backfill` produce the existing placeholder-bearing migration, with the placeholder in the operation that consumes the value. The pending state, the empty `ops.json` and the re-emit path are unchanged. 10. **Destructive means data loss.** Drops of indexes, unique and foreign-key constraints, checks, row-level-security policies, defaults and native enum types, and disabling row-level security, are widening on every target. A type change the family knows to be widening needs no statement. 11. **Framework stays family-agnostic.** The statement grammar, parser and resolver carry no family or target vocabulary, and each family documents what every verb means for its storage. -12. **Built for the interactive prompt to drop in.** The refusal is a structured list of destructive operations, each with its entity in domain coordinates and the statements that would resolve it; the error text is rendered from that list. Statement strings are parsed, resolved and applied through one entry point that does not know whether they came from the command line or from an answer typed at a prompt, so statements can be added after the command has started. See [`plan.md`](./plan.md) § Stretch goal. +12. **Built for the interactive prompt to drop in.** The refusal is a structured list of destructive operations, each with its entity in domain coordinates and the statements that would resolve it; the error text is rendered from that list. Statement strings are parsed, resolved and applied through one entry point that does not know whether they came from the command line or from an answer typed at a prompt, so statements can be added after the command has started. See [`plan.md`](./plan.md) § Delivered: the interactive prompt. ## Transitional-shape constraints @@ -121,7 +121,7 @@ In addition to the team DoD floor in [`drive/calibration/dod.md`](../../drive/ca - The same change through `db update` with the same statements on both targets produces the same database state and no consent prompt. Running the same `db update` again with the same statements fails on the first statement. - The same change with `--delete` omitted makes both commands refuse, and the refusal text contains the exact statement that then succeeds. - A type change through `migration plan --convert` writes a migration whose only placeholder is in the Postgres `using` slot; filling it and re-running the file produces an applicable migration that converts the rows. The same change without a statement is refused by both commands, and `--delete` on the field plans the direct alter. -- A new required field on a non-empty table through `migration plan --backfill` writes the scaffolded backfill transform; without the statement both commands plan the temporary-default recipe. +- A new required field on an existing table through `migration plan --backfill` writes the scaffolded backfill transform; without the statement both commands plan the temporary-default recipe. - On Mongo, the rename journey above with a collection and a document field, through both commands, and a value object field rename that rewrites subdocuments. - A `db update` against a database whose marker hash has no local snapshot fails every rename and convert statement with one error naming the hash, and succeeds with `--delete` alone. - `--confirm` is gone from both commands, the CLI README describes the statements and the consent model, and the upgrade fragments for the refusal and the consent change are validated by execution.