TML-3474: Plan the rest of the data-loss statements work, and specify --convert, --backfill and MongoDB renames

projects/migration-statements/deferred.md

@@ -65,3 +65,50 @@ Adding a required field to a populated collection, or making an optional field r
 
 `CLI.CONSENT_REQUIRED` lists one `user-choice` next action per flag form ("Run the command again with --delete Legacy", another for the rename form), not grouped by subject and not a complete runnable command, and its summary says "N subjects need a statement", which is project vocabulary. Grouping the forms per subject and printing a full command would need the engine to know the invocation; raise it on prisma/prisma-cli once the statement prompt has shipped.
 
+## SQL variants stored in their own tables
+
+**Found:** slice 4a design (2026-10-09). Will: SQL will need to handle multi-table storage of variants in the future.
+
+On MongoDB a variant's field rename touches only that variant's documents. On SQL today a variant's field is a column in the shared table, and a rename applies to every row. When SQL supports variants stored in their own tables, a variant's field statements must act on that variant's table only.
+
+
+## MongoDB `@map` accepts field names MongoDB cannot validate
+
+**Found:** slice 4a review of dispatch 3 (2026-10-09). For the MongoDB authoring owners, not this project.
+
+`@map("o.d")` or `@map("$x")` is accepted in a MongoDB schema. MongoDB reads such a name as a path or an operator, and an insert of `{ "o.d": ... }` already fails the validator Prisma writes. Slice 4a skips such fields when planning statements. The fix belongs in authoring: reject the name with a diagnostic. When it lands, remove the framework resolver's fallback that reads `User.o.d` as field `o.d` (added in slice 4a), and the planner's silent skip of such fields.
+
+## A repeated run reports outdated statements one at a time
+
+**Found:** slice 4a manual QA (2026-10-09), F3. Applies to every family.
+
+Running the same `db update` again after it applied fails on the first outdated `--rename` or `--delete`, so a user with three outdated flags needs three runs to learn about all of them. The resolver could report every statement that no longer resolves in one error.
+
+## A collection name MongoDB rejects reaches the runner
+
+**Found:** slice 4a manual QA (2026-10-09), F9. For the MongoDB authoring owners.
+
+`@@map("system.x")` is accepted in a MongoDB schema. A rename onto it is refused with a reason, but a plan that creates or drops such a collection fails at apply with `CLI.UNEXPECTED`. Authoring should reject collection names MongoDB rejects.
+
+## `db migrate` of a migration with an unfilled placeholder gives advice that does not mention the slot
+
+**Found:** slice 3a review (2026-10-12). Older than this project's slices.
+
+A migration written with a placeholder has `ops.json` as `[]`. Running `db migrate` before filling the slot applies nothing and then fails at the destination check with general advice. It should say that the migration still has an unfilled slot, name the slot, and say to fill it and run `node migration.ts`.
+
+## `db migrate` applies an `ops.json` older than its `migration.ts`
+
+**Found:** slice 4a review (2026-10-12). Older than this project.
+
+The Migration System doc lists emit-drift detection (re-emit `migration.ts` in memory and compare with `ops.json` before applying) as architecturally required and not yet implemented. Without it, a user who edits `migration.ts` and forgets to run it applies the old operations; in slice 4a's review that turned a wrong advice text into a dropped collection. Worth a ticket when this project closes, if none exists.
+
+## A `--convert` expression can leave values the field's type cannot hold
+
+**Found:** slice 3a review (2026-10-12).
+
+A SQLite `--convert` rebuild runs the user's expression without the storage-class check the direct rebuild gets, so `trim("n")` on a field changed to `Int` leaves `'abc'` as text, and the ORM returns `"abc"` for an `Int` field. Postgres refuses such a value itself. Consider running the same check after a converted copy.
+
+## `migration status --json` prints `{bin}` unreplaced
+
+**Found:** slice 3a review (2026-10-12). Outside this project. The summary text in the JSON output contains a raw `{bin}` placeholder.
+

projects/migration-statements/plan.md

@@ -1,82 +1,106 @@
 # Project plan — Migration statements
 
-**Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474) · **Shaping PR:** prisma/orm#30604 · **Working branch:** `tml-3474-migration-statements`
+**Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474)
 
-## Summary
+## At a glance
 
-Four slices. The first three stack: the statement surface with renames, then the refusal and consent model, then the scaffolded verbs and the remaining nouns. The fourth, Mongo, builds on the second and runs in parallel with the third. Each slice is one PR, stacked on the working branch until the shaping PR merges and then retargeted to `main`.
+| Slice | What it delivers | Builds on | State |
+| --- | --- | --- | --- |
+| 1 | `--rename` for models and fields, Postgres and SQLite | — | Merged, prisma/orm#30638 |
+| 2 | Data loss refused until answered; `--delete`, `--allow`; the interactive prompt | 1 | Merged, prisma/orm#30648 |
+| 3a | `--convert` and `--backfill`, Postgres and SQLite | 2 | Spec: [`slices/convert-backfill/`](./slices/convert-backfill/spec.md) |
+| 4a | Renames and deletes on MongoDB | 2 | Spec: [`slices/mongo-renames/`](./slices/mongo-renames/spec.md) |
+| 4p | MongoDB contracts written in TypeScript get the same validator as PSL; removes 4a's temporary refusal | 4a | Next after 4a |
+| 3b | The remaining SQL renames and deletes | 3a | Not started |
+| 4b | `--convert`, `--backfill` and value object renames on MongoDB | 3b, 4a | Not started |
 
-## Slices
+```text
+1 → 2 ─┬─ 3a ── 3b ─┐
+       └─ 4a ───────┴─ 4b → close-out
+```
 
-### Slice 1 — Statements on the command line, and renames of models and fields on Postgres and SQLite
+- 3a and 4a run in parallel now.
+- Each slice is one pull request against `main`.
+- Slices 3 and 4 were split on 2026-10-08. Each was too large to review as one pull request.
 
-**Linear:** [TML-3475](https://linear.app/prisma-company/issue/TML-3475) · **Folder:** `slices/renames/`
+## Slices still to build
 
-**Outcome.** Both commands accept `--rename old:new` for models and fields, with a model rename across namespaces resolved but refused until slice 3. The framework parses each statement, resolves it against the origin and destination contracts, and applies it in order to a working copy of the origin. The SQL planners emit the table or column rename and every companion rename, with names from the destination contract. `db update` resolves its origin contract from the marker hash through the snapshot store and fails every rename when it cannot. An unusable statement is an error. Non-data drops are widening. The written migration is what a user could write by hand.
+### 3a — `--convert` and `--backfill` on Postgres and SQLite
 
-**Builds on.** The planner substrate from prisma/orm#30570, without its contract section.
+**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Spec:** [`slices/convert-backfill/spec.md`](./slices/convert-backfill/spec.md)
 
-**Hands to.** A statement type the framework owns, parsed and resolved, delivered to every family planner as resolved entities; the working-copy mechanism statements apply to; the origin-contract resolution for `db update`; a column rename operation on both SQL targets.
+- A lossy type change is answered with `--convert` (fill in the conversion) or `--delete`.
+- `--backfill` writes a slot to fill existing rows of a field that became required.
+- `migration plan` stops writing placeholders nobody asked for.
+- A new required field gets a temporary value on both targets. SQLite lacks this today.
+- The `SET NOT NULL` failure names the NULL rows (TML-3517).
 
-### Slice 2 — Both commands refuse data loss by default, and `--delete` is the per-operation consent
+**Hands to 3b and 4b:** the two flags and how their questions and placeholders work.
 
-**Linear:** [TML-3476](https://linear.app/prisma-company/issue/TML-3476) · **Folder:** `slices/refusal/`
+### 4a — Renames and deletes on MongoDB
 
-**Outcome.** `migration plan` refuses any plan that loses data, with the error `db update` uses. The refusal lists each destructive operation with the statements that resolve it. `--delete` consents to one operation, for namespaces, models and fields, and replaces `--confirm` on both commands. The terminal consent question asks per operation. Upgrade fragments record both changes; the CLI README describes statements and consent.
+**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Spec:** [`slices/mongo-renames/spec.md`](./slices/mongo-renames/spec.md)
 
-**Builds on.** Slice 1. Two things slice 1 leaves for it: the missing-origin check in the CLI statement resolver runs over the whole input and must move inside the per-statement loop so that `--delete` works without an origin contract; and the `@@map`-only rename gap in `deferred.md` must be decided before the refusal text is written. Decided 2026-10-07 on prisma/orm#30638: dropping a row-level-security policy and disabling row-level security stay `widening` (they lose no data), and slice 2 adds a separate consent question before `db update` applies an operation that widens who can read or write rows, answered per operation like the data-loss consent and refused in a non-interactive run unless consented.
+- A model rename renames the collection. A field rename rewrites the documents.
+- Removing a field is asked about. `--delete` removes it from every document.
+- Both commands, including `db update`.
+- The temporary setting that makes MongoDB refuse renames is deleted.
 
-**Design settled 2026-10-07 (Will).** Consent is a statement the user answers with, not a token copied back: the CLI engine gains `ctx.prompt.statement`/`statements`, answered on the command line by verb flags (`--delete Legacy`, `--rename Legacy:Archive`, `--allow User`) and interactively by typing the statement when a human runs the command. The engine change is its own slice, built in prisma/prisma-cli first: [`slices/engine-statement-prompt/spec.md`](./slices/engine-statement-prompt/spec.md). Slice 2's spec is [`slices/refusal/spec.md`](./slices/refusal/spec.md). The `@@map`-only gap gets no statement (see `deferred.md`). Operations that lose no data (`SET NOT NULL`, safe type widenings, MongoDB validator and index changes) stop being destructive.
+**Hands to 4b:** a MongoDB planner that takes statements.
 
-**Hands to.** The refusal shape every later verb hooks its statements into; the per-operation consent model; `--confirm` no longer consents to data loss.
+### 4p — TypeScript MongoDB contracts get the PSL validator
 
-### Slice 3 — Convert and backfill scaffold the placeholder migration, and the remaining nouns
+**Linear:** [TML-3553](https://linear.app/prisma-company/issue/TML-3553) · **Folder:** [`slices/mongo-ts-validator/`](./slices/mongo-ts-validator/investigation.md)
 
-**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/convert-backfill/`
+Decided by Will on 2026-10-12: contracts written in either form must behave the same, and this slice comes right after 4a, before 3b and 4b. Two pull requests:
 
-**Outcome.** `--convert` scaffolds the type change with the placeholder in the slot that carries the conversion, and `--backfill` the backfill transform; both refused on `db update`; the scaffolding stops being automatic. `--rename` on enum values, namespaces and value object fields, a model move across namespaces (`alter table set schema` on Postgres, deferred from slice 1 on 2026-10-06 because no operation existed for it), and `--convert` on a variant, plan the row updates, the JSON rewrites and the schema rename. `--delete` on an enum value nulls where nullable, else refuses.
+1. The validator derivation moves into `@internal/mongo-contract`. No output changes.
+2. The TypeScript builder calls it. 4a's temporary refusal goes. An upgrade fragment covers the closed validator TypeScript users gain.
 
-**Builds on.** Slice 2.
+**Why now:** without a validator, a TypeScript contract's field names are not in its storage hash, so a MongoDB field rename on it gives a migration `db migrate` refuses. 4a refuses that case for now.
 
-**Hands to.** Project close-out for the SQL targets.
+### 3b — The remaining renames and deletes on Postgres and SQLite
 
-### Slice 4 — The same statements on MongoDB
+**Linear:** [TML-3537](https://linear.app/prisma-company/issue/TML-3537) · **Folder:** `slices/remaining-nouns/`
 
-**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo/`
+- `--rename` for enum values, namespaces and fields inside value objects. This slice builds the statement support for value objects, which 4b reuses.
+- Moving a model to another namespace (`alter table set schema` on Postgres).
+- `--convert` on a variant whose discriminator value changed.
+- `--delete` on an enum value: sets it to NULL where the field is optional, otherwise refuses.
+- `--delete` on a variant stored in its base's table: deletes its rows and drops its columns. The refusal says rows are deleted.
+- `--delete <namespace>` answers every question for the models in it.
 
-**Outcome.** The Mongo planner takes the same resolved statements: collection rename, document rewrites for field and value object field renames, drops and unsets for deletes, a data transform scaffold for convert, under the slice 2 refusal and consent model. No family vocabulary enters the framework.
+**Risk to settle in its spec:** a value object can contain itself (ADR 178's `NavItem.children`). Renaming a field inside it means rewriting to any depth, which a fixed SQL JSON expression or MongoDB update cannot do.
 
-**Builds on.** Slice 2. Slice 2 left a temporary member for it: `TargetMigrationsCapability.refusesRenameStatements`, set by MongoDB so a data-loss question offers no `rename` there. Slice 4 deletes the member, its one CLI read and its doc mention when the Mongo planner carries out renames.
+**Hands to:** close-out for Postgres and SQLite.
 
-**Hands to.** Project close-out for Mongo.
+### 4b — `--convert`, `--backfill` and value object renames on MongoDB
 
-## Stretch goal — interactive statements when a human runs the command
+**Linear:** [TML-3538](https://linear.app/prisma-company/issue/TML-3538) · **Folder:** `slices/mongo-convert-backfill/`
 
-Recorded 2026-10-06 at the operator's request. This is the eventual direction, and the slices build with it in mind so that it drops in on top of their mechanisms; it is not a slice of its own, and it is built once the mechanisms it needs exist (after slice 2 at the earliest).
+- The same two flags as 3a, writing a data transform with a placeholder.
+- Making a field required on a collection with documents is no longer silent. It is answered with `--backfill`, or the command says the existing documents need the field.
+- Renaming a field inside a value object rewrites the embedded documents, including lists, dictionaries and unions.
+- Removing a variant model is data loss: today its documents stay and fail every update. `--delete <Variant>` deletes the documents with that discriminator value, as 3b does for SQL rows.
 
-When `migration plan` or `db update` detects a destructive operation and no statement covers it, the command today refuses and prints the statements that would resolve it. The stretch goal: when the command is run by a human in a terminal (stdin and stdout are a TTY, and no `--json` or similar non-interactive flag is set), the refusal becomes an interactive prompt that asks, per destructive operation, what the user means: rename it to a name they type, or delete it. Each answer is exactly the statement the refusal would have printed, applied in order, and the plan then proceeds as if the statements had been given on the command line. In a script, an agent, or CI (no TTY) the behaviour stays as it is: refuse and print the statements.
+**Hands to:** close-out for MongoDB.
 
-Rules that carry over unchanged: the prompt never guesses or proposes a rename candidate; it asks. The statements are not recorded anywhere; the migration file or the database is the record. The consent model of slice 2 (`--delete` per operation) is what the "delete" answer maps to.
+### Alongside — a syntax for naming storage objects
 
-What the slices do now so this drops in later (see the spec's cross-cutting requirement 12):
+There is no statement for a model that keeps its name but changes its table (`@@map`). Will asked on 2026-10-07 for a critical discussion and a look at how established tools name tables on the command line before any syntax is chosen. `--rename table/users:app_users` was an off-the-cuff example, not a decision.
 
-- The refusal is a structured value first and text second: a list of destructive operations, each with the entity it touches in domain coordinates and the statements that would resolve it, rendered to the error message by the command. The prompt consumes the same list.
-- Statement text is parsed, resolved and applied through one function the command calls, which takes the statement strings and the two contracts and does not care whether the strings came from the command line or from an answer typed at a prompt. Statements can therefore be added after the command has started and the plan re-run.
-- The per-operation consent question that `db update` already asks in a terminal is the seed of this prompt, so slice 2 keeps it as a per-operation question rather than folding it into a yes/no over the whole plan.
+- The write-up goes to Will for a decision.
+- It does not block close-out.
+- Building the chosen syntax is not in this project unless Will adds it.
 
-## Sequencing
+## Delivered: the interactive prompt
 
-- **Stack:** slice 1 → slice 2 → slice 3.
-- **Parallel:** slice 4 runs beside slice 3 once slice 2 has merged.
-
-## Dependencies
-
-- Nothing external. The shelved prisma/orm#30570 is a source to copy from, not a dependency; its branch stays as a draft.
-- Users get the feature through the next published CLI release after slice 2 merges; slices 1 and 2 together are the minimum that changes behaviour a user sees.
+Recorded on 2026-10-06 as the stretch goal: when a human runs the command in a terminal, the refusal becomes a question per operation, and the answer is the same statement the flags take. Slice 2 built it on the CLI engine's statement prompt (prisma/prisma-cli#337). Scripts, agents and CI get the refusal that names the flags.
 
 ## Close-out (required)
 
 - [ ] Verify every project DoD item in [`spec.md`](./spec.md)
+- [ ] Final retro
 - [ ] Write the ADR and amend ADR 001, ADR 028 and the Data Contract and Migration System subsystem docs
 - [ ] Migrate long-lived docs into `docs/`
 - [ ] Strip repo-wide references to `projects/migration-statements/**`