From 0d63f59ebf258bd56f984d522b55cc9c841f7645 Mon Sep 17 00:00:00 2001 From: willbot Date: Thu, 8 Oct 2026 19:18:59 +0200 Subject: [PATCH 01/26] TML-3474: re-plan the remaining slices; specs for 3a and 4a Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/plan.md | 53 ++++++-- .../slices/convert-backfill/spec.md | 121 ++++++++++++++++++ .../slices/mongo-renames/spec.md | 96 ++++++++++++++ projects/migration-statements/spec.md | 2 +- 4 files changed, 260 insertions(+), 12 deletions(-) create mode 100644 projects/migration-statements/slices/convert-backfill/spec.md create mode 100644 projects/migration-statements/slices/mongo-renames/spec.md diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index 0090e6f8d0d8..3365ffefe043 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -1,10 +1,16 @@ # Project plan — Migration statements -**Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474) · **Shaping PR:** prisma/orm#30604 · **Working branch:** `tml-3474-migration-statements` +**Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474) ## Summary -Four slices. The first three stack: the statement surface with renames, then the refusal and consent model, then the scaffolded verbs and the remaining nouns. The fourth, Mongo, builds on the second and runs in parallel with the third. Each slice is one PR, stacked on the working branch until the shaping PR merges and then retargeted to `main`. +Six slices. Slices 1 and 2 are merged. The rest, re-planned on 2026-10-08 when slices 3 and 4 proved too large for one review each: + +- **3a** (SQL `--convert` and `--backfill`) and **4a** (MongoDB renames and deletes) run in parallel. +- **3b** (the remaining SQL nouns) follows 3a. **4b** (MongoDB `--convert`, `--backfill` and value object renames) follows 3b and 4a, and reuses their design. +- The storage-name syntax survey runs beside the slices and goes to Will for a decision. It does not block close-out. + +Each slice is one PR against `main`. ## Slices @@ -30,25 +36,49 @@ Four slices. The first three stack: the statement surface with renames, then the **Hands to.** The refusal shape every later verb hooks its statements into; the per-operation consent model; `--confirm` no longer consents to data loss. -### Slice 3 — Convert and backfill scaffold the placeholder migration, and the remaining nouns +### Slice 3a — `--convert` and `--backfill` on Postgres and SQLite **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/convert-backfill/` -**Outcome.** `--convert` scaffolds the type change with the placeholder in the slot that carries the conversion, and `--backfill` the backfill transform; both refused on `db update`; the scaffolding stops being automatic. `--rename` on enum values, namespaces and value object fields, a model move across namespaces (`alter table set schema` on Postgres, deferred from slice 1 on 2026-10-06 because no operation existed for it), and `--convert` on a variant, plan the row updates, the JSON rewrites and the schema rename. `--delete` on an enum value nulls where nullable, else refuses. +**Outcome.** `migration plan --convert User.age` writes the type change with a `placeholder()` where the conversion goes, and `--backfill User.email` writes the backfill transform. A lossy type change is answered with `convert` or `delete`. Neither verb is accepted by `db update`. The planner no longer scaffolds without being asked; without `--backfill`, a required field on a populated table gets the temporary-default recipe on both SQL targets, which SQLite lacks today. The Postgres runner's advice when `SET NOT NULL` meets NULLs names the NULLs (TML-3517). **Builds on.** Slice 2. +**Hands to.** The convert and backfill verbs, their questions and their scaffold rule, for 3b (variants) and 4b (MongoDB). + +### Slice 3b — The remaining nouns on Postgres and SQLite + +**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/remaining-nouns/` + +**Outcome.** `--rename` on enum values, namespaces and value object fields (building the framework's value object statement surface: statement entity, subject kind, grammar, resolver), a model move across namespaces (`alter table set schema` on Postgres), and `--convert` on a variant whose discriminator value changed, each planning the row updates, JSON rewrites or schema rename. `--delete` on an enum value nulls it where nullable, else refuses. `--delete ` answers every question for the models in it. + +**Builds on.** Slice 3a. + **Hands to.** Project close-out for the SQL targets. -### Slice 4 — The same statements on MongoDB +### Slice 4a — Renames and deletes on MongoDB + +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo-renames/` + +**Outcome.** The MongoDB planner carries out `--rename` for models (collection rename) and fields (document rewrite), with the validator and indexes kept consistent, through both commands. Removing a field becomes data loss, answered by `--delete` with an `$unset`. Document rewrites run under `db update`. The temporary capability member `renameStatements: { refused: true; keepDataByHand }`, its three CLI reads and its doc mention are deleted. Statements name MongoDB fields the same way they name SQL fields. + +**Builds on.** Slice 2. + +**Hands to.** A MongoDB planner that takes resolved statements, for 4b. + +### Slice 4b — `--convert`, `--backfill` and value object renames on MongoDB + +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo-convert-backfill/` + +**Outcome.** The same two verbs as 3a on MongoDB, scaffolding a data transform with a placeholder. Value object field renames rewrite subdocuments, including lists, dictionaries and unions, using the statement surface from 3b. A validator change that makes a field required on a populated collection is no longer silent: it is answered with `--backfill`, or the command says that existing documents need the field. -**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo/` +**Builds on.** Slices 3a, 3b and 4a. -**Outcome.** The Mongo planner takes the same resolved statements: collection rename, document rewrites for field and value object field renames, drops and unsets for deletes, a data transform scaffold for convert, under the slice 2 refusal and consent model. No family vocabulary enters the framework. +**Hands to.** Project close-out for MongoDB. -**Builds on.** Slice 2. Slice 2 left a temporary member for it: `TargetMigrationsCapability.refusesRenameStatements`, set by MongoDB so a data-loss question offers no `rename` there. Slice 4 deletes the member, its one CLI read and its doc mention when the Mongo planner carries out renames. +### Alongside — A syntax for naming storage objects -**Hands to.** Project close-out for Mongo. +Will asked (2026-10-07) for a critical discussion and a survey of how established tools name tables and other storage objects on the command line, before any syntax is chosen for the `@@map`-only rename gap (`deferred.md`). The write-up goes to Will for a decision. Building the chosen syntax is not in this project unless Will adds it. ## Stretch goal — interactive statements when a human runs the command @@ -66,8 +96,9 @@ What the slices do now so this drops in later (see the spec's cross-cutting requ ## Sequencing -- **Stack:** slice 1 → slice 2 → slice 3. -- **Parallel:** slice 4 runs beside slice 3 once slice 2 has merged. +- **Merged:** slice 1 → slice 2. +- **Parallel now:** 3a and 4a. +- **Then:** 3b after 3a; 4b after 3b and 4a. ## Dependencies diff --git a/projects/migration-statements/slices/convert-backfill/spec.md b/projects/migration-statements/slices/convert-backfill/spec.md new file mode 100644 index 000000000000..1fbf81be20bf --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/spec.md @@ -0,0 +1,121 @@ +# Slice spec — `--convert` and `--backfill` write the placeholder migration only when asked + +**Project:** [`projects/migration-statements/`](../../spec.md) · **Slice 3a** · **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477), [TML-3517](https://linear.app/prisma-company/issue/TML-3517) · **Branch:** `tml-3477-convert-backfill` · **Builds on:** slice 2 (merged). + +## At a glance + +A user changes `User.age` from `String` to `Int` on Postgres: + +```text +$ prisma migration plan --name age-to-int +✖ [CLI.CONSENT_REQUIRED] 1 subject needs a statement, and the session is not interactive. + why: Change type of column "User"."age" would lose the data of field "User.age". +→ Pass --convert User.age +→ Pass --delete User.age + +$ prisma migration plan --name age-to-int --convert User.age +⚠ migration.ts has a placeholder: replace it with the conversion, then run the file to re-emit. +``` + +`migration.ts` then holds one placeholder, where the conversion goes: + +```ts +this.alterColumnType({ + table: 'User', + column: 'age', + type: 'int4', + using: placeholder('User.age:using'), +}), +``` + +The user writes `using: '"age"::integer'` (or any SQL expression), runs the file, and gets an applicable migration. On SQLite the placeholder sits in the table rebuild's copy step for that column. + +Adding a required `email` field to an existing `User` table plans a temporary default by default: the column is added with a fill value and the default is dropped, and nothing is asked. To write the fill yourself, run `prisma migration plan --backfill User.email`. That writes the nullable column, a data transform with a placeholder, and `SET NOT NULL`. + +Today `migration plan` writes these placeholders on its own for every type change and every required field on an existing table. Users who didn't want them get a migration they can't apply until they edit it. After this slice it writes them only when asked. + +## Chosen design + +### The two verbs + +- **`--convert `** answers a type change that would lose data. It is accepted by `migration plan` only. It names the field by its destination name, like the new side of a rename: after `--rename User.age:User.years`, write `--convert User.years`. It needs the origin contract, like `rename`, and fails with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN` without one. It must name a field that exists on both sides and whose type changes. +- **`--backfill `** is an opt-in that answers no question. It is accepted by `migration plan` only. It names a field that is required in the destination and is either new on an existing table or optional in the origin. Anything else is an error that names what was found. +- `db update` declares both verbs only to refuse them, with `MIGRATION.STATEMENT_NEEDS_MIGRATION_FILE`. The error says `db update` has no migration file to fill and names `migration plan`. It doesn't fall through to the engine's generic "no flag registered" error, which doesn't explain the rule. The shared verb table in `orm/statement-verbs.ts` declares the verbs per command, so neither command accepts a verb by accident. + +### What the planners write + +| Diff | No statement | `--convert` | `--backfill` | +| --- | --- | --- | --- | +| Type change the target knows is safe (`int4`→`int8`) | direct alter, `widening`, nothing asked | error: nothing to convert | — | +| Any other type change | refused: question offers `convert` and `delete`. `--delete` plans the direct alter, `destructive` | Postgres: `alterColumnType` with `using: placeholder(…)`. SQLite: `recreateTable` whose copy step takes `placeholder(…)` for the column. Class `data` | — | +| Required field added to an existing table | temporary-default recipe on both targets, both commands, `additive` | — | nullable add, `dataTransform` with placeholders, then `SET NOT NULL` (SQLite: a tightening rebuild) | +| Optional field made required | direct `SET NOT NULL` (SQLite: tightening rebuild), `widening`. Fails at apply if NULLs exist, with advice that names `--backfill` | — | `dataTransform` with placeholders, then `SET NOT NULL` | + +The four automatic placeholder strategies (Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`) fire only for a field a statement names. Statements reach them as destination coordinates through the strategy context. They change no names in the working schema, so they don't go through the rename path in `planStatements`. A converted type change carries no separate NULL-handling transform; the conversion expression can map NULL itself. + +A converted alter is classed `data`, because the user writes its data step. So it isn't a `destructive` operation and leaves `dataLoss`. A typed `convert` at the prompt re-plans like a typed rename, and the type-change question isn't asked again. + +### The questions know why data is lost + +Each `dataLoss` entry gains `loss: 'drop' | 'typeChange'`, set by each target's operation-subjects function. The CLI stops guessing from whether the destination still has the field. A `typeChange` question offers `convert` and `delete` in `migration plan`, and `delete` only in `db update`. A `drop` question offers `rename` and `delete`, as today. The re-plan loop (`askPlanQuestions`) carries every planned statement, not only renames. + +`ResolvedMigrationStatement` gains a `kind`, which is `rename`, `convert` or `backfill`. Every reader that assumes a rename now switches on `kind` first. One example is the SQL family's `StatementPlanner`, which today plans any non-model statement as a column rename. MongoDB keeps refusing statements it can't carry out, until slices 4a and 4b. + +### The temporary default becomes a real migration call + +The Postgres temporary-default and direct-add calls render as `rawSql({ id, label, operationClass })` with no SQL. Re-running such a `migration.ts` doesn't reproduce `ops.json` (cross-cutting requirement 7). They now render as a facade call that builds the same operation. SQLite gets the recipe as a rebuild in which the new column's copy value is a literal. This needs a new per-column copy-expression option on `recreateTable`, which `--convert` also uses, and a SQLite fill-value map (integer `0`, real `0`, text `''`, blob `X''`, and the values its boolean and date-time codecs store). It is refused on the same primary-key, unique and foreign-key conditions as Postgres. This closes the deferred item "SQLite `db update` cannot add a required field to a table that has rows". + +### `alterColumnType` takes plain options + +The Postgres facade's `alterColumnType` options are reshaped so the rendered call shows a readable shape with an optional `using`, and no internal names (`qualifiedTargetType`, `formatTypeExpected`, `rawTargetTypeForLabel`). The `using` slot renders as code, so it can hold `placeholder(…)`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result the CLI already handles. Upgrade instructions are recorded for hand-written migrations. + +### The NOT NULL failure names the NULLs (TML-3517) + +When `SET NOT NULL` or a SQLite tightening rebuild meets NULL values, the runner reports how many rows hold NULL in which column. It says to fill them, make the field optional, or plan with `--backfill`. `MigrationRunnerFailure` gains a `fix`, and the `db update`, `db init` and `migrate` error mappers use it instead of the fixed advice about schema drift. + +## Coherence rationale + +Everything in this slice serves one change: the placeholder migration is written only when a statement asks for it. The verbs, the loss kind, the statement `kind`, the temporary default becoming real on both targets, the `alterColumnType` reshape and the NULL advice are each what that change needs to work on Postgres and SQLite. The reviewer reads one rule and its consequences. + +## Scope + +**In:** +- `--convert` and `--backfill` on `migration plan`, for Postgres and SQLite, refused on `db update`. +- The loss kind on `dataLoss` entries, and the statement `kind`. +- Ending automatic scaffolding. +- The temporary default as a real call on both targets, including the SQLite recipe. +- The `alterColumnType` reshape. +- TML-3517. +- The project spec's wording: "an existing table", not "a non-empty table", since `migration plan` can't see rows. +- Docs: Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`. +- An app upgrade fragment: type changes ask, scaffolds are opt-in, and the temporary default applies under `migration plan`. +- Upgrade instructions for the facade change. + +**Deliberately out:** +- `--convert` on a variant (slice 3b). +- MongoDB (slice 4b). +- A temporary default for a field in a new unique or foreign key, which stays refused as today. +- Detecting rows offline. + +## Pre-investigated edge cases + +| Case | Handling | +| --- | --- | +| A temporary value in a unique column fails on the second row | The recipe is refused for new unique, primary-key and foreign-key columns, as on Postgres today | +| A convert after a rename in the same run | The convert uses the destination name, which the diff issues already use | +| A placeholder operation has no position until re-emit | Applied-statement positions for a convert are empty until the file is re-run, as for every placeholder today | + +## Slice-specific done conditions + +- The project DoD's convert and backfill lines pass on Postgres and SQLite, reading "an existing table". +- Re-running every `migration.ts` the planners write in this slice's tests reproduces its `ops.json`. + +## Open questions + +None. + +## References + +- Grounding report (local, gitignored): `wip/grounding-3a.md`. +- [`../../design-notes.md`](../../design-notes.md): the scaffold rule and the convert and backfill rows. +- [ADR 200](../../../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md): the placeholder flow. diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md new file mode 100644 index 000000000000..aa3f4f79a822 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -0,0 +1,96 @@ +# Slice spec — MongoDB carries out renames and deletes of models and fields + +**Project:** [`projects/migration-statements/`](../../spec.md) · **Slice 4a** · **Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Branch:** `tml-3478-mongo-renames` · **Builds on:** slice 2 (merged). + +## At a glance + +A MongoDB user renames the `Profile` model to `User` (collection `profiles` to `users`), changes `name` to `fullName`, and removes the `nickname` field: + +```text +$ prisma migration plan --name tidy --rename Profile:User --rename User.name:User.fullName --delete User.nickname +✔ Planned 6 operation(s) + Rename collection "profiles" to "users" widening + Drop index users.name_1 widening + Rename field "name" to "fullName" in collection "users" widening + Remove field "nickname" from collection "users" destructive + Create index users.fullName_1 additive + Update validator on users widening +``` + +`migration.ts` carries `renameCollection(...)`, `renameField(...)` and `unsetField(...)` calls, and running it reproduces `ops.json`. `prisma db update` with the same statements does the same against the live database. Today MongoDB refuses every `--rename` and prints steps to run by hand in `mongosh`, and removing a field asks nothing, even though every document that still holds the field then fails each later update against the closed validator. + +Without `--delete User.nickname`, both commands refuse, as they do on SQL: + +```text +✖ [CLI.CONSENT_REQUIRED] 1 subject needs a statement, and the session is not interactive. + why: Remove field "nickname" from collection "users" would lose the data of field "User.nickname". +→ Pass --rename 'User.nickname:' +→ Pass --delete User.nickname +``` + +## Chosen design + +### Statements name fields by their contract key + +A statement names a field as the contract and the generated client name it: the key in the model's `fields`. On SQL that key is the field name in the schema; on MongoDB the contract keys fields by the stored name (`id @map("_id")` is `User._id`), and the client uses the same name. The resolver already behaves this way, so nothing changes in it. On MongoDB a change to `@map` alone is a field rename in contract terms and is stated with `--rename`; a change to the schema field name alone changes nothing stored and needs no statement. This settles the deferred item "Which field name a MongoDB statement uses" and is recorded in the Migration System doc. + +### Collection rename + +A new DDL operation, `renameCollection`, added through the MongoDB stack the way the other collection operations are: query AST command and visitor, wire command and lowering, driver, operation serializer, preview text, factory and call class. Precheck: the source collection exists and the target does not. Postcheck: the target exists. Class `widening`. A model rename renames the collection in the working copy of the origin schema before the diff, so the indexes and validator that MongoDB keeps through a rename are not dropped and recreated. + +### Field rename and field removal rewrite the documents + +- A field rename plans `renameField(collection, from, to, { filter })`: one `updateMany` with `$rename`, run with validation bypassed. Filter: documents that hold the old field, and for a variant's field also the variant's discriminator value. Precheck: no matching document already holds the new field, since `$rename` would overwrite it. Postcheck: no matching document still holds the old field, so a re-run skips it. Class `widening`. +- Removing a field from a model is data loss, matching the spec's rule that dropping a field is destructive. The planner plans `unsetField(collection, field, { filter })`, an `updateMany` with `$unset`, class `destructive`, and reports a `field` subject. The question offers `rename` and `delete`, as on SQL. A field removed together with its collection gets no unset; the collection drop covers it. +- Order within a plan: collection renames, collection creates, index drops, document rewrites (renames and unsets), index creates, validator changes, option changes, collection drops. Rewrites come after index drops so a unique index on the old field does not fail once documents lose it, and before index creates so a unique index on the new field sees the moved values. Validation is bypassed for the rewrite because the origin validator requires the old field and the destination validator may require a field the same plan adds. Plans without statements and without field removals keep their current operations and order. +- The rewrite command gains a `bypassDocumentValidation` option through its wire form and the driver. + +### Document rewrites run under `db update` + +`db update` allows `additive`, `widening` and `destructive`, not `data`, so a rewrite classed `data` would be refused there. The rewrites above carry the class of what they do to data (`widening` for a rename, `destructive` for an unset), and the MongoDB runner and operation serializer tell a rewrite from a DDL command by its shape, not by `operationClass === 'data'`. `data` stays the class of transforms a user writes or a scaffold produces; `db update` keeps excluding it. Slice 3b's SQL JSON rewrites follow the same rule. + +### What is refused + +The MongoDB planner refuses, as a `statementRejected` conflict naming the reason: a rename of `_id`; a collection rename whose target exists in the working schema; a rename of a field stored on one side only; and any statement when the control policy is not `managed`, as SQL does. A document that already holds the new name fails at apply time by precheck, before anything is written. + +### The temporary refusal goes + +`TargetMigrationsCapability.renameStatements` and `keepDataByHand` are deleted: the member in `control-migration-types.ts`, MongoDB's setting in `control-target.ts`, `keepDataByHand` and `keepTheData` in `mongo-planner.ts`, `keepDataByHandFor` and its parameter in `plan-questions.ts`, its reads in `db-run.ts` and `migration-plan.ts`, and the Statements paragraph that describes it in the Migration System doc. The unrelated `renameStatements()` helper in `statement-text.ts` stays. The member shipped in 8.0.0-rc.17, so an extension upgrade fragment tells target authors to remove it. + +## Coherence rationale + +One reviewer can hold this: it is one family learning the two verbs the SQL targets already carry out, through one new DDL operation and one new rewrite shape, with the refusal member removed as the visible proof. The collection rename's six-package plumbing is mechanical and mirrors existing collection operations. + +## Scope + +**In:** model rename (collection rename); top-level field rename, including a variant's field; field removal as data loss with `$unset` under `--delete`; both commands; the `bypassDocumentValidation` option; runner and serializer telling rewrites apart by shape; deleting `renameStatements`; docs (Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`); an app upgrade fragment (MongoDB renames work; removing a field now asks) and an extension upgrade fragment (the capability member is gone). + +**Deliberately out:** +- Value object field renames on MongoDB. They need the framework's value object statement surface (statement entity, subject kind, grammar, resolver), which slice 3b builds for SQL JSON rewrites, and a recursive update pipeline for lists, dictionaries and unions. Moved to slice 4b. +- `--convert` and `--backfill` on MongoDB, and the required-field validator gap: slice 4b. +- Batching large rewrites. One `updateMany` per field, as today's hand-written transforms do. + +## Pre-investigated edge cases + +| Case | Handling | +| --- | --- | +| Slice 1 manual QA (F3): a unique index on the old field fails the rewrite once two documents lose it; one on the new field fails if created first | Order above: index drops, rewrite, index creates | +| The origin validator requires the old field and forbids the new one | Rewrite runs with validation bypassed | +| Two variants in one collection store a field with the same name | Rewrite filters on the discriminator value | +| The runner is not transactional | Postcheck makes a re-run skip a finished rewrite | +| `bypassDocumentValidation` needs a privilege on hosted clusters | Runner failure names the privilege | + +## Slice-specific done conditions + +- The project DoD's MongoDB journey without the value object part: a model rename and a field rename (with a unique index on the field) and a field removal under `--delete`, through `migration plan` then `migrate` and through `db update`, on collections with documents. Afterwards the documents are under the new names, the removed field is gone from every document, every document accepts an update, a further plan is empty, and `db verify --schema-only` is clean. +- `git grep -n "keepDataByHand\|TargetMigrationsCapability\['renameStatements'\]"` finds nothing. + +## Open questions + +None. + +## References + +- Grounding report for this slice (local, gitignored): `wip/grounding-4a.md`. +- [`../../deferred.md`](../../deferred.md) — "Which field name a MongoDB statement uses". +- Slice 1 manual QA, F3: [`../../manual-qa-reports/2026-10-07-qa-opus.md`](../../manual-qa-reports/2026-10-07-qa-opus.md). diff --git a/projects/migration-statements/spec.md b/projects/migration-statements/spec.md index 746c6e454693..25683c4676b8 100644 --- a/projects/migration-statements/spec.md +++ b/projects/migration-statements/spec.md @@ -121,7 +121,7 @@ In addition to the team DoD floor in [`drive/calibration/dod.md`](../../drive/ca - The same change through `db update` with the same statements on both targets produces the same database state and no consent prompt. Running the same `db update` again with the same statements fails on the first statement. - The same change with `--delete` omitted makes both commands refuse, and the refusal text contains the exact statement that then succeeds. - A type change through `migration plan --convert` writes a migration whose only placeholder is in the Postgres `using` slot; filling it and re-running the file produces an applicable migration that converts the rows. The same change without a statement is refused by both commands, and `--delete` on the field plans the direct alter. -- A new required field on a non-empty table through `migration plan --backfill` writes the scaffolded backfill transform; without the statement both commands plan the temporary-default recipe. +- A new required field on an existing table through `migration plan --backfill` writes the scaffolded backfill transform; without the statement both commands plan the temporary-default recipe. - On Mongo, the rename journey above with a collection and a document field, through both commands, and a value object field rename that rewrites subdocuments. - A `db update` against a database whose marker hash has no local snapshot fails every rename and convert statement with one error naming the hash, and succeeds with `--delete` alone. - `--confirm` is gone from both commands, the CLI README describes the statements and the consent model, and the upgrade fragments for the refusal and the consent change are validated by execution. From b5a51bbec829f34b58ccdd5850e6f3bc49dccf83 Mon Sep 17 00:00:00 2001 From: willbot Date: Thu, 8 Oct 2026 19:19:06 +0200 Subject: [PATCH 02/26] TML-3474: the spec describes scaffolding as it is today Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/migration-statements/spec.md b/projects/migration-statements/spec.md index 25683c4676b8..783bd6fb856c 100644 --- a/projects/migration-statements/spec.md +++ b/projects/migration-statements/spec.md @@ -71,7 +71,7 @@ The reasoning and the alternatives rejected are in [`design-notes.md`](./design- - **Planner inputs.** Every planner takes a destination contract and an origin schema: derived from a contract snapshot for `migration plan`, introspected for `db update` and `db init`, and one differ runs over two schema IRs ([ADR 235](../../docs/architecture%20docs/adrs/ADR%20235%20-%20The%20schema%20differ%20walks%20two%20derived%20schema%20IRs.md)). `migration plan` also hands the SQL planner the origin contract as `fromContract` (`packages/1-framework/3-tooling/cli/src/control-api/operations/plan-resolution.ts`); the `db update` path passes `fromContract: null` (`packages/1-framework/3-tooling/migration/src/aggregate/strategies/plan-from-diff.ts`). Statements become the third input, and `db update` learns to supply the origin contract. - **Where the origin contract comes from for `db update`.** The database marker stores the contract hash the database was last updated to ([ADR 021](../../docs/architecture%20docs/adrs/ADR%20021%20-%20Contract%20Marker%20Storage.md)), and `db update` advances the `db` ref to the same hash ([ADR 218](../../docs/architecture%20docs/adrs/ADR%20218%20-%20Refs%20with%20paired%20contract%20snapshots%20and%20universal%20graph-node%20invariant.md)). The content-addressed snapshot store ([ADR 240](../../docs/architecture%20docs/adrs/ADR%20240%20-%20Contract%20snapshots%20live%20in%20a%20content-addressed%20store.md)) holds the contract for that hash when the project has it. - **Consent today.** `db update` pre-plans and refuses with `MIGRATION.DESTRUCTIVE_CHANGES` until the user types the database name or passes the engine's `--confirm ` token (`packages/1-framework/3-tooling/cli/src/orm/db/consent.ts`, `control-api/operations/db-update.ts`). `migration plan` applies the same refusal to auto-baselines only, with the project directory as the token (`control-api/operations/migration-plan.ts`). The CLI README's `db update` section still describes a `--yes` flag that no longer exists. -- **The placeholder flow.** `migration plan` already scaffolds data transforms with `placeholder()` slots for a required column on a non-empty table, a nullable-to-required change and an unsafe type change, writes `ops.json` as `[]` with `pendingPlaceholders`, and the user fills the slots and re-runs `migration.ts` to re-emit ([ADR 200](../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md), [ADR 196](../../docs/architecture%20docs/adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md), `packages/3-targets/3-targets/postgres/src/core/migrations/planner-strategies.ts`). Under `db update`, whose policy excludes the `data` class, those strategies stand down and direct DDL is emitted. The Postgres `alterColumnType` operation already has a `using` option the planner never fills. +- **The placeholder flow.** `migration plan` already scaffolds data transforms with `placeholder()` slots on Postgres for a required column on an existing table, a nullable-to-required change and an unsafe type change (SQLite only for nullable-to-required), writes `ops.json` as `[]` with `pendingPlaceholders`, and the user fills the slots and re-runs `migration.ts` to re-emit ([ADR 200](../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md), [ADR 196](../../docs/architecture%20docs/adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md), `packages/3-targets/3-targets/postgres/src/core/migrations/planner-strategies.ts`). Under `db update`, whose policy excludes the `data` class, those strategies stand down and direct DDL is emitted. The Postgres `alterColumnType` operation already has a `using` option the planner never fills. - **Rename substrate.** `renameTable` exists on Postgres and SQLite from prisma/orm#30331. The shelved prisma/orm#30570 refactored it onto a working schema shared by the planner and the hand-written facade, with companion constraint names taken from the destination contract and foreign keys paired by their own and referenced columns, and reclassified non-data drops as widening. That code is the starting point; the contract `hints` section it also added is not. - **Operation classes.** Additive, widening, destructive and data (`packages/2-sql/9-family/src/core/migrations/types.ts`). Every Postgres type change is destructive today except a short list of safe widenings; Mongo classifies by validator comparison (`packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts`). - **Coordinates.** A contract entity is addressed by plane, namespace, kind and name ([ADR 221](../../docs/architecture%20docs/adrs/ADR%20221%20-%20Contract%20IR%20two%20planes%20with%20uniform%20entity%20coordinate%20and%20pack-contributed%20entity%20kinds.md), [ADR 224](../../docs/architecture%20docs/adrs/ADR%20224%20-%20Namespace%20concretions%20address%20entities%20by%20coordinate.md)); the contract source declares namespaces as `namespace { ... }` blocks; the query surface resolves a bare name through the sole namespace ([ADR 223](../../docs/architecture%20docs/adrs/ADR%20223%20-%20Target-owned%20default%20namespace.md)). Variants and discriminators are [ADR 173](../../docs/architecture%20docs/adrs/ADR%20173%20-%20Polymorphism%20via%20discriminator%20and%20variants.md); value objects are [ADR 178](../../docs/architecture%20docs/adrs/ADR%20178%20-%20Value%20objects%20in%20the%20contract.md), stored as one JSONB column on SQL and a subdocument on Mongo. From 2808c74992a563b04c961a68cd5181c207159ced Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 08:21:35 +0200 Subject: [PATCH 03/26] TML-3474: specs name the ADR amendments, variant deletes and the type-change delete wording Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/plan.md | 2 +- .../migration-statements/slices/convert-backfill/spec.md | 4 +++- projects/migration-statements/slices/mongo-renames/spec.md | 5 +++-- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index 3365ffefe043..28a03fda63cc 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -50,7 +50,7 @@ Each slice is one PR against `main`. **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/remaining-nouns/` -**Outcome.** `--rename` on enum values, namespaces and value object fields (building the framework's value object statement surface: statement entity, subject kind, grammar, resolver), a model move across namespaces (`alter table set schema` on Postgres), and `--convert` on a variant whose discriminator value changed, each planning the row updates, JSON rewrites or schema rename. `--delete` on an enum value nulls it where nullable, else refuses. `--delete ` answers every question for the models in it. +**Outcome.** `--rename` on enum values, namespaces and value object fields (building the framework's value object statement surface: statement entity, subject kind, grammar, resolver), a model move across namespaces (`alter table set schema` on Postgres), and `--convert` on a variant whose discriminator value changed, each planning the row updates, JSON rewrites or schema rename. `--delete` on an enum value nulls it where nullable, else refuses. `--delete` on a variant stored in its base's table deletes the rows with that discriminator value and drops its columns, and the refusal says it deletes rows. `--delete ` answers every question for the models in it. Risk to settle in the spec: a value object can contain itself (ADR 178's `NavItem.children`), and a field rename inside such a type must rewrite to any depth, which a fixed SQL JSON expression or MongoDB update pipeline cannot do. **Builds on.** Slice 3a. diff --git a/projects/migration-statements/slices/convert-backfill/spec.md b/projects/migration-statements/slices/convert-backfill/spec.md index 1fbf81be20bf..48bfe8e4a426 100644 --- a/projects/migration-statements/slices/convert-backfill/spec.md +++ b/projects/migration-statements/slices/convert-backfill/spec.md @@ -53,7 +53,7 @@ Today `migration plan` writes these placeholders on its own for every type chang The four automatic placeholder strategies (Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`) fire only for a field a statement names. Statements reach them as destination coordinates through the strategy context. They change no names in the working schema, so they don't go through the rename path in `planStatements`. A converted type change carries no separate NULL-handling transform; the conversion expression can map NULL itself. -A converted alter is classed `data`, because the user writes its data step. So it isn't a `destructive` operation and leaves `dataLoss`. A typed `convert` at the prompt re-plans like a typed rename, and the type-change question isn't asked again. +A `delete` answer to a type-change question keeps planning the direct alter, and the statement is reported as giving up the field's values in a type change (`delete values of field "User.age" (type change)`), not as deleting the field; slice 2's manual QA found the old wording misleading (F14). A converted alter is classed `data`, because the user writes its data step. So it isn't a `destructive` operation and leaves `dataLoss`. A typed `convert` at the prompt re-plans like a typed rename, and the type-change question isn't asked again. ### The questions know why data is lost @@ -87,6 +87,8 @@ Everything in this slice serves one change: the placeholder migration is written - The `alterColumnType` reshape. - TML-3517. - The project spec's wording: "an existing table", not "a non-empty table", since `migration plan` can't see rows. +- An amendment to ADR 200, which says only a data transform holds a placeholder. +- The Migration System doc's claim that the Postgres planner writes no placeholder data transforms, which the code contradicts. - Docs: Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`. - An app upgrade fragment: type changes ask, scaffolds are opt-in, and the temporary default applies under `migration plan`. - Upgrade instructions for the facade change. diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md index aa3f4f79a822..0952fd89bb7c 100644 --- a/projects/migration-statements/slices/mongo-renames/spec.md +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -47,7 +47,7 @@ A new DDL operation, `renameCollection`, added through the MongoDB stack the way ### Document rewrites run under `db update` -`db update` allows `additive`, `widening` and `destructive`, not `data`, so a rewrite classed `data` would be refused there. The rewrites above carry the class of what they do to data (`widening` for a rename, `destructive` for an unset), and the MongoDB runner and operation serializer tell a rewrite from a DDL command by its shape, not by `operationClass === 'data'`. `data` stays the class of transforms a user writes or a scaffold produces; `db update` keeps excluding it. Slice 3b's SQL JSON rewrites follow the same rule. +`db update` allows `additive`, `widening` and `destructive`, not `data`, so a rewrite classed `data` would be refused there. The rewrites above carry the class of what they do to data (`widening` for a rename, `destructive` for an unset), and the MongoDB runner and operation serializer tell a rewrite from a DDL command by its shape, not by `operationClass === 'data'`. `data` stays the class of transforms a user writes or a scaffold produces; `db update` keeps excluding it. Slice 3b's SQL JSON rewrites follow the same rule. This follows the Migration System doc's rule that only the planner's own classification promises an operation loses data exactly when it is `destructive`, and it changes ADR 188, which says every MongoDB data transform is `data`: the slice amends ADR 188, and ADR 264's line that MongoDB has no rename operation. ### What is refused @@ -63,7 +63,7 @@ One reviewer can hold this: it is one family learning the two verbs the SQL targ ## Scope -**In:** model rename (collection rename); top-level field rename, including a variant's field; field removal as data loss with `$unset` under `--delete`; both commands; the `bypassDocumentValidation` option; runner and serializer telling rewrites apart by shape; deleting `renameStatements`; docs (Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`); an app upgrade fragment (MongoDB renames work; removing a field now asks) and an extension upgrade fragment (the capability member is gone). +**In:** amendments to ADR 188 (planner-written rewrites carry the class of their effect) and ADR 264 (MongoDB has a collection rename); model rename (collection rename); top-level field rename, including a variant's field; field removal as data loss with `$unset` under `--delete`; both commands; the `bypassDocumentValidation` option; runner and serializer telling rewrites apart by shape; deleting `renameStatements`; docs (Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`); an app upgrade fragment (MongoDB renames work; removing a field now asks) and an extension upgrade fragment (the capability member is gone). **Deliberately out:** - Value object field renames on MongoDB. They need the framework's value object statement surface (statement entity, subject kind, grammar, resolver), which slice 3b builds for SQL JSON rewrites, and a recursive update pipeline for lists, dictionaries and unions. Moved to slice 4b. @@ -79,6 +79,7 @@ One reviewer can hold this: it is one family learning the two verbs the SQL targ | Two variants in one collection store a field with the same name | Rewrite filters on the discriminator value | | The runner is not transactional | Postcheck makes a re-run skip a finished rewrite | | `bypassDocumentValidation` needs a privilege on hosted clusters | Runner failure names the privilege | +| TML-2447 (open): the planner creates separate collections and incomplete validators for variants that share a collection | The first dispatch checks how variants are stored today before the variant field rename is built on it | ## Slice-specific done conditions From e44518aa65ced4f0c13e91b209f08bb863573167 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 08:25:42 +0200 Subject: [PATCH 04/26] TML-3474: the plan and the 3a and 4a specs lead with tables and short lists Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/plan.md | 128 +++++++-------- .../slices/convert-backfill/spec.md | 148 ++++++++++-------- .../slices/engine-statement-prompt/spec.md | 2 +- .../slices/mongo-renames/spec.md | 145 +++++++++++------ projects/migration-statements/spec.md | 4 +- 5 files changed, 229 insertions(+), 198 deletions(-) diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index 28a03fda63cc..7d3276433912 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -2,112 +2,92 @@ **Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474) -## Summary +## At a glance -Six slices. Slices 1 and 2 are merged. The rest, re-planned on 2026-10-08 when slices 3 and 4 proved too large for one review each: +| Slice | What it delivers | Builds on | State | +| --- | --- | --- | --- | +| 1 | `--rename` for models and fields, Postgres and SQLite | — | Merged, prisma/orm#30638 | +| 2 | Data loss refused until answered; `--delete`, `--allow`; the interactive prompt | 1 | Merged, prisma/orm#30648 | +| 3a | `--convert` and `--backfill`, Postgres and SQLite | 2 | Spec: [`slices/convert-backfill/`](./slices/convert-backfill/spec.md) | +| 4a | Renames and deletes on MongoDB | 2 | Spec: [`slices/mongo-renames/`](./slices/mongo-renames/spec.md) | +| 3b | The remaining SQL renames and deletes | 3a | Not started | +| 4b | `--convert`, `--backfill` and value object renames on MongoDB | 3b, 4a | Not started | -- **3a** (SQL `--convert` and `--backfill`) and **4a** (MongoDB renames and deletes) run in parallel. -- **3b** (the remaining SQL nouns) follows 3a. **4b** (MongoDB `--convert`, `--backfill` and value object renames) follows 3b and 4a, and reuses their design. -- The storage-name syntax survey runs beside the slices and goes to Will for a decision. It does not block close-out. +```text +1 → 2 ─┬─ 3a ── 3b ─┐ + └─ 4a ───────┴─ 4b → close-out +``` -Each slice is one PR against `main`. +- 3a and 4a run in parallel now. +- Each slice is one pull request against `main`. +- Slices 3 and 4 were split on 2026-10-08. Each was too large to review as one pull request. -## Slices +## Slices still to build -### Slice 1 — Statements on the command line, and renames of models and fields on Postgres and SQLite +### 3a — `--convert` and `--backfill` on Postgres and SQLite -**Linear:** [TML-3475](https://linear.app/prisma-company/issue/TML-3475) · **Folder:** `slices/renames/` +**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Spec:** [`slices/convert-backfill/spec.md`](./slices/convert-backfill/spec.md) -**Outcome.** Both commands accept `--rename old:new` for models and fields, with a model rename across namespaces resolved but refused until slice 3. The framework parses each statement, resolves it against the origin and destination contracts, and applies it in order to a working copy of the origin. The SQL planners emit the table or column rename and every companion rename, with names from the destination contract. `db update` resolves its origin contract from the marker hash through the snapshot store and fails every rename when it cannot. An unusable statement is an error. Non-data drops are widening. The written migration is what a user could write by hand. +- A lossy type change is answered with `--convert` (fill in the conversion) or `--delete`. +- `--backfill` writes a slot to fill existing rows of a field that became required. +- `migration plan` stops writing placeholders nobody asked for. +- A new required field gets a temporary value on both targets. SQLite lacks this today. +- The `SET NOT NULL` failure names the NULL rows (TML-3517). -**Builds on.** The planner substrate from prisma/orm#30570, without its contract section. +**Hands to 3b and 4b:** the two flags and how their questions and placeholders work. -**Hands to.** A statement type the framework owns, parsed and resolved, delivered to every family planner as resolved entities; the working-copy mechanism statements apply to; the origin-contract resolution for `db update`; a column rename operation on both SQL targets. +### 4a — Renames and deletes on MongoDB -### Slice 2 — Both commands refuse data loss by default, and `--delete` is the per-operation consent +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Spec:** [`slices/mongo-renames/spec.md`](./slices/mongo-renames/spec.md) -**Linear:** [TML-3476](https://linear.app/prisma-company/issue/TML-3476) · **Folder:** `slices/refusal/` +- A model rename renames the collection. A field rename rewrites the documents. +- Removing a field is asked about. `--delete` removes it from every document. +- Both commands, including `db update`. +- The temporary setting that makes MongoDB refuse renames is deleted. -**Outcome.** `migration plan` refuses any plan that loses data, with the error `db update` uses. The refusal lists each destructive operation with the statements that resolve it. `--delete` consents to one operation, for namespaces, models and fields, and replaces `--confirm` on both commands. The terminal consent question asks per operation. Upgrade fragments record both changes; the CLI README describes statements and consent. +**Hands to 4b:** a MongoDB planner that takes statements. -**Builds on.** Slice 1. Two things slice 1 leaves for it: the missing-origin check in the CLI statement resolver runs over the whole input and must move inside the per-statement loop so that `--delete` works without an origin contract; and the `@@map`-only rename gap in `deferred.md` must be decided before the refusal text is written. Decided 2026-10-07 on prisma/orm#30638: dropping a row-level-security policy and disabling row-level security stay `widening` (they lose no data), and slice 2 adds a separate consent question before `db update` applies an operation that widens who can read or write rows, answered per operation like the data-loss consent and refused in a non-interactive run unless consented. - -**Design settled 2026-10-07 (Will).** Consent is a statement the user answers with, not a token copied back: the CLI engine gains `ctx.prompt.statement`/`statements`, answered on the command line by verb flags (`--delete Legacy`, `--rename Legacy:Archive`, `--allow User`) and interactively by typing the statement when a human runs the command. The engine change is its own slice, built in prisma/prisma-cli first: [`slices/engine-statement-prompt/spec.md`](./slices/engine-statement-prompt/spec.md). Slice 2's spec is [`slices/refusal/spec.md`](./slices/refusal/spec.md). The `@@map`-only gap gets no statement (see `deferred.md`). Operations that lose no data (`SET NOT NULL`, safe type widenings, MongoDB validator and index changes) stop being destructive. - -**Hands to.** The refusal shape every later verb hooks its statements into; the per-operation consent model; `--confirm` no longer consents to data loss. - -### Slice 3a — `--convert` and `--backfill` on Postgres and SQLite - -**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/convert-backfill/` - -**Outcome.** `migration plan --convert User.age` writes the type change with a `placeholder()` where the conversion goes, and `--backfill User.email` writes the backfill transform. A lossy type change is answered with `convert` or `delete`. Neither verb is accepted by `db update`. The planner no longer scaffolds without being asked; without `--backfill`, a required field on a populated table gets the temporary-default recipe on both SQL targets, which SQLite lacks today. The Postgres runner's advice when `SET NOT NULL` meets NULLs names the NULLs (TML-3517). - -**Builds on.** Slice 2. - -**Hands to.** The convert and backfill verbs, their questions and their scaffold rule, for 3b (variants) and 4b (MongoDB). - -### Slice 3b — The remaining nouns on Postgres and SQLite +### 3b — The remaining renames and deletes on Postgres and SQLite **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/remaining-nouns/` -**Outcome.** `--rename` on enum values, namespaces and value object fields (building the framework's value object statement surface: statement entity, subject kind, grammar, resolver), a model move across namespaces (`alter table set schema` on Postgres), and `--convert` on a variant whose discriminator value changed, each planning the row updates, JSON rewrites or schema rename. `--delete` on an enum value nulls it where nullable, else refuses. `--delete` on a variant stored in its base's table deletes the rows with that discriminator value and drops its columns, and the refusal says it deletes rows. `--delete ` answers every question for the models in it. Risk to settle in the spec: a value object can contain itself (ADR 178's `NavItem.children`), and a field rename inside such a type must rewrite to any depth, which a fixed SQL JSON expression or MongoDB update pipeline cannot do. - -**Builds on.** Slice 3a. - -**Hands to.** Project close-out for the SQL targets. - -### Slice 4a — Renames and deletes on MongoDB +- `--rename` for enum values, namespaces and fields inside value objects. This slice builds the statement support for value objects, which 4b reuses. +- Moving a model to another namespace (`alter table set schema` on Postgres). +- `--convert` on a variant whose discriminator value changed. +- `--delete` on an enum value: sets it to NULL where the field is optional, otherwise refuses. +- `--delete` on a variant stored in its base's table: deletes its rows and drops its columns. The refusal says rows are deleted. +- `--delete ` answers every question for the models in it. -**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo-renames/` +**Risk to settle in its spec:** a value object can contain itself (ADR 178's `NavItem.children`). Renaming a field inside it means rewriting to any depth, which a fixed SQL JSON expression or MongoDB update cannot do. -**Outcome.** The MongoDB planner carries out `--rename` for models (collection rename) and fields (document rewrite), with the validator and indexes kept consistent, through both commands. Removing a field becomes data loss, answered by `--delete` with an `$unset`. Document rewrites run under `db update`. The temporary capability member `renameStatements: { refused: true; keepDataByHand }`, its three CLI reads and its doc mention are deleted. Statements name MongoDB fields the same way they name SQL fields. +**Hands to:** close-out for Postgres and SQLite. -**Builds on.** Slice 2. - -**Hands to.** A MongoDB planner that takes resolved statements, for 4b. - -### Slice 4b — `--convert`, `--backfill` and value object renames on MongoDB +### 4b — `--convert`, `--backfill` and value object renames on MongoDB **Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo-convert-backfill/` -**Outcome.** The same two verbs as 3a on MongoDB, scaffolding a data transform with a placeholder. Value object field renames rewrite subdocuments, including lists, dictionaries and unions, using the statement surface from 3b. A validator change that makes a field required on a populated collection is no longer silent: it is answered with `--backfill`, or the command says that existing documents need the field. - -**Builds on.** Slices 3a, 3b and 4a. - -**Hands to.** Project close-out for MongoDB. - -### Alongside — A syntax for naming storage objects - -Will asked (2026-10-07) for a critical discussion and a survey of how established tools name tables and other storage objects on the command line, before any syntax is chosen for the `@@map`-only rename gap (`deferred.md`). The write-up goes to Will for a decision. Building the chosen syntax is not in this project unless Will adds it. - -## Stretch goal — interactive statements when a human runs the command - -Recorded 2026-10-06 at the operator's request. This is the eventual direction, and the slices build with it in mind so that it drops in on top of their mechanisms; it is not a slice of its own, and it is built once the mechanisms it needs exist (after slice 2 at the earliest). - -When `migration plan` or `db update` detects a destructive operation and no statement covers it, the command today refuses and prints the statements that would resolve it. The stretch goal: when the command is run by a human in a terminal (stdin and stdout are a TTY, and no `--json` or similar non-interactive flag is set), the refusal becomes an interactive prompt that asks, per destructive operation, what the user means: rename it to a name they type, or delete it. Each answer is exactly the statement the refusal would have printed, applied in order, and the plan then proceeds as if the statements had been given on the command line. In a script, an agent, or CI (no TTY) the behaviour stays as it is: refuse and print the statements. - -Rules that carry over unchanged: the prompt never guesses or proposes a rename candidate; it asks. The statements are not recorded anywhere; the migration file or the database is the record. The consent model of slice 2 (`--delete` per operation) is what the "delete" answer maps to. +- The same two flags as 3a, writing a data transform with a placeholder. +- Making a field required on a collection with documents is no longer silent. It is answered with `--backfill`, or the command says the existing documents need the field. +- Renaming a field inside a value object rewrites the embedded documents, including lists, dictionaries and unions. -What the slices do now so this drops in later (see the spec's cross-cutting requirement 12): +**Hands to:** close-out for MongoDB. -- The refusal is a structured value first and text second: a list of destructive operations, each with the entity it touches in domain coordinates and the statements that would resolve it, rendered to the error message by the command. The prompt consumes the same list. -- Statement text is parsed, resolved and applied through one function the command calls, which takes the statement strings and the two contracts and does not care whether the strings came from the command line or from an answer typed at a prompt. Statements can therefore be added after the command has started and the plan re-run. -- The per-operation consent question that `db update` already asks in a terminal is the seed of this prompt, so slice 2 keeps it as a per-operation question rather than folding it into a yes/no over the whole plan. +### Alongside — a syntax for naming storage objects -## Sequencing +There is no statement for a model that keeps its name but changes its table (`@@map`). Will asked on 2026-10-07 for a critical discussion and a look at how established tools name tables on the command line before any syntax is chosen. `--rename table/users:app_users` was an off-the-cuff example, not a decision. -- **Merged:** slice 1 → slice 2. -- **Parallel now:** 3a and 4a. -- **Then:** 3b after 3a; 4b after 3b and 4a. +- The write-up goes to Will for a decision. +- It does not block close-out. +- Building the chosen syntax is not in this project unless Will adds it. -## Dependencies +## Delivered: the interactive prompt -- Nothing external. The shelved prisma/orm#30570 is a source to copy from, not a dependency; its branch stays as a draft. -- Users get the feature through the next published CLI release after slice 2 merges; slices 1 and 2 together are the minimum that changes behaviour a user sees. +Recorded on 2026-10-06 as the stretch goal: when a human runs the command in a terminal, the refusal becomes a question per operation, and the answer is the same statement the flags take. Slice 2 built it on the CLI engine's statement prompt (prisma/prisma-cli#337). Scripts, agents and CI get the refusal that names the flags. ## Close-out (required) - [ ] Verify every project DoD item in [`spec.md`](./spec.md) +- [ ] Final retro - [ ] Write the ADR and amend ADR 001, ADR 028 and the Data Contract and Migration System subsystem docs - [ ] Migrate long-lived docs into `docs/` - [ ] Strip repo-wide references to `projects/migration-statements/**` diff --git a/projects/migration-statements/slices/convert-backfill/spec.md b/projects/migration-statements/slices/convert-backfill/spec.md index 48bfe8e4a426..e48b659f09db 100644 --- a/projects/migration-statements/slices/convert-backfill/spec.md +++ b/projects/migration-statements/slices/convert-backfill/spec.md @@ -1,10 +1,10 @@ -# Slice spec — `--convert` and `--backfill` write the placeholder migration only when asked +# Slice 3a — `--convert` and `--backfill` -**Project:** [`projects/migration-statements/`](../../spec.md) · **Slice 3a** · **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477), [TML-3517](https://linear.app/prisma-company/issue/TML-3517) · **Branch:** `tml-3477-convert-backfill` · **Builds on:** slice 2 (merged). +**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477), [TML-3517](https://linear.app/prisma-company/issue/TML-3517) · **Branch:** `tml-3477-convert-backfill` · **Builds on:** slice 2 (merged) · **Project:** [`../../spec.md`](../../spec.md) -## At a glance +## In one example -A user changes `User.age` from `String` to `Int` on Postgres: +`User.age` changes from `String` to `Int` on Postgres. ```text $ prisma migration plan --name age-to-int @@ -17,7 +17,7 @@ $ prisma migration plan --name age-to-int --convert User.age ⚠ migration.ts has a placeholder: replace it with the conversion, then run the file to re-emit. ``` -`migration.ts` then holds one placeholder, where the conversion goes: +`migration.ts` then has one slot to fill: ```ts this.alterColumnType({ @@ -28,96 +28,106 @@ this.alterColumnType({ }), ``` -The user writes `using: '"age"::integer'` (or any SQL expression), runs the file, and gets an applicable migration. On SQLite the placeholder sits in the table rebuild's copy step for that column. +The user writes `using: '"age"::integer'`, runs the file, and applies it. -Adding a required `email` field to an existing `User` table plans a temporary default by default: the column is added with a fill value and the default is dropped, and nothing is asked. To write the fill yourself, run `prisma migration plan --backfill User.email`. That writes the nullable column, a data transform with a placeholder, and `SET NOT NULL`. +## What changes for users -Today `migration plan` writes these placeholders on its own for every type change and every required field on an existing table. Users who didn't want them get a migration they can't apply until they edit it. After this slice it writes them only when asked. +| Change in the schema | Today | After this slice | +| --- | --- | --- | +| Type change that loses data | Placeholder migration, written without asking | Refused. Answer `--convert` (fill in the conversion) or `--delete` (accept the loss) | +| Type change that keeps every value (`int4` → `int8`) | Planned, nothing asked | Same | +| Required field added to an existing table | Placeholder migration, written without asking | Filled with a temporary value (`''`, `0`, …), nothing asked. `--backfill` writes a slot to fill instead | +| Optional field made required | Placeholder migration, written without asking | `SET NOT NULL`, nothing asked. Fails if NULLs exist, and says how many. `--backfill` writes a slot to fill instead | +| Any of the above through `db update` | No placeholders | Same. `--convert` and `--backfill` are refused: `db update` has no file to fill | -## Chosen design +SQLite works the same way. Two SQLite gaps close on the way: +- `db update` can add a required field to a table with rows. Today it fails. +- A type change gets a slot for the conversion. Today SQLite just rebuilds the table. -### The two verbs +## Decisions to check -- **`--convert `** answers a type change that would lose data. It is accepted by `migration plan` only. It names the field by its destination name, like the new side of a rename: after `--rename User.age:User.years`, write `--convert User.years`. It needs the origin contract, like `rename`, and fails with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN` without one. It must name a field that exists on both sides and whose type changes. -- **`--backfill `** is an opt-in that answers no question. It is accepted by `migration plan` only. It names a field that is required in the destination and is either new on an existing table or optional in the origin. Anything else is an error that names what was found. -- `db update` declares both verbs only to refuse them, with `MIGRATION.STATEMENT_NEEDS_MIGRATION_FILE`. The error says `db update` has no migration file to fill and names `migration plan`. It doesn't fall through to the engine's generic "no flag registered" error, which doesn't explain the rule. The shared verb table in `orm/statement-verbs.ts` declares the verbs per command, so neither command accepts a verb by accident. +1. **Temporary values without asking.** Without `--backfill`, `migration plan` writes fill values such as `''` or `0` into existing rows. The project spec already decided this. It is the biggest behaviour change in the slice. +2. **`alterColumnType` changes shape.** Its options become `{ table, column, type, using? }`. Today they expose internal names. Hand-written migrations that call it need the upgrade instructions. +3. **A type change answered with `delete` is reported honestly.** The output says `delete values of field "User.age" (type change)`, not "delete field". Slice 2's QA found the old wording misleading. -### What the planners write +## The two flags -| Diff | No statement | `--convert` | `--backfill` | -| --- | --- | --- | --- | -| Type change the target knows is safe (`int4`→`int8`) | direct alter, `widening`, nothing asked | error: nothing to convert | — | -| Any other type change | refused: question offers `convert` and `delete`. `--delete` plans the direct alter, `destructive` | Postgres: `alterColumnType` with `using: placeholder(…)`. SQLite: `recreateTable` whose copy step takes `placeholder(…)` for the column. Class `data` | — | -| Required field added to an existing table | temporary-default recipe on both targets, both commands, `additive` | — | nullable add, `dataTransform` with placeholders, then `SET NOT NULL` (SQLite: a tightening rebuild) | -| Optional field made required | direct `SET NOT NULL` (SQLite: tightening rebuild), `widening`. Fails at apply if NULLs exist, with advice that names `--backfill` | — | `dataTransform` with placeholders, then `SET NOT NULL` | +**`--convert `** +- Answers a type change that would lose data. +- `migration plan` only. +- Names the field by its new name. After `--rename User.age:User.years`, write `--convert User.years`. +- Needs the old contract, like `--rename`. -The four automatic placeholder strategies (Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`) fire only for a field a statement names. Statements reach them as destination coordinates through the strategy context. They change no names in the working schema, so they don't go through the rename path in `planStatements`. A converted type change carries no separate NULL-handling transform; the conversion expression can map NULL itself. +**`--backfill `** +- Optional. Answers no question. +- `migration plan` only. +- The field must be required now, and either new on an existing table or optional before. -A `delete` answer to a type-change question keeps planning the direct alter, and the statement is reported as giving up the field's values in a type change (`delete values of field "User.age" (type change)`), not as deleting the field; slice 2's manual QA found the old wording misleading (F14). A converted alter is classed `data`, because the user writes its data step. So it isn't a `destructive` operation and leaves `dataLoss`. A typed `convert` at the prompt re-plans like a typed rename, and the type-change question isn't asked again. +**On `db update`,** both flags fail with `MIGRATION.STATEMENT_NEEDS_MIGRATION_FILE`, which points to `migration plan`. -### The questions know why data is lost +## What the planners write -Each `dataLoss` entry gains `loss: 'drop' | 'typeChange'`, set by each target's operation-subjects function. The CLI stops guessing from whether the destination still has the field. A `typeChange` question offers `convert` and `delete` in `migration plan`, and `delete` only in `db update`. A `drop` question offers `rename` and `delete`, as today. The re-plan loop (`askPlanQuestions`) carries every planned statement, not only renames. +| Case | Postgres | SQLite | +| --- | --- | --- | +| `--convert` | `alterColumnType` with `using: placeholder(…)` | Table rebuild whose copy step has `placeholder(…)` for the column | +| `--backfill`, new field | Add nullable column, data transform with placeholders, `SET NOT NULL` | Add nullable column, data transform, tightening rebuild | +| `--backfill`, optional made required | Data transform with placeholders, `SET NOT NULL` | Data transform, tightening rebuild | +| New required field, no flag | Add with temporary default, drop the default | Rebuild that copies the temporary value into the new column | -`ResolvedMigrationStatement` gains a `kind`, which is `rename`, `convert` or `backfill`. Every reader that assumes a rename now switches on `kind` first. One example is the SQL family's `StatementPlanner`, which today plans any non-model statement as a column rename. MongoDB keeps refusing statements it can't carry out, until slices 4a and 4b. +## How the pieces fit -### The temporary default becomes a real migration call +- **Questions know why data is lost.** Each data-loss entry says `drop` or `typeChange`. A `typeChange` question offers `convert` and `delete`; a `drop` question offers `rename` and `delete`. +- **A converted change stops being data loss.** It is classed `data`, because the user writes the conversion. So it is not asked about again. +- **A typed `convert` at the prompt re-plans,** the same way a typed rename does. +- **Placeholders appear only when a flag asks.** The four strategies that write them today run only for a field a flag names. +- **The temporary default becomes a real call in `migration.ts`.** Today it renders as an empty `rawSql(...)`, so re-running the file does not reproduce `ops.json`. +- **The NOT NULL failure names the NULLs (TML-3517).** It says how many rows hold NULL in which column, and suggests filling them, keeping the field optional, or `--backfill`. It no longer talks about schema drift. -The Postgres temporary-default and direct-add calls render as `rawSql({ id, label, operationClass })` with no SQL. Re-running such a `migration.ts` doesn't reproduce `ops.json` (cross-cutting requirement 7). They now render as a facade call that builds the same operation. SQLite gets the recipe as a rebuild in which the new column's copy value is a literal. This needs a new per-column copy-expression option on `recreateTable`, which `--convert` also uses, and a SQLite fill-value map (integer `0`, real `0`, text `''`, blob `X''`, and the values its boolean and date-time codecs store). It is refused on the same primary-key, unique and foreign-key conditions as Postgres. This closes the deferred item "SQLite `db update` cannot add a required field to a table that has rows". +## Why one pull request -### `alterColumnType` takes plain options - -The Postgres facade's `alterColumnType` options are reshaped so the rendered call shows a readable shape with an optional `using`, and no internal names (`qualifiedTargetType`, `formatTypeExpected`, `rawTargetTypeForLabel`). The `using` slot renders as code, so it can hold `placeholder(…)`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result the CLI already handles. Upgrade instructions are recorded for hand-written migrations. - -### The NOT NULL failure names the NULLs (TML-3517) - -When `SET NOT NULL` or a SQLite tightening rebuild meets NULL values, the runner reports how many rows hold NULL in which column. It says to fill them, make the field optional, or plan with `--backfill`. `MigrationRunnerFailure` gains a `fix`, and the `db update`, `db init` and `migrate` error mappers use it instead of the fixed advice about schema drift. - -## Coherence rationale - -Everything in this slice serves one change: the placeholder migration is written only when a statement asks for it. The verbs, the loss kind, the statement `kind`, the temporary default becoming real on both targets, the `alterColumnType` reshape and the NULL advice are each what that change needs to work on Postgres and SQLite. The reviewer reads one rule and its consequences. +Everything here serves one rule: a placeholder migration is written only when a flag asks for it. The reviewer reads that rule and its consequences. ## Scope -**In:** -- `--convert` and `--backfill` on `migration plan`, for Postgres and SQLite, refused on `db update`. -- The loss kind on `dataLoss` entries, and the statement `kind`. -- Ending automatic scaffolding. -- The temporary default as a real call on both targets, including the SQLite recipe. -- The `alterColumnType` reshape. +**In** +- Both flags on Postgres and SQLite; refused on `db update`. +- Ending automatic placeholders. +- The temporary default on both targets, as a real call. +- The `alterColumnType` reshape, with upgrade instructions. - TML-3517. -- The project spec's wording: "an existing table", not "a non-empty table", since `migration plan` can't see rows. -- An amendment to ADR 200, which says only a data transform holds a placeholder. -- The Migration System doc's claim that the Postgres planner writes no placeholder data transforms, which the code contradicts. -- Docs: Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`. -- An app upgrade fragment: type changes ask, scaffolds are opt-in, and the temporary default applies under `migration plan`. -- Upgrade instructions for the facade change. +- Docs: + - ADR 200: it says only a data transform can hold a placeholder. + - The Migration System doc: it says the Postgres planner never writes placeholders. + - Migration System § Statements, the error reference, the CLI README, `skills/prisma-8/references/migrations.md`. +- An app upgrade fragment for the behaviour changes. +- The project spec: "an existing table", not "a non-empty table". `migration plan` cannot see rows. -**Deliberately out:** -- `--convert` on a variant (slice 3b). -- MongoDB (slice 4b). -- A temporary default for a field in a new unique or foreign key, which stays refused as today. -- Detecting rows offline. +**Out** +- `--convert` on a variant: slice 3b. +- MongoDB: slice 4b. +- A temporary value for a new unique, primary-key or foreign-key column. It stays refused, as on Postgres today. -## Pre-investigated edge cases +## Edge cases already known | Case | Handling | | --- | --- | -| A temporary value in a unique column fails on the second row | The recipe is refused for new unique, primary-key and foreign-key columns, as on Postgres today | -| A convert after a rename in the same run | The convert uses the destination name, which the diff issues already use | -| A placeholder operation has no position until re-emit | Applied-statement positions for a convert are empty until the file is re-run, as for every placeholder today | - -## Slice-specific done conditions - -- The project DoD's convert and backfill lines pass on Postgres and SQLite, reading "an existing table". -- Re-running every `migration.ts` the planners write in this slice's tests reproduces its `ops.json`. +| A temporary value in a unique column fails on the second row | Refused for new unique, primary-key and foreign-key columns | +| `--convert` after a rename in the same run | Uses the new name, which the planner already uses | +| A placeholder operation has no position until the file is re-run | The convert's reported positions are empty until then, as for every placeholder today | -## Open questions +## Done when -None. +- The project's convert and backfill checks pass on Postgres and SQLite. +- Re-running every `migration.ts` that this slice's tests write reproduces its `ops.json`. -## References +## Implementation notes +- `dataLoss` entries gain `loss: 'drop' | 'typeChange'`, set by each target's operation-subjects function. `dataLossQuestion` stops inferring it from whether the destination has the field. +- `ResolvedMigrationStatement` gains `kind: 'rename' | 'convert' | 'backfill'`. Every reader that assumes a rename switches on `kind` first, including the SQL `StatementPlanner`, which today plans any non-model statement as a column rename. +- `askPlanQuestions` re-plans with every planned statement, not only renames. +- Convert and backfill reach the strategies as destination coordinates through the strategy context. They do not go through `planStatements`, because they rename nothing. +- Strategies gated: Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`. +- `AlterColumnTypeClass` gains `data`. The `using` slot renders as code, imports `placeholder`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result. +- `recreateTable` gains a per-column copy expression, used by convert and by the SQLite temporary value. SQLite gets a fill-value map: integer `0`, real `0`, text `''`, blob `X''`, and what its boolean and date-time codecs store. +- `MigrationRunnerFailure` gains `fix`. The `db update`, `db init` and `migrate` error mappers prefer it. - Grounding report (local, gitignored): `wip/grounding-3a.md`. -- [`../../design-notes.md`](../../design-notes.md): the scaffold rule and the convert and backfill rows. -- [ADR 200](../../../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md): the placeholder flow. diff --git a/projects/migration-statements/slices/engine-statement-prompt/spec.md b/projects/migration-statements/slices/engine-statement-prompt/spec.md index c2b105039d2d..32929d9ea75c 100644 --- a/projects/migration-statements/slices/engine-statement-prompt/spec.md +++ b/projects/migration-statements/slices/engine-statement-prompt/spec.md @@ -105,6 +105,6 @@ None. Will decided on 2026-10-07: consents carry a verb; the answer is a free-te ## References -- Project spec [`../../spec.md`](../../spec.md) decisions 2 and 6, requirement 12; [`../../plan.md`](../../plan.md) § Stretch goal. +- Project spec [`../../spec.md`](../../spec.md) decisions 2 and 6, requirement 12; [`../../plan.md`](../../plan.md) § Delivered: the interactive prompt. - prisma-cli: `packages/cli-engine/src/execution/prompts.ts`, `shared-flags.ts`, `clack-renderer.ts`, `src/context.ts`; `docs/product/cli-style-guide.md` § consent; ADR 0004 (engine version pinning). - `wip/slice-2-consent-options.md`: the option evaluation Will chose from. diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md index 0952fd89bb7c..0344713b340b 100644 --- a/projects/migration-statements/slices/mongo-renames/spec.md +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -1,10 +1,10 @@ -# Slice spec — MongoDB carries out renames and deletes of models and fields +# Slice 4a — Renames and deletes on MongoDB -**Project:** [`projects/migration-statements/`](../../spec.md) · **Slice 4a** · **Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Branch:** `tml-3478-mongo-renames` · **Builds on:** slice 2 (merged). +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Branch:** `tml-3478-mongo-renames` · **Builds on:** slice 2 (merged) · **Project:** [`../../spec.md`](../../spec.md) -## At a glance +## In one example -A MongoDB user renames the `Profile` model to `User` (collection `profiles` to `users`), changes `name` to `fullName`, and removes the `nickname` field: +A MongoDB user renames model `Profile` to `User`, renames field `name` to `fullName`, and removes field `nickname`. ```text $ prisma migration plan --name tidy --rename Profile:User --rename User.name:User.fullName --delete User.nickname @@ -17,7 +17,7 @@ $ prisma migration plan --name tidy --rename Profile:User --rename User.name:Use Update validator on users widening ``` -`migration.ts` carries `renameCollection(...)`, `renameField(...)` and `unsetField(...)` calls, and running it reproduces `ops.json`. `prisma db update` with the same statements does the same against the live database. Today MongoDB refuses every `--rename` and prints steps to run by hand in `mongosh`, and removing a field asks nothing, even though every document that still holds the field then fails each later update against the closed validator. +`db update` with the same flags does the same against the live database. Without `--delete User.nickname`, both commands refuse, as they do on SQL: @@ -28,70 +28,111 @@ Without `--delete User.nickname`, both commands refuse, as they do on SQL: → Pass --delete User.nickname ``` -## Chosen design +## What changes for users -### Statements name fields by their contract key +| Change in the schema | Today | After this slice | +| --- | --- | --- | +| Model renamed, `--rename` given | Refused, with steps to run by hand in `mongosh` | Collection renamed; documents, indexes and validator kept | +| Field renamed, `--rename` given | Refused, with steps to run by hand | Every document's field renamed; indexes moved to the new name | +| Field removed | Validator updated, nothing asked. Every document that still holds the field then fails each update | Asked about. `--delete` removes the field from every document; `--rename` keeps its values | +| Model removed | Asked about; `--delete` drops the collection | Same, and `--rename` is offered too | -A statement names a field as the contract and the generated client name it: the key in the model's `fields`. On SQL that key is the field name in the schema; on MongoDB the contract keys fields by the stored name (`id @map("_id")` is `User._id`), and the client uses the same name. The resolver already behaves this way, so nothing changes in it. On MongoDB a change to `@map` alone is a field rename in contract terms and is stated with `--rename`; a change to the schema field name alone changes nothing stored and needs no statement. This settles the deferred item "Which field name a MongoDB statement uses" and is recorded in the Migration System doc. +## Decisions to check -### Collection rename +1. **Removing a field now asks.** Today it is silent, but MongoDB's validator then rejects every update to a document that still holds the field. +2. **Fields are named as stored.** `id @map("_id")` is `User._id` in a statement. The contract and the generated client already use that name. Naming the schema field instead would change the contract and its hashes. +3. **The rewrites run under `db update`.** They are classed by what they do to data: a rename is `widening`, a removal is `destructive`. `db update` refuses the `data` class, so classing them `data`, as ADR 188 says today, would block `db update`. This slice amends ADR 188. -A new DDL operation, `renameCollection`, added through the MongoDB stack the way the other collection operations are: query AST command and visitor, wire command and lowering, driver, operation serializer, preview text, factory and call class. Precheck: the source collection exists and the target does not. Postcheck: the target exists. Class `widening`. A model rename renames the collection in the working copy of the origin schema before the diff, so the indexes and validator that MongoDB keeps through a rename are not dropped and recreated. +## How it works -### Field rename and field removal rewrite the documents +**Collection rename** +- A new `renameCollection` operation. +- Before: the source exists and the target does not. After: the target exists. +- The rename happens before the diff, so MongoDB keeps the indexes and validator, and nothing is dropped and recreated. -- A field rename plans `renameField(collection, from, to, { filter })`: one `updateMany` with `$rename`, run with validation bypassed. Filter: documents that hold the old field, and for a variant's field also the variant's discriminator value. Precheck: no matching document already holds the new field, since `$rename` would overwrite it. Postcheck: no matching document still holds the old field, so a re-run skips it. Class `widening`. -- Removing a field from a model is data loss, matching the spec's rule that dropping a field is destructive. The planner plans `unsetField(collection, field, { filter })`, an `updateMany` with `$unset`, class `destructive`, and reports a `field` subject. The question offers `rename` and `delete`, as on SQL. A field removed together with its collection gets no unset; the collection drop covers it. -- Order within a plan: collection renames, collection creates, index drops, document rewrites (renames and unsets), index creates, validator changes, option changes, collection drops. Rewrites come after index drops so a unique index on the old field does not fail once documents lose it, and before index creates so a unique index on the new field sees the moved values. Validation is bypassed for the rewrite because the origin validator requires the old field and the destination validator may require a field the same plan adds. Plans without statements and without field removals keep their current operations and order. -- The rewrite command gains a `bypassDocumentValidation` option through its wire form and the driver. +**Field rename** +- One `updateMany` with `$rename`, written as `renameField(...)` in `migration.ts`. +- Touches only documents that hold the old field. For a variant's field, only documents with that variant's discriminator value. +- Before: no document already holds the new name, which `$rename` would overwrite. After: no document holds the old name, so a re-run skips it. -### Document rewrites run under `db update` +**Field removal** +- One `updateMany` with `$unset`, written as `unsetField(...)`. +- No unset when the whole collection is dropped. -`db update` allows `additive`, `widening` and `destructive`, not `data`, so a rewrite classed `data` would be refused there. The rewrites above carry the class of what they do to data (`widening` for a rename, `destructive` for an unset), and the MongoDB runner and operation serializer tell a rewrite from a DDL command by its shape, not by `operationClass === 'data'`. `data` stays the class of transforms a user writes or a scaffold produces; `db update` keeps excluding it. Slice 3b's SQL JSON rewrites follow the same rule. This follows the Migration System doc's rule that only the planner's own classification promises an operation loses data exactly when it is `destructive`, and it changes ADR 188, which says every MongoDB data transform is `data`: the slice amends ADR 188, and ADR 264's line that MongoDB has no rename operation. +**Order in a plan** -### What is refused +1. Collection renames +2. Collection creates +3. Index drops +4. Document rewrites (renames and removals) +5. Index creates +6. Validator changes +7. Option changes +8. Collection drops -The MongoDB planner refuses, as a `statementRejected` conflict naming the reason: a rename of `_id`; a collection rename whose target exists in the working schema; a rename of a field stored on one side only; and any statement when the control policy is not `managed`, as SQL does. A document that already holds the new name fails at apply time by precheck, before anything is written. +- Rewrites run after index drops, so a unique index on the old field doesn't fail once documents lose the field. +- They run before index creates, so a unique index on the new field sees the moved values. +- Rewrites skip validation. The old validator requires the old field, and the new one may require a field the same plan adds. +- Plans with no statements and no removed fields come out exactly as today. -### The temporary refusal goes +**Refused, with the reason named** +- Renaming `_id`. +- Renaming a collection onto one that already exists. +- Renaming a field that is stored on only one side. +- Any statement when the model's control policy is not `managed`, as on SQL. -`TargetMigrationsCapability.renameStatements` and `keepDataByHand` are deleted: the member in `control-migration-types.ts`, MongoDB's setting in `control-target.ts`, `keepDataByHand` and `keepTheData` in `mongo-planner.ts`, `keepDataByHandFor` and its parameter in `plan-questions.ts`, its reads in `db-run.ts` and `migration-plan.ts`, and the Statements paragraph that describes it in the Migration System doc. The unrelated `renameStatements()` helper in `statement-text.ts` stays. The member shipped in 8.0.0-rc.17, so an extension upgrade fragment tells target authors to remove it. +**The temporary refusal goes.** MongoDB's `renameStatements: { refused: true, keepDataByHand }` setting, and all the code that reads it, is deleted. -## Coherence rationale +## Why one pull request -One reviewer can hold this: it is one family learning the two verbs the SQL targets already carry out, through one new DDL operation and one new rewrite shape, with the refusal member removed as the visible proof. The collection rename's six-package plumbing is mechanical and mirrors existing collection operations. +It is one database learning two flags the SQL targets already support. It adds one new operation and one new rewrite. The collection rename touches six packages, but the change is the same in each and copies the existing collection operations. ## Scope -**In:** amendments to ADR 188 (planner-written rewrites carry the class of their effect) and ADR 264 (MongoDB has a collection rename); model rename (collection rename); top-level field rename, including a variant's field; field removal as data loss with `$unset` under `--delete`; both commands; the `bypassDocumentValidation` option; runner and serializer telling rewrites apart by shape; deleting `renameStatements`; docs (Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`); an app upgrade fragment (MongoDB renames work; removing a field now asks) and an extension upgrade fragment (the capability member is gone). - -**Deliberately out:** -- Value object field renames on MongoDB. They need the framework's value object statement surface (statement entity, subject kind, grammar, resolver), which slice 3b builds for SQL JSON rewrites, and a recursive update pipeline for lists, dictionaries and unions. Moved to slice 4b. -- `--convert` and `--backfill` on MongoDB, and the required-field validator gap: slice 4b. -- Batching large rewrites. One `updateMany` per field, as today's hand-written transforms do. - -## Pre-investigated edge cases +**In** +- Model rename, field rename (including a variant's field), field removal. +- Both commands. +- Deleting `renameStatements` and `keepDataByHand`. The unrelated `renameStatements()` helper in `statement-text.ts` stays. +- ADR amendments: + - ADR 188: planner-written rewrites carry the class of their effect. + - ADR 264: MongoDB now has a collection rename. +- Docs: Migration System § Statements, the error reference, the CLI README, `skills/prisma-8/references/migrations.md`. +- Upgrade fragments: + - App: renames work; removing a field now asks. + - Extension: the capability member is gone. It shipped in 8.0.0-rc.17. + +**Out** +- Renaming a field inside a value object. It needs statement support that slice 3b builds. Moved to 4b. +- `--convert`, `--backfill`, and the validator problem when a field becomes required: slice 4b. +- Batching rewrites on large collections. + +## Edge cases already known | Case | Handling | | --- | --- | -| Slice 1 manual QA (F3): a unique index on the old field fails the rewrite once two documents lose it; one on the new field fails if created first | Order above: index drops, rewrite, index creates | -| The origin validator requires the old field and forbids the new one | Rewrite runs with validation bypassed | -| Two variants in one collection store a field with the same name | Rewrite filters on the discriminator value | -| The runner is not transactional | Postcheck makes a re-run skip a finished rewrite | -| `bypassDocumentValidation` needs a privilege on hosted clusters | Runner failure names the privilege | -| TML-2447 (open): the planner creates separate collections and incomplete validators for variants that share a collection | The first dispatch checks how variants are stored today before the variant field rename is built on it | - -## Slice-specific done conditions - -- The project DoD's MongoDB journey without the value object part: a model rename and a field rename (with a unique index on the field) and a field removal under `--delete`, through `migration plan` then `migrate` and through `db update`, on collections with documents. Afterwards the documents are under the new names, the removed field is gone from every document, every document accepts an update, a further plan is empty, and `db verify --schema-only` is clean. -- `git grep -n "keepDataByHand\|TargetMigrationsCapability\['renameStatements'\]"` finds nothing. - -## Open questions - -None. - -## References - -- Grounding report for this slice (local, gitignored): `wip/grounding-4a.md`. -- [`../../deferred.md`](../../deferred.md) — "Which field name a MongoDB statement uses". -- Slice 1 manual QA, F3: [`../../manual-qa-reports/2026-10-07-qa-opus.md`](../../manual-qa-reports/2026-10-07-qa-opus.md). +| A unique index on the old field fails the rewrite once two documents lose it (slice 1 QA, F3) | Index drops run before the rewrite | +| A unique index on the new field fails if created before the rewrite | Index creates run after it | +| The old validator requires the old field and forbids the new one | The rewrite skips validation | +| Two variants in one collection store a field with the same name | The rewrite filters on the discriminator value | +| MongoDB migrations are not transactional | The after-check lets a re-run skip a finished rewrite | +| Skipping validation needs a privilege on hosted clusters | The failure names the privilege | +| TML-2447 (open): variants that share a collection get their own collection and an incomplete validator | The first dispatch checks how variants are stored today, before building on it | + +## Done when + +- On collections with documents, through `migration plan` then `migrate` and through `db update`: a model rename, a field rename with a unique index, and a field removal with `--delete`. Afterwards: + - the documents are under the new names; + - the removed field is gone from every document; + - every document accepts an update; + - a further plan is empty; + - `db verify --schema-only` is clean. +- `git grep -n "keepDataByHand"` finds nothing. + +## Implementation notes + +- `renameCollection` goes through the stack like the other collection operations: query AST command and visitor, wire command and lowering, driver, operation serializer, preview text, factory, call class. +- The raw update command gains `bypassDocumentValidation`, through its wire form and the driver. +- The runner and the operation serializer tell a rewrite from a DDL command by its shape (`run` against `execute`), not by `operationClass === 'data'`. +- Removed fields are reported as `field` subjects in `dataLoss`. +- Code to delete: the member in `control-migration-types.ts`; MongoDB's setting in `control-target.ts`; `keepDataByHand` and `keepTheData` in `mongo-planner.ts`; `keepDataByHandFor` and its parameter in `plan-questions.ts`; the reads in `db-run.ts` and `migration-plan.ts`; the paragraph in the Migration System doc. +- Grounding report (local, gitignored): `wip/grounding-4a.md`. diff --git a/projects/migration-statements/spec.md b/projects/migration-statements/spec.md index 783bd6fb856c..6603c7b15683 100644 --- a/projects/migration-statements/spec.md +++ b/projects/migration-statements/spec.md @@ -45,7 +45,7 @@ $ prisma migration plan --name age-to-int --convert User.age The reasoning and the alternatives rejected are in [`design-notes.md`](./design-notes.md). 1. **Data loss is refused by default, in both commands.** An operation is destructive only when it loses data: dropping a model or field, deleting a value rows still carry, or changing a field's type in a way the database cannot keep. Drops of indexes, constraints, checks, policies and defaults lose nothing and are widening. `db update` already refuses through its consent prompt; `migration plan` gains the same refusal for every plan, not only auto-baselines. -2. **Intent is stated on the command line, up front.** There is no prompt that guesses whether a drop is a rename, and no detection of candidate pairs. The refusal is where the user learns what to type, and it prints each statement ready to paste. The eventual direction (see [`plan.md`](./plan.md) § Stretch goal and requirement 12 below) is that when a human runs the command in a terminal, the refusal becomes an interactive prompt that asks the user to state what each destructive operation is; the answers are the same statements. "No prompt" was the initial design stance, not a rule; the rule is that nothing is guessed. +2. **Intent is stated on the command line, up front.** There is no prompt that guesses whether a drop is a rename, and no detection of candidate pairs. The refusal is where the user learns what to type, and it prints each statement ready to paste. The eventual direction (see [`plan.md`](./plan.md) § Delivered: the interactive prompt and requirement 12 below) is that when a human runs the command in a terminal, the refusal becomes an interactive prompt that asks the user to state what each destructive operation is; the answers are the same statements. "No prompt" was the initial design stance, not a rule; the rule is that nothing is guessed. 3. **Four verbs, four nouns.** The verbs are `rename`, `delete`, `convert` and `backfill`. The nouns are namespace, model, field and value, addressed by contract coordinates such as `User`, `User.name`, `Status.ARCHIVED`, `Bug` and `Address.street`, with the namespace prefixed when the contract has more than one. Rename and delete apply to all four nouns; a model rename across namespaces is a move. Convert applies to fields and to variants whose discriminator value changed. Backfill applies to fields. 4. **Statements are intent, not hints.** A statement that cannot be resolved against the origin and destination contracts, or cannot be applied, is an error and nothing is planned. There is no "spent" or "ignored" statement. A statement that resolves and applies but changes nothing in storage, such as renaming a value object or a relation field, is reported as applied with no operations. 5. **Everything is in the application domain.** The command line never carries a table, column, collection, SQL expression or literal value. The framework resolves coordinates against the two contracts; the family turns the resolved statement into storage operations. @@ -91,7 +91,7 @@ The reasoning and the alternatives rejected are in [`design-notes.md`](./design- 9. **Scaffolds reuse the placeholder flow.** `--convert` and `--backfill` produce the existing placeholder-bearing migration, with the placeholder in the operation that consumes the value. The pending state, the empty `ops.json` and the re-emit path are unchanged. 10. **Destructive means data loss.** Drops of indexes, unique and foreign-key constraints, checks, row-level-security policies, defaults and native enum types, and disabling row-level security, are widening on every target. A type change the family knows to be widening needs no statement. 11. **Framework stays family-agnostic.** The statement grammar, parser and resolver carry no family or target vocabulary, and each family documents what every verb means for its storage. -12. **Built for the interactive prompt to drop in.** The refusal is a structured list of destructive operations, each with its entity in domain coordinates and the statements that would resolve it; the error text is rendered from that list. Statement strings are parsed, resolved and applied through one entry point that does not know whether they came from the command line or from an answer typed at a prompt, so statements can be added after the command has started. See [`plan.md`](./plan.md) § Stretch goal. +12. **Built for the interactive prompt to drop in.** The refusal is a structured list of destructive operations, each with its entity in domain coordinates and the statements that would resolve it; the error text is rendered from that list. Statement strings are parsed, resolved and applied through one entry point that does not know whether they came from the command line or from an answer typed at a prompt, so statements can be added after the command has started. See [`plan.md`](./plan.md) § Delivered: the interactive prompt. ## Transitional-shape constraints From e86e483a89c9fa3034df73501a162e53d066da35 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 09:43:28 +0200 Subject: [PATCH 05/26] TML-3478: the 4a spec records Will's decisions on field removal, names, classes and variants Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/deferred.md | 6 ++++++ .../migration-statements/slices/mongo-renames/spec.md | 10 ++++++---- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index 9d0718e7c125..5f6948efc015 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -65,3 +65,9 @@ Adding a required field to a populated collection, or making an optional field r `CLI.CONSENT_REQUIRED` lists one `user-choice` next action per flag form ("Run the command again with --delete Legacy", another for the rename form), not grouped by subject and not a complete runnable command, and its summary says "N subjects need a statement", which is project vocabulary. Grouping the forms per subject and printing a full command would need the engine to know the invocation; raise it on prisma/prisma-cli once the statement prompt has shipped. +## SQL variants stored in their own tables + +**Found:** slice 4a design (2026-10-09). Will: SQL will need to handle multi-table storage of variants in the future. + +On MongoDB a variant's field rename touches only that variant's documents. On SQL today a variant's field is a column in the shared table, and a rename applies to every row. When SQL supports variants stored in their own tables, a variant's field statements must act on that variant's table only. + diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md index 0344713b340b..07af0f73aa93 100644 --- a/projects/migration-statements/slices/mongo-renames/spec.md +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -37,11 +37,13 @@ Without `--delete User.nickname`, both commands refuse, as they do on SQL: | Field removed | Validator updated, nothing asked. Every document that still holds the field then fails each update | Asked about. `--delete` removes the field from every document; `--rename` keeps its values | | Model removed | Asked about; `--delete` drops the collection | Same, and `--rename` is offered too | -## Decisions to check +## Decisions (settled with Will, 2026-10-09) -1. **Removing a field now asks.** Today it is silent, but MongoDB's validator then rejects every update to a document that still holds the field. -2. **Fields are named as stored.** `id @map("_id")` is `User._id` in a statement. The contract and the generated client already use that name. Naming the schema field instead would change the contract and its hashes. -3. **The rewrites run under `db update`.** They are classed by what they do to data: a rename is `widening`, a removal is `destructive`. `db update` refuses the `data` class, so classing them `data`, as ADR 188 says today, would block `db update`. This slice amends ADR 188. +1. **Removing a field is data loss.** It is asked about, and `--delete` removes it from every document. Today it is silent, and MongoDB's validator then rejects every update to a document that still holds the field. +2. **Fields are named by their contract key, as on SQL.** MongoDB's contract keys fields by their stored name, so `id @map("_id")` is `User._id` in a statement. Naming the schema field instead would need MongoDB's contract to keep a field-to-stored-name map, as SQL's does. That is outside this project. +3. **Rewrites are classed by what they do to data, as on SQL.** A field rename is `widening`, like a column rename. A field removal is `destructive`, like a column drop. The planner writes a `data` operation only as a placeholder scaffold. So `db update` runs these rewrites, as it runs SQL renames. ADR 188 changes one sentence: today it says every MongoDB data transform is `data`. +4. **Without the old contract, a removed field is named `.`,** for example `users.nickname`. SQL uses `..`, or `
.` on SQLite. +5. **A variant's field rename touches only that variant's documents.** The rewrite filters on the variant's discriminator value, so another variant in the same collection that stores a field with the same name keeps it. This differs from SQL single-table storage, where a rename renames the shared column for every row. ## How it works From 9d2922e672573f52f119f6dd14d8489e80a03746 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 09:47:01 +0200 Subject: [PATCH 06/26] TML-3474: dispatch plans for 3a and 4a, first briefs, and the new tickets Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/plan.md | 4 +- .../01-statements-reach-the-planner.md | 36 ++++++++++++++++++ .../slices/convert-backfill/plan.md | 37 ++++++++++++++++++ .../01-rename-collection-operation.md | 38 +++++++++++++++++++ .../slices/mongo-renames/plan.md | 35 +++++++++++++++++ 5 files changed, 148 insertions(+), 2 deletions(-) create mode 100644 projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md create mode 100644 projects/migration-statements/slices/convert-backfill/plan.md create mode 100644 projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md create mode 100644 projects/migration-statements/slices/mongo-renames/plan.md diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index 7d3276433912..b08e4a85240a 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -49,7 +49,7 @@ ### 3b — The remaining renames and deletes on Postgres and SQLite -**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/remaining-nouns/` +**Linear:** [TML-3537](https://linear.app/prisma-company/issue/TML-3537) · **Folder:** `slices/remaining-nouns/` - `--rename` for enum values, namespaces and fields inside value objects. This slice builds the statement support for value objects, which 4b reuses. - Moving a model to another namespace (`alter table set schema` on Postgres). @@ -64,7 +64,7 @@ ### 4b — `--convert`, `--backfill` and value object renames on MongoDB -**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo-convert-backfill/` +**Linear:** [TML-3538](https://linear.app/prisma-company/issue/TML-3538) · **Folder:** `slices/mongo-convert-backfill/` - The same two flags as 3a, writing a data transform with a placeholder. - Making a field required on a collection with documents is no longer silent. It is answered with `--backfill`, or the command says the existing documents need the field. diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md b/projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md new file mode 100644 index 000000000000..340b3935c712 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md @@ -0,0 +1,36 @@ +# Dispatch 1 — `--convert` and `--backfill` reach the planner, and questions know why data is lost + +**Slice:** [`../spec.md`](../spec.md) § The two flags, § How the pieces fit, § Implementation notes · **Plan:** [`../plan.md`](../plan.md) dispatch 1 · **Builds on:** slice 2 (merged on `main`) + +## Outcome + +Both flags are parsed, resolved against the two contracts, and handed to the SQL planners as destination coordinates. The planners do nothing new with them yet. Each data-loss entry says whether it is a `drop` or a `typeChange`, and the questions offer the verbs that fit. `db update` refuses both flags with a clear error. + +## What to build + +1. **Verbs per command.** `orm/statement-verbs.ts` declares verbs per command. `migration plan`: `rename`, `delete`, `convert`, `backfill`. `db update`: `rename`, `delete`, `allow`, plus `convert` and `backfill` declared only so they can be refused: any value fails with the new `MIGRATION.STATEMENT_NEEDS_MIGRATION_FILE`, whose message says `db update` has no migration file to fill and names `migration plan`. Error reference entry. +2. **Resolved statements gain a `kind`.** `ResolvedMigrationStatement` (`packages/1-framework/1-core/framework-components/src/control/migration-statements.ts`) becomes `kind: 'rename' | 'convert' | 'backfill'`. Every reader that assumes a rename switches on `kind` first: `describeMigrationStatement`, `migrationStatementJson`, `sameCoordinate` and `renamedModel` in `plan-questions.ts`, `originTable` and `originColumn` in the SQL family `operation-subjects.ts`, `StatementPlanner.plan` in `statement-planning.ts` (today it plans any non-model statement as a column rename), and the MongoDB planner (keeps refusing every statement it cannot carry out). +3. **Resolution rules** (in `resolve-statements.ts`): + - `convert Model.field`: names the field by its destination name (after `--rename User.age:User.years`, `--convert User.years`); the field exists in the destination and, through any earlier rename, in the origin; its type changes. Needs the origin contract, else `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`. + - `backfill Model.field`: the field is required in the destination and is either new on a model that exists in the origin or optional in the origin. + - Anything else is `MIGRATION.STATEMENT_UNRESOLVED` naming what was found. +4. **Handed to the planners.** `migration plan` takes `convert` and `backfill` from the run before planning, like `rename`. The SQL planners receive them as destination coordinates (namespace, table, column) on the strategy context (Postgres `planner-strategies.ts` around `StrategyContext`; SQLite equivalent). They do not go through `planStatements`, which applies renames to the working schema. No strategy reads them yet. +5. **The loss kind.** Each `dataLoss` entry gains `loss: 'drop' | 'typeChange'`, set by each target's operation-subjects function. `dataLossQuestion` stops inferring from whether the destination still has the field. A `typeChange` question offers `convert` and `delete` on `migration plan` and `delete` only on `db update`. A `drop` question offers `rename` and `delete`, as today. +6. **Re-plan with every statement.** `askPlanQuestions` re-plans with all planned statements, not only renames, so a typed `convert` re-plans. +7. **Reporting a delete on a type change.** A `delete` answering a `typeChange` question is described as `delete values of field "User.age" (type change)`, not `delete field`. + +## Not in this dispatch + +Any planner output change (dispatches 2–4). The temporary default. Docs beyond the error reference entry. Journeys. + +## Tests + +Unit tests in the CLI and framework-components packages: each resolution rule, both pass and fail; `convert` after a rename uses the new name; `convert` with no origin contract; the verbs offered per loss kind and per command; `db update --convert` fails with the new code before planning; a typed `convert` triggers a re-plan; the delete-on-type-change description; every `kind` switch (a `convert` statement is never planned as a column rename — write the test that would have caught it). + +## Halt conditions + +Stop and report if the engine cannot declare a verb on `db update` that the command refuses with its own error; or if `dataLoss` producers outside the SQL family (extension spaces applying recorded migrations, MongoDB) cannot set `loss` without a design decision. + +## Gate + +The plan's gate, plus `pnpm check:error-reference` and `pnpm lint:framework-vocabulary`. diff --git a/projects/migration-statements/slices/convert-backfill/plan.md b/projects/migration-statements/slices/convert-backfill/plan.md new file mode 100644 index 000000000000..5b419702253a --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/plan.md @@ -0,0 +1,37 @@ +# Slice plan — 3a, `--convert` and `--backfill` + +**Spec:** [`spec.md`](./spec.md) · **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Branch:** `tml-3477-convert-backfill` + +One implementer and one reviewer, both on Opus, resumed across every dispatch. Briefs are in `dispatches/`. + +## Gate for every dispatch + +- `pnpm typecheck`; `pnpm --filter lint`; `pnpm --filter test`. +- `pnpm lint:deps` when imports change; `pnpm fixtures:check` when rendering or classes change; `pnpm check:error-reference` when error codes change; `pnpm lint:framework-vocabulary` when `packages/1-framework` changes; `pnpm lint:throws` and `pnpm lint:casts` (CI runs them, `lint:agent` skips them). +- Integration and e2e: only named files, run as `pnpm test `. Never `pnpm test:integration`, `test:e2e` or `test:all`. +- The reviewer runs code, not only reads it (`drive/calibration/failure-modes.md` F34): SQL the slice renders is compiled and run on PGlite and SQLite, and every printed advice is followed (F46). + +## Dispatches + +| # | Outcome | Builds on | +| --- | --- | --- | +| 1 | `--convert` and `--backfill` are parsed, resolved and handed to the planners; each data-loss entry says `drop` or `typeChange`; questions offer the right verbs; `db update` refuses both flags | slice 2 | +| 2 | Postgres: a lossy type change plans nothing without a statement; `--convert` writes `alterColumnType` with a `using` placeholder; `alterColumnType` takes plain options | 1 | +| 3 | Postgres: `--backfill` writes the backfill scaffold; without it a new required field gets the temporary default as a real call in `migration.ts`; optional-to-required is a direct `SET NOT NULL` | 1 | +| 4 | SQLite: the rebuild takes a per-column copy expression; `--convert`, `--backfill` and the temporary value work as on Postgres | 2, 3 | +| 5 | The NOT NULL failure names the NULL rows and suggests a fix, on Postgres and SQLite (TML-3517) | 3, 4 | +| 6 | Journeys for the slice's done conditions; docs; app upgrade fragment and facade upgrade instructions; every printed advice followed | 1–5 | + +**Hand-offs** +- 1 → 2, 3: planners receive convert and backfill as destination coordinates, and a converted change leaves `dataLoss`. +- 2 → 4: the converted alter is `data`; the `using` slot renders as code and blocks `toOp()` while unfilled. +- 3 → 4: the temporary default renders as a call that reproduces `ops.json`. +- 6 → slice DoD. + +## Coordination with 4a + +4a runs at the same time on `tml-3478-mongo-renames`. Both touch `plan-questions.ts` and the resolved statement type. Whichever merges second merges `main` and adapts. The MongoDB planner must keep refusing `convert` and `backfill` statements until 4b. + +## Open items + +- ADR 200 amendment: written by the orchestrator in dispatch 6's branch, not by the implementer. diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md b/projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md new file mode 100644 index 000000000000..d9a3de8aad32 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md @@ -0,0 +1,38 @@ +# Dispatch 1 — A `renameCollection` operation, end to end + +**Slice:** [`../spec.md`](../spec.md) § How it works, § Implementation notes · **Plan:** [`../plan.md`](../plan.md) dispatch 1 · **Builds on:** slice 2 (merged on `main`) + +## Outcome + +A hand-written MongoDB migration can call `renameCollection('profiles', 'users')`, and running it renames the collection with its documents, indexes and validator. The planner does not use it yet. You also report how variants are stored today. + +## What to build + +1. **The operation through the stack**, the way `createCollection` and `dropCollection` go: + - a `RenameCollectionCommand` and its visitor method in `packages/2-mongo-family/4-query/query-ast` (`ddl-commands.ts`, `ddl-visitors.ts`); + - its wire command in `packages/2-mongo-family/6-transport/mongo-lowering` and lowering in `packages/3-mongo-target/2-mongo-adapter`; + - a driver case in `packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts`; + - serializer and deserializer in the target's operation serializer; + - preview text in `packages/2-mongo-family/9-family/src/core/operation-preview.ts`; + - a `renameCollection` factory in `migration-factories.ts`, a `RenameCollectionCall` in `op-factory-call.ts` that renders `renameCollection('profiles', 'users')`, and the export from `src/exports/migration.ts`. +2. **Checks.** Before: the source collection exists and the target does not (`listCollections`). After: the target exists and the source does not. A re-run after success skips through the after-check. +3. **Class:** `widening`. +4. **Variant report (TML-2447).** Plan a contract with a base model and two variants that share one collection (`@@base`, `@@discriminator`) on current `main`, and report: which collections the planner creates, what the validator holds, and whether a variant's documents can be told apart by the discriminator value in the stored documents. Do not fix anything here; the orchestrator decides from the report. + +## Not in this dispatch + +Planning the rename (dispatch 3). Document rewrites (dispatch 2). Removing `renameStatements`. + +## Tests + +- Unit: factory, call rendering, serializer round trip, preview text. +- Against mongodb-memory-server (existing target-runner tests under `test/integration/test/mongo/target-runner/` show the setup): a collection with documents, a unique index and a closed validator is renamed, and all three are kept; re-running skips; renaming onto an existing collection fails at the before-check with nothing changed. +- `render-typescript` round trip: running the rendered `migration.ts` reproduces the operation in `ops.json`. + +## Halt conditions + +Stop and report if the driver's rename needs admin privileges that mongodb-memory-server does not give; or if a DDL command cannot carry a second collection name without changing the shared command shape. + +## Gate + +The plan's gate, plus the new target-runner test file. diff --git a/projects/migration-statements/slices/mongo-renames/plan.md b/projects/migration-statements/slices/mongo-renames/plan.md new file mode 100644 index 000000000000..7a9597d0c7c2 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/plan.md @@ -0,0 +1,35 @@ +# Slice plan — 4a, renames and deletes on MongoDB + +**Spec:** [`spec.md`](./spec.md) · **Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Branch:** `tml-3478-mongo-renames` + +One implementer and one reviewer, both on Opus, resumed across every dispatch. Briefs are in `dispatches/`. + +## Gate for every dispatch + +- `pnpm typecheck`; `pnpm --filter lint`; `pnpm --filter test`. +- `pnpm lint:deps` when imports change; `pnpm fixtures:check` when rendering or classes change; `pnpm check:error-reference` when error codes change; `pnpm lint:framework-vocabulary` when `packages/1-framework` changes; `pnpm lint:throws` and `pnpm lint:casts`. +- Integration and e2e: only named files, run as `pnpm test `. Never `pnpm test:integration`, `test:e2e` or `test:all`. +- The reviewer runs code against mongodb-memory-server, not only reads it (F34), and follows every printed advice (F46). + +## Dispatches + +| # | Outcome | Builds on | +| --- | --- | --- | +| 1 | A `renameCollection` operation exists end to end: factory, call, serializer, runner, preview text, with before and after checks. Also reports how variants are stored today (TML-2447) | slice 2 | +| 2 | `renameField` and `unsetField` rewrites exist end to end, skip validation, carry `widening` and `destructive`, and the runner and serializer tell them from DDL by shape | 1 | +| 3 | The planner carries out statements: collection rename on the working schema, field rename (filtered by discriminator for a variant), field removal as data loss, the operation order, the refusals. `renameStatements` and `keepDataByHand` are deleted | 2 | +| 4 | Journeys for the slice's done conditions through both commands; docs; app and extension upgrade fragments; every printed advice followed | 3 | + +**Hand-offs** +- 1 → 3: `renameCollection` can be planned and rendered. +- 2 → 3: `renameField` and `unsetField` can be planned and rendered, and run under `db update`'s classes. +- 3 → 4: both commands rename and delete on MongoDB. + +## Coordination with 3a + +3a runs at the same time on `tml-3477-convert-backfill` and adds a `kind` to resolved statements. Whichever merges second merges `main` and adapts. After 3a, the MongoDB planner handles `kind: 'rename'` and refuses `convert` and `backfill`. + +## Open items + +- ADR 188 and ADR 264 amendments: written by the orchestrator in dispatch 4's branch. +- If dispatch 1 finds that TML-2447 makes variant field renames wrong, they move to 4b, and Will is told. From f560017c3cdf4f5a478852527cbc99fd1726276d Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 09:48:26 +0200 Subject: [PATCH 07/26] TML-3474: second briefs for 3a and 4a Signed-off-by: willbot Signed-off-by: Will Madden --- .../dispatches/02-postgres-convert.md | 34 +++++++++++++++++++ .../dispatches/02-document-rewrites.md | 34 +++++++++++++++++++ 2 files changed, 68 insertions(+) create mode 100644 projects/migration-statements/slices/convert-backfill/dispatches/02-postgres-convert.md create mode 100644 projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/02-postgres-convert.md b/projects/migration-statements/slices/convert-backfill/dispatches/02-postgres-convert.md new file mode 100644 index 000000000000..229db0040e60 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/02-postgres-convert.md @@ -0,0 +1,34 @@ +# Dispatch 2 — Postgres: `--convert` writes the conversion slot + +**Slice:** [`../spec.md`](../spec.md) § What changes for users, § What the planners write · **Plan:** [`../plan.md`](../plan.md) dispatch 2 · **Builds on:** dispatch 1 + +## Outcome + +On Postgres, a lossy type change plans no placeholder unless `--convert` names the field. With `--convert User.age`, `migration plan` writes one `alterColumnType` whose `using` is `placeholder(...)`, classed `data`. `alterColumnType` takes plain options. + +## What to build + +1. **Gate the type-change scaffold.** `typeChangeCallStrategy` (`packages/3-targets/3-targets/postgres/src/core/migrations/planner-strategies.ts`) runs only for a field that a `convert` statement names. Without one, the type change falls through to the direct alter: `widening` for a known-safe widening, `destructive` otherwise, which dispatch 1's questions already ask about. +2. **One slot, in the alter.** With `convert`, the strategy writes a single `alterColumnType` with `using: placeholder('.:using')` and no separate data transform. No separate NULL transform either: when the field also becomes required, `SET NOT NULL` follows the alter, and the conversion expression can map NULL. +3. **Class `data`.** `AlterColumnTypeClass` gains `data`. A converted alter is `data`, so it is not in `dataLoss`, and a typed `convert` at the prompt resolves the question on re-plan. +4. **Plain options.** The facade's `alterColumnType` (`postgres-migration.ts`) takes `{ table, column, type, using? }` (plus a namespace where the facade needs one; match the neighbouring facade calls). The rendered call shows that shape, with no `qualifiedTargetType`, `formatTypeExpected` or `rawTargetTypeForLabel`. The `using` slot renders as code, imports `placeholder`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result (`unfilledPlaceholderOperation`), so the CLI writes `ops.json` as `[]` with `pendingPlaceholders`. +5. **Upgrade instructions** for hand-written migrations that call `alterColumnType`, using the `record-upgrade-instructions` skill. + +## Not in this dispatch + +Backfill and the temporary default (dispatch 3). SQLite (dispatch 4). Docs beyond the upgrade instructions. + +## Tests + +- Planner: a lossy change with no statement plans the direct `destructive` alter; with `convert` plans one `data` alter with the slot; a safe widening ignores `convert` (dispatch 1's resolver already refuses a convert with nothing to convert — keep that). +- Rendering: the rendered `migration.ts` for a convert; filling the slot with `'"age"::integer'` and running the file produces an `ops.json` whose SQL is `ALTER TABLE ... TYPE integer USING "age"::integer`; that SQL runs on PGlite against rows holding numeric strings and converts them. +- CLI: `migration plan --convert User.age` reports `pendingPlaceholders`; a typed `convert` at the prompt re-plans and asks nothing more. +- Update `render-typescript.test.ts`, `postgres-issue-planner.test.ts` and any test that asserted the old automatic scaffold. + +## Halt conditions + +Stop and report if the `using` slot cannot render as code without changing the shared call-rendering helper for every call; or if any example migration under `examples/` calls `alterColumnType` with the old options in a way the upgrade instructions cannot describe. + +## Gate + +The plan's gate, plus `pnpm fixtures:check`, `pnpm check:upgrade-coverage --mode pr --prev $(git merge-base HEAD origin/main) --head HEAD`, and `test/integration/test/cli-journeys/data-transform-strategies.e2e.test.ts` (its type-change scenario changes). diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md b/projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md new file mode 100644 index 000000000000..517003989065 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md @@ -0,0 +1,34 @@ +# Dispatch 2 — `renameField` and `unsetField` rewrites, end to end + +**Slice:** [`../spec.md`](../spec.md) § How it works, § Decisions 3 and 5 · **Plan:** [`../plan.md`](../plan.md) dispatch 2 · **Builds on:** dispatch 1 + +## Outcome + +A hand-written MongoDB migration can call `renameField(...)` and `unsetField(...)`. Each runs one `updateMany` that skips validation, carries the class of its effect, and runs under `db update`'s classes. The planner does not use them yet. + +## What to build + +1. **Skipping validation.** The raw update command (`RawUpdateManyCommand`, `packages/2-mongo-family/4-query/query-ast/src/raw-commands.ts`) gains `bypassDocumentValidation`, through its wire form and the driver (`mongo-driver.ts` passes only `upsert` today). +2. **Two factories and calls.** + - `renameField(collection, from, to, { filter? })`: `$rename` on documents that hold `from` and match `filter`. Before: no matching document holds `to`. After: no matching document holds `from`. Class `widening`. + - `unsetField(collection, field, { filter? })`: `$unset` on documents that hold `field` and match `filter`. After: no matching document holds `field`. Class `destructive`. + - `filter` carries a variant's discriminator condition (`{ type: 'bug' }`); dispatch 3 supplies it. + - Calls render as `renameField('users', 'name', 'fullName')` and `unsetField(...)`, exported from the target's migration module, so a re-run of `migration.ts` reproduces `ops.json`. +3. **Telling rewrites from DDL by shape.** The runner (`mongo-runner.ts`) and the operation serializer recognise these by their shape (`run` against `execute`), not by `operationClass === 'data'`. `data` stays the class of user-written and scaffolded transforms. A `db update`-style policy without `data` accepts these two and still refuses a `data` transform. +4. **Storage name.** The family helper that names an operation's storage (`mongoStorageNameOf`, `packages/2-mongo-family/9-family/src/core/operation-storage-name.ts`) handles the new shape: `.`. + +## Not in this dispatch + +Planning them (dispatch 3). Value object fields (slice 4b). + +## Tests + +Against mongodb-memory-server: a collection with a closed validator that requires `name` and a unique index on `name`. Drop the index, `renameField` to `fullName`, create the index on `fullName`: every document moved, the unique index builds, nothing failed validation. `renameField` when a document already holds `fullName` fails at the before-check with nothing changed. A re-run after success skips. `unsetField` removes the field from every matching document. A `filter` limits both to matching documents. Under a policy of `additive`, `widening`, `destructive`, both run and a `data` transform is refused. Unit: factories, rendering, serializer round trip, storage names. + +## Halt conditions + +Stop and report if skipping validation needs a server privilege mongodb-memory-server lacks; or if the serializer cannot tell the shapes apart without a new discriminating field in `ops.json` (that changes the file format and needs a decision). + +## Gate + +The plan's gate, plus the target-runner test files you add or change. From 25afd128dbb6f16b3c41d80c92962cb9e2988584 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:05:53 +0200 Subject: [PATCH 08/26] TML-3478: variants share their base's collection on main; record it in the plan Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/slices/mongo-renames/plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/migration-statements/slices/mongo-renames/plan.md b/projects/migration-statements/slices/mongo-renames/plan.md index 7a9597d0c7c2..371a75301b66 100644 --- a/projects/migration-statements/slices/mongo-renames/plan.md +++ b/projects/migration-statements/slices/mongo-renames/plan.md @@ -32,4 +32,4 @@ One implementer and one reviewer, both on Opus, resumed across every dispatch. B ## Open items - ADR 188 and ADR 264 amendments: written by the orchestrator in dispatch 4's branch. -- If dispatch 1 finds that TML-2447 makes variant field renames wrong, they move to 4b, and Will is told. +- Dispatch 1 found TML-2447's gaps fixed on `main`: variants share their base's collection, and the validator has a `oneOf` branch per variant pinned to its discriminator value. Variant field renames stay in 4a. Dispatch 4 removes the stale "Known gaps" paragraph from the MongoDB Family doc § Polymorphic variants. From f68220468be95ed6cf3ce49147206c7fdc73d43d Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:09:45 +0200 Subject: [PATCH 09/26] TML-3478: review of dispatch 1 adds preview escaping and the time-series refusal Signed-off-by: willbot Signed-off-by: Will Madden --- .../mongo-renames/dispatches/02-document-rewrites.md | 2 ++ projects/migration-statements/slices/mongo-renames/spec.md | 7 ++++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md b/projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md index 517003989065..f73944df2973 100644 --- a/projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md +++ b/projects/migration-statements/slices/mongo-renames/dispatches/02-document-rewrites.md @@ -17,6 +17,8 @@ A hand-written MongoDB migration can call `renameField(...)` and `unsetField(... 3. **Telling rewrites from DDL by shape.** The runner (`mongo-runner.ts`) and the operation serializer recognise these by their shape (`run` against `execute`), not by `operationClass === 'data'`. `data` stays the class of user-written and scaffolded transforms. A `db update`-style policy without `data` accepts these two and still refuses a `data` transform. 4. **Storage name.** The family helper that names an operation's storage (`mongoStorageNameOf`, `packages/2-mongo-family/9-family/src/core/operation-storage-name.ts`) handles the new shape: `.`. +5. **Preview text that mongosh accepts (review finding D1-1).** Every MongoDB preview line in `packages/2-mongo-family/9-family/src/core/operation-preview.ts` writes the collection as `db.getCollection("")`, with the name escaped as a string, so a name such as `my-coll` or one containing quotes is valid mongosh. This covers the existing lines (`drop`, `createIndex`, and the rest) and the new rename and rewrite lines. + ## Not in this dispatch Planning them (dispatch 3). Value object fields (slice 4b). diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md index 07af0f73aa93..efba9a14ddbc 100644 --- a/projects/migration-statements/slices/mongo-renames/spec.md +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -9,10 +9,10 @@ A MongoDB user renames model `Profile` to `User`, renames field `name` to `fullN ```text $ prisma migration plan --name tidy --rename Profile:User --rename User.name:User.fullName --delete User.nickname ✔ Planned 6 operation(s) - Rename collection "profiles" to "users" widening + Rename collection profiles to users widening Drop index users.name_1 widening - Rename field "name" to "fullName" in collection "users" widening - Remove field "nickname" from collection "users" destructive + Rename field name to fullName in users widening + Remove field nickname from users destructive Create index users.fullName_1 additive Update validator on users widening ``` @@ -78,6 +78,7 @@ Without `--delete User.nickname`, both commands refuse, as they do on SQL: - Plans with no statements and no removed fields come out exactly as today. **Refused, with the reason named** +- Renaming a model stored in a time-series collection, or in a view. MongoDB cannot rename them. - Renaming `_id`. - Renaming a collection onto one that already exists. - Renaming a field that is stored on only one side. From 1559a8250a3f68a32f1b7e779c81bf874c985272 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:18:40 +0200 Subject: [PATCH 10/26] TML-3474: third briefs for 3a and 4a Signed-off-by: willbot Signed-off-by: Will Madden --- ...postgres-backfill-and-temporary-default.md | 36 +++++++++++++++++++ .../03-planner-carries-out-statements.md | 36 +++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100644 projects/migration-statements/slices/convert-backfill/dispatches/03-postgres-backfill-and-temporary-default.md create mode 100644 projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/03-postgres-backfill-and-temporary-default.md b/projects/migration-statements/slices/convert-backfill/dispatches/03-postgres-backfill-and-temporary-default.md new file mode 100644 index 000000000000..1e9439fccf7f --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/03-postgres-backfill-and-temporary-default.md @@ -0,0 +1,36 @@ +# Dispatch 3 — Postgres: `--backfill`, and the temporary default as a real call + +**Slice:** [`../spec.md`](../spec.md) § What changes for users, § What the planners write · **Plan:** [`../plan.md`](../plan.md) dispatch 3 · **Builds on:** dispatch 1 (dispatch 2 may land first; nothing here depends on it) + +## Outcome + +On Postgres, `migration plan` writes a backfill placeholder only for a field `--backfill` names. Without it, a new required field on an existing table gets the temporary default under `migration plan` as well as `db update`, and an optional field made required is a direct `SET NOT NULL`. The temporary default renders as a call that reproduces `ops.json`. + +## What to build + +1. **Gate the backfill scaffolds.** `notNullBackfillCallStrategy` and `nullableTighteningCallStrategy` (`planner-strategies.ts`) run only for a field a `backfill` statement names. + - New required field with `--backfill`: nullable add, `dataTransform` with placeholders, `SET NOT NULL` (today's recipe). + - Optional made required with `--backfill`: `dataTransform` with placeholders, `SET NOT NULL`. +2. **No flag.** + - New required field on an existing table: the temporary-default recipe (`notNullAddColumnCallStrategy`), now under `migration plan` too. It keeps today's refusal for a column in a new primary key, unique or foreign key, falling back to the empty-table direct add as today. + - Optional made required: a direct `SET NOT NULL`, `widening`. +3. **A real call.** `AddNotNullColumnWithTempDefaultCall` and `AddNotNullColumnDirectCall` render today as `rawSql({ id, label, operationClass })` with no SQL (`op-factory-call.ts`), so re-running `migration.ts` does not reproduce `ops.json`. Render each as a facade call (a new facade method on the Postgres migration class, or an existing one with the right options) that builds the same operation. Running the written `migration.ts` reproduces `ops.json` byte for byte. + +## Not in this dispatch + +SQLite (dispatch 4). The NOT NULL failure text (dispatch 5). Docs and upgrade fragment (dispatch 6). + +## Tests + +- Planner: each row of the spec's table for Postgres, with and without `--backfill`, under `migration plan` and `db update`. +- Rendering round trip: the temporary default and the direct add, written by `migration plan`, re-run, `ops.json` identical. +- On PGlite: the temporary-default migration applied to a table with rows leaves the column `NOT NULL` with no default and the fill value in old rows; the backfill scaffold, filled with an `UPDATE`, applies. +- Update `render-typescript.test.ts` (its pinned `rawSql` output changes), `nullability-backfill` and any test that asserted the automatic scaffold; `test/integration/test/cli-journeys/invariant-routing.e2e.test.ts` (around lines 130–160) needs `--backfill`. + +## Halt conditions + +Stop and report if the temporary default cannot be expressed as a facade call without exposing internal operation fields in `migration.ts`; or if a codec's temporary value cannot be rendered as code. + +## Gate + +The plan's gate, plus `pnpm fixtures:check` and the named journeys `data-transform-strategies`, `invariant-routing`, and any other file under `test/integration` or `test/e2e` that reaches `migration plan` or `db update` with a new required field (find them through the helpers in `test/integration/test/utils/journey-test-helpers.ts` as well as the command names). diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md b/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md new file mode 100644 index 000000000000..81c8d24ee980 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md @@ -0,0 +1,36 @@ +# Dispatch 3 — The MongoDB planner carries out renames and deletes + +**Slice:** [`../spec.md`](../spec.md) § Decisions, § How it works · **Plan:** [`../plan.md`](../plan.md) dispatch 3 · **Builds on:** dispatches 1 and 2 + +## Outcome + +`migration plan` and `db update` on MongoDB rename collections and fields from `--rename`, and ask about a removed field, which `--delete` answers by removing it from every document. The temporary refusal (`renameStatements`, `keepDataByHand`) is gone. + +## What to build + +1. **Collection rename.** A model statement whose collection name changes renames the collection in the working copy of the origin schema before the diff (`mongo-planner.ts`, `planCalls`), and plans one `renameCollection`. Indexes and the validator then diff as usual. +2. **Field rename.** A field statement plans one `renameField`. For a variant's field, `filter` is `{ : }`, read from the base model (`discriminator.field`, `variants..value`; a variant's `fields` hold only its own fields). A statement whose stored names do not change plans nothing and reports applied with no operations. +3. **Field removal is data loss.** A field present in the origin model and absent from the destination, on a collection that is not dropped, plans one `unsetField` (with the variant filter for a variant's field) and reports a `field` subject in `dataLoss` (with `loss: 'drop'` once slice 3a's loss kinds are on `main`). Without an origin contract the subject is `{ kind: 'storage', name: '.' }`. The question offers `rename` and `delete`. +4. **Order.** Collection renames, collection creates, index drops, rewrites (renames and removals), index creates, validator changes, option changes, collection drops. Plans with no statements and no removed fields are unchanged; prove it with the existing planner tests passing untouched. +5. **Refusals** (`statementRejected`, reason named): renaming `_id`; renaming a model stored in a time-series collection or a view; a collection rename onto a name the working schema already has; a field stored on one side only; any statement when the control policy is not `managed`. +6. **`appliedStatements`** count only the rename or rewrite each statement accounts for. +7. **Delete the temporary refusal.** The `renameStatements` member in `control-migration-types.ts`; MongoDB's setting in `control-target.ts`; `keepDataByHand`, `keepTheData` and `NOT_IN_THIS_RELEASE` in `mongo-planner.ts`; `keepDataByHandFor` and its parameter in `plan-questions.ts`; the reads in `db-run.ts` and `migration-plan.ts`. Keep the unrelated `renameStatements()` helper in `statement-text.ts`. +8. **After slice 3a merges** (if it has by then): resolved statements have a `kind`. Carry out `kind: 'rename'`; refuse `convert` and `backfill` as 3a's MongoDB code does today. + +## Not in this dispatch + +Journeys, docs, ADR amendments and upgrade fragments (dispatch 4). + +## Tests + +- Planner unit tests for each item above, rewriting `mongo-planner.statements.test.ts` and `db-update-statements.test.ts` (both assert the old refusal), and the `keepDataByHand` cases in `mongo-planner.data-loss.test.ts`. +- CLI tests that used `keepDataByHand` or `renameStatements` (listed in `wip/grounding-4a.md` § 7 in the parent worktree). +- Against mongodb-memory-server, through the planner and runner: a model rename and a field rename with a unique index on the field and a closed validator; a field removal; a variant field rename where another variant stores a field with the same name, which keeps its value. + +## Halt conditions + +Stop and report if the working-schema rename cannot happen before the diff without changing the shared schema IR; or if removing a field from a variant cannot be told apart from removing it from the base in the diff. + +## Gate + +The plan's gate, plus `pnpm lint:framework-vocabulary`, `pnpm check:error-reference`, and the named files `test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts` and `mongo-db-update-consent.e2e.test.ts` (S4 of the first now succeeds instead of refusing). From 82de10cafcb3e701b7db372b598c518c88ea4f4f Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:28:39 +0200 Subject: [PATCH 11/26] TML-3478: the planner refuses field names MongoDB reads as paths or operators Signed-off-by: willbot Signed-off-by: Will Madden --- .../dispatches/03-planner-carries-out-statements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md b/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md index 81c8d24ee980..b65cbacb5ddc 100644 --- a/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md +++ b/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md @@ -12,7 +12,7 @@ 2. **Field rename.** A field statement plans one `renameField`. For a variant's field, `filter` is `{ : }`, read from the base model (`discriminator.field`, `variants..value`; a variant's `fields` hold only its own fields). A statement whose stored names do not change plans nothing and reports applied with no operations. 3. **Field removal is data loss.** A field present in the origin model and absent from the destination, on a collection that is not dropped, plans one `unsetField` (with the variant filter for a variant's field) and reports a `field` subject in `dataLoss` (with `loss: 'drop'` once slice 3a's loss kinds are on `main`). Without an origin contract the subject is `{ kind: 'storage', name: '.' }`. The question offers `rename` and `delete`. 4. **Order.** Collection renames, collection creates, index drops, rewrites (renames and removals), index creates, validator changes, option changes, collection drops. Plans with no statements and no removed fields are unchanged; prove it with the existing planner tests passing untouched. -5. **Refusals** (`statementRejected`, reason named): renaming `_id`; renaming a model stored in a time-series collection or a view; a collection rename onto a name the working schema already has; a field stored on one side only; any statement when the control policy is not `managed`. +5. **Refusals** (`statementRejected`, reason named): renaming or removing `_id`; a field whose stored name contains `.` or starts with `$` (MongoDB reads it as a path or an operator; `@map` can produce one); renaming a model stored in a time-series collection or a view; a collection rename onto a name the working schema already has; a field stored on one side only; any statement when the control policy is not `managed`. 6. **`appliedStatements`** count only the rename or rewrite each statement accounts for. 7. **Delete the temporary refusal.** The `renameStatements` member in `control-migration-types.ts`; MongoDB's setting in `control-target.ts`; `keepDataByHand`, `keepTheData` and `NOT_IN_THIS_RELEASE` in `mongo-planner.ts`; `keepDataByHandFor` and its parameter in `plan-questions.ts`; the reads in `db-run.ts` and `migration-plan.ts`. Keep the unrelated `renameStatements()` helper in `statement-text.ts`. 8. **After slice 3a merges** (if it has by then): resolved statements have a `kind`. Carry out `kind: 'rename'`; refuse `convert` and `backfill` as 3a's MongoDB code does today. From 9b5f4e64526674067501801073214bd32a572e7b Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:36:10 +0200 Subject: [PATCH 12/26] TML-3477: alterColumnType derives its after-check text from the written type Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/slices/convert-backfill/spec.md | 1 + 1 file changed, 1 insertion(+) diff --git a/projects/migration-statements/slices/convert-backfill/spec.md b/projects/migration-statements/slices/convert-backfill/spec.md index e48b659f09db..e9c341b723aa 100644 --- a/projects/migration-statements/slices/convert-backfill/spec.md +++ b/projects/migration-statements/slices/convert-backfill/spec.md @@ -127,6 +127,7 @@ Everything here serves one rule: a placeholder migration is written only when a - `askPlanQuestions` re-plans with every planned statement, not only renames. - Convert and backfill reach the strategies as destination coordinates through the strategy context. They do not go through `planStatements`, because they rename nothing. - Strategies gated: Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`. +- `alterColumnType`'s after-check compares the column's `format_type` text, which `migration.ts` no longer carries. The facade derives it from `type` through the control stack's data types (any of a data type's texts, rendered as its `format_type` text; a type that names its own kind, such as an enum, from its quoted name), and the planner derives it the same way, so `ops.json` is unchanged by a re-run. A `type` the stack cannot read fails with a structured error. A test runs every Postgres data type through both and checks they agree. Decided 2026-10-09 by the orchestrator: comparing only the type OID would let a length-only change look already applied, so the runner would skip it. - `AlterColumnTypeClass` gains `data`. The `using` slot renders as code, imports `placeholder`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result. - `recreateTable` gains a per-column copy expression, used by convert and by the SQLite temporary value. SQLite gets a fill-value map: integer `0`, real `0`, text `''`, blob `X''`, and what its boolean and date-time codecs store. - `MigrationRunnerFailure` gains `fix`. The `db update`, `db init` and `migrate` error mappers prefer it. From ad7937720721840e08019428141fab617d2671e3 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:36:37 +0200 Subject: [PATCH 13/26] TML-3478: the planner refuses a collection rename onto a name MongoDB rejects Signed-off-by: willbot Signed-off-by: Will Madden --- .../dispatches/03-planner-carries-out-statements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md b/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md index b65cbacb5ddc..12f33854b9ea 100644 --- a/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md +++ b/projects/migration-statements/slices/mongo-renames/dispatches/03-planner-carries-out-statements.md @@ -12,7 +12,7 @@ 2. **Field rename.** A field statement plans one `renameField`. For a variant's field, `filter` is `{ : }`, read from the base model (`discriminator.field`, `variants..value`; a variant's `fields` hold only its own fields). A statement whose stored names do not change plans nothing and reports applied with no operations. 3. **Field removal is data loss.** A field present in the origin model and absent from the destination, on a collection that is not dropped, plans one `unsetField` (with the variant filter for a variant's field) and reports a `field` subject in `dataLoss` (with `loss: 'drop'` once slice 3a's loss kinds are on `main`). Without an origin contract the subject is `{ kind: 'storage', name: '.' }`. The question offers `rename` and `delete`. 4. **Order.** Collection renames, collection creates, index drops, rewrites (renames and removals), index creates, validator changes, option changes, collection drops. Plans with no statements and no removed fields are unchanged; prove it with the existing planner tests passing untouched. -5. **Refusals** (`statementRejected`, reason named): renaming or removing `_id`; a field whose stored name contains `.` or starts with `$` (MongoDB reads it as a path or an operator; `@map` can produce one); renaming a model stored in a time-series collection or a view; a collection rename onto a name the working schema already has; a field stored on one side only; any statement when the control policy is not `managed`. +5. **Refusals** (`statementRejected`, reason named): renaming or removing `_id`; a field whose stored name contains `.` or starts with `$` (MongoDB reads it as a path or an operator; `@map` can produce one); renaming a model stored in a time-series collection or a view; a collection rename onto a name the working schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, starting with `system.`); a field stored on one side only; any statement when the control policy is not `managed`. 6. **`appliedStatements`** count only the rename or rewrite each statement accounts for. 7. **Delete the temporary refusal.** The `renameStatements` member in `control-migration-types.ts`; MongoDB's setting in `control-target.ts`; `keepDataByHand`, `keepTheData` and `NOT_IN_THIS_RELEASE` in `mongo-planner.ts`; `keepDataByHandFor` and its parameter in `plan-questions.ts`; the reads in `db-run.ts` and `migration-plan.ts`. Keep the unrelated `renameStatements()` helper in `statement-text.ts`. 8. **After slice 3a merges** (if it has by then): resolved statements have a `kind`. Carry out `kind: 'rename'`; refuse `convert` and `backfill` as 3a's MongoDB code does today. From 6eff6a3735c89b87f175b60102b55c637ae51d96 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:38:54 +0200 Subject: [PATCH 14/26] TML-3474: remaining briefs for 3a and 4a Signed-off-by: willbot Signed-off-by: Will Madden --- .../convert-backfill/dispatches/04-sqlite.md | 31 ++++++++++++++++ .../dispatches/05-not-null-failure-advice.md | 37 +++++++++++++++++++ .../dispatches/06-journeys-docs-fragments.md | 29 +++++++++++++++ .../dispatches/04-journeys-docs-fragments.md | 33 +++++++++++++++++ 4 files changed, 130 insertions(+) create mode 100644 projects/migration-statements/slices/convert-backfill/dispatches/04-sqlite.md create mode 100644 projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md create mode 100644 projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md create mode 100644 projects/migration-statements/slices/mongo-renames/dispatches/04-journeys-docs-fragments.md diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/04-sqlite.md b/projects/migration-statements/slices/convert-backfill/dispatches/04-sqlite.md new file mode 100644 index 000000000000..49799b641153 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/04-sqlite.md @@ -0,0 +1,31 @@ +# Dispatch 4 — SQLite: `--convert`, `--backfill` and the temporary value + +**Slice:** [`../spec.md`](../spec.md) § What the planners write, § Implementation notes · **Plan:** [`../plan.md`](../plan.md) dispatch 4 · **Builds on:** dispatches 2 and 3 + +## Outcome + +SQLite behaves like Postgres for every row of the spec's table. A table rebuild can copy a column through an expression, which carries the conversion slot and the temporary value. + +## What to build + +1. **Copy expressions.** `recreateTable` (`packages/3-targets/3-targets/sqlite/src/core/migrations/operations/tables.ts`) takes a per-column copy expression for the `INSERT INTO … SELECT` step. `RecreateTableCall` renders it as code, so it can hold `placeholder(...)`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result. +2. **`--convert`.** A lossy type change plans nothing without a statement (the destructive rebuild that dispatch 1's questions ask about). With `--convert`, the rebuild's copy expression for the column is `placeholder('.:using')`, and the rebuild is classed `data`. +3. **`--backfill`.** New required field: add the column nullable, a `dataTransform` with placeholders, then a tightening rebuild. Optional made required: `dataTransform`, then the tightening rebuild. `nullabilityTighteningBackfillStrategy` runs only for a field `--backfill` names. +4. **No flag.** Optional made required: the tightening rebuild, `widening`. New required field on an existing table: a rebuild whose copy expression for the new column is a fill value, under both commands. Refused for a column in a new primary key, unique or foreign key, as on Postgres. +5. **Fill values.** A SQLite fill-value map, consulted after the codec hook as on Postgres: integer `0`, real `0`, text `''`, blob `X''`, and whatever SQLite's boolean and date-time codecs store (find them in the SQLite target's codecs; do not guess). + +## Tests + +- Planner: each row of the spec's table for SQLite, with and without each flag, under `migration plan` and `db update`. +- Rendering round trip for the rebuild with a copy expression and with a placeholder. +- On SQLite with rows: the conversion slot filled with `CAST("age" AS INTEGER)` converts the values; the temporary value lands in old rows and the column is `NOT NULL`; the backfill scaffold, filled, applies. +- `delete-statements-migration.sqlite.e2e.test.ts` journey S3 now adds the required field to a table with rows (it used an empty table because of this gap). +- The deferred item "SQLite `db update` cannot add a required field to a table that has rows" is closed: remove it from `projects/migration-statements/deferred.md`. + +## Halt conditions + +Stop and report if a SQLite codec stores booleans or dates in a form that has no safe fill value; or if the copy expression cannot be added without changing the rebuild's postcheck SQL for plans that do not use it (existing `ops.json` must not change). + +## Gate + +The plan's gate, plus `pnpm fixtures:check` and every file under `test/integration` or `test/e2e` that runs `migration plan` or `db update` against SQLite with a type change or a new required field (directly or through `journey-test-helpers.ts`, F44). diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md b/projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md new file mode 100644 index 000000000000..c2bfc2225f47 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md @@ -0,0 +1,37 @@ +# Dispatch 5 — The NOT NULL failure names the NULL rows (TML-3517) + +**Slice:** [`../spec.md`](../spec.md) § How the pieces fit · **Plan:** [`../plan.md`](../plan.md) dispatch 5 · **Builds on:** dispatches 3 and 4 + +## Outcome + +When `SET NOT NULL` on Postgres, or a tightening rebuild on SQLite, meets NULL values, the command says how many rows hold NULL in which column, and how to fix it. It no longer talks about schema drift. + +Today, on Postgres: + +```text +✖ [MIGRATION.RUNNER_FAILED] Operation alterNullability.setNotNull.Item.note failed during precheck: ensure no NULL values in "note" + why: Migration runner failed +→ Inspect the reported conflict, reconcile schema drift if needed, then re-run `prisma db update` +``` + +After: + +```text +✖ [MIGRATION.RUNNER_FAILED] 3 rows of "Item" hold NULL in "note", so it cannot become required. +→ Set a value in those rows, or keep the field optional. +→ Or plan it with `prisma migration plan --backfill Item.note` and fill the backfill step. +``` + +## What to build + +1. **The runner reports the count.** When the precheck that refuses NULLs fails (Postgres `runner.ts` around lines 331–355), the runner counts the NULL rows and returns a `why` with the table, the column and the count. SQLite: the tightening rebuild's copy fails with `NOT NULL constraint failed`; turn that into the same report, with the count. +2. **A `fix` on the failure.** `MigrationRunnerFailure` gains an optional `fix`. The `db update`, `db init` and `migrate` error mappers (`db-update-failure.ts` around lines 23–27, and the matching spots for the other two) use it when present, instead of the fixed drift advice. +3. **The fix names a flag only where it applies.** `--backfill` is suggested for `db update` and `migration plan`, written in the field's contract name when the origin contract is known, and through `statementFlag` so it is shell-quoted. Under `migrate`, the advice says to fix the rows and re-run, since the migration is already written. + +## Tests + +On PGlite and SQLite, a table with NULLs in a field made required: each command prints the count and the fix; following the fix (set the values, re-run) applies; following the `--backfill` advice plans a backfill scaffold. The previous text no longer appears. + +## Gate + +The plan's gate, plus `pnpm check:error-reference` if a code or entry changes, and every journey that asserts the old drift advice. diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md b/projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md new file mode 100644 index 000000000000..1ebac0664c98 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md @@ -0,0 +1,29 @@ +# Dispatch 6 — Journeys, docs and upgrade fragments + +**Slice:** [`../spec.md`](../spec.md) § Done when, § Scope · **Plan:** [`../plan.md`](../plan.md) dispatch 6 · **Builds on:** dispatches 1–5 + +## Outcome + +The slice's done conditions hold through journeys on Postgres and SQLite, and the docs, error reference, skill and upgrade fragment describe the new behaviour. + +## What to build + +1. **Journeys**, on tables with rows, on both targets: + - a type change: refused without a flag; `--convert`, fill the slot, run `migration.ts`, `migrate`, rows converted; `--delete` plans the direct change; + - a new required field: no flag gives the temporary value; `--backfill`, fill the step, `migrate`; + - an optional field made required with NULLs present: the failure names the rows; after setting them, it applies; + - `db update --convert` refused, and its printed `migration plan` command run as printed; + - afterwards a further plan is empty and `db verify --schema-only` is clean. +2. **Follow every printed advice to its end** (F46), on each target, including the `db update` refusal through `migrate` on a database created by `db init`. +3. **Docs.** + - Migration System doc § Statements: `--convert` and `--backfill`; the loss kinds; the questions per command; remove the sentence saying the Postgres planner writes no placeholder data transforms (§ Authoring intermediate state); § NOT NULL columns without defaults now applies to `migration plan` too. + - `docs/reference/error-reference.md`: every code added or changed. + - The CLI README sections for `migration plan` and `db update`. + - `skills/prisma-8/references/migrations.md`: type changes and new required fields. +4. **App upgrade fragment** (`record-upgrade-instructions` skill): lossy type changes now ask; placeholders only with `--convert` or `--backfill`; a new required field gets a temporary value under `migration plan`; `alterColumnType`'s options (if dispatch 2's instructions do not already cover it). + +The orchestrator writes the ADR 200 amendment; do not edit that file. + +## Gate + +The plan's gate, plus `pnpm check:upgrade-coverage --mode pr --prev $(git merge-base HEAD origin/main) --head HEAD`, `pnpm lint:docs`, the journey files, and every other file under `test/integration` or `test/e2e` that runs `migration plan` or `db update` on Postgres or SQLite, directly or through `journey-test-helpers.ts` (F44). diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/04-journeys-docs-fragments.md b/projects/migration-statements/slices/mongo-renames/dispatches/04-journeys-docs-fragments.md new file mode 100644 index 000000000000..e51e046d2afe --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/dispatches/04-journeys-docs-fragments.md @@ -0,0 +1,33 @@ +# Dispatch 4 — Journeys, docs and upgrade fragments + +**Slice:** [`../spec.md`](../spec.md) § Done when, § Scope · **Plan:** [`../plan.md`](../plan.md) dispatch 4 · **Builds on:** dispatch 3 + +## Outcome + +The slice's done conditions hold through journeys on real MongoDB, and the docs, error reference, skill and upgrade fragments describe what MongoDB now does. + +## What to build + +1. **Journeys** (in `test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts` or a sibling file), on collections with documents, through `migration plan` then `migrate` and through `db update`: + - a model rename, a field rename with a unique index on the field, and a field removal with `--delete`, in one run; + - a variant's field rename in a collection where another variant stores a field with the same name; + - afterwards: documents under the new names; the removed field gone from every document; every document accepts an update; a further plan is empty; `db verify --schema-only` is clean; + - without `--delete`, both commands refuse, and the printed flags then succeed when run as printed; + - a second `db update` with the same statements fails on the first statement; + - `db update` with no snapshot: a removed field is named `.`, and `--delete .` works. +2. **Follow every printed advice to its end** (`drive/calibration/failure-modes.md` F46): each refusal, error and dry-run suggestion this slice adds or changes is run as printed in a journey or a probe, and the end state checked. +3. **Docs.** + - `docs/architecture docs/subsystems/7. Migration System.md` § Statements: MongoDB carries out renames and deletes; remove the `renameStatements` paragraph and the "MongoDB refuses every statement" limit; say how a field is named (contract key, the stored name on MongoDB) and that a variant's field rename touches only that variant's documents. + - `docs/architecture docs/subsystems/10. MongoDB Family.md`: the operation list gains `renameCollection`, `renameField`, `unsetField`; remove the stale "Known gaps" paragraph under § Polymorphic variants (TML-2447's gaps are fixed on `main`). + - `docs/reference/error-reference.md`: every code this slice adds or changes. + - The CLI README sections for `migration plan` and `db update`. + - `skills/prisma-8/references/migrations.md`: renames and deletes on MongoDB. +4. **Upgrade fragments** (`record-upgrade-instructions` skill): + - app: MongoDB renames work; removing a field now asks and `--delete` removes it from every document; + - extension: `TargetMigrationsCapability.renameStatements` and `keepDataByHand` are gone (shipped in 8.0.0-rc.17). + +The orchestrator writes the ADR 188 and ADR 264 amendments; do not edit those two files. + +## Gate + +The plan's gate, plus `pnpm check:upgrade-coverage --mode pr --prev $(git merge-base HEAD origin/main) --head HEAD`, `pnpm lint:docs`, the journey files, and every other file under `test/integration` or `test/e2e` that runs `migration plan` or `db update` against MongoDB, directly or through the helpers in `test/integration/test/utils/journey-test-helpers.ts` (F44). From c396de10b94ff34fcc2f5655eadef512869474ff Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 11:22:34 +0200 Subject: [PATCH 15/26] TML-3478: record that @map accepts field names MongoDB cannot validate Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/deferred.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index 5f6948efc015..81d35c213dbf 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -71,3 +71,9 @@ Adding a required field to a populated collection, or making an optional field r On MongoDB a variant's field rename touches only that variant's documents. On SQL today a variant's field is a column in the shared table, and a rename applies to every row. When SQL supports variants stored in their own tables, a variant's field statements must act on that variant's table only. + +## MongoDB `@map` accepts field names MongoDB cannot validate + +**Found:** slice 4a review of dispatch 3 (2026-10-09). For the MongoDB authoring owners, not this project. + +`@map("o.d")` or `@map("$x")` is accepted in a MongoDB schema. MongoDB reads such a name as a path or an operator, and an insert of `{ "o.d": ... }` already fails the validator Prisma writes. Slice 4a skips such fields when planning statements. The fix belongs in authoring: reject the name with a diagnostic. From 5c2fa716320d8b9255e219b6d93fc480a3f25e97 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:23:16 +0200 Subject: [PATCH 16/26] TML-3477: dispatch 5 also gives the empty-table add its own advice Signed-off-by: willbot Signed-off-by: Will Madden --- .../convert-backfill/dispatches/05-not-null-failure-advice.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md b/projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md index c2bfc2225f47..fab087fafef0 100644 --- a/projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md +++ b/projects/migration-statements/slices/convert-backfill/dispatches/05-not-null-failure-advice.md @@ -26,7 +26,8 @@ After: 1. **The runner reports the count.** When the precheck that refuses NULLs fails (Postgres `runner.ts` around lines 331–355), the runner counts the NULL rows and returns a `why` with the table, the column and the count. SQLite: the tightening rebuild's copy fails with `NOT NULL constraint failed`; turn that into the same report, with the count. 2. **A `fix` on the failure.** `MigrationRunnerFailure` gains an optional `fix`. The `db update`, `db init` and `migrate` error mappers (`db-update-failure.ts` around lines 23–27, and the matching spots for the other two) use it when present, instead of the fixed drift advice. -3. **The fix names a flag only where it applies.** `--backfill` is suggested for `db update` and `migration plan`, written in the field's contract name when the origin contract is known, and through `statementFlag` so it is shell-quoted. Under `migrate`, the advice says to fix the rows and re-run, since the migration is already written. +3. **A required column added to an empty table only** (a new unique, primary-key or foreign-key column, or a type with no temporary value such as a native enum) fails its empty-table precheck on a table with rows. Today `db migrate` says "Fix the issue and re-run" and `db update` says to reconcile schema drift. Give it its own advice naming the field: plan it with `migration plan --backfill ` and fill the step, make the field optional, or give it a default. On `db update`, point to `migration plan`. (Review finding D3-1 of dispatch 3; the reviewer confirmed the `--backfill` route applies for an enum column and a unique column.) +4. **The fix names a flag only where it applies.** `--backfill` is suggested for `db update` and `migration plan`, written in the field's contract name when the origin contract is known, and through `statementFlag` so it is shell-quoted. Under `migrate`, the advice says to fix the rows and re-run, since the migration is already written. ## Tests From 3de51d177b629c1076e1bbb421bf2c731d3d5eaa Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:34:56 +0200 Subject: [PATCH 17/26] TML-3478: variant removal moves to 4b; the dotted-name fallback goes when authoring refuses such names Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/deferred.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index 81d35c213dbf..6976c81f8621 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -76,4 +76,4 @@ On MongoDB a variant's field rename touches only that variant's documents. On SQ **Found:** slice 4a review of dispatch 3 (2026-10-09). For the MongoDB authoring owners, not this project. -`@map("o.d")` or `@map("$x")` is accepted in a MongoDB schema. MongoDB reads such a name as a path or an operator, and an insert of `{ "o.d": ... }` already fails the validator Prisma writes. Slice 4a skips such fields when planning statements. The fix belongs in authoring: reject the name with a diagnostic. +`@map("o.d")` or `@map("$x")` is accepted in a MongoDB schema. MongoDB reads such a name as a path or an operator, and an insert of `{ "o.d": ... }` already fails the validator Prisma writes. Slice 4a skips such fields when planning statements. The fix belongs in authoring: reject the name with a diagnostic. When it lands, remove the framework resolver's fallback that reads `User.o.d` as field `o.d` (added in slice 4a), and the planner's silent skip of such fields. From e5eaf9da2efbd9d05da9b83360828beabdcaf92e Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:35:20 +0200 Subject: [PATCH 18/26] TML-3478: 4b deletes a removed variant's documents Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/plan.md | 1 + 1 file changed, 1 insertion(+) diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index b08e4a85240a..b6beacad002a 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -69,6 +69,7 @@ - The same two flags as 3a, writing a data transform with a placeholder. - Making a field required on a collection with documents is no longer silent. It is answered with `--backfill`, or the command says the existing documents need the field. - Renaming a field inside a value object rewrites the embedded documents, including lists, dictionaries and unions. +- Removing a variant model is data loss: today its documents stay and fail every update. `--delete ` deletes the documents with that discriminator value, as 3b does for SQL rows. **Hands to:** close-out for MongoDB. From 24b3b8c526a23170a012129c397856e410405cdc Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 13:17:59 +0200 Subject: [PATCH 19/26] TML-3478: record two manual QA findings outside the slice Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/deferred.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index 6976c81f8621..1ae094464714 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -77,3 +77,15 @@ On MongoDB a variant's field rename touches only that variant's documents. On SQ **Found:** slice 4a review of dispatch 3 (2026-10-09). For the MongoDB authoring owners, not this project. `@map("o.d")` or `@map("$x")` is accepted in a MongoDB schema. MongoDB reads such a name as a path or an operator, and an insert of `{ "o.d": ... }` already fails the validator Prisma writes. Slice 4a skips such fields when planning statements. The fix belongs in authoring: reject the name with a diagnostic. When it lands, remove the framework resolver's fallback that reads `User.o.d` as field `o.d` (added in slice 4a), and the planner's silent skip of such fields. + +## A repeated run reports outdated statements one at a time + +**Found:** slice 4a manual QA (2026-10-09), F3. Applies to every family. + +Running the same `db update` again after it applied fails on the first outdated `--rename` or `--delete`, so a user with three outdated flags needs three runs to learn about all of them. The resolver could report every statement that no longer resolves in one error. + +## A collection name MongoDB rejects reaches the runner + +**Found:** slice 4a manual QA (2026-10-09), F9. For the MongoDB authoring owners. + +`@@map("system.x")` is accepted in a MongoDB schema. A rename onto it is refused with a reason, but a plan that creates or drops such a collection fails at apply with `CLI.UNEXPECTED`. Authoring should reject collection names MongoDB rejects. From 1f3401e12b37b9f28e1c533763c85f550a1813cf Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 12:21:31 +0200 Subject: [PATCH 20/26] TML-3477: record the unfilled-slot migrate advice gap Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/deferred.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index 1ae094464714..a8c33c26c718 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -89,3 +89,9 @@ Running the same `db update` again after it applied fails on the first outdated **Found:** slice 4a manual QA (2026-10-09), F9. For the MongoDB authoring owners. `@@map("system.x")` is accepted in a MongoDB schema. A rename onto it is refused with a reason, but a plan that creates or drops such a collection fails at apply with `CLI.UNEXPECTED`. Authoring should reject collection names MongoDB rejects. + +## `db migrate` of a migration with an unfilled placeholder gives advice that does not mention the slot + +**Found:** slice 3a review (2026-10-12). Older than this project's slices. + +A migration written with a placeholder has `ops.json` as `[]`. Running `db migrate` before filling the slot applies nothing and then fails at the destination check with general advice. It should say that the migration still has an unfilled slot, name the slot, and say to fill it and run `node migration.ts`. From 4d9570aad7a3c7d737b4804126fb24f415b2bebf Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 12:26:27 +0200 Subject: [PATCH 21/26] TML-3477: a delete answer to a type change runs the database's cast, reported as accepting the lossy change Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/slices/convert-backfill/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/migration-statements/slices/convert-backfill/spec.md b/projects/migration-statements/slices/convert-backfill/spec.md index e9c341b723aa..12895c44a223 100644 --- a/projects/migration-statements/slices/convert-backfill/spec.md +++ b/projects/migration-statements/slices/convert-backfill/spec.md @@ -48,7 +48,7 @@ SQLite works the same way. Two SQLite gaps close on the way: 1. **Temporary values without asking.** Without `--backfill`, `migration plan` writes fill values such as `''` or `0` into existing rows. The project spec already decided this. It is the biggest behaviour change in the slice. 2. **`alterColumnType` changes shape.** Its options become `{ table, column, type, using? }`. Today they expose internal names. Hand-written migrations that call it need the upgrade instructions. -3. **A type change answered with `delete` is reported honestly.** The output says `delete values of field "User.age" (type change)`, not "delete field". Slice 2's QA found the old wording misleading. +3. **A type change answered with `delete` runs the database's cast** (decided by Will, 2026-10-12). Values the database can cast are kept; a value it cannot cast fails the migration, and the failure points to `--convert`. The statement is reported as `accept lossy type change of field "User.age"`, not as deleting the field or its values. ## The two flags From b049c68c4f3f3b6e79c93c0a83bac4a178c5c65e Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 12:28:27 +0200 Subject: [PATCH 22/26] TML-3474: both slices add MigrationDataLoss; the second to merge combines them Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/slices/convert-backfill/plan.md | 1 + projects/migration-statements/slices/mongo-renames/plan.md | 1 + 2 files changed, 2 insertions(+) diff --git a/projects/migration-statements/slices/convert-backfill/plan.md b/projects/migration-statements/slices/convert-backfill/plan.md index 5b419702253a..7d86a0aea322 100644 --- a/projects/migration-statements/slices/convert-backfill/plan.md +++ b/projects/migration-statements/slices/convert-backfill/plan.md @@ -35,3 +35,4 @@ One implementer and one reviewer, both on Opus, resumed across every dispatch. B ## Open items - ADR 200 amendment: written by the orchestrator in dispatch 6's branch, not by the implementer. +- Both 3a and 4a add a framework type `MigrationDataLoss` (3a with an optional `label`, 4a with an optional `keepDataAdvice`). Whichever merges second combines them into one type. diff --git a/projects/migration-statements/slices/mongo-renames/plan.md b/projects/migration-statements/slices/mongo-renames/plan.md index 371a75301b66..113fae5dffcf 100644 --- a/projects/migration-statements/slices/mongo-renames/plan.md +++ b/projects/migration-statements/slices/mongo-renames/plan.md @@ -33,3 +33,4 @@ One implementer and one reviewer, both on Opus, resumed across every dispatch. B - ADR 188 and ADR 264 amendments: written by the orchestrator in dispatch 4's branch. - Dispatch 1 found TML-2447's gaps fixed on `main`: variants share their base's collection, and the validator has a `oneOf` branch per variant pinned to its discriminator value. Variant field renames stay in 4a. Dispatch 4 removes the stale "Known gaps" paragraph from the MongoDB Family doc § Polymorphic variants. +- Both 3a and 4a add a framework type `MigrationDataLoss` (3a with an optional `label`, 4a with an optional `keepDataAdvice`). Whichever merges second combines them into one type. From b256803f777d71766ef96313b98ffb493f3921b8 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 12:32:22 +0200 Subject: [PATCH 23/26] TML-3477: a SQLite rebuild fails on a value its retyped column cannot hold Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/slices/convert-backfill/spec.md | 1 + 1 file changed, 1 insertion(+) diff --git a/projects/migration-statements/slices/convert-backfill/spec.md b/projects/migration-statements/slices/convert-backfill/spec.md index 12895c44a223..7f320106fd87 100644 --- a/projects/migration-statements/slices/convert-backfill/spec.md +++ b/projects/migration-statements/slices/convert-backfill/spec.md @@ -128,6 +128,7 @@ Everything here serves one rule: a placeholder migration is written only when a - Convert and backfill reach the strategies as destination coordinates through the strategy context. They do not go through `planStatements`, because they rename nothing. - Strategies gated: Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`. - `alterColumnType`'s after-check compares the column's `format_type` text, which `migration.ts` no longer carries. The facade derives it from `type` through the control stack's data types (any of a data type's texts, rendered as its `format_type` text; a type that names its own kind, such as an enum, from its quoted name), and the planner derives it the same way, so `ops.json` is unchanged by a re-run. A `type` the stack cannot read fails with a structured error. A test runs every Postgres data type through both and checks they agree. Decided 2026-10-09 by the orchestrator: comparing only the type OID would let a length-only change look already applied, so the runner would skip it. +- SQLite does not cast on a table rebuild: a value its new column cannot hold stays in its old storage class. So a direct rebuild of a retyped column (no statement, or `--delete`) checks, after the copy and inside the rebuild, that every value's storage class fits the new column, and fails naming the field with `--convert` advice otherwise. A `--convert` rebuild runs the user's expression unchecked. Decided 2026-10-12 by the orchestrator, applying Will's rule for `--delete` on type changes. - `AlterColumnTypeClass` gains `data`. The `using` slot renders as code, imports `placeholder`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result. - `recreateTable` gains a per-column copy expression, used by convert and by the SQLite temporary value. SQLite gets a fill-value map: integer `0`, real `0`, text `''`, blob `X''`, and what its boolean and date-time codecs store. - `MigrationRunnerFailure` gains `fix`. The `db update`, `db init` and `migrate` error mappers prefer it. From d427532a0a31fa7fc5e302b8fc9a16f0a3e7d1a2 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 12:36:15 +0200 Subject: [PATCH 24/26] TML-3478: record that db migrate applies an ops.json older than its migration.ts Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/deferred.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index a8c33c26c718..75b1441e6b64 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -95,3 +95,9 @@ Running the same `db update` again after it applied fails on the first outdated **Found:** slice 3a review (2026-10-12). Older than this project's slices. A migration written with a placeholder has `ops.json` as `[]`. Running `db migrate` before filling the slot applies nothing and then fails at the destination check with general advice. It should say that the migration still has an unfilled slot, name the slot, and say to fill it and run `node migration.ts`. + +## `db migrate` applies an `ops.json` older than its `migration.ts` + +**Found:** slice 4a review (2026-10-12). Older than this project. + +The Migration System doc lists emit-drift detection (re-emit `migration.ts` in memory and compare with `ops.json` before applying) as architecturally required and not yet implemented. Without it, a user who edits `migration.ts` and forgets to run it applies the old operations; in slice 4a's review that turned a wrong advice text into a dropped collection. Worth a ticket when this project closes, if none exists. From c47b8b47878eb3a2f4526c75e3f27f2a38295fe8 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 12:43:50 +0200 Subject: [PATCH 25/26] TML-3553: TypeScript MongoDB validator parity is the next slice; 4a refuses the equal-hash case until then Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/plan.md | 12 +++ .../slices/mongo-renames/spec.md | 1 + .../mongo-ts-validator/investigation.md | 100 ++++++++++++++++++ 3 files changed, 113 insertions(+) create mode 100644 projects/migration-statements/slices/mongo-ts-validator/investigation.md diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index b6beacad002a..91c59007bc8c 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -10,6 +10,7 @@ | 2 | Data loss refused until answered; `--delete`, `--allow`; the interactive prompt | 1 | Merged, prisma/orm#30648 | | 3a | `--convert` and `--backfill`, Postgres and SQLite | 2 | Spec: [`slices/convert-backfill/`](./slices/convert-backfill/spec.md) | | 4a | Renames and deletes on MongoDB | 2 | Spec: [`slices/mongo-renames/`](./slices/mongo-renames/spec.md) | +| 4p | MongoDB contracts written in TypeScript get the same validator as PSL; removes 4a's temporary refusal | 4a | Next after 4a | | 3b | The remaining SQL renames and deletes | 3a | Not started | | 4b | `--convert`, `--backfill` and value object renames on MongoDB | 3b, 4a | Not started | @@ -47,6 +48,17 @@ **Hands to 4b:** a MongoDB planner that takes statements. +### 4p — TypeScript MongoDB contracts get the PSL validator + +**Linear:** [TML-3553](https://linear.app/prisma-company/issue/TML-3553) · **Folder:** [`slices/mongo-ts-validator/`](./slices/mongo-ts-validator/investigation.md) + +Decided by Will on 2026-10-12: contracts written in either form must behave the same, and this slice comes right after 4a, before 3b and 4b. Two pull requests: + +1. The validator derivation moves into `@internal/mongo-contract`. No output changes. +2. The TypeScript builder calls it. 4a's temporary refusal goes. An upgrade fragment covers the closed validator TypeScript users gain. + +**Why now:** without a validator, a TypeScript contract's field names are not in its storage hash, so a MongoDB field rename on it gives a migration `db migrate` refuses. 4a refuses that case for now. + ### 3b — The remaining renames and deletes on Postgres and SQLite **Linear:** [TML-3537](https://linear.app/prisma-company/issue/TML-3537) · **Folder:** `slices/remaining-nouns/` diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md index efba9a14ddbc..15a19ed225ef 100644 --- a/projects/migration-statements/slices/mongo-renames/spec.md +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -44,6 +44,7 @@ Without `--delete User.nickname`, both commands refuse, as they do on SQL: 3. **Rewrites are classed by what they do to data, as on SQL.** A field rename is `widening`, like a column rename. A field removal is `destructive`, like a column drop. The planner writes a `data` operation only as a placeholder scaffold. So `db update` runs these rewrites, as it runs SQL renames. ADR 188 changes one sentence: today it says every MongoDB data transform is `data`. 4. **Without the old contract, a removed field is named `.`,** for example `users.nickname`. SQL uses `.
.`, or `
.` on SQLite. 5. **A variant's field rename touches only that variant's documents.** The rewrite filters on the variant's discriminator value, so another variant in the same collection that stores a field with the same name keeps it. This differs from SQL single-table storage, where a rename renames the shared column for every row. +6. **Contracts without a validator get a temporary refusal** (Will, 2026-10-12). A MongoDB contract written in TypeScript has no validator today, so its field names are not in its storage hash, and a field rename or removal would give a migration whose start and end hashes are equal, which `db migrate` refuses. `migration plan` refuses such a change and says why; `db update` is unaffected. TML-3553, the next slice, gives TypeScript contracts the PSL validator and removes this refusal. ## How it works diff --git a/projects/migration-statements/slices/mongo-ts-validator/investigation.md b/projects/migration-statements/slices/mongo-ts-validator/investigation.md new file mode 100644 index 000000000000..76f96f2cd4c8 --- /dev/null +++ b/projects/migration-statements/slices/mongo-ts-validator/investigation.md @@ -0,0 +1,100 @@ +# Grounding: `$jsonSchema` validators for TypeScript-authored MongoDB contracts + +Read-only investigation. Paths are repo-relative. No tests were run. + +## 1. The gap is real + +- PSL: the interpreter sets `coll['validator']` for every non-variant model's collection (`packages/2-mongo-family/2-authoring/contract-psl/src/interpreter.ts:1703-1733`), then passes it into `buildMongoStorage`, which computes the storage hash (`interpreter.ts:1737-1751`). +- TypeScript: `buildCollections` writes only `indexes`, `options` and `control` (`packages/2-mongo-family/2-authoring/contract-ts/src/contract-builder.ts:2395-2416`). `defineContract` hashes that body (`contract-builder.ts:2553-2606`). There is no validator anywhere in `contract-ts/src`. +- Committed emitted output confirms it: + - TS-authored `test/integration/test/mongo/fixtures/generated/contract.json`: `tasks` has only `kind`; `users` has `indexes`, `kind`, `options`. + - TS-authored `examples/bundle-size/src/mongo/generated/contract.json`: `notes` has only `kind`. + - PSL-authored `test/integration/test/mongo/temporal-presets/_fixture/generated/contract.json`: every collection has `kind` and `validator`. +- Side-by-side fixture `test/integration/test/authoring/side-by-side/mongo/` builds the same schema both ways. The test strips `validator` and `storageHash` before comparing (`test/integration/test/authoring/side-by-side-contracts.test.ts:273-292`, `306-327`). Everything else is equal, so the validator is the only storage difference. The stripping exists since commit `3d9d8bf1e9` (2026-04-10). +- The consequence for renames: a self-edge (`from === to`) migration is refused unless it runs a data transform (`packages/1-framework/3-tooling/migration/src/errors.ts:218-226`); `migration new` reports `MIGRATION.NO_CHANGES` (`packages/1-framework/3-tooling/cli/src/control-api/operations/migration-new.ts:189-190`). + +## 2. How PSL derives the validator + +- Code: `packages/2-mongo-family/2-authoring/contract-psl/src/derive-json-schema.ts` (196 lines). Two exports: `deriveJsonSchema` and `derivePolymorphicJsonSchema`. +- Inputs are all contract-level, not PSL-level: + - domain fields (`ContractField` from `@internal/contract/types`), including `nullable`, `many` (with `elementNullable`), scalar `codecId`, `valueSet`, and value-object references; + - `MongoTypeLookups` = codec lookup (`descriptorFor`) + data type lookup (`get`), defined in `packages/2-mongo-family/1-foundation/mongo-contract/src/mongo-data-type.ts:38-55`. `bsonTypesOfCodec` reads `bsonTypes` from the codec's data type; + - value objects (`ContractValueObject`), recursed into as nested closed objects; + - storage value sets (for the `enum` keyword); + - for polymorphism: base fields, discriminator field name, and per-variant `{ discriminatorValue, fields }`, producing `oneOf` branches. +- Behaviour: every object level is closed (`additionalProperties: false`); non-nullable fields go in `required`; an empty `bsonTypes` list (`Bson`) means "any value"; a codec with no Mongo data type returns `undefined` and the field is left out of `properties`. Level and action are fixed: `strict` / `error` (`derive-json-schema.ts:128-132`, `191-195`). +- Imports: only `@internal/contract/types` and `@internal/mongo-contract`. No PSL AST, no diagnostics. +- The per-collection loop (pick base models, build variant entries, choose plain or polymorphic) lives in the PSL interpreter (`interpreter.ts:1703-1733`). It reads only the resolved domain models (`base`, `discriminator`, `variants`, `fields`, `storage.collection`). It is also surface-neutral. +- Current callers: the PSL interpreter, its tests, and `packages/3-extensions/mongo/test/mongo.enum.e2e.test.ts:10` (through the `@internal/mongo-contract-psl` export). +- Where it can live: + - `@internal/mongo-contract` (`packages/2-mongo-family/1-foundation/mongo-contract`) is layer `foundation`, plane `shared` (`architecture.config.json:676-680`). It already owns `MongoValidator`, `bsonTypesOfCodec` and `buildMongoStorage`. Both authoring packages already depend on it. This is the right home. + - `9-family` is not possible: authoring (`shared` plane, lower layer) cannot import it. + - `contract-ts` is possible (`contract-psl` already depends on `@internal/mongo-contract-ts`), but it would make the TS builder the owner of a rule both surfaces share. + +## 3. Is the omission deliberate? + +- No ADR, doc or commit gives a reason. The original plan (`9d07bdb93f`, `projects/mongo-schema-migrations/plan.md`) said "Emitter: auto-derive `$jsonSchema` from model fields". The work landed as "PSL authoring + emitter" (`92d9841fef`, `9e1f160aeb`), and the TS builder was never wired. Closing the validators (`48f2f73a5e`, TML-2730) again touched only PSL. +- The gap is known and has been documented as behaviour since then: + - `skills/prisma-8/references/contract.md:313`: "A TypeScript Mongo contract has no collection validator and accepts these enums." + - Shipped upgrade instructions tell users to use a TS contract as a workaround: `upgrade-instructions/releases/8.0.0-rc.16-to-8.0.0-rc.17/sources/enum-members-hold-read-values/app/instructions.md:19` and `:78` ("author the enum ... in a TypeScript contract, which carries no collection validator"). Same wording in `skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md`. + - `upgrade-instructions/releases/8.0.0-rc.12-to-8.0.0-rc.13/sources/mongo-json-and-bson/app/instructions.md:39` and `:46` describe the TS case as having no validator. + - `test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts:84` switched to a PSL project because TS fixtures emit no validator (also `wip/beowulf-40-handbacks.txt:2117`). +- The Prisma 6 reader omits validators on purpose: `packages/2-mongo-family/2-authoring/contract-prisma6/README.md:9` ("Prisma 6 databases have none"). Its parity test strips them (`contract-prisma6/test/parity.test.ts:24-60`). +- No opt-out exists on either surface. Level and action are hard-coded. `validationLevel: 'off'` is not in the type (`mongo-contract/src/ir/mongo-validator.ts:3-4`). + +## 4. Blast radius + +TS-authored MongoDB contracts whose `contract.json` and storage hash would change: + +- Committed emitted fixtures (checked by `pnpm fixtures:check` / `emit:check`): + - `examples/bundle-size/src/mongo/generated/contract.{json,d.ts}` + - `test/integration/test/mongo/fixtures/generated/contract.{json,d.ts}` (emitted with `|| true` in `test/integration/package.json:7`) +- TS contracts emitted at test time (no committed JSON, but assertions may change): + - `test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-*.ts` (used by `cli-journeys/mongo-migration.e2e.test.ts`, `mongo-db-update-consent.e2e.test.ts`, `delete-statements-migration.mongo.e2e.test.ts`). `mongo-migration.e2e.test.ts:235-238` asserts every op is `additive`; the first plan will now also create each collection with its validator. + - `test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/control-policy/mongo/contract.ts` (used by `cli.control-policy.mongo.e2e.test.ts`). `tolerated`, `external` and `observed` collections gain an expected validator, so verify outcomes may change. + - `test/integration/test/fixtures/cli/cli-integration-test-app/fixtures/emit-command/contract.mongo.ts` (`cli.emit-command.test.ts`). + - `test/integration/test/contract-space-fixture-mongo/contract.ts`, `test/integration/test/mongo-runtime/query-builder.test.ts`, `test/integration/test/enum-members/enum-members.mongo-ts.test.ts`, `test/integration/test/authoring/mongo-pack-composition.test.ts`, `test/integration/test/ports/engines/queries/data_types/enum_type/enum_type.test.ts`, `packages/1-framework/3-tooling/cli/test/control-api/contract-space-seed-phase.mongo.test.ts`. +- Unit tests in `packages/2-mongo-family/2-authoring/contract-ts/test/`: + - `contract-builder.storage-hash-pin.test.ts` pins a hash; it changes. + - `contract-builder.dsl.test.ts:90-91, 307` assert collections equal `{ kind: 'mongo-collection' }`. + - Many tests use fake target packs with no codecs or data types (for example `storage-hash-pin.test.ts:13-20`). With today's derivation every field would be dropped and the validator would be a closed empty object. +- Side-by-side test: the validator strip can be removed, which is the parity check we want (`side-by-side-contracts.test.ts:273-327`). +- Prisma 6 reader: produces no validator by design and stays that way; its parity test keeps stripping. +- `contract infer`: Postgres only (`packages/3-targets/3-targets/postgres/src/core/psl-infer/`). No MongoDB infer exists. +- Examples: `mongo-demo`, `mongo-blog-leaderboard`, `retail-store` are PSL; they do not change. + +Three behaviours the TS builder would need that PSL already has: + +- **`_id` requirement.** PSL refuses a model without an `objectId` `_id` (`PSL_MONGO_ID_REQUIRED`, `interpreter.ts:1563-1580`). The TS builder has no such rule. A closed validator for a model without `_id` rejects every insert, because MongoDB adds `_id`. Several TS unit-test models have no `_id`. +- **Enums the validator cannot list.** PSL refuses enums whose codec stores a BSON type JSON cannot represent, or non-finite doubles (`packages/2-mongo-family/9-family/src/core/authoring-entity-types.ts:113, 151`). That check is in `9-family`, which the TS builder cannot import. Without it, a TS enum over `mongo/int64@1` would get a validator that refuses every write, which is exactly the bug rc.17 fixed for PSL. +- **Complete lookups.** The TS builder builds its codec lookup from the target only (`contract-builder.ts:2563`). PSL gets codec and data type lookups from the full control stack (`contract-psl/src/provider.ts:45-46`). The target pack carries both codecs and `dataTypes` (`packages/3-mongo-target/1-mongo-target/src/core/descriptor-meta-runtime.ts:22-27`); the adapter and family contribute none. Extension packs must be included, or an extension-typed field is dropped and then rejected by `additionalProperties: false`. + +## 5. What changes for an existing user of a TS contract + +- After upgrading and re-emitting, the storage hash changes. The runtime marker check warns once per runtime (`verifyMarker: 'onFirstUse'`). +- `db verify` fails: a validator the contract declares but the database lacks is a `fail` issue (`packages/2-mongo-family/9-family/src/core/schema-diff.ts:198-212`), subject to the collection's control policy. +- `migration plan` / `db update` plan one `collMod` per existing collection, id `validator..add`, class `widening` (`packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts:439-460`; ADR 195 note dated 2026-10-07). `db init` refuses when a collection holds data and needs a validator, and points at `db update` (`test/integration/test/cli-journeys/mongo-db-update-consent.e2e.test.ts:178-204`). +- Existing documents: MongoDB does not check existing documents when `collMod` sets a validator. With `validationLevel: 'strict'` and `validationAction: 'error'`, every later insert, and every update to any document, must produce a valid document. A stored document that does not match (a missing required field, a wrong BSON type, or any extra field, because the schema is closed) can no longer be updated unless the update makes it valid. Writes from other applications that add fields outside the contract start failing. This is the main user risk, and it is larger for TS users, who have run with an open schema until now. +- Precedent for upgrade instructions: + - `skills/upgrade/prisma-next-upgrade/upgrades/0.11-to-0.12/instructions.md`, change `re-emit-closed-mongo-contracts`: re-emit, then apply the validator migration with `db update`. + - ADR 258 (`docs/architecture docs/adrs/ADR 258 - List cardinality has independent container and element nullability.md:104`): re-emit, then apply a new validator migration; runtime deployment alone does not change stored validators. + - Current location for new instructions: `upgrade-instructions/pending//app/instructions.md` (see `upgrade-instructions/pending/`). + +## 6. Size + +One PR is possible but heavy. It would contain: moving the derivation and the per-collection loop to `@internal/mongo-contract`; moving the enum-listability check down from `9-family`; adding an `_id` rule to the TS builder; building full lookups in the TS builder; regenerating two committed fixtures; updating TS unit tests and several e2e assertions; removing the side-by-side strip; upgrade instructions; and correcting `skills/prisma-8/references/contract.md:313` plus the advice in shipped rc.17 instructions (a new pending note, since shipped notes do not change). The e2e changes (control policy, mongo migration journeys) are the least predictable part. + +A clean split is two PRs: + +1. Move `derive-json-schema.ts`, the per-collection loop and the enum-listability check to `@internal/mongo-contract`. PSL calls them. No output change, so no fixture change. +2. The TS builder calls them, adds the `_id` rule, builds lookups from target plus extensions, regenerates fixtures, updates tests, and adds upgrade instructions. + +## Design questions and recommendations + +1. **Where does the derivation live?** Recommend `@internal/mongo-contract`, as one function taking domain models, value objects, value sets and `MongoTypeLookups`, returning validators per collection. Both surfaces call it. +2. **Should the TS builder require an `objectId` `_id`?** Recommend yes, with the same rule as PSL, raised as a contract error. The alternative (always allow `_id` in the schema) would make the two surfaces differ again. +3. **What happens when a field's codec has no Mongo data type?** Today the field is silently dropped, then rejected by the closed schema. Recommend making it an error in the shared function. This also forces TS unit tests with fake packs to use the real target pack, or a pack with data types. +4. **Enums the validator cannot list.** Recommend moving the check to the shared package and applying it on both surfaces. Write a pending upgrade note that withdraws the rc.17 "use a TypeScript contract" workaround. +5. **Opt-out (`validator: false` or `validationLevel: 'moderate'`)?** Recommend no opt-out in this change. Neither surface has one. Adding one to TS only would recreate the asymmetry. If users need it, add it to both surfaces later. +6. **Prisma 6 reader?** Recommend leaving it without validators. Its README states the reason, and it describes an existing Prisma 6 database. +7. **One PR or two?** Recommend two: a no-output-change move first, then the TS behaviour change with fixtures and upgrade instructions. From 50d1706225aa667359b78993bd5f7937234d24fd Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 13:05:03 +0200 Subject: [PATCH 26/26] TML-3477: dispatch 6 documents the lossy cast; record two review notes Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/deferred.md | 11 +++++++++++ .../dispatches/06-journeys-docs-fragments.md | 1 + 2 files changed, 12 insertions(+) diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index 75b1441e6b64..e79746a3f208 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -101,3 +101,14 @@ A migration written with a placeholder has `ops.json` as `[]`. Running `db migra **Found:** slice 4a review (2026-10-12). Older than this project. The Migration System doc lists emit-drift detection (re-emit `migration.ts` in memory and compare with `ops.json` before applying) as architecturally required and not yet implemented. Without it, a user who edits `migration.ts` and forgets to run it applies the old operations; in slice 4a's review that turned a wrong advice text into a dropped collection. Worth a ticket when this project closes, if none exists. + +## A `--convert` expression can leave values the field's type cannot hold + +**Found:** slice 3a review (2026-10-12). + +A SQLite `--convert` rebuild runs the user's expression without the storage-class check the direct rebuild gets, so `trim("n")` on a field changed to `Int` leaves `'abc'` as text, and the ORM returns `"abc"` for an `Int` field. Postgres refuses such a value itself. Consider running the same check after a converted copy. + +## `migration status --json` prints `{bin}` unreplaced + +**Found:** slice 3a review (2026-10-12). Outside this project. The summary text in the JSON output contains a raw `{bin}` placeholder. + diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md b/projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md index 1ebac0664c98..43fd3bd9660a 100644 --- a/projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md +++ b/projects/migration-statements/slices/convert-backfill/dispatches/06-journeys-docs-fragments.md @@ -17,6 +17,7 @@ The slice's done conditions hold through journeys on Postgres and SQLite, and th 2. **Follow every printed advice to its end** (F46), on each target, including the `db update` refusal through `migrate` on a database created by `db init`. 3. **Docs.** - Migration System doc § Statements: `--convert` and `--backfill`; the loss kinds; the questions per command; remove the sentence saying the Postgres planner writes no placeholder data transforms (§ Authoring intermediate state); § NOT NULL columns without defaults now applies to `migration plan` too. + - Migration System doc: what a `--delete` answer to a type change does (the database's cast on Postgres; the storage-class check after a SQLite rebuild), how it is reported, and the `--convert` advice when a cast fails. - `docs/reference/error-reference.md`: every code added or changed. - The CLI README sections for `migration plan` and `db update`. - `skills/prisma-8/references/migrations.md`: type changes and new required fields.