From 0d63f59ebf258bd56f984d522b55cc9c841f7645 Mon Sep 17 00:00:00 2001 From: willbot Date: Thu, 8 Oct 2026 19:18:59 +0200 Subject: [PATCH 01/57] TML-3474: re-plan the remaining slices; specs for 3a and 4a Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/plan.md | 53 ++++++-- .../slices/convert-backfill/spec.md | 121 ++++++++++++++++++ .../slices/mongo-renames/spec.md | 96 ++++++++++++++ projects/migration-statements/spec.md | 2 +- 4 files changed, 260 insertions(+), 12 deletions(-) create mode 100644 projects/migration-statements/slices/convert-backfill/spec.md create mode 100644 projects/migration-statements/slices/mongo-renames/spec.md diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index 0090e6f8d0d8..3365ffefe043 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -1,10 +1,16 @@ # Project plan — Migration statements -**Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474) · **Shaping PR:** prisma/orm#30604 · **Working branch:** `tml-3474-migration-statements` +**Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474) ## Summary -Four slices. The first three stack: the statement surface with renames, then the refusal and consent model, then the scaffolded verbs and the remaining nouns. The fourth, Mongo, builds on the second and runs in parallel with the third. Each slice is one PR, stacked on the working branch until the shaping PR merges and then retargeted to `main`. +Six slices. Slices 1 and 2 are merged. The rest, re-planned on 2026-10-08 when slices 3 and 4 proved too large for one review each: + +- **3a** (SQL `--convert` and `--backfill`) and **4a** (MongoDB renames and deletes) run in parallel. +- **3b** (the remaining SQL nouns) follows 3a. **4b** (MongoDB `--convert`, `--backfill` and value object renames) follows 3b and 4a, and reuses their design. +- The storage-name syntax survey runs beside the slices and goes to Will for a decision. It does not block close-out. + +Each slice is one PR against `main`. ## Slices @@ -30,25 +36,49 @@ Four slices. The first three stack: the statement surface with renames, then the **Hands to.** The refusal shape every later verb hooks its statements into; the per-operation consent model; `--confirm` no longer consents to data loss. -### Slice 3 — Convert and backfill scaffold the placeholder migration, and the remaining nouns +### Slice 3a — `--convert` and `--backfill` on Postgres and SQLite **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/convert-backfill/` -**Outcome.** `--convert` scaffolds the type change with the placeholder in the slot that carries the conversion, and `--backfill` the backfill transform; both refused on `db update`; the scaffolding stops being automatic. `--rename` on enum values, namespaces and value object fields, a model move across namespaces (`alter table set schema` on Postgres, deferred from slice 1 on 2026-10-06 because no operation existed for it), and `--convert` on a variant, plan the row updates, the JSON rewrites and the schema rename. `--delete` on an enum value nulls where nullable, else refuses. +**Outcome.** `migration plan --convert User.age` writes the type change with a `placeholder()` where the conversion goes, and `--backfill User.email` writes the backfill transform. A lossy type change is answered with `convert` or `delete`. Neither verb is accepted by `db update`. The planner no longer scaffolds without being asked; without `--backfill`, a required field on a populated table gets the temporary-default recipe on both SQL targets, which SQLite lacks today. The Postgres runner's advice when `SET NOT NULL` meets NULLs names the NULLs (TML-3517). **Builds on.** Slice 2. +**Hands to.** The convert and backfill verbs, their questions and their scaffold rule, for 3b (variants) and 4b (MongoDB). + +### Slice 3b — The remaining nouns on Postgres and SQLite + +**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/remaining-nouns/` + +**Outcome.** `--rename` on enum values, namespaces and value object fields (building the framework's value object statement surface: statement entity, subject kind, grammar, resolver), a model move across namespaces (`alter table set schema` on Postgres), and `--convert` on a variant whose discriminator value changed, each planning the row updates, JSON rewrites or schema rename. `--delete` on an enum value nulls it where nullable, else refuses. `--delete ` answers every question for the models in it. + +**Builds on.** Slice 3a. + **Hands to.** Project close-out for the SQL targets. -### Slice 4 — The same statements on MongoDB +### Slice 4a — Renames and deletes on MongoDB + +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo-renames/` + +**Outcome.** The MongoDB planner carries out `--rename` for models (collection rename) and fields (document rewrite), with the validator and indexes kept consistent, through both commands. Removing a field becomes data loss, answered by `--delete` with an `$unset`. Document rewrites run under `db update`. The temporary capability member `renameStatements: { refused: true; keepDataByHand }`, its three CLI reads and its doc mention are deleted. Statements name MongoDB fields the same way they name SQL fields. + +**Builds on.** Slice 2. + +**Hands to.** A MongoDB planner that takes resolved statements, for 4b. + +### Slice 4b — `--convert`, `--backfill` and value object renames on MongoDB + +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo-convert-backfill/` + +**Outcome.** The same two verbs as 3a on MongoDB, scaffolding a data transform with a placeholder. Value object field renames rewrite subdocuments, including lists, dictionaries and unions, using the statement surface from 3b. A validator change that makes a field required on a populated collection is no longer silent: it is answered with `--backfill`, or the command says that existing documents need the field. -**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo/` +**Builds on.** Slices 3a, 3b and 4a. -**Outcome.** The Mongo planner takes the same resolved statements: collection rename, document rewrites for field and value object field renames, drops and unsets for deletes, a data transform scaffold for convert, under the slice 2 refusal and consent model. No family vocabulary enters the framework. +**Hands to.** Project close-out for MongoDB. -**Builds on.** Slice 2. Slice 2 left a temporary member for it: `TargetMigrationsCapability.refusesRenameStatements`, set by MongoDB so a data-loss question offers no `rename` there. Slice 4 deletes the member, its one CLI read and its doc mention when the Mongo planner carries out renames. +### Alongside — A syntax for naming storage objects -**Hands to.** Project close-out for Mongo. +Will asked (2026-10-07) for a critical discussion and a survey of how established tools name tables and other storage objects on the command line, before any syntax is chosen for the `@@map`-only rename gap (`deferred.md`). The write-up goes to Will for a decision. Building the chosen syntax is not in this project unless Will adds it. ## Stretch goal — interactive statements when a human runs the command @@ -66,8 +96,9 @@ What the slices do now so this drops in later (see the spec's cross-cutting requ ## Sequencing -- **Stack:** slice 1 → slice 2 → slice 3. -- **Parallel:** slice 4 runs beside slice 3 once slice 2 has merged. +- **Merged:** slice 1 → slice 2. +- **Parallel now:** 3a and 4a. +- **Then:** 3b after 3a; 4b after 3b and 4a. ## Dependencies diff --git a/projects/migration-statements/slices/convert-backfill/spec.md b/projects/migration-statements/slices/convert-backfill/spec.md new file mode 100644 index 000000000000..1fbf81be20bf --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/spec.md @@ -0,0 +1,121 @@ +# Slice spec — `--convert` and `--backfill` write the placeholder migration only when asked + +**Project:** [`projects/migration-statements/`](../../spec.md) · **Slice 3a** · **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477), [TML-3517](https://linear.app/prisma-company/issue/TML-3517) · **Branch:** `tml-3477-convert-backfill` · **Builds on:** slice 2 (merged). + +## At a glance + +A user changes `User.age` from `String` to `Int` on Postgres: + +```text +$ prisma migration plan --name age-to-int +✖ [CLI.CONSENT_REQUIRED] 1 subject needs a statement, and the session is not interactive. + why: Change type of column "User"."age" would lose the data of field "User.age". +→ Pass --convert User.age +→ Pass --delete User.age + +$ prisma migration plan --name age-to-int --convert User.age +⚠ migration.ts has a placeholder: replace it with the conversion, then run the file to re-emit. +``` + +`migration.ts` then holds one placeholder, where the conversion goes: + +```ts +this.alterColumnType({ + table: 'User', + column: 'age', + type: 'int4', + using: placeholder('User.age:using'), +}), +``` + +The user writes `using: '"age"::integer'` (or any SQL expression), runs the file, and gets an applicable migration. On SQLite the placeholder sits in the table rebuild's copy step for that column. + +Adding a required `email` field to an existing `User` table plans a temporary default by default: the column is added with a fill value and the default is dropped, and nothing is asked. To write the fill yourself, run `prisma migration plan --backfill User.email`. That writes the nullable column, a data transform with a placeholder, and `SET NOT NULL`. + +Today `migration plan` writes these placeholders on its own for every type change and every required field on an existing table. Users who didn't want them get a migration they can't apply until they edit it. After this slice it writes them only when asked. + +## Chosen design + +### The two verbs + +- **`--convert `** answers a type change that would lose data. It is accepted by `migration plan` only. It names the field by its destination name, like the new side of a rename: after `--rename User.age:User.years`, write `--convert User.years`. It needs the origin contract, like `rename`, and fails with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN` without one. It must name a field that exists on both sides and whose type changes. +- **`--backfill `** is an opt-in that answers no question. It is accepted by `migration plan` only. It names a field that is required in the destination and is either new on an existing table or optional in the origin. Anything else is an error that names what was found. +- `db update` declares both verbs only to refuse them, with `MIGRATION.STATEMENT_NEEDS_MIGRATION_FILE`. The error says `db update` has no migration file to fill and names `migration plan`. It doesn't fall through to the engine's generic "no flag registered" error, which doesn't explain the rule. The shared verb table in `orm/statement-verbs.ts` declares the verbs per command, so neither command accepts a verb by accident. + +### What the planners write + +| Diff | No statement | `--convert` | `--backfill` | +| --- | --- | --- | --- | +| Type change the target knows is safe (`int4`→`int8`) | direct alter, `widening`, nothing asked | error: nothing to convert | — | +| Any other type change | refused: question offers `convert` and `delete`. `--delete` plans the direct alter, `destructive` | Postgres: `alterColumnType` with `using: placeholder(…)`. SQLite: `recreateTable` whose copy step takes `placeholder(…)` for the column. Class `data` | — | +| Required field added to an existing table | temporary-default recipe on both targets, both commands, `additive` | — | nullable add, `dataTransform` with placeholders, then `SET NOT NULL` (SQLite: a tightening rebuild) | +| Optional field made required | direct `SET NOT NULL` (SQLite: tightening rebuild), `widening`. Fails at apply if NULLs exist, with advice that names `--backfill` | — | `dataTransform` with placeholders, then `SET NOT NULL` | + +The four automatic placeholder strategies (Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`) fire only for a field a statement names. Statements reach them as destination coordinates through the strategy context. They change no names in the working schema, so they don't go through the rename path in `planStatements`. A converted type change carries no separate NULL-handling transform; the conversion expression can map NULL itself. + +A converted alter is classed `data`, because the user writes its data step. So it isn't a `destructive` operation and leaves `dataLoss`. A typed `convert` at the prompt re-plans like a typed rename, and the type-change question isn't asked again. + +### The questions know why data is lost + +Each `dataLoss` entry gains `loss: 'drop' | 'typeChange'`, set by each target's operation-subjects function. The CLI stops guessing from whether the destination still has the field. A `typeChange` question offers `convert` and `delete` in `migration plan`, and `delete` only in `db update`. A `drop` question offers `rename` and `delete`, as today. The re-plan loop (`askPlanQuestions`) carries every planned statement, not only renames. + +`ResolvedMigrationStatement` gains a `kind`, which is `rename`, `convert` or `backfill`. Every reader that assumes a rename now switches on `kind` first. One example is the SQL family's `StatementPlanner`, which today plans any non-model statement as a column rename. MongoDB keeps refusing statements it can't carry out, until slices 4a and 4b. + +### The temporary default becomes a real migration call + +The Postgres temporary-default and direct-add calls render as `rawSql({ id, label, operationClass })` with no SQL. Re-running such a `migration.ts` doesn't reproduce `ops.json` (cross-cutting requirement 7). They now render as a facade call that builds the same operation. SQLite gets the recipe as a rebuild in which the new column's copy value is a literal. This needs a new per-column copy-expression option on `recreateTable`, which `--convert` also uses, and a SQLite fill-value map (integer `0`, real `0`, text `''`, blob `X''`, and the values its boolean and date-time codecs store). It is refused on the same primary-key, unique and foreign-key conditions as Postgres. This closes the deferred item "SQLite `db update` cannot add a required field to a table that has rows". + +### `alterColumnType` takes plain options + +The Postgres facade's `alterColumnType` options are reshaped so the rendered call shows a readable shape with an optional `using`, and no internal names (`qualifiedTargetType`, `formatTypeExpected`, `rawTargetTypeForLabel`). The `using` slot renders as code, so it can hold `placeholder(…)`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result the CLI already handles. Upgrade instructions are recorded for hand-written migrations. + +### The NOT NULL failure names the NULLs (TML-3517) + +When `SET NOT NULL` or a SQLite tightening rebuild meets NULL values, the runner reports how many rows hold NULL in which column. It says to fill them, make the field optional, or plan with `--backfill`. `MigrationRunnerFailure` gains a `fix`, and the `db update`, `db init` and `migrate` error mappers use it instead of the fixed advice about schema drift. + +## Coherence rationale + +Everything in this slice serves one change: the placeholder migration is written only when a statement asks for it. The verbs, the loss kind, the statement `kind`, the temporary default becoming real on both targets, the `alterColumnType` reshape and the NULL advice are each what that change needs to work on Postgres and SQLite. The reviewer reads one rule and its consequences. + +## Scope + +**In:** +- `--convert` and `--backfill` on `migration plan`, for Postgres and SQLite, refused on `db update`. +- The loss kind on `dataLoss` entries, and the statement `kind`. +- Ending automatic scaffolding. +- The temporary default as a real call on both targets, including the SQLite recipe. +- The `alterColumnType` reshape. +- TML-3517. +- The project spec's wording: "an existing table", not "a non-empty table", since `migration plan` can't see rows. +- Docs: Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`. +- An app upgrade fragment: type changes ask, scaffolds are opt-in, and the temporary default applies under `migration plan`. +- Upgrade instructions for the facade change. + +**Deliberately out:** +- `--convert` on a variant (slice 3b). +- MongoDB (slice 4b). +- A temporary default for a field in a new unique or foreign key, which stays refused as today. +- Detecting rows offline. + +## Pre-investigated edge cases + +| Case | Handling | +| --- | --- | +| A temporary value in a unique column fails on the second row | The recipe is refused for new unique, primary-key and foreign-key columns, as on Postgres today | +| A convert after a rename in the same run | The convert uses the destination name, which the diff issues already use | +| A placeholder operation has no position until re-emit | Applied-statement positions for a convert are empty until the file is re-run, as for every placeholder today | + +## Slice-specific done conditions + +- The project DoD's convert and backfill lines pass on Postgres and SQLite, reading "an existing table". +- Re-running every `migration.ts` the planners write in this slice's tests reproduces its `ops.json`. + +## Open questions + +None. + +## References + +- Grounding report (local, gitignored): `wip/grounding-3a.md`. +- [`../../design-notes.md`](../../design-notes.md): the scaffold rule and the convert and backfill rows. +- [ADR 200](../../../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md): the placeholder flow. diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md new file mode 100644 index 000000000000..aa3f4f79a822 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -0,0 +1,96 @@ +# Slice spec — MongoDB carries out renames and deletes of models and fields + +**Project:** [`projects/migration-statements/`](../../spec.md) · **Slice 4a** · **Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Branch:** `tml-3478-mongo-renames` · **Builds on:** slice 2 (merged). + +## At a glance + +A MongoDB user renames the `Profile` model to `User` (collection `profiles` to `users`), changes `name` to `fullName`, and removes the `nickname` field: + +```text +$ prisma migration plan --name tidy --rename Profile:User --rename User.name:User.fullName --delete User.nickname +✔ Planned 6 operation(s) + Rename collection "profiles" to "users" widening + Drop index users.name_1 widening + Rename field "name" to "fullName" in collection "users" widening + Remove field "nickname" from collection "users" destructive + Create index users.fullName_1 additive + Update validator on users widening +``` + +`migration.ts` carries `renameCollection(...)`, `renameField(...)` and `unsetField(...)` calls, and running it reproduces `ops.json`. `prisma db update` with the same statements does the same against the live database. Today MongoDB refuses every `--rename` and prints steps to run by hand in `mongosh`, and removing a field asks nothing, even though every document that still holds the field then fails each later update against the closed validator. + +Without `--delete User.nickname`, both commands refuse, as they do on SQL: + +```text +✖ [CLI.CONSENT_REQUIRED] 1 subject needs a statement, and the session is not interactive. + why: Remove field "nickname" from collection "users" would lose the data of field "User.nickname". +→ Pass --rename 'User.nickname:' +→ Pass --delete User.nickname +``` + +## Chosen design + +### Statements name fields by their contract key + +A statement names a field as the contract and the generated client name it: the key in the model's `fields`. On SQL that key is the field name in the schema; on MongoDB the contract keys fields by the stored name (`id @map("_id")` is `User._id`), and the client uses the same name. The resolver already behaves this way, so nothing changes in it. On MongoDB a change to `@map` alone is a field rename in contract terms and is stated with `--rename`; a change to the schema field name alone changes nothing stored and needs no statement. This settles the deferred item "Which field name a MongoDB statement uses" and is recorded in the Migration System doc. + +### Collection rename + +A new DDL operation, `renameCollection`, added through the MongoDB stack the way the other collection operations are: query AST command and visitor, wire command and lowering, driver, operation serializer, preview text, factory and call class. Precheck: the source collection exists and the target does not. Postcheck: the target exists. Class `widening`. A model rename renames the collection in the working copy of the origin schema before the diff, so the indexes and validator that MongoDB keeps through a rename are not dropped and recreated. + +### Field rename and field removal rewrite the documents + +- A field rename plans `renameField(collection, from, to, { filter })`: one `updateMany` with `$rename`, run with validation bypassed. Filter: documents that hold the old field, and for a variant's field also the variant's discriminator value. Precheck: no matching document already holds the new field, since `$rename` would overwrite it. Postcheck: no matching document still holds the old field, so a re-run skips it. Class `widening`. +- Removing a field from a model is data loss, matching the spec's rule that dropping a field is destructive. The planner plans `unsetField(collection, field, { filter })`, an `updateMany` with `$unset`, class `destructive`, and reports a `field` subject. The question offers `rename` and `delete`, as on SQL. A field removed together with its collection gets no unset; the collection drop covers it. +- Order within a plan: collection renames, collection creates, index drops, document rewrites (renames and unsets), index creates, validator changes, option changes, collection drops. Rewrites come after index drops so a unique index on the old field does not fail once documents lose it, and before index creates so a unique index on the new field sees the moved values. Validation is bypassed for the rewrite because the origin validator requires the old field and the destination validator may require a field the same plan adds. Plans without statements and without field removals keep their current operations and order. +- The rewrite command gains a `bypassDocumentValidation` option through its wire form and the driver. + +### Document rewrites run under `db update` + +`db update` allows `additive`, `widening` and `destructive`, not `data`, so a rewrite classed `data` would be refused there. The rewrites above carry the class of what they do to data (`widening` for a rename, `destructive` for an unset), and the MongoDB runner and operation serializer tell a rewrite from a DDL command by its shape, not by `operationClass === 'data'`. `data` stays the class of transforms a user writes or a scaffold produces; `db update` keeps excluding it. Slice 3b's SQL JSON rewrites follow the same rule. + +### What is refused + +The MongoDB planner refuses, as a `statementRejected` conflict naming the reason: a rename of `_id`; a collection rename whose target exists in the working schema; a rename of a field stored on one side only; and any statement when the control policy is not `managed`, as SQL does. A document that already holds the new name fails at apply time by precheck, before anything is written. + +### The temporary refusal goes + +`TargetMigrationsCapability.renameStatements` and `keepDataByHand` are deleted: the member in `control-migration-types.ts`, MongoDB's setting in `control-target.ts`, `keepDataByHand` and `keepTheData` in `mongo-planner.ts`, `keepDataByHandFor` and its parameter in `plan-questions.ts`, its reads in `db-run.ts` and `migration-plan.ts`, and the Statements paragraph that describes it in the Migration System doc. The unrelated `renameStatements()` helper in `statement-text.ts` stays. The member shipped in 8.0.0-rc.17, so an extension upgrade fragment tells target authors to remove it. + +## Coherence rationale + +One reviewer can hold this: it is one family learning the two verbs the SQL targets already carry out, through one new DDL operation and one new rewrite shape, with the refusal member removed as the visible proof. The collection rename's six-package plumbing is mechanical and mirrors existing collection operations. + +## Scope + +**In:** model rename (collection rename); top-level field rename, including a variant's field; field removal as data loss with `$unset` under `--delete`; both commands; the `bypassDocumentValidation` option; runner and serializer telling rewrites apart by shape; deleting `renameStatements`; docs (Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`); an app upgrade fragment (MongoDB renames work; removing a field now asks) and an extension upgrade fragment (the capability member is gone). + +**Deliberately out:** +- Value object field renames on MongoDB. They need the framework's value object statement surface (statement entity, subject kind, grammar, resolver), which slice 3b builds for SQL JSON rewrites, and a recursive update pipeline for lists, dictionaries and unions. Moved to slice 4b. +- `--convert` and `--backfill` on MongoDB, and the required-field validator gap: slice 4b. +- Batching large rewrites. One `updateMany` per field, as today's hand-written transforms do. + +## Pre-investigated edge cases + +| Case | Handling | +| --- | --- | +| Slice 1 manual QA (F3): a unique index on the old field fails the rewrite once two documents lose it; one on the new field fails if created first | Order above: index drops, rewrite, index creates | +| The origin validator requires the old field and forbids the new one | Rewrite runs with validation bypassed | +| Two variants in one collection store a field with the same name | Rewrite filters on the discriminator value | +| The runner is not transactional | Postcheck makes a re-run skip a finished rewrite | +| `bypassDocumentValidation` needs a privilege on hosted clusters | Runner failure names the privilege | + +## Slice-specific done conditions + +- The project DoD's MongoDB journey without the value object part: a model rename and a field rename (with a unique index on the field) and a field removal under `--delete`, through `migration plan` then `migrate` and through `db update`, on collections with documents. Afterwards the documents are under the new names, the removed field is gone from every document, every document accepts an update, a further plan is empty, and `db verify --schema-only` is clean. +- `git grep -n "keepDataByHand\|TargetMigrationsCapability\['renameStatements'\]"` finds nothing. + +## Open questions + +None. + +## References + +- Grounding report for this slice (local, gitignored): `wip/grounding-4a.md`. +- [`../../deferred.md`](../../deferred.md) — "Which field name a MongoDB statement uses". +- Slice 1 manual QA, F3: [`../../manual-qa-reports/2026-10-07-qa-opus.md`](../../manual-qa-reports/2026-10-07-qa-opus.md). diff --git a/projects/migration-statements/spec.md b/projects/migration-statements/spec.md index 746c6e454693..25683c4676b8 100644 --- a/projects/migration-statements/spec.md +++ b/projects/migration-statements/spec.md @@ -121,7 +121,7 @@ In addition to the team DoD floor in [`drive/calibration/dod.md`](../../drive/ca - The same change through `db update` with the same statements on both targets produces the same database state and no consent prompt. Running the same `db update` again with the same statements fails on the first statement. - The same change with `--delete` omitted makes both commands refuse, and the refusal text contains the exact statement that then succeeds. - A type change through `migration plan --convert` writes a migration whose only placeholder is in the Postgres `using` slot; filling it and re-running the file produces an applicable migration that converts the rows. The same change without a statement is refused by both commands, and `--delete` on the field plans the direct alter. -- A new required field on a non-empty table through `migration plan --backfill` writes the scaffolded backfill transform; without the statement both commands plan the temporary-default recipe. +- A new required field on an existing table through `migration plan --backfill` writes the scaffolded backfill transform; without the statement both commands plan the temporary-default recipe. - On Mongo, the rename journey above with a collection and a document field, through both commands, and a value object field rename that rewrites subdocuments. - A `db update` against a database whose marker hash has no local snapshot fails every rename and convert statement with one error naming the hash, and succeeds with `--delete` alone. - `--confirm` is gone from both commands, the CLI README describes the statements and the consent model, and the upgrade fragments for the refusal and the consent change are validated by execution. From b5a51bbec829f34b58ccdd5850e6f3bc49dccf83 Mon Sep 17 00:00:00 2001 From: willbot Date: Thu, 8 Oct 2026 19:19:06 +0200 Subject: [PATCH 02/57] TML-3474: the spec describes scaffolding as it is today Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/migration-statements/spec.md b/projects/migration-statements/spec.md index 25683c4676b8..783bd6fb856c 100644 --- a/projects/migration-statements/spec.md +++ b/projects/migration-statements/spec.md @@ -71,7 +71,7 @@ The reasoning and the alternatives rejected are in [`design-notes.md`](./design- - **Planner inputs.** Every planner takes a destination contract and an origin schema: derived from a contract snapshot for `migration plan`, introspected for `db update` and `db init`, and one differ runs over two schema IRs ([ADR 235](../../docs/architecture%20docs/adrs/ADR%20235%20-%20The%20schema%20differ%20walks%20two%20derived%20schema%20IRs.md)). `migration plan` also hands the SQL planner the origin contract as `fromContract` (`packages/1-framework/3-tooling/cli/src/control-api/operations/plan-resolution.ts`); the `db update` path passes `fromContract: null` (`packages/1-framework/3-tooling/migration/src/aggregate/strategies/plan-from-diff.ts`). Statements become the third input, and `db update` learns to supply the origin contract. - **Where the origin contract comes from for `db update`.** The database marker stores the contract hash the database was last updated to ([ADR 021](../../docs/architecture%20docs/adrs/ADR%20021%20-%20Contract%20Marker%20Storage.md)), and `db update` advances the `db` ref to the same hash ([ADR 218](../../docs/architecture%20docs/adrs/ADR%20218%20-%20Refs%20with%20paired%20contract%20snapshots%20and%20universal%20graph-node%20invariant.md)). The content-addressed snapshot store ([ADR 240](../../docs/architecture%20docs/adrs/ADR%20240%20-%20Contract%20snapshots%20live%20in%20a%20content-addressed%20store.md)) holds the contract for that hash when the project has it. - **Consent today.** `db update` pre-plans and refuses with `MIGRATION.DESTRUCTIVE_CHANGES` until the user types the database name or passes the engine's `--confirm ` token (`packages/1-framework/3-tooling/cli/src/orm/db/consent.ts`, `control-api/operations/db-update.ts`). `migration plan` applies the same refusal to auto-baselines only, with the project directory as the token (`control-api/operations/migration-plan.ts`). The CLI README's `db update` section still describes a `--yes` flag that no longer exists. -- **The placeholder flow.** `migration plan` already scaffolds data transforms with `placeholder()` slots for a required column on a non-empty table, a nullable-to-required change and an unsafe type change, writes `ops.json` as `[]` with `pendingPlaceholders`, and the user fills the slots and re-runs `migration.ts` to re-emit ([ADR 200](../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md), [ADR 196](../../docs/architecture%20docs/adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md), `packages/3-targets/3-targets/postgres/src/core/migrations/planner-strategies.ts`). Under `db update`, whose policy excludes the `data` class, those strategies stand down and direct DDL is emitted. The Postgres `alterColumnType` operation already has a `using` option the planner never fills. +- **The placeholder flow.** `migration plan` already scaffolds data transforms with `placeholder()` slots on Postgres for a required column on an existing table, a nullable-to-required change and an unsafe type change (SQLite only for nullable-to-required), writes `ops.json` as `[]` with `pendingPlaceholders`, and the user fills the slots and re-runs `migration.ts` to re-emit ([ADR 200](../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md), [ADR 196](../../docs/architecture%20docs/adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md), `packages/3-targets/3-targets/postgres/src/core/migrations/planner-strategies.ts`). Under `db update`, whose policy excludes the `data` class, those strategies stand down and direct DDL is emitted. The Postgres `alterColumnType` operation already has a `using` option the planner never fills. - **Rename substrate.** `renameTable` exists on Postgres and SQLite from prisma/orm#30331. The shelved prisma/orm#30570 refactored it onto a working schema shared by the planner and the hand-written facade, with companion constraint names taken from the destination contract and foreign keys paired by their own and referenced columns, and reclassified non-data drops as widening. That code is the starting point; the contract `hints` section it also added is not. - **Operation classes.** Additive, widening, destructive and data (`packages/2-sql/9-family/src/core/migrations/types.ts`). Every Postgres type change is destructive today except a short list of safe widenings; Mongo classifies by validator comparison (`packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts`). - **Coordinates.** A contract entity is addressed by plane, namespace, kind and name ([ADR 221](../../docs/architecture%20docs/adrs/ADR%20221%20-%20Contract%20IR%20two%20planes%20with%20uniform%20entity%20coordinate%20and%20pack-contributed%20entity%20kinds.md), [ADR 224](../../docs/architecture%20docs/adrs/ADR%20224%20-%20Namespace%20concretions%20address%20entities%20by%20coordinate.md)); the contract source declares namespaces as `namespace { ... }` blocks; the query surface resolves a bare name through the sole namespace ([ADR 223](../../docs/architecture%20docs/adrs/ADR%20223%20-%20Target-owned%20default%20namespace.md)). Variants and discriminators are [ADR 173](../../docs/architecture%20docs/adrs/ADR%20173%20-%20Polymorphism%20via%20discriminator%20and%20variants.md); value objects are [ADR 178](../../docs/architecture%20docs/adrs/ADR%20178%20-%20Value%20objects%20in%20the%20contract.md), stored as one JSONB column on SQL and a subdocument on Mongo. From 2808c74992a563b04c961a68cd5181c207159ced Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 08:21:35 +0200 Subject: [PATCH 03/57] TML-3474: specs name the ADR amendments, variant deletes and the type-change delete wording Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/plan.md | 2 +- .../migration-statements/slices/convert-backfill/spec.md | 4 +++- projects/migration-statements/slices/mongo-renames/spec.md | 5 +++-- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index 3365ffefe043..28a03fda63cc 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -50,7 +50,7 @@ Each slice is one PR against `main`. **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/remaining-nouns/` -**Outcome.** `--rename` on enum values, namespaces and value object fields (building the framework's value object statement surface: statement entity, subject kind, grammar, resolver), a model move across namespaces (`alter table set schema` on Postgres), and `--convert` on a variant whose discriminator value changed, each planning the row updates, JSON rewrites or schema rename. `--delete` on an enum value nulls it where nullable, else refuses. `--delete ` answers every question for the models in it. +**Outcome.** `--rename` on enum values, namespaces and value object fields (building the framework's value object statement surface: statement entity, subject kind, grammar, resolver), a model move across namespaces (`alter table set schema` on Postgres), and `--convert` on a variant whose discriminator value changed, each planning the row updates, JSON rewrites or schema rename. `--delete` on an enum value nulls it where nullable, else refuses. `--delete` on a variant stored in its base's table deletes the rows with that discriminator value and drops its columns, and the refusal says it deletes rows. `--delete ` answers every question for the models in it. Risk to settle in the spec: a value object can contain itself (ADR 178's `NavItem.children`), and a field rename inside such a type must rewrite to any depth, which a fixed SQL JSON expression or MongoDB update pipeline cannot do. **Builds on.** Slice 3a. diff --git a/projects/migration-statements/slices/convert-backfill/spec.md b/projects/migration-statements/slices/convert-backfill/spec.md index 1fbf81be20bf..48bfe8e4a426 100644 --- a/projects/migration-statements/slices/convert-backfill/spec.md +++ b/projects/migration-statements/slices/convert-backfill/spec.md @@ -53,7 +53,7 @@ Today `migration plan` writes these placeholders on its own for every type chang The four automatic placeholder strategies (Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`) fire only for a field a statement names. Statements reach them as destination coordinates through the strategy context. They change no names in the working schema, so they don't go through the rename path in `planStatements`. A converted type change carries no separate NULL-handling transform; the conversion expression can map NULL itself. -A converted alter is classed `data`, because the user writes its data step. So it isn't a `destructive` operation and leaves `dataLoss`. A typed `convert` at the prompt re-plans like a typed rename, and the type-change question isn't asked again. +A `delete` answer to a type-change question keeps planning the direct alter, and the statement is reported as giving up the field's values in a type change (`delete values of field "User.age" (type change)`), not as deleting the field; slice 2's manual QA found the old wording misleading (F14). A converted alter is classed `data`, because the user writes its data step. So it isn't a `destructive` operation and leaves `dataLoss`. A typed `convert` at the prompt re-plans like a typed rename, and the type-change question isn't asked again. ### The questions know why data is lost @@ -87,6 +87,8 @@ Everything in this slice serves one change: the placeholder migration is written - The `alterColumnType` reshape. - TML-3517. - The project spec's wording: "an existing table", not "a non-empty table", since `migration plan` can't see rows. +- An amendment to ADR 200, which says only a data transform holds a placeholder. +- The Migration System doc's claim that the Postgres planner writes no placeholder data transforms, which the code contradicts. - Docs: Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`. - An app upgrade fragment: type changes ask, scaffolds are opt-in, and the temporary default applies under `migration plan`. - Upgrade instructions for the facade change. diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md index aa3f4f79a822..0952fd89bb7c 100644 --- a/projects/migration-statements/slices/mongo-renames/spec.md +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -47,7 +47,7 @@ A new DDL operation, `renameCollection`, added through the MongoDB stack the way ### Document rewrites run under `db update` -`db update` allows `additive`, `widening` and `destructive`, not `data`, so a rewrite classed `data` would be refused there. The rewrites above carry the class of what they do to data (`widening` for a rename, `destructive` for an unset), and the MongoDB runner and operation serializer tell a rewrite from a DDL command by its shape, not by `operationClass === 'data'`. `data` stays the class of transforms a user writes or a scaffold produces; `db update` keeps excluding it. Slice 3b's SQL JSON rewrites follow the same rule. +`db update` allows `additive`, `widening` and `destructive`, not `data`, so a rewrite classed `data` would be refused there. The rewrites above carry the class of what they do to data (`widening` for a rename, `destructive` for an unset), and the MongoDB runner and operation serializer tell a rewrite from a DDL command by its shape, not by `operationClass === 'data'`. `data` stays the class of transforms a user writes or a scaffold produces; `db update` keeps excluding it. Slice 3b's SQL JSON rewrites follow the same rule. This follows the Migration System doc's rule that only the planner's own classification promises an operation loses data exactly when it is `destructive`, and it changes ADR 188, which says every MongoDB data transform is `data`: the slice amends ADR 188, and ADR 264's line that MongoDB has no rename operation. ### What is refused @@ -63,7 +63,7 @@ One reviewer can hold this: it is one family learning the two verbs the SQL targ ## Scope -**In:** model rename (collection rename); top-level field rename, including a variant's field; field removal as data loss with `$unset` under `--delete`; both commands; the `bypassDocumentValidation` option; runner and serializer telling rewrites apart by shape; deleting `renameStatements`; docs (Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`); an app upgrade fragment (MongoDB renames work; removing a field now asks) and an extension upgrade fragment (the capability member is gone). +**In:** amendments to ADR 188 (planner-written rewrites carry the class of their effect) and ADR 264 (MongoDB has a collection rename); model rename (collection rename); top-level field rename, including a variant's field; field removal as data loss with `$unset` under `--delete`; both commands; the `bypassDocumentValidation` option; runner and serializer telling rewrites apart by shape; deleting `renameStatements`; docs (Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`); an app upgrade fragment (MongoDB renames work; removing a field now asks) and an extension upgrade fragment (the capability member is gone). **Deliberately out:** - Value object field renames on MongoDB. They need the framework's value object statement surface (statement entity, subject kind, grammar, resolver), which slice 3b builds for SQL JSON rewrites, and a recursive update pipeline for lists, dictionaries and unions. Moved to slice 4b. @@ -79,6 +79,7 @@ One reviewer can hold this: it is one family learning the two verbs the SQL targ | Two variants in one collection store a field with the same name | Rewrite filters on the discriminator value | | The runner is not transactional | Postcheck makes a re-run skip a finished rewrite | | `bypassDocumentValidation` needs a privilege on hosted clusters | Runner failure names the privilege | +| TML-2447 (open): the planner creates separate collections and incomplete validators for variants that share a collection | The first dispatch checks how variants are stored today before the variant field rename is built on it | ## Slice-specific done conditions From e44518aa65ced4f0c13e91b209f08bb863573167 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 08:25:42 +0200 Subject: [PATCH 04/57] TML-3474: the plan and the 3a and 4a specs lead with tables and short lists Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/plan.md | 128 +++++++-------- .../slices/convert-backfill/spec.md | 148 ++++++++++-------- .../slices/engine-statement-prompt/spec.md | 2 +- .../slices/mongo-renames/spec.md | 145 +++++++++++------ projects/migration-statements/spec.md | 4 +- 5 files changed, 229 insertions(+), 198 deletions(-) diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index 28a03fda63cc..7d3276433912 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -2,112 +2,92 @@ **Spec:** [`spec.md`](./spec.md) · **Linear:** [Destructive changes need stated intent](https://linear.app/prisma-company/project/destructive-changes-need-stated-intent-7626c0107cd9), plan issue [TML-3474](https://linear.app/prisma-company/issue/TML-3474) -## Summary +## At a glance -Six slices. Slices 1 and 2 are merged. The rest, re-planned on 2026-10-08 when slices 3 and 4 proved too large for one review each: +| Slice | What it delivers | Builds on | State | +| --- | --- | --- | --- | +| 1 | `--rename` for models and fields, Postgres and SQLite | — | Merged, prisma/orm#30638 | +| 2 | Data loss refused until answered; `--delete`, `--allow`; the interactive prompt | 1 | Merged, prisma/orm#30648 | +| 3a | `--convert` and `--backfill`, Postgres and SQLite | 2 | Spec: [`slices/convert-backfill/`](./slices/convert-backfill/spec.md) | +| 4a | Renames and deletes on MongoDB | 2 | Spec: [`slices/mongo-renames/`](./slices/mongo-renames/spec.md) | +| 3b | The remaining SQL renames and deletes | 3a | Not started | +| 4b | `--convert`, `--backfill` and value object renames on MongoDB | 3b, 4a | Not started | -- **3a** (SQL `--convert` and `--backfill`) and **4a** (MongoDB renames and deletes) run in parallel. -- **3b** (the remaining SQL nouns) follows 3a. **4b** (MongoDB `--convert`, `--backfill` and value object renames) follows 3b and 4a, and reuses their design. -- The storage-name syntax survey runs beside the slices and goes to Will for a decision. It does not block close-out. +```text +1 → 2 ─┬─ 3a ── 3b ─┐ + └─ 4a ───────┴─ 4b → close-out +``` -Each slice is one PR against `main`. +- 3a and 4a run in parallel now. +- Each slice is one pull request against `main`. +- Slices 3 and 4 were split on 2026-10-08. Each was too large to review as one pull request. -## Slices +## Slices still to build -### Slice 1 — Statements on the command line, and renames of models and fields on Postgres and SQLite +### 3a — `--convert` and `--backfill` on Postgres and SQLite -**Linear:** [TML-3475](https://linear.app/prisma-company/issue/TML-3475) · **Folder:** `slices/renames/` +**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Spec:** [`slices/convert-backfill/spec.md`](./slices/convert-backfill/spec.md) -**Outcome.** Both commands accept `--rename old:new` for models and fields, with a model rename across namespaces resolved but refused until slice 3. The framework parses each statement, resolves it against the origin and destination contracts, and applies it in order to a working copy of the origin. The SQL planners emit the table or column rename and every companion rename, with names from the destination contract. `db update` resolves its origin contract from the marker hash through the snapshot store and fails every rename when it cannot. An unusable statement is an error. Non-data drops are widening. The written migration is what a user could write by hand. +- A lossy type change is answered with `--convert` (fill in the conversion) or `--delete`. +- `--backfill` writes a slot to fill existing rows of a field that became required. +- `migration plan` stops writing placeholders nobody asked for. +- A new required field gets a temporary value on both targets. SQLite lacks this today. +- The `SET NOT NULL` failure names the NULL rows (TML-3517). -**Builds on.** The planner substrate from prisma/orm#30570, without its contract section. +**Hands to 3b and 4b:** the two flags and how their questions and placeholders work. -**Hands to.** A statement type the framework owns, parsed and resolved, delivered to every family planner as resolved entities; the working-copy mechanism statements apply to; the origin-contract resolution for `db update`; a column rename operation on both SQL targets. +### 4a — Renames and deletes on MongoDB -### Slice 2 — Both commands refuse data loss by default, and `--delete` is the per-operation consent +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Spec:** [`slices/mongo-renames/spec.md`](./slices/mongo-renames/spec.md) -**Linear:** [TML-3476](https://linear.app/prisma-company/issue/TML-3476) · **Folder:** `slices/refusal/` +- A model rename renames the collection. A field rename rewrites the documents. +- Removing a field is asked about. `--delete` removes it from every document. +- Both commands, including `db update`. +- The temporary setting that makes MongoDB refuse renames is deleted. -**Outcome.** `migration plan` refuses any plan that loses data, with the error `db update` uses. The refusal lists each destructive operation with the statements that resolve it. `--delete` consents to one operation, for namespaces, models and fields, and replaces `--confirm` on both commands. The terminal consent question asks per operation. Upgrade fragments record both changes; the CLI README describes statements and consent. +**Hands to 4b:** a MongoDB planner that takes statements. -**Builds on.** Slice 1. Two things slice 1 leaves for it: the missing-origin check in the CLI statement resolver runs over the whole input and must move inside the per-statement loop so that `--delete` works without an origin contract; and the `@@map`-only rename gap in `deferred.md` must be decided before the refusal text is written. Decided 2026-10-07 on prisma/orm#30638: dropping a row-level-security policy and disabling row-level security stay `widening` (they lose no data), and slice 2 adds a separate consent question before `db update` applies an operation that widens who can read or write rows, answered per operation like the data-loss consent and refused in a non-interactive run unless consented. - -**Design settled 2026-10-07 (Will).** Consent is a statement the user answers with, not a token copied back: the CLI engine gains `ctx.prompt.statement`/`statements`, answered on the command line by verb flags (`--delete Legacy`, `--rename Legacy:Archive`, `--allow User`) and interactively by typing the statement when a human runs the command. The engine change is its own slice, built in prisma/prisma-cli first: [`slices/engine-statement-prompt/spec.md`](./slices/engine-statement-prompt/spec.md). Slice 2's spec is [`slices/refusal/spec.md`](./slices/refusal/spec.md). The `@@map`-only gap gets no statement (see `deferred.md`). Operations that lose no data (`SET NOT NULL`, safe type widenings, MongoDB validator and index changes) stop being destructive. - -**Hands to.** The refusal shape every later verb hooks its statements into; the per-operation consent model; `--confirm` no longer consents to data loss. - -### Slice 3a — `--convert` and `--backfill` on Postgres and SQLite - -**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/convert-backfill/` - -**Outcome.** `migration plan --convert User.age` writes the type change with a `placeholder()` where the conversion goes, and `--backfill User.email` writes the backfill transform. A lossy type change is answered with `convert` or `delete`. Neither verb is accepted by `db update`. The planner no longer scaffolds without being asked; without `--backfill`, a required field on a populated table gets the temporary-default recipe on both SQL targets, which SQLite lacks today. The Postgres runner's advice when `SET NOT NULL` meets NULLs names the NULLs (TML-3517). - -**Builds on.** Slice 2. - -**Hands to.** The convert and backfill verbs, their questions and their scaffold rule, for 3b (variants) and 4b (MongoDB). - -### Slice 3b — The remaining nouns on Postgres and SQLite +### 3b — The remaining renames and deletes on Postgres and SQLite **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/remaining-nouns/` -**Outcome.** `--rename` on enum values, namespaces and value object fields (building the framework's value object statement surface: statement entity, subject kind, grammar, resolver), a model move across namespaces (`alter table set schema` on Postgres), and `--convert` on a variant whose discriminator value changed, each planning the row updates, JSON rewrites or schema rename. `--delete` on an enum value nulls it where nullable, else refuses. `--delete` on a variant stored in its base's table deletes the rows with that discriminator value and drops its columns, and the refusal says it deletes rows. `--delete ` answers every question for the models in it. Risk to settle in the spec: a value object can contain itself (ADR 178's `NavItem.children`), and a field rename inside such a type must rewrite to any depth, which a fixed SQL JSON expression or MongoDB update pipeline cannot do. - -**Builds on.** Slice 3a. - -**Hands to.** Project close-out for the SQL targets. - -### Slice 4a — Renames and deletes on MongoDB +- `--rename` for enum values, namespaces and fields inside value objects. This slice builds the statement support for value objects, which 4b reuses. +- Moving a model to another namespace (`alter table set schema` on Postgres). +- `--convert` on a variant whose discriminator value changed. +- `--delete` on an enum value: sets it to NULL where the field is optional, otherwise refuses. +- `--delete` on a variant stored in its base's table: deletes its rows and drops its columns. The refusal says rows are deleted. +- `--delete ` answers every question for the models in it. -**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo-renames/` +**Risk to settle in its spec:** a value object can contain itself (ADR 178's `NavItem.children`). Renaming a field inside it means rewriting to any depth, which a fixed SQL JSON expression or MongoDB update cannot do. -**Outcome.** The MongoDB planner carries out `--rename` for models (collection rename) and fields (document rewrite), with the validator and indexes kept consistent, through both commands. Removing a field becomes data loss, answered by `--delete` with an `$unset`. Document rewrites run under `db update`. The temporary capability member `renameStatements: { refused: true; keepDataByHand }`, its three CLI reads and its doc mention are deleted. Statements name MongoDB fields the same way they name SQL fields. +**Hands to:** close-out for Postgres and SQLite. -**Builds on.** Slice 2. - -**Hands to.** A MongoDB planner that takes resolved statements, for 4b. - -### Slice 4b — `--convert`, `--backfill` and value object renames on MongoDB +### 4b — `--convert`, `--backfill` and value object renames on MongoDB **Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo-convert-backfill/` -**Outcome.** The same two verbs as 3a on MongoDB, scaffolding a data transform with a placeholder. Value object field renames rewrite subdocuments, including lists, dictionaries and unions, using the statement surface from 3b. A validator change that makes a field required on a populated collection is no longer silent: it is answered with `--backfill`, or the command says that existing documents need the field. - -**Builds on.** Slices 3a, 3b and 4a. - -**Hands to.** Project close-out for MongoDB. - -### Alongside — A syntax for naming storage objects - -Will asked (2026-10-07) for a critical discussion and a survey of how established tools name tables and other storage objects on the command line, before any syntax is chosen for the `@@map`-only rename gap (`deferred.md`). The write-up goes to Will for a decision. Building the chosen syntax is not in this project unless Will adds it. - -## Stretch goal — interactive statements when a human runs the command - -Recorded 2026-10-06 at the operator's request. This is the eventual direction, and the slices build with it in mind so that it drops in on top of their mechanisms; it is not a slice of its own, and it is built once the mechanisms it needs exist (after slice 2 at the earliest). - -When `migration plan` or `db update` detects a destructive operation and no statement covers it, the command today refuses and prints the statements that would resolve it. The stretch goal: when the command is run by a human in a terminal (stdin and stdout are a TTY, and no `--json` or similar non-interactive flag is set), the refusal becomes an interactive prompt that asks, per destructive operation, what the user means: rename it to a name they type, or delete it. Each answer is exactly the statement the refusal would have printed, applied in order, and the plan then proceeds as if the statements had been given on the command line. In a script, an agent, or CI (no TTY) the behaviour stays as it is: refuse and print the statements. - -Rules that carry over unchanged: the prompt never guesses or proposes a rename candidate; it asks. The statements are not recorded anywhere; the migration file or the database is the record. The consent model of slice 2 (`--delete` per operation) is what the "delete" answer maps to. +- The same two flags as 3a, writing a data transform with a placeholder. +- Making a field required on a collection with documents is no longer silent. It is answered with `--backfill`, or the command says the existing documents need the field. +- Renaming a field inside a value object rewrites the embedded documents, including lists, dictionaries and unions. -What the slices do now so this drops in later (see the spec's cross-cutting requirement 12): +**Hands to:** close-out for MongoDB. -- The refusal is a structured value first and text second: a list of destructive operations, each with the entity it touches in domain coordinates and the statements that would resolve it, rendered to the error message by the command. The prompt consumes the same list. -- Statement text is parsed, resolved and applied through one function the command calls, which takes the statement strings and the two contracts and does not care whether the strings came from the command line or from an answer typed at a prompt. Statements can therefore be added after the command has started and the plan re-run. -- The per-operation consent question that `db update` already asks in a terminal is the seed of this prompt, so slice 2 keeps it as a per-operation question rather than folding it into a yes/no over the whole plan. +### Alongside — a syntax for naming storage objects -## Sequencing +There is no statement for a model that keeps its name but changes its table (`@@map`). Will asked on 2026-10-07 for a critical discussion and a look at how established tools name tables on the command line before any syntax is chosen. `--rename table/users:app_users` was an off-the-cuff example, not a decision. -- **Merged:** slice 1 → slice 2. -- **Parallel now:** 3a and 4a. -- **Then:** 3b after 3a; 4b after 3b and 4a. +- The write-up goes to Will for a decision. +- It does not block close-out. +- Building the chosen syntax is not in this project unless Will adds it. -## Dependencies +## Delivered: the interactive prompt -- Nothing external. The shelved prisma/orm#30570 is a source to copy from, not a dependency; its branch stays as a draft. -- Users get the feature through the next published CLI release after slice 2 merges; slices 1 and 2 together are the minimum that changes behaviour a user sees. +Recorded on 2026-10-06 as the stretch goal: when a human runs the command in a terminal, the refusal becomes a question per operation, and the answer is the same statement the flags take. Slice 2 built it on the CLI engine's statement prompt (prisma/prisma-cli#337). Scripts, agents and CI get the refusal that names the flags. ## Close-out (required) - [ ] Verify every project DoD item in [`spec.md`](./spec.md) +- [ ] Final retro - [ ] Write the ADR and amend ADR 001, ADR 028 and the Data Contract and Migration System subsystem docs - [ ] Migrate long-lived docs into `docs/` - [ ] Strip repo-wide references to `projects/migration-statements/**` diff --git a/projects/migration-statements/slices/convert-backfill/spec.md b/projects/migration-statements/slices/convert-backfill/spec.md index 48bfe8e4a426..e48b659f09db 100644 --- a/projects/migration-statements/slices/convert-backfill/spec.md +++ b/projects/migration-statements/slices/convert-backfill/spec.md @@ -1,10 +1,10 @@ -# Slice spec — `--convert` and `--backfill` write the placeholder migration only when asked +# Slice 3a — `--convert` and `--backfill` -**Project:** [`projects/migration-statements/`](../../spec.md) · **Slice 3a** · **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477), [TML-3517](https://linear.app/prisma-company/issue/TML-3517) · **Branch:** `tml-3477-convert-backfill` · **Builds on:** slice 2 (merged). +**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477), [TML-3517](https://linear.app/prisma-company/issue/TML-3517) · **Branch:** `tml-3477-convert-backfill` · **Builds on:** slice 2 (merged) · **Project:** [`../../spec.md`](../../spec.md) -## At a glance +## In one example -A user changes `User.age` from `String` to `Int` on Postgres: +`User.age` changes from `String` to `Int` on Postgres. ```text $ prisma migration plan --name age-to-int @@ -17,7 +17,7 @@ $ prisma migration plan --name age-to-int --convert User.age ⚠ migration.ts has a placeholder: replace it with the conversion, then run the file to re-emit. ``` -`migration.ts` then holds one placeholder, where the conversion goes: +`migration.ts` then has one slot to fill: ```ts this.alterColumnType({ @@ -28,96 +28,106 @@ this.alterColumnType({ }), ``` -The user writes `using: '"age"::integer'` (or any SQL expression), runs the file, and gets an applicable migration. On SQLite the placeholder sits in the table rebuild's copy step for that column. +The user writes `using: '"age"::integer'`, runs the file, and applies it. -Adding a required `email` field to an existing `User` table plans a temporary default by default: the column is added with a fill value and the default is dropped, and nothing is asked. To write the fill yourself, run `prisma migration plan --backfill User.email`. That writes the nullable column, a data transform with a placeholder, and `SET NOT NULL`. +## What changes for users -Today `migration plan` writes these placeholders on its own for every type change and every required field on an existing table. Users who didn't want them get a migration they can't apply until they edit it. After this slice it writes them only when asked. +| Change in the schema | Today | After this slice | +| --- | --- | --- | +| Type change that loses data | Placeholder migration, written without asking | Refused. Answer `--convert` (fill in the conversion) or `--delete` (accept the loss) | +| Type change that keeps every value (`int4` → `int8`) | Planned, nothing asked | Same | +| Required field added to an existing table | Placeholder migration, written without asking | Filled with a temporary value (`''`, `0`, …), nothing asked. `--backfill` writes a slot to fill instead | +| Optional field made required | Placeholder migration, written without asking | `SET NOT NULL`, nothing asked. Fails if NULLs exist, and says how many. `--backfill` writes a slot to fill instead | +| Any of the above through `db update` | No placeholders | Same. `--convert` and `--backfill` are refused: `db update` has no file to fill | -## Chosen design +SQLite works the same way. Two SQLite gaps close on the way: +- `db update` can add a required field to a table with rows. Today it fails. +- A type change gets a slot for the conversion. Today SQLite just rebuilds the table. -### The two verbs +## Decisions to check -- **`--convert `** answers a type change that would lose data. It is accepted by `migration plan` only. It names the field by its destination name, like the new side of a rename: after `--rename User.age:User.years`, write `--convert User.years`. It needs the origin contract, like `rename`, and fails with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN` without one. It must name a field that exists on both sides and whose type changes. -- **`--backfill `** is an opt-in that answers no question. It is accepted by `migration plan` only. It names a field that is required in the destination and is either new on an existing table or optional in the origin. Anything else is an error that names what was found. -- `db update` declares both verbs only to refuse them, with `MIGRATION.STATEMENT_NEEDS_MIGRATION_FILE`. The error says `db update` has no migration file to fill and names `migration plan`. It doesn't fall through to the engine's generic "no flag registered" error, which doesn't explain the rule. The shared verb table in `orm/statement-verbs.ts` declares the verbs per command, so neither command accepts a verb by accident. +1. **Temporary values without asking.** Without `--backfill`, `migration plan` writes fill values such as `''` or `0` into existing rows. The project spec already decided this. It is the biggest behaviour change in the slice. +2. **`alterColumnType` changes shape.** Its options become `{ table, column, type, using? }`. Today they expose internal names. Hand-written migrations that call it need the upgrade instructions. +3. **A type change answered with `delete` is reported honestly.** The output says `delete values of field "User.age" (type change)`, not "delete field". Slice 2's QA found the old wording misleading. -### What the planners write +## The two flags -| Diff | No statement | `--convert` | `--backfill` | -| --- | --- | --- | --- | -| Type change the target knows is safe (`int4`→`int8`) | direct alter, `widening`, nothing asked | error: nothing to convert | — | -| Any other type change | refused: question offers `convert` and `delete`. `--delete` plans the direct alter, `destructive` | Postgres: `alterColumnType` with `using: placeholder(…)`. SQLite: `recreateTable` whose copy step takes `placeholder(…)` for the column. Class `data` | — | -| Required field added to an existing table | temporary-default recipe on both targets, both commands, `additive` | — | nullable add, `dataTransform` with placeholders, then `SET NOT NULL` (SQLite: a tightening rebuild) | -| Optional field made required | direct `SET NOT NULL` (SQLite: tightening rebuild), `widening`. Fails at apply if NULLs exist, with advice that names `--backfill` | — | `dataTransform` with placeholders, then `SET NOT NULL` | +**`--convert `** +- Answers a type change that would lose data. +- `migration plan` only. +- Names the field by its new name. After `--rename User.age:User.years`, write `--convert User.years`. +- Needs the old contract, like `--rename`. -The four automatic placeholder strategies (Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`) fire only for a field a statement names. Statements reach them as destination coordinates through the strategy context. They change no names in the working schema, so they don't go through the rename path in `planStatements`. A converted type change carries no separate NULL-handling transform; the conversion expression can map NULL itself. +**`--backfill `** +- Optional. Answers no question. +- `migration plan` only. +- The field must be required now, and either new on an existing table or optional before. -A `delete` answer to a type-change question keeps planning the direct alter, and the statement is reported as giving up the field's values in a type change (`delete values of field "User.age" (type change)`), not as deleting the field; slice 2's manual QA found the old wording misleading (F14). A converted alter is classed `data`, because the user writes its data step. So it isn't a `destructive` operation and leaves `dataLoss`. A typed `convert` at the prompt re-plans like a typed rename, and the type-change question isn't asked again. +**On `db update`,** both flags fail with `MIGRATION.STATEMENT_NEEDS_MIGRATION_FILE`, which points to `migration plan`. -### The questions know why data is lost +## What the planners write -Each `dataLoss` entry gains `loss: 'drop' | 'typeChange'`, set by each target's operation-subjects function. The CLI stops guessing from whether the destination still has the field. A `typeChange` question offers `convert` and `delete` in `migration plan`, and `delete` only in `db update`. A `drop` question offers `rename` and `delete`, as today. The re-plan loop (`askPlanQuestions`) carries every planned statement, not only renames. +| Case | Postgres | SQLite | +| --- | --- | --- | +| `--convert` | `alterColumnType` with `using: placeholder(…)` | Table rebuild whose copy step has `placeholder(…)` for the column | +| `--backfill`, new field | Add nullable column, data transform with placeholders, `SET NOT NULL` | Add nullable column, data transform, tightening rebuild | +| `--backfill`, optional made required | Data transform with placeholders, `SET NOT NULL` | Data transform, tightening rebuild | +| New required field, no flag | Add with temporary default, drop the default | Rebuild that copies the temporary value into the new column | -`ResolvedMigrationStatement` gains a `kind`, which is `rename`, `convert` or `backfill`. Every reader that assumes a rename now switches on `kind` first. One example is the SQL family's `StatementPlanner`, which today plans any non-model statement as a column rename. MongoDB keeps refusing statements it can't carry out, until slices 4a and 4b. +## How the pieces fit -### The temporary default becomes a real migration call +- **Questions know why data is lost.** Each data-loss entry says `drop` or `typeChange`. A `typeChange` question offers `convert` and `delete`; a `drop` question offers `rename` and `delete`. +- **A converted change stops being data loss.** It is classed `data`, because the user writes the conversion. So it is not asked about again. +- **A typed `convert` at the prompt re-plans,** the same way a typed rename does. +- **Placeholders appear only when a flag asks.** The four strategies that write them today run only for a field a flag names. +- **The temporary default becomes a real call in `migration.ts`.** Today it renders as an empty `rawSql(...)`, so re-running the file does not reproduce `ops.json`. +- **The NOT NULL failure names the NULLs (TML-3517).** It says how many rows hold NULL in which column, and suggests filling them, keeping the field optional, or `--backfill`. It no longer talks about schema drift. -The Postgres temporary-default and direct-add calls render as `rawSql({ id, label, operationClass })` with no SQL. Re-running such a `migration.ts` doesn't reproduce `ops.json` (cross-cutting requirement 7). They now render as a facade call that builds the same operation. SQLite gets the recipe as a rebuild in which the new column's copy value is a literal. This needs a new per-column copy-expression option on `recreateTable`, which `--convert` also uses, and a SQLite fill-value map (integer `0`, real `0`, text `''`, blob `X''`, and the values its boolean and date-time codecs store). It is refused on the same primary-key, unique and foreign-key conditions as Postgres. This closes the deferred item "SQLite `db update` cannot add a required field to a table that has rows". +## Why one pull request -### `alterColumnType` takes plain options - -The Postgres facade's `alterColumnType` options are reshaped so the rendered call shows a readable shape with an optional `using`, and no internal names (`qualifiedTargetType`, `formatTypeExpected`, `rawTargetTypeForLabel`). The `using` slot renders as code, so it can hold `placeholder(…)`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result the CLI already handles. Upgrade instructions are recorded for hand-written migrations. - -### The NOT NULL failure names the NULLs (TML-3517) - -When `SET NOT NULL` or a SQLite tightening rebuild meets NULL values, the runner reports how many rows hold NULL in which column. It says to fill them, make the field optional, or plan with `--backfill`. `MigrationRunnerFailure` gains a `fix`, and the `db update`, `db init` and `migrate` error mappers use it instead of the fixed advice about schema drift. - -## Coherence rationale - -Everything in this slice serves one change: the placeholder migration is written only when a statement asks for it. The verbs, the loss kind, the statement `kind`, the temporary default becoming real on both targets, the `alterColumnType` reshape and the NULL advice are each what that change needs to work on Postgres and SQLite. The reviewer reads one rule and its consequences. +Everything here serves one rule: a placeholder migration is written only when a flag asks for it. The reviewer reads that rule and its consequences. ## Scope -**In:** -- `--convert` and `--backfill` on `migration plan`, for Postgres and SQLite, refused on `db update`. -- The loss kind on `dataLoss` entries, and the statement `kind`. -- Ending automatic scaffolding. -- The temporary default as a real call on both targets, including the SQLite recipe. -- The `alterColumnType` reshape. +**In** +- Both flags on Postgres and SQLite; refused on `db update`. +- Ending automatic placeholders. +- The temporary default on both targets, as a real call. +- The `alterColumnType` reshape, with upgrade instructions. - TML-3517. -- The project spec's wording: "an existing table", not "a non-empty table", since `migration plan` can't see rows. -- An amendment to ADR 200, which says only a data transform holds a placeholder. -- The Migration System doc's claim that the Postgres planner writes no placeholder data transforms, which the code contradicts. -- Docs: Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`. -- An app upgrade fragment: type changes ask, scaffolds are opt-in, and the temporary default applies under `migration plan`. -- Upgrade instructions for the facade change. +- Docs: + - ADR 200: it says only a data transform can hold a placeholder. + - The Migration System doc: it says the Postgres planner never writes placeholders. + - Migration System § Statements, the error reference, the CLI README, `skills/prisma-8/references/migrations.md`. +- An app upgrade fragment for the behaviour changes. +- The project spec: "an existing table", not "a non-empty table". `migration plan` cannot see rows. -**Deliberately out:** -- `--convert` on a variant (slice 3b). -- MongoDB (slice 4b). -- A temporary default for a field in a new unique or foreign key, which stays refused as today. -- Detecting rows offline. +**Out** +- `--convert` on a variant: slice 3b. +- MongoDB: slice 4b. +- A temporary value for a new unique, primary-key or foreign-key column. It stays refused, as on Postgres today. -## Pre-investigated edge cases +## Edge cases already known | Case | Handling | | --- | --- | -| A temporary value in a unique column fails on the second row | The recipe is refused for new unique, primary-key and foreign-key columns, as on Postgres today | -| A convert after a rename in the same run | The convert uses the destination name, which the diff issues already use | -| A placeholder operation has no position until re-emit | Applied-statement positions for a convert are empty until the file is re-run, as for every placeholder today | - -## Slice-specific done conditions - -- The project DoD's convert and backfill lines pass on Postgres and SQLite, reading "an existing table". -- Re-running every `migration.ts` the planners write in this slice's tests reproduces its `ops.json`. +| A temporary value in a unique column fails on the second row | Refused for new unique, primary-key and foreign-key columns | +| `--convert` after a rename in the same run | Uses the new name, which the planner already uses | +| A placeholder operation has no position until the file is re-run | The convert's reported positions are empty until then, as for every placeholder today | -## Open questions +## Done when -None. +- The project's convert and backfill checks pass on Postgres and SQLite. +- Re-running every `migration.ts` that this slice's tests write reproduces its `ops.json`. -## References +## Implementation notes +- `dataLoss` entries gain `loss: 'drop' | 'typeChange'`, set by each target's operation-subjects function. `dataLossQuestion` stops inferring it from whether the destination has the field. +- `ResolvedMigrationStatement` gains `kind: 'rename' | 'convert' | 'backfill'`. Every reader that assumes a rename switches on `kind` first, including the SQL `StatementPlanner`, which today plans any non-model statement as a column rename. +- `askPlanQuestions` re-plans with every planned statement, not only renames. +- Convert and backfill reach the strategies as destination coordinates through the strategy context. They do not go through `planStatements`, because they rename nothing. +- Strategies gated: Postgres `notNullBackfillCallStrategy`, `typeChangeCallStrategy`, `nullableTighteningCallStrategy`; SQLite `nullabilityTighteningBackfillStrategy`. +- `AlterColumnTypeClass` gains `data`. The `using` slot renders as code, imports `placeholder`, and an unfilled slot makes `toOp()` return the unfilled-placeholder result. +- `recreateTable` gains a per-column copy expression, used by convert and by the SQLite temporary value. SQLite gets a fill-value map: integer `0`, real `0`, text `''`, blob `X''`, and what its boolean and date-time codecs store. +- `MigrationRunnerFailure` gains `fix`. The `db update`, `db init` and `migrate` error mappers prefer it. - Grounding report (local, gitignored): `wip/grounding-3a.md`. -- [`../../design-notes.md`](../../design-notes.md): the scaffold rule and the convert and backfill rows. -- [ADR 200](../../../../docs/architecture%20docs/adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md): the placeholder flow. diff --git a/projects/migration-statements/slices/engine-statement-prompt/spec.md b/projects/migration-statements/slices/engine-statement-prompt/spec.md index c2b105039d2d..32929d9ea75c 100644 --- a/projects/migration-statements/slices/engine-statement-prompt/spec.md +++ b/projects/migration-statements/slices/engine-statement-prompt/spec.md @@ -105,6 +105,6 @@ None. Will decided on 2026-10-07: consents carry a verb; the answer is a free-te ## References -- Project spec [`../../spec.md`](../../spec.md) decisions 2 and 6, requirement 12; [`../../plan.md`](../../plan.md) § Stretch goal. +- Project spec [`../../spec.md`](../../spec.md) decisions 2 and 6, requirement 12; [`../../plan.md`](../../plan.md) § Delivered: the interactive prompt. - prisma-cli: `packages/cli-engine/src/execution/prompts.ts`, `shared-flags.ts`, `clack-renderer.ts`, `src/context.ts`; `docs/product/cli-style-guide.md` § consent; ADR 0004 (engine version pinning). - `wip/slice-2-consent-options.md`: the option evaluation Will chose from. diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md index 0952fd89bb7c..0344713b340b 100644 --- a/projects/migration-statements/slices/mongo-renames/spec.md +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -1,10 +1,10 @@ -# Slice spec — MongoDB carries out renames and deletes of models and fields +# Slice 4a — Renames and deletes on MongoDB -**Project:** [`projects/migration-statements/`](../../spec.md) · **Slice 4a** · **Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Branch:** `tml-3478-mongo-renames` · **Builds on:** slice 2 (merged). +**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Branch:** `tml-3478-mongo-renames` · **Builds on:** slice 2 (merged) · **Project:** [`../../spec.md`](../../spec.md) -## At a glance +## In one example -A MongoDB user renames the `Profile` model to `User` (collection `profiles` to `users`), changes `name` to `fullName`, and removes the `nickname` field: +A MongoDB user renames model `Profile` to `User`, renames field `name` to `fullName`, and removes field `nickname`. ```text $ prisma migration plan --name tidy --rename Profile:User --rename User.name:User.fullName --delete User.nickname @@ -17,7 +17,7 @@ $ prisma migration plan --name tidy --rename Profile:User --rename User.name:Use Update validator on users widening ``` -`migration.ts` carries `renameCollection(...)`, `renameField(...)` and `unsetField(...)` calls, and running it reproduces `ops.json`. `prisma db update` with the same statements does the same against the live database. Today MongoDB refuses every `--rename` and prints steps to run by hand in `mongosh`, and removing a field asks nothing, even though every document that still holds the field then fails each later update against the closed validator. +`db update` with the same flags does the same against the live database. Without `--delete User.nickname`, both commands refuse, as they do on SQL: @@ -28,70 +28,111 @@ Without `--delete User.nickname`, both commands refuse, as they do on SQL: → Pass --delete User.nickname ``` -## Chosen design +## What changes for users -### Statements name fields by their contract key +| Change in the schema | Today | After this slice | +| --- | --- | --- | +| Model renamed, `--rename` given | Refused, with steps to run by hand in `mongosh` | Collection renamed; documents, indexes and validator kept | +| Field renamed, `--rename` given | Refused, with steps to run by hand | Every document's field renamed; indexes moved to the new name | +| Field removed | Validator updated, nothing asked. Every document that still holds the field then fails each update | Asked about. `--delete` removes the field from every document; `--rename` keeps its values | +| Model removed | Asked about; `--delete` drops the collection | Same, and `--rename` is offered too | -A statement names a field as the contract and the generated client name it: the key in the model's `fields`. On SQL that key is the field name in the schema; on MongoDB the contract keys fields by the stored name (`id @map("_id")` is `User._id`), and the client uses the same name. The resolver already behaves this way, so nothing changes in it. On MongoDB a change to `@map` alone is a field rename in contract terms and is stated with `--rename`; a change to the schema field name alone changes nothing stored and needs no statement. This settles the deferred item "Which field name a MongoDB statement uses" and is recorded in the Migration System doc. +## Decisions to check -### Collection rename +1. **Removing a field now asks.** Today it is silent, but MongoDB's validator then rejects every update to a document that still holds the field. +2. **Fields are named as stored.** `id @map("_id")` is `User._id` in a statement. The contract and the generated client already use that name. Naming the schema field instead would change the contract and its hashes. +3. **The rewrites run under `db update`.** They are classed by what they do to data: a rename is `widening`, a removal is `destructive`. `db update` refuses the `data` class, so classing them `data`, as ADR 188 says today, would block `db update`. This slice amends ADR 188. -A new DDL operation, `renameCollection`, added through the MongoDB stack the way the other collection operations are: query AST command and visitor, wire command and lowering, driver, operation serializer, preview text, factory and call class. Precheck: the source collection exists and the target does not. Postcheck: the target exists. Class `widening`. A model rename renames the collection in the working copy of the origin schema before the diff, so the indexes and validator that MongoDB keeps through a rename are not dropped and recreated. +## How it works -### Field rename and field removal rewrite the documents +**Collection rename** +- A new `renameCollection` operation. +- Before: the source exists and the target does not. After: the target exists. +- The rename happens before the diff, so MongoDB keeps the indexes and validator, and nothing is dropped and recreated. -- A field rename plans `renameField(collection, from, to, { filter })`: one `updateMany` with `$rename`, run with validation bypassed. Filter: documents that hold the old field, and for a variant's field also the variant's discriminator value. Precheck: no matching document already holds the new field, since `$rename` would overwrite it. Postcheck: no matching document still holds the old field, so a re-run skips it. Class `widening`. -- Removing a field from a model is data loss, matching the spec's rule that dropping a field is destructive. The planner plans `unsetField(collection, field, { filter })`, an `updateMany` with `$unset`, class `destructive`, and reports a `field` subject. The question offers `rename` and `delete`, as on SQL. A field removed together with its collection gets no unset; the collection drop covers it. -- Order within a plan: collection renames, collection creates, index drops, document rewrites (renames and unsets), index creates, validator changes, option changes, collection drops. Rewrites come after index drops so a unique index on the old field does not fail once documents lose it, and before index creates so a unique index on the new field sees the moved values. Validation is bypassed for the rewrite because the origin validator requires the old field and the destination validator may require a field the same plan adds. Plans without statements and without field removals keep their current operations and order. -- The rewrite command gains a `bypassDocumentValidation` option through its wire form and the driver. +**Field rename** +- One `updateMany` with `$rename`, written as `renameField(...)` in `migration.ts`. +- Touches only documents that hold the old field. For a variant's field, only documents with that variant's discriminator value. +- Before: no document already holds the new name, which `$rename` would overwrite. After: no document holds the old name, so a re-run skips it. -### Document rewrites run under `db update` +**Field removal** +- One `updateMany` with `$unset`, written as `unsetField(...)`. +- No unset when the whole collection is dropped. -`db update` allows `additive`, `widening` and `destructive`, not `data`, so a rewrite classed `data` would be refused there. The rewrites above carry the class of what they do to data (`widening` for a rename, `destructive` for an unset), and the MongoDB runner and operation serializer tell a rewrite from a DDL command by its shape, not by `operationClass === 'data'`. `data` stays the class of transforms a user writes or a scaffold produces; `db update` keeps excluding it. Slice 3b's SQL JSON rewrites follow the same rule. This follows the Migration System doc's rule that only the planner's own classification promises an operation loses data exactly when it is `destructive`, and it changes ADR 188, which says every MongoDB data transform is `data`: the slice amends ADR 188, and ADR 264's line that MongoDB has no rename operation. +**Order in a plan** -### What is refused +1. Collection renames +2. Collection creates +3. Index drops +4. Document rewrites (renames and removals) +5. Index creates +6. Validator changes +7. Option changes +8. Collection drops -The MongoDB planner refuses, as a `statementRejected` conflict naming the reason: a rename of `_id`; a collection rename whose target exists in the working schema; a rename of a field stored on one side only; and any statement when the control policy is not `managed`, as SQL does. A document that already holds the new name fails at apply time by precheck, before anything is written. +- Rewrites run after index drops, so a unique index on the old field doesn't fail once documents lose the field. +- They run before index creates, so a unique index on the new field sees the moved values. +- Rewrites skip validation. The old validator requires the old field, and the new one may require a field the same plan adds. +- Plans with no statements and no removed fields come out exactly as today. -### The temporary refusal goes +**Refused, with the reason named** +- Renaming `_id`. +- Renaming a collection onto one that already exists. +- Renaming a field that is stored on only one side. +- Any statement when the model's control policy is not `managed`, as on SQL. -`TargetMigrationsCapability.renameStatements` and `keepDataByHand` are deleted: the member in `control-migration-types.ts`, MongoDB's setting in `control-target.ts`, `keepDataByHand` and `keepTheData` in `mongo-planner.ts`, `keepDataByHandFor` and its parameter in `plan-questions.ts`, its reads in `db-run.ts` and `migration-plan.ts`, and the Statements paragraph that describes it in the Migration System doc. The unrelated `renameStatements()` helper in `statement-text.ts` stays. The member shipped in 8.0.0-rc.17, so an extension upgrade fragment tells target authors to remove it. +**The temporary refusal goes.** MongoDB's `renameStatements: { refused: true, keepDataByHand }` setting, and all the code that reads it, is deleted. -## Coherence rationale +## Why one pull request -One reviewer can hold this: it is one family learning the two verbs the SQL targets already carry out, through one new DDL operation and one new rewrite shape, with the refusal member removed as the visible proof. The collection rename's six-package plumbing is mechanical and mirrors existing collection operations. +It is one database learning two flags the SQL targets already support. It adds one new operation and one new rewrite. The collection rename touches six packages, but the change is the same in each and copies the existing collection operations. ## Scope -**In:** amendments to ADR 188 (planner-written rewrites carry the class of their effect) and ADR 264 (MongoDB has a collection rename); model rename (collection rename); top-level field rename, including a variant's field; field removal as data loss with `$unset` under `--delete`; both commands; the `bypassDocumentValidation` option; runner and serializer telling rewrites apart by shape; deleting `renameStatements`; docs (Migration System § Statements, error reference, CLI README, `skills/prisma-8/references/migrations.md`); an app upgrade fragment (MongoDB renames work; removing a field now asks) and an extension upgrade fragment (the capability member is gone). - -**Deliberately out:** -- Value object field renames on MongoDB. They need the framework's value object statement surface (statement entity, subject kind, grammar, resolver), which slice 3b builds for SQL JSON rewrites, and a recursive update pipeline for lists, dictionaries and unions. Moved to slice 4b. -- `--convert` and `--backfill` on MongoDB, and the required-field validator gap: slice 4b. -- Batching large rewrites. One `updateMany` per field, as today's hand-written transforms do. - -## Pre-investigated edge cases +**In** +- Model rename, field rename (including a variant's field), field removal. +- Both commands. +- Deleting `renameStatements` and `keepDataByHand`. The unrelated `renameStatements()` helper in `statement-text.ts` stays. +- ADR amendments: + - ADR 188: planner-written rewrites carry the class of their effect. + - ADR 264: MongoDB now has a collection rename. +- Docs: Migration System § Statements, the error reference, the CLI README, `skills/prisma-8/references/migrations.md`. +- Upgrade fragments: + - App: renames work; removing a field now asks. + - Extension: the capability member is gone. It shipped in 8.0.0-rc.17. + +**Out** +- Renaming a field inside a value object. It needs statement support that slice 3b builds. Moved to 4b. +- `--convert`, `--backfill`, and the validator problem when a field becomes required: slice 4b. +- Batching rewrites on large collections. + +## Edge cases already known | Case | Handling | | --- | --- | -| Slice 1 manual QA (F3): a unique index on the old field fails the rewrite once two documents lose it; one on the new field fails if created first | Order above: index drops, rewrite, index creates | -| The origin validator requires the old field and forbids the new one | Rewrite runs with validation bypassed | -| Two variants in one collection store a field with the same name | Rewrite filters on the discriminator value | -| The runner is not transactional | Postcheck makes a re-run skip a finished rewrite | -| `bypassDocumentValidation` needs a privilege on hosted clusters | Runner failure names the privilege | -| TML-2447 (open): the planner creates separate collections and incomplete validators for variants that share a collection | The first dispatch checks how variants are stored today before the variant field rename is built on it | - -## Slice-specific done conditions - -- The project DoD's MongoDB journey without the value object part: a model rename and a field rename (with a unique index on the field) and a field removal under `--delete`, through `migration plan` then `migrate` and through `db update`, on collections with documents. Afterwards the documents are under the new names, the removed field is gone from every document, every document accepts an update, a further plan is empty, and `db verify --schema-only` is clean. -- `git grep -n "keepDataByHand\|TargetMigrationsCapability\['renameStatements'\]"` finds nothing. - -## Open questions - -None. - -## References - -- Grounding report for this slice (local, gitignored): `wip/grounding-4a.md`. -- [`../../deferred.md`](../../deferred.md) — "Which field name a MongoDB statement uses". -- Slice 1 manual QA, F3: [`../../manual-qa-reports/2026-10-07-qa-opus.md`](../../manual-qa-reports/2026-10-07-qa-opus.md). +| A unique index on the old field fails the rewrite once two documents lose it (slice 1 QA, F3) | Index drops run before the rewrite | +| A unique index on the new field fails if created before the rewrite | Index creates run after it | +| The old validator requires the old field and forbids the new one | The rewrite skips validation | +| Two variants in one collection store a field with the same name | The rewrite filters on the discriminator value | +| MongoDB migrations are not transactional | The after-check lets a re-run skip a finished rewrite | +| Skipping validation needs a privilege on hosted clusters | The failure names the privilege | +| TML-2447 (open): variants that share a collection get their own collection and an incomplete validator | The first dispatch checks how variants are stored today, before building on it | + +## Done when + +- On collections with documents, through `migration plan` then `migrate` and through `db update`: a model rename, a field rename with a unique index, and a field removal with `--delete`. Afterwards: + - the documents are under the new names; + - the removed field is gone from every document; + - every document accepts an update; + - a further plan is empty; + - `db verify --schema-only` is clean. +- `git grep -n "keepDataByHand"` finds nothing. + +## Implementation notes + +- `renameCollection` goes through the stack like the other collection operations: query AST command and visitor, wire command and lowering, driver, operation serializer, preview text, factory, call class. +- The raw update command gains `bypassDocumentValidation`, through its wire form and the driver. +- The runner and the operation serializer tell a rewrite from a DDL command by its shape (`run` against `execute`), not by `operationClass === 'data'`. +- Removed fields are reported as `field` subjects in `dataLoss`. +- Code to delete: the member in `control-migration-types.ts`; MongoDB's setting in `control-target.ts`; `keepDataByHand` and `keepTheData` in `mongo-planner.ts`; `keepDataByHandFor` and its parameter in `plan-questions.ts`; the reads in `db-run.ts` and `migration-plan.ts`; the paragraph in the Migration System doc. +- Grounding report (local, gitignored): `wip/grounding-4a.md`. diff --git a/projects/migration-statements/spec.md b/projects/migration-statements/spec.md index 783bd6fb856c..6603c7b15683 100644 --- a/projects/migration-statements/spec.md +++ b/projects/migration-statements/spec.md @@ -45,7 +45,7 @@ $ prisma migration plan --name age-to-int --convert User.age The reasoning and the alternatives rejected are in [`design-notes.md`](./design-notes.md). 1. **Data loss is refused by default, in both commands.** An operation is destructive only when it loses data: dropping a model or field, deleting a value rows still carry, or changing a field's type in a way the database cannot keep. Drops of indexes, constraints, checks, policies and defaults lose nothing and are widening. `db update` already refuses through its consent prompt; `migration plan` gains the same refusal for every plan, not only auto-baselines. -2. **Intent is stated on the command line, up front.** There is no prompt that guesses whether a drop is a rename, and no detection of candidate pairs. The refusal is where the user learns what to type, and it prints each statement ready to paste. The eventual direction (see [`plan.md`](./plan.md) § Stretch goal and requirement 12 below) is that when a human runs the command in a terminal, the refusal becomes an interactive prompt that asks the user to state what each destructive operation is; the answers are the same statements. "No prompt" was the initial design stance, not a rule; the rule is that nothing is guessed. +2. **Intent is stated on the command line, up front.** There is no prompt that guesses whether a drop is a rename, and no detection of candidate pairs. The refusal is where the user learns what to type, and it prints each statement ready to paste. The eventual direction (see [`plan.md`](./plan.md) § Delivered: the interactive prompt and requirement 12 below) is that when a human runs the command in a terminal, the refusal becomes an interactive prompt that asks the user to state what each destructive operation is; the answers are the same statements. "No prompt" was the initial design stance, not a rule; the rule is that nothing is guessed. 3. **Four verbs, four nouns.** The verbs are `rename`, `delete`, `convert` and `backfill`. The nouns are namespace, model, field and value, addressed by contract coordinates such as `User`, `User.name`, `Status.ARCHIVED`, `Bug` and `Address.street`, with the namespace prefixed when the contract has more than one. Rename and delete apply to all four nouns; a model rename across namespaces is a move. Convert applies to fields and to variants whose discriminator value changed. Backfill applies to fields. 4. **Statements are intent, not hints.** A statement that cannot be resolved against the origin and destination contracts, or cannot be applied, is an error and nothing is planned. There is no "spent" or "ignored" statement. A statement that resolves and applies but changes nothing in storage, such as renaming a value object or a relation field, is reported as applied with no operations. 5. **Everything is in the application domain.** The command line never carries a table, column, collection, SQL expression or literal value. The framework resolves coordinates against the two contracts; the family turns the resolved statement into storage operations. @@ -91,7 +91,7 @@ The reasoning and the alternatives rejected are in [`design-notes.md`](./design- 9. **Scaffolds reuse the placeholder flow.** `--convert` and `--backfill` produce the existing placeholder-bearing migration, with the placeholder in the operation that consumes the value. The pending state, the empty `ops.json` and the re-emit path are unchanged. 10. **Destructive means data loss.** Drops of indexes, unique and foreign-key constraints, checks, row-level-security policies, defaults and native enum types, and disabling row-level security, are widening on every target. A type change the family knows to be widening needs no statement. 11. **Framework stays family-agnostic.** The statement grammar, parser and resolver carry no family or target vocabulary, and each family documents what every verb means for its storage. -12. **Built for the interactive prompt to drop in.** The refusal is a structured list of destructive operations, each with its entity in domain coordinates and the statements that would resolve it; the error text is rendered from that list. Statement strings are parsed, resolved and applied through one entry point that does not know whether they came from the command line or from an answer typed at a prompt, so statements can be added after the command has started. See [`plan.md`](./plan.md) § Stretch goal. +12. **Built for the interactive prompt to drop in.** The refusal is a structured list of destructive operations, each with its entity in domain coordinates and the statements that would resolve it; the error text is rendered from that list. Statement strings are parsed, resolved and applied through one entry point that does not know whether they came from the command line or from an answer typed at a prompt, so statements can be added after the command has started. See [`plan.md`](./plan.md) § Delivered: the interactive prompt. ## Transitional-shape constraints From e86e483a89c9fa3034df73501a162e53d066da35 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 09:43:28 +0200 Subject: [PATCH 05/57] TML-3478: the 4a spec records Will's decisions on field removal, names, classes and variants Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/deferred.md | 6 ++++++ .../migration-statements/slices/mongo-renames/spec.md | 10 ++++++---- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index 9d0718e7c125..5f6948efc015 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -65,3 +65,9 @@ Adding a required field to a populated collection, or making an optional field r `CLI.CONSENT_REQUIRED` lists one `user-choice` next action per flag form ("Run the command again with --delete Legacy", another for the rename form), not grouped by subject and not a complete runnable command, and its summary says "N subjects need a statement", which is project vocabulary. Grouping the forms per subject and printing a full command would need the engine to know the invocation; raise it on prisma/prisma-cli once the statement prompt has shipped. +## SQL variants stored in their own tables + +**Found:** slice 4a design (2026-10-09). Will: SQL will need to handle multi-table storage of variants in the future. + +On MongoDB a variant's field rename touches only that variant's documents. On SQL today a variant's field is a column in the shared table, and a rename applies to every row. When SQL supports variants stored in their own tables, a variant's field statements must act on that variant's table only. + diff --git a/projects/migration-statements/slices/mongo-renames/spec.md b/projects/migration-statements/slices/mongo-renames/spec.md index 0344713b340b..07af0f73aa93 100644 --- a/projects/migration-statements/slices/mongo-renames/spec.md +++ b/projects/migration-statements/slices/mongo-renames/spec.md @@ -37,11 +37,13 @@ Without `--delete User.nickname`, both commands refuse, as they do on SQL: | Field removed | Validator updated, nothing asked. Every document that still holds the field then fails each update | Asked about. `--delete` removes the field from every document; `--rename` keeps its values | | Model removed | Asked about; `--delete` drops the collection | Same, and `--rename` is offered too | -## Decisions to check +## Decisions (settled with Will, 2026-10-09) -1. **Removing a field now asks.** Today it is silent, but MongoDB's validator then rejects every update to a document that still holds the field. -2. **Fields are named as stored.** `id @map("_id")` is `User._id` in a statement. The contract and the generated client already use that name. Naming the schema field instead would change the contract and its hashes. -3. **The rewrites run under `db update`.** They are classed by what they do to data: a rename is `widening`, a removal is `destructive`. `db update` refuses the `data` class, so classing them `data`, as ADR 188 says today, would block `db update`. This slice amends ADR 188. +1. **Removing a field is data loss.** It is asked about, and `--delete` removes it from every document. Today it is silent, and MongoDB's validator then rejects every update to a document that still holds the field. +2. **Fields are named by their contract key, as on SQL.** MongoDB's contract keys fields by their stored name, so `id @map("_id")` is `User._id` in a statement. Naming the schema field instead would need MongoDB's contract to keep a field-to-stored-name map, as SQL's does. That is outside this project. +3. **Rewrites are classed by what they do to data, as on SQL.** A field rename is `widening`, like a column rename. A field removal is `destructive`, like a column drop. The planner writes a `data` operation only as a placeholder scaffold. So `db update` runs these rewrites, as it runs SQL renames. ADR 188 changes one sentence: today it says every MongoDB data transform is `data`. +4. **Without the old contract, a removed field is named `.`,** for example `users.nickname`. SQL uses `..`, or `
.` on SQLite. +5. **A variant's field rename touches only that variant's documents.** The rewrite filters on the variant's discriminator value, so another variant in the same collection that stores a field with the same name keeps it. This differs from SQL single-table storage, where a rename renames the shared column for every row. ## How it works From 9d2922e672573f52f119f6dd14d8489e80a03746 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 09:47:01 +0200 Subject: [PATCH 06/57] TML-3474: dispatch plans for 3a and 4a, first briefs, and the new tickets Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/plan.md | 4 +- .../01-statements-reach-the-planner.md | 36 ++++++++++++++++++ .../slices/convert-backfill/plan.md | 37 ++++++++++++++++++ .../01-rename-collection-operation.md | 38 +++++++++++++++++++ .../slices/mongo-renames/plan.md | 35 +++++++++++++++++ 5 files changed, 148 insertions(+), 2 deletions(-) create mode 100644 projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md create mode 100644 projects/migration-statements/slices/convert-backfill/plan.md create mode 100644 projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md create mode 100644 projects/migration-statements/slices/mongo-renames/plan.md diff --git a/projects/migration-statements/plan.md b/projects/migration-statements/plan.md index 7d3276433912..b08e4a85240a 100644 --- a/projects/migration-statements/plan.md +++ b/projects/migration-statements/plan.md @@ -49,7 +49,7 @@ ### 3b — The remaining renames and deletes on Postgres and SQLite -**Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Folder:** `slices/remaining-nouns/` +**Linear:** [TML-3537](https://linear.app/prisma-company/issue/TML-3537) · **Folder:** `slices/remaining-nouns/` - `--rename` for enum values, namespaces and fields inside value objects. This slice builds the statement support for value objects, which 4b reuses. - Moving a model to another namespace (`alter table set schema` on Postgres). @@ -64,7 +64,7 @@ ### 4b — `--convert`, `--backfill` and value object renames on MongoDB -**Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Folder:** `slices/mongo-convert-backfill/` +**Linear:** [TML-3538](https://linear.app/prisma-company/issue/TML-3538) · **Folder:** `slices/mongo-convert-backfill/` - The same two flags as 3a, writing a data transform with a placeholder. - Making a field required on a collection with documents is no longer silent. It is answered with `--backfill`, or the command says the existing documents need the field. diff --git a/projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md b/projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md new file mode 100644 index 000000000000..340b3935c712 --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/dispatches/01-statements-reach-the-planner.md @@ -0,0 +1,36 @@ +# Dispatch 1 — `--convert` and `--backfill` reach the planner, and questions know why data is lost + +**Slice:** [`../spec.md`](../spec.md) § The two flags, § How the pieces fit, § Implementation notes · **Plan:** [`../plan.md`](../plan.md) dispatch 1 · **Builds on:** slice 2 (merged on `main`) + +## Outcome + +Both flags are parsed, resolved against the two contracts, and handed to the SQL planners as destination coordinates. The planners do nothing new with them yet. Each data-loss entry says whether it is a `drop` or a `typeChange`, and the questions offer the verbs that fit. `db update` refuses both flags with a clear error. + +## What to build + +1. **Verbs per command.** `orm/statement-verbs.ts` declares verbs per command. `migration plan`: `rename`, `delete`, `convert`, `backfill`. `db update`: `rename`, `delete`, `allow`, plus `convert` and `backfill` declared only so they can be refused: any value fails with the new `MIGRATION.STATEMENT_NEEDS_MIGRATION_FILE`, whose message says `db update` has no migration file to fill and names `migration plan`. Error reference entry. +2. **Resolved statements gain a `kind`.** `ResolvedMigrationStatement` (`packages/1-framework/1-core/framework-components/src/control/migration-statements.ts`) becomes `kind: 'rename' | 'convert' | 'backfill'`. Every reader that assumes a rename switches on `kind` first: `describeMigrationStatement`, `migrationStatementJson`, `sameCoordinate` and `renamedModel` in `plan-questions.ts`, `originTable` and `originColumn` in the SQL family `operation-subjects.ts`, `StatementPlanner.plan` in `statement-planning.ts` (today it plans any non-model statement as a column rename), and the MongoDB planner (keeps refusing every statement it cannot carry out). +3. **Resolution rules** (in `resolve-statements.ts`): + - `convert Model.field`: names the field by its destination name (after `--rename User.age:User.years`, `--convert User.years`); the field exists in the destination and, through any earlier rename, in the origin; its type changes. Needs the origin contract, else `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`. + - `backfill Model.field`: the field is required in the destination and is either new on a model that exists in the origin or optional in the origin. + - Anything else is `MIGRATION.STATEMENT_UNRESOLVED` naming what was found. +4. **Handed to the planners.** `migration plan` takes `convert` and `backfill` from the run before planning, like `rename`. The SQL planners receive them as destination coordinates (namespace, table, column) on the strategy context (Postgres `planner-strategies.ts` around `StrategyContext`; SQLite equivalent). They do not go through `planStatements`, which applies renames to the working schema. No strategy reads them yet. +5. **The loss kind.** Each `dataLoss` entry gains `loss: 'drop' | 'typeChange'`, set by each target's operation-subjects function. `dataLossQuestion` stops inferring from whether the destination still has the field. A `typeChange` question offers `convert` and `delete` on `migration plan` and `delete` only on `db update`. A `drop` question offers `rename` and `delete`, as today. +6. **Re-plan with every statement.** `askPlanQuestions` re-plans with all planned statements, not only renames, so a typed `convert` re-plans. +7. **Reporting a delete on a type change.** A `delete` answering a `typeChange` question is described as `delete values of field "User.age" (type change)`, not `delete field`. + +## Not in this dispatch + +Any planner output change (dispatches 2–4). The temporary default. Docs beyond the error reference entry. Journeys. + +## Tests + +Unit tests in the CLI and framework-components packages: each resolution rule, both pass and fail; `convert` after a rename uses the new name; `convert` with no origin contract; the verbs offered per loss kind and per command; `db update --convert` fails with the new code before planning; a typed `convert` triggers a re-plan; the delete-on-type-change description; every `kind` switch (a `convert` statement is never planned as a column rename — write the test that would have caught it). + +## Halt conditions + +Stop and report if the engine cannot declare a verb on `db update` that the command refuses with its own error; or if `dataLoss` producers outside the SQL family (extension spaces applying recorded migrations, MongoDB) cannot set `loss` without a design decision. + +## Gate + +The plan's gate, plus `pnpm check:error-reference` and `pnpm lint:framework-vocabulary`. diff --git a/projects/migration-statements/slices/convert-backfill/plan.md b/projects/migration-statements/slices/convert-backfill/plan.md new file mode 100644 index 000000000000..5b419702253a --- /dev/null +++ b/projects/migration-statements/slices/convert-backfill/plan.md @@ -0,0 +1,37 @@ +# Slice plan — 3a, `--convert` and `--backfill` + +**Spec:** [`spec.md`](./spec.md) · **Linear:** [TML-3477](https://linear.app/prisma-company/issue/TML-3477) · **Branch:** `tml-3477-convert-backfill` + +One implementer and one reviewer, both on Opus, resumed across every dispatch. Briefs are in `dispatches/`. + +## Gate for every dispatch + +- `pnpm typecheck`; `pnpm --filter lint`; `pnpm --filter test`. +- `pnpm lint:deps` when imports change; `pnpm fixtures:check` when rendering or classes change; `pnpm check:error-reference` when error codes change; `pnpm lint:framework-vocabulary` when `packages/1-framework` changes; `pnpm lint:throws` and `pnpm lint:casts` (CI runs them, `lint:agent` skips them). +- Integration and e2e: only named files, run as `pnpm test `. Never `pnpm test:integration`, `test:e2e` or `test:all`. +- The reviewer runs code, not only reads it (`drive/calibration/failure-modes.md` F34): SQL the slice renders is compiled and run on PGlite and SQLite, and every printed advice is followed (F46). + +## Dispatches + +| # | Outcome | Builds on | +| --- | --- | --- | +| 1 | `--convert` and `--backfill` are parsed, resolved and handed to the planners; each data-loss entry says `drop` or `typeChange`; questions offer the right verbs; `db update` refuses both flags | slice 2 | +| 2 | Postgres: a lossy type change plans nothing without a statement; `--convert` writes `alterColumnType` with a `using` placeholder; `alterColumnType` takes plain options | 1 | +| 3 | Postgres: `--backfill` writes the backfill scaffold; without it a new required field gets the temporary default as a real call in `migration.ts`; optional-to-required is a direct `SET NOT NULL` | 1 | +| 4 | SQLite: the rebuild takes a per-column copy expression; `--convert`, `--backfill` and the temporary value work as on Postgres | 2, 3 | +| 5 | The NOT NULL failure names the NULL rows and suggests a fix, on Postgres and SQLite (TML-3517) | 3, 4 | +| 6 | Journeys for the slice's done conditions; docs; app upgrade fragment and facade upgrade instructions; every printed advice followed | 1–5 | + +**Hand-offs** +- 1 → 2, 3: planners receive convert and backfill as destination coordinates, and a converted change leaves `dataLoss`. +- 2 → 4: the converted alter is `data`; the `using` slot renders as code and blocks `toOp()` while unfilled. +- 3 → 4: the temporary default renders as a call that reproduces `ops.json`. +- 6 → slice DoD. + +## Coordination with 4a + +4a runs at the same time on `tml-3478-mongo-renames`. Both touch `plan-questions.ts` and the resolved statement type. Whichever merges second merges `main` and adapts. The MongoDB planner must keep refusing `convert` and `backfill` statements until 4b. + +## Open items + +- ADR 200 amendment: written by the orchestrator in dispatch 6's branch, not by the implementer. diff --git a/projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md b/projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md new file mode 100644 index 000000000000..d9a3de8aad32 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/dispatches/01-rename-collection-operation.md @@ -0,0 +1,38 @@ +# Dispatch 1 — A `renameCollection` operation, end to end + +**Slice:** [`../spec.md`](../spec.md) § How it works, § Implementation notes · **Plan:** [`../plan.md`](../plan.md) dispatch 1 · **Builds on:** slice 2 (merged on `main`) + +## Outcome + +A hand-written MongoDB migration can call `renameCollection('profiles', 'users')`, and running it renames the collection with its documents, indexes and validator. The planner does not use it yet. You also report how variants are stored today. + +## What to build + +1. **The operation through the stack**, the way `createCollection` and `dropCollection` go: + - a `RenameCollectionCommand` and its visitor method in `packages/2-mongo-family/4-query/query-ast` (`ddl-commands.ts`, `ddl-visitors.ts`); + - its wire command in `packages/2-mongo-family/6-transport/mongo-lowering` and lowering in `packages/3-mongo-target/2-mongo-adapter`; + - a driver case in `packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts`; + - serializer and deserializer in the target's operation serializer; + - preview text in `packages/2-mongo-family/9-family/src/core/operation-preview.ts`; + - a `renameCollection` factory in `migration-factories.ts`, a `RenameCollectionCall` in `op-factory-call.ts` that renders `renameCollection('profiles', 'users')`, and the export from `src/exports/migration.ts`. +2. **Checks.** Before: the source collection exists and the target does not (`listCollections`). After: the target exists and the source does not. A re-run after success skips through the after-check. +3. **Class:** `widening`. +4. **Variant report (TML-2447).** Plan a contract with a base model and two variants that share one collection (`@@base`, `@@discriminator`) on current `main`, and report: which collections the planner creates, what the validator holds, and whether a variant's documents can be told apart by the discriminator value in the stored documents. Do not fix anything here; the orchestrator decides from the report. + +## Not in this dispatch + +Planning the rename (dispatch 3). Document rewrites (dispatch 2). Removing `renameStatements`. + +## Tests + +- Unit: factory, call rendering, serializer round trip, preview text. +- Against mongodb-memory-server (existing target-runner tests under `test/integration/test/mongo/target-runner/` show the setup): a collection with documents, a unique index and a closed validator is renamed, and all three are kept; re-running skips; renaming onto an existing collection fails at the before-check with nothing changed. +- `render-typescript` round trip: running the rendered `migration.ts` reproduces the operation in `ops.json`. + +## Halt conditions + +Stop and report if the driver's rename needs admin privileges that mongodb-memory-server does not give; or if a DDL command cannot carry a second collection name without changing the shared command shape. + +## Gate + +The plan's gate, plus the new target-runner test file. diff --git a/projects/migration-statements/slices/mongo-renames/plan.md b/projects/migration-statements/slices/mongo-renames/plan.md new file mode 100644 index 000000000000..7a9597d0c7c2 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/plan.md @@ -0,0 +1,35 @@ +# Slice plan — 4a, renames and deletes on MongoDB + +**Spec:** [`spec.md`](./spec.md) · **Linear:** [TML-3478](https://linear.app/prisma-company/issue/TML-3478) · **Branch:** `tml-3478-mongo-renames` + +One implementer and one reviewer, both on Opus, resumed across every dispatch. Briefs are in `dispatches/`. + +## Gate for every dispatch + +- `pnpm typecheck`; `pnpm --filter lint`; `pnpm --filter test`. +- `pnpm lint:deps` when imports change; `pnpm fixtures:check` when rendering or classes change; `pnpm check:error-reference` when error codes change; `pnpm lint:framework-vocabulary` when `packages/1-framework` changes; `pnpm lint:throws` and `pnpm lint:casts`. +- Integration and e2e: only named files, run as `pnpm test `. Never `pnpm test:integration`, `test:e2e` or `test:all`. +- The reviewer runs code against mongodb-memory-server, not only reads it (F34), and follows every printed advice (F46). + +## Dispatches + +| # | Outcome | Builds on | +| --- | --- | --- | +| 1 | A `renameCollection` operation exists end to end: factory, call, serializer, runner, preview text, with before and after checks. Also reports how variants are stored today (TML-2447) | slice 2 | +| 2 | `renameField` and `unsetField` rewrites exist end to end, skip validation, carry `widening` and `destructive`, and the runner and serializer tell them from DDL by shape | 1 | +| 3 | The planner carries out statements: collection rename on the working schema, field rename (filtered by discriminator for a variant), field removal as data loss, the operation order, the refusals. `renameStatements` and `keepDataByHand` are deleted | 2 | +| 4 | Journeys for the slice's done conditions through both commands; docs; app and extension upgrade fragments; every printed advice followed | 3 | + +**Hand-offs** +- 1 → 3: `renameCollection` can be planned and rendered. +- 2 → 3: `renameField` and `unsetField` can be planned and rendered, and run under `db update`'s classes. +- 3 → 4: both commands rename and delete on MongoDB. + +## Coordination with 3a + +3a runs at the same time on `tml-3477-convert-backfill` and adds a `kind` to resolved statements. Whichever merges second merges `main` and adapts. After 3a, the MongoDB planner handles `kind: 'rename'` and refuses `convert` and `backfill`. + +## Open items + +- ADR 188 and ADR 264 amendments: written by the orchestrator in dispatch 4's branch. +- If dispatch 1 finds that TML-2447 makes variant field renames wrong, they move to 4b, and Will is told. From 0b4a2220e645a45dbdf6fdd0972bee9149444d4d Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 09:53:46 +0200 Subject: [PATCH 07/57] TML-3478: a renameCollection operation runs end to end on MongoDB A hand-written migration can call renameCollection('profiles', 'users'). The command goes through the query AST, wire command, lowering, driver, operation serializer and preview text. It checks that the source exists and the target does not, and a re-run skips once the target exists without the source. The planner does not use it yet. Signed-off-by: willbot Signed-off-by: Will Madden --- .../4-query/query-ast/src/ddl-commands.ts | 18 ++ .../4-query/query-ast/src/ddl-visitors.ts | 2 + .../4-query/query-ast/src/exports/control.ts | 1 + .../query-ast/test/ddl-commands.test.ts | 25 +++ .../mongo-wire/src/exports/index.ts | 1 + .../mongo-wire/src/wire-commands.ts | 13 ++ .../9-family/src/core/operation-preview.ts | 5 + .../test/operation-preview-format.test.ts | 18 ++ .../core/migrations/migration-factories.ts | 27 +++ .../core/migrations/mongo-ops-serializer.ts | 11 + .../src/core/migrations/op-factory-call.ts | 26 +++ .../1-mongo-target/src/exports/control.ts | 1 + .../1-mongo-target/src/exports/migration.ts | 1 + .../1-mongo-target/test/migration-e2e.test.ts | 25 +++ .../test/migration-factories.test.ts | 39 ++++ .../test/mongo-ops-serializer.test.ts | 9 + .../test/op-factory-call.test.ts | 24 ++ .../2-mongo-adapter/src/mongo-adapter.ts | 3 + .../test/command-executor.test.ts | 16 +- .../test/ddl-lowering-oracle.test.ts | 13 ++ .../3-mongo-driver/src/mongo-driver.ts | 7 + .../3-mongo-driver/test/mongo-driver.test.ts | 25 +++ .../runner-rename-collection.test.ts | 212 ++++++++++++++++++ 23 files changed, 521 insertions(+), 1 deletion(-) create mode 100644 test/integration/test/mongo/target-runner/runner-rename-collection.test.ts diff --git a/packages/2-mongo-family/4-query/query-ast/src/ddl-commands.ts b/packages/2-mongo-family/4-query/query-ast/src/ddl-commands.ts index 2c39038759df..feac8cfd7d9b 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/ddl-commands.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/ddl-commands.ts @@ -162,6 +162,23 @@ export class DropCollectionCommand extends MongoAstNode { } } +export class RenameCollectionCommand extends MongoAstNode { + readonly kind = 'renameCollection' as const; + readonly collection: string; + readonly to: string; + + constructor(collection: string, to: string) { + super(); + this.collection = collection; + this.to = to; + this.freeze(); + } + + accept(visitor: MongoDdlCommandVisitor): R { + return visitor.renameCollection(this); + } +} + export interface CollModOptions { readonly validator?: Record; readonly validationLevel?: 'strict' | 'moderate'; @@ -198,4 +215,5 @@ export type AnyMongoDdlCommand = | DropIndexCommand | CreateCollectionCommand | DropCollectionCommand + | RenameCollectionCommand | CollModCommand; diff --git a/packages/2-mongo-family/4-query/query-ast/src/ddl-visitors.ts b/packages/2-mongo-family/4-query/query-ast/src/ddl-visitors.ts index a53730f93f14..5beb098b9b78 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/ddl-visitors.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/ddl-visitors.ts @@ -4,6 +4,7 @@ import type { CreateIndexCommand, DropCollectionCommand, DropIndexCommand, + RenameCollectionCommand, } from './ddl-commands'; import type { ListCollectionsCommand, ListIndexesCommand } from './inspection-commands'; @@ -12,6 +13,7 @@ export interface MongoDdlCommandVisitor { dropIndex(command: DropIndexCommand): R; createCollection(command: CreateCollectionCommand): R; dropCollection(command: DropCollectionCommand): R; + renameCollection(command: RenameCollectionCommand): R; collMod(command: CollModCommand): R; } diff --git a/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts b/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts index 3e043827f515..998135c7b6ea 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts @@ -11,6 +11,7 @@ export { CreateIndexCommand, DropCollectionCommand, DropIndexCommand, + RenameCollectionCommand, } from '../ddl-commands'; export type { MongoDdlCommandVisitor, MongoInspectionCommandVisitor } from '../ddl-visitors'; export type { MongoFilterExpr } from '../filter-expressions'; diff --git a/packages/2-mongo-family/4-query/query-ast/test/ddl-commands.test.ts b/packages/2-mongo-family/4-query/query-ast/test/ddl-commands.test.ts index 6e15e7108e78..2b75e4234f19 100644 --- a/packages/2-mongo-family/4-query/query-ast/test/ddl-commands.test.ts +++ b/packages/2-mongo-family/4-query/query-ast/test/ddl-commands.test.ts @@ -5,6 +5,7 @@ import { CreateIndexCommand, DropCollectionCommand, DropIndexCommand, + RenameCollectionCommand, } from '../src/ddl-commands'; import type { MongoDdlCommandVisitor, MongoInspectionCommandVisitor } from '../src/ddl-visitors'; import { ListCollectionsCommand, ListIndexesCommand } from '../src/inspection-commands'; @@ -17,6 +18,7 @@ function makeDdlVisitor( dropIndex: () => 'dropIndex', createCollection: () => 'createCollection', dropCollection: () => 'dropCollection', + renameCollection: () => 'renameCollection', collMod: () => 'collMod', ...overrides, }; @@ -252,6 +254,29 @@ describe('DropCollectionCommand', () => { }); }); +describe('RenameCollectionCommand', () => { + it('constructs with the source collection and the new name', () => { + const cmd = new RenameCollectionCommand('profiles', 'users'); + expect({ kind: cmd.kind, collection: cmd.collection, to: cmd.to }).toEqual({ + kind: 'renameCollection', + collection: 'profiles', + to: 'users', + }); + }); + + it('is frozen', () => { + expect(Object.isFrozen(new RenameCollectionCommand('profiles', 'users'))).toBe(true); + }); + + it('dispatches via DDL visitor', () => { + const cmd = new RenameCollectionCommand('profiles', 'users'); + const visitor = makeDdlVisitor({ + renameCollection: (c) => `rename:${c.collection}->${c.to}`, + }); + expect(cmd.accept(visitor)).toBe('rename:profiles->users'); + }); +}); + describe('CollModCommand', () => { it('constructs with validator', () => { const cmd = new CollModCommand('users', { diff --git a/packages/2-mongo-family/6-transport/mongo-wire/src/exports/index.ts b/packages/2-mongo-family/6-transport/mongo-wire/src/exports/index.ts index 9290c92a15e2..698515eacaad 100644 --- a/packages/2-mongo-family/6-transport/mongo-wire/src/exports/index.ts +++ b/packages/2-mongo-family/6-transport/mongo-wire/src/exports/index.ts @@ -25,6 +25,7 @@ export { InsertManyWireCommand, InsertOneWireCommand, isDdlWireCommand, + RenameCollectionWireCommand, UpdateManyWireCommand, UpdateOneWireCommand, } from '../wire-commands'; diff --git a/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts b/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts index 36ffe626bf7d..470f512d28ce 100644 --- a/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts +++ b/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts @@ -250,6 +250,17 @@ export class DropCollectionWireCommand extends MongoWireCommand { } } +export class RenameCollectionWireCommand extends MongoWireCommand { + readonly kind = 'renameCollection' as const; + readonly to: string; + + constructor(collection: string, to: string) { + super(collection); + this.to = to; + this.freeze(); + } +} + export class DropIndexWireCommand extends MongoWireCommand { readonly kind = 'dropIndex' as const; readonly name: string; @@ -294,6 +305,7 @@ export type AnyMongoDdlWireCommand = | CreateCollectionWireCommand | CreateIndexWireCommand | DropCollectionWireCommand + | RenameCollectionWireCommand | DropIndexWireCommand | CollModWireCommand; @@ -303,6 +315,7 @@ const DDL_KINDS: ReadonlySet = new Set([ 'createCollection', 'createIndex', 'dropCollection', + 'renameCollection', 'dropIndex', 'collMod', ]); diff --git a/packages/2-mongo-family/9-family/src/core/operation-preview.ts b/packages/2-mongo-family/9-family/src/core/operation-preview.ts index 0303816b0853..1959fda7778e 100644 --- a/packages/2-mongo-family/9-family/src/core/operation-preview.ts +++ b/packages/2-mongo-family/9-family/src/core/operation-preview.ts @@ -10,6 +10,7 @@ import type { DropIndexCommand, MongoDdlCommandVisitor, MongoIndexKey, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; function formatKeySpec(keys: ReadonlyArray): string { @@ -59,6 +60,10 @@ class MongoDdlCommandFormatter implements MongoDdlCommandVisitor { return `db.${cmd.collection}.drop()`; } + renameCollection(cmd: RenameCollectionCommand): string { + return `db.${cmd.collection}.renameCollection(${JSON.stringify(cmd.to)})`; + } + collMod(cmd: CollModCommand): string { const parts: string[] = [`collMod: ${JSON.stringify(cmd.collection)}`]; for (const [key, value] of Object.entries(cmd)) { diff --git a/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts b/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts index 3ea37b482383..88b66369601c 100644 --- a/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts +++ b/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts @@ -5,6 +5,7 @@ import { DropCollectionCommand, DropIndexCommand, type MongoMigrationPlanOperation, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import { describe, expect, it } from 'vitest'; import { formatMongoOperations } from '../src/core/operation-preview'; @@ -246,6 +247,23 @@ describe('formatMongoOperations', () => { expect(result).toEqual(['db.events.drop()']); }); + it('formats renameCollection', () => { + const op: MongoMigrationPlanOperation = { + id: 'collection.profiles.rename', + label: 'Rename collection', + operationClass: 'widening', + precheck: [], + execute: [ + { + description: 'rename collection', + command: new RenameCollectionCommand('profiles', 'users'), + }, + ], + postcheck: [], + }; + expect(formatMongoOperations([op])).toEqual(['db.profiles.renameCollection("users")']); + }); + it('formats collMod with validator', () => { const op: MongoMigrationPlanOperation = { id: 'test', diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts index 87c5dcf53702..159014771199 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts @@ -20,6 +20,7 @@ import { MongoExistsExpr, MongoFieldFilter, type MongoMigrationPlanOperation, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import type { MongoQueryPlan } from '@internal/mongo-query-ast/execution'; import { createFieldAccessor } from '@internal/mongo-query-builder'; @@ -249,6 +250,32 @@ export function dropCollection(collectionName: string): MongoMigrationPlanOperat }; } +function collectionCheck(name: string, expect: 'exists' | 'notExists') { + return { + description: + expect === 'exists' ? `collection ${name} exists` : `collection ${name} does not exist`, + source: new ListCollectionsCommand(), + filter: MongoFieldFilter.eq('name', name), + expect, + }; +} + +export function renameCollection(from: string, to: string): MongoMigrationPlanOperation { + return { + id: `collection.${from}.rename(${to})`, + label: `Rename collection ${from} to ${to}`, + operationClass: 'widening', + precheck: [collectionCheck(from, 'exists'), collectionCheck(to, 'notExists')], + execute: [ + { + description: `rename collection ${from} to ${to}`, + command: new RenameCollectionCommand(from, to), + }, + ], + postcheck: [collectionCheck(to, 'exists'), collectionCheck(from, 'notExists')], + }; +} + export function setValidation( collectionName: string, schema: Record, diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts index f4e005e11475..6fcee0370f04 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts @@ -21,6 +21,7 @@ import { type MongoMigrationStep, MongoNotExpr, MongoOrExpr, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import { AggregateCommand, @@ -115,6 +116,12 @@ const DropCollectionJson = type({ collection: 'string', }); +const RenameCollectionJson = type({ + kind: '"renameCollection"', + collection: 'string', + to: 'string', +}); + const CollModJson = type({ kind: '"collMod"', collection: 'string', @@ -678,6 +685,10 @@ function deserializeDdlCommand(json: unknown): AnyMongoDdlCommand { const data = validate(DropCollectionJson, json, 'dropCollection command'); return new DropCollectionCommand(data.collection); } + case 'renameCollection': { + const data = validate(RenameCollectionJson, json, 'renameCollection command'); + return new RenameCollectionCommand(data.collection, data.to); + } case 'collMod': { const data = validate(CollModJson, json, 'collMod command'); return new CollModCommand(data.collection, { diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts index 49e30382823a..b867ae077d3f 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts @@ -42,6 +42,7 @@ import { createIndex, dropCollection, dropIndex, + renameCollection, } from './migration-factories'; export interface CollModMeta { @@ -182,6 +183,30 @@ export class DropCollectionCall extends OpFactoryCallNode { } } +export class RenameCollectionCall extends OpFactoryCallNode { + readonly factoryName = 'renameCollection' as const; + readonly operationClass = 'widening' as const; + readonly from: string; + readonly to: string; + readonly label: string; + + constructor(from: string, to: string) { + super(); + this.from = from; + this.to = to; + this.label = `Rename collection ${from} to ${to}`; + this.freeze(); + } + + toOp(): MongoMigrationPlanOperation { + return renameCollection(this.from, this.to); + } + + renderTypeScript(): string { + return `renameCollection(${jsonToTsSource(this.from)}, ${jsonToTsSource(this.to)})`; + } +} + export class CollModCall extends OpFactoryCallNode { readonly factoryName = 'collMod' as const; readonly collection: string; @@ -216,6 +241,7 @@ export type OpFactoryCall = | DropIndexCall | CreateCollectionCall | DropCollectionCall + | RenameCollectionCall | CollModCall; export function schemaIndexToCreateIndexOptions(index: MongoSchemaIndex): CreateIndexOptions { diff --git a/packages/3-mongo-target/1-mongo-target/src/exports/control.ts b/packages/3-mongo-target/1-mongo-target/src/exports/control.ts index ee1b7b1e2f33..54609cfb607b 100644 --- a/packages/3-mongo-target/1-mongo-target/src/exports/control.ts +++ b/packages/3-mongo-target/1-mongo-target/src/exports/control.ts @@ -19,6 +19,7 @@ export { CreateIndexCall, DropCollectionCall, DropIndexCall, + RenameCollectionCall, schemaCollectionToCreateCollectionOptions, schemaIndexToCreateIndexOptions, } from '../core/migrations/op-factory-call'; diff --git a/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts b/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts index 363dd0a6c152..fb18e090eff0 100644 --- a/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts +++ b/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts @@ -19,6 +19,7 @@ export { dataTransform, dropCollection, dropIndex, + renameCollection, setValidation, validatedCollection, } from '../core/migrations/migration-factories'; diff --git a/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts b/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts index cfdfa6f9cdbb..da4d6c4c299b 100644 --- a/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts @@ -310,6 +310,31 @@ describe('migration file E2E', () => { expect(ops[0].execute[0].command.kind).toBe('collMod'); }); + it('reproduces the renameCollection operation in ops.json', async () => { + const { renderCallsToTypeScript } = await import('../src/core/migrations/render-typescript'); + const { RenameCollectionCall } = await import('../src/core/migrations/op-factory-call'); + const { deserializeMongoOps } = await import('../src/core/migrations/mongo-ops-serializer'); + const call = new RenameCollectionCall('profiles', 'users'); + + const tsSource = renderCallsToTypeScript([call], defaultMeta); + expect(tsSource).toContain('renameCollection("profiles", "users")'); + const resolvedSource = tsSource.replace( + "'@internal/target-mongo/migration'", + `'${factoryExport}'`, + ); + await writeFile(join(tmpDir, 'migration.ts'), resolvedSource); + await writeContractFixtures(defaultMeta); + + const result = await runFile('migration.ts'); + expect({ exitCode: result.exitCode, stderr: result.stderr }).toEqual({ + exitCode: 0, + stderr: '', + }); + + const ops = JSON.parse(await readFile(join(tmpDir, 'ops.json'), 'utf-8')); + expect(deserializeMongoOps(ops)).toEqual([call.toOp()]); + }); + it('round-trips describe() meta through TypeScript execution', async () => { const { renderCallsToTypeScript } = await import('../src/core/migrations/render-typescript'); const { DropCollectionCall } = await import('../src/core/migrations/op-factory-call'); diff --git a/packages/3-mongo-target/1-mongo-target/test/migration-factories.test.ts b/packages/3-mongo-target/1-mongo-target/test/migration-factories.test.ts index 88d001d20a0a..38e02af6a911 100644 --- a/packages/3-mongo-target/1-mongo-target/test/migration-factories.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/migration-factories.test.ts @@ -12,6 +12,7 @@ import { MongoAndExpr, MongoFieldFilter, type MongoMigrationPlanOperation, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import { MongoSchemaCollection, MongoSchemaCollectionOptions } from '@internal/mongo-schema-ir'; import { describe, expect, it } from 'vitest'; @@ -21,6 +22,7 @@ import { createIndex, dropCollection, dropIndex, + renameCollection, setValidation, validatedCollection, } from '../src/core/migrations/migration-factories'; @@ -159,6 +161,43 @@ describe('dropIndex', () => { }); }); +describe('renameCollection', () => { + it('renames when the source exists and the target does not, and skips once the target exists without the source', () => { + const sourceIs = (expect: 'exists' | 'notExists') => ({ + description: 'collection profiles exists', + source: new ListCollectionsCommand(), + filter: MongoFieldFilter.eq('name', 'profiles'), + expect, + }); + const targetIs = (expect: 'exists' | 'notExists') => ({ + description: 'collection users exists', + source: new ListCollectionsCommand(), + filter: MongoFieldFilter.eq('name', 'users'), + expect, + }); + + expect(renameCollection('profiles', 'users')).toEqual({ + id: 'collection.profiles.rename(users)', + label: 'Rename collection profiles to users', + operationClass: 'widening', + precheck: [ + sourceIs('exists'), + { ...targetIs('notExists'), description: 'collection users does not exist' }, + ], + execute: [ + { + description: 'rename collection profiles to users', + command: new RenameCollectionCommand('profiles', 'users'), + }, + ], + postcheck: [ + targetIs('exists'), + { ...sourceIs('notExists'), description: 'collection profiles does not exist' }, + ], + }); + }); +}); + describe('createCollection', () => { it('produces correct operation structure', () => { const op = createCollection('users'); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-ops-serializer.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-ops-serializer.test.ts index c127697b9a0a..305bd6ff9993 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-ops-serializer.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-ops-serializer.test.ts @@ -15,6 +15,7 @@ import { } from '@internal/mongo-query-ast/control'; import { isStructuredError } from '@internal/utils/structured-error'; import { describe, expect, it } from 'vitest'; +import { renameCollection } from '../src/core/migrations/migration-factories'; import { deserializeMongoOps, serializeMongoOps, @@ -634,6 +635,14 @@ describe('serializeMongoOps / deserializeMongoOps', () => { expect(cmd.collection).toBe('events'); }); + it('round-trips a renameCollection operation', () => { + const op = renameCollection('profiles', 'users'); + + const deserialized = deserializeMongoOps(JSON.parse(serializeMongoOps([op])) as unknown[]); + + expect(deserialized).toEqual([op]); + }); + it('round-trips collMod command', () => { const op: MongoMigrationPlanOperation = { id: 'validator.users.update', diff --git a/packages/3-mongo-target/1-mongo-target/test/op-factory-call.test.ts b/packages/3-mongo-target/1-mongo-target/test/op-factory-call.test.ts index db2f8cc600ca..3c375d7f9623 100644 --- a/packages/3-mongo-target/1-mongo-target/test/op-factory-call.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/op-factory-call.test.ts @@ -20,6 +20,7 @@ import { createIndex, dropCollection, dropIndex, + renameCollection, } from '../src/core/migrations/migration-factories'; import { CollModCall, @@ -27,6 +28,7 @@ import { CreateIndexCall, DropCollectionCall, DropIndexCall, + RenameCollectionCall, } from '../src/core/migrations/op-factory-call'; describe('Mongo call classes', () => { @@ -87,6 +89,19 @@ describe('Mongo call classes', () => { expect(call.toOp()).toEqual(dropCollection('users')); }); + it('RenameCollectionCall freezes, labels as widening, and lowers to renameCollection(...)', () => { + const call = new RenameCollectionCall('profiles', 'users'); + + expect(Object.isFrozen(call)).toBe(true); + expect(call).toMatchObject({ + factoryName: 'renameCollection', + operationClass: 'widening', + label: 'Rename collection profiles to users', + }); + + expect(call.toOp()).toEqual(renameCollection('profiles', 'users')); + }); + it('CollModCall defaults operationClass to widening, uses a default label, and lowers to collMod(...)', () => { const call = new CollModCall('users', { validator: { $jsonSchema: { type: 'object' } } }); @@ -183,6 +198,15 @@ describe('Mongo call classes', () => { ]); }); + it('RenameCollectionCall emits the source and target names and imports renameCollection only', () => { + const call = new RenameCollectionCall('profiles', 'users'); + + expect(call.renderTypeScript()).toBe('renameCollection("profiles", "users")'); + expect(call.importRequirements()).toEqual([ + { moduleSpecifier: '@internal/target-mongo/migration', symbol: 'renameCollection' }, + ]); + }); + it('CollModCall omits the trailing meta argument when no meta supplied', () => { const call = new CollModCall('users', { validator: { $jsonSchema: { type: 'object' } } }); diff --git a/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts b/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts index 6d5fee67aab5..1f458ad3e6ad 100644 --- a/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts +++ b/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts @@ -26,6 +26,7 @@ import { FindOneAndUpdateWireCommand, InsertManyWireCommand, InsertOneWireCommand, + RenameCollectionWireCommand, UpdateManyWireCommand, UpdateOneWireCommand, } from '@internal/mongo-wire'; @@ -66,6 +67,8 @@ function lowerDdlCommand(command: AnyMongoDdlCommand): AnyMongoDdlWireCommand { return new CreateIndexWireCommand(command.collection, keysToKeySpec(command.keys), command); case 'dropCollection': return new DropCollectionWireCommand(command.collection); + case 'renameCollection': + return new RenameCollectionWireCommand(command.collection, command.to); case 'dropIndex': return new DropIndexWireCommand(command.collection, command.name); case 'collMod': diff --git a/packages/3-mongo-target/2-mongo-adapter/test/command-executor.test.ts b/packages/3-mongo-target/2-mongo-adapter/test/command-executor.test.ts index e8df92781691..da158e65a5a1 100644 --- a/packages/3-mongo-target/2-mongo-adapter/test/command-executor.test.ts +++ b/packages/3-mongo-target/2-mongo-adapter/test/command-executor.test.ts @@ -2,7 +2,7 @@ * Behavioral tests for DDL execution via the adapter-owned seam * (`controlAdapter.executeDdl(driver, command)` → `driver.run(wire)`). * - * Each test exercises one of the five DDL command kinds against mongodb-memory-server + * Each test exercises one of the DDL command kinds against mongodb-memory-server * and asserts the resulting database state. */ import { MongoDriverImpl } from '@internal/driver-mongo'; @@ -14,6 +14,7 @@ import { DropIndexCommand, ListCollectionsCommand, ListIndexesCommand, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import { type Db, MongoClient } from 'mongodb'; import { MongoMemoryReplSet } from 'mongodb-memory-server'; @@ -55,6 +56,7 @@ async function execDdl( | CreateIndexCommand | DropCollectionCommand | DropIndexCommand + | RenameCollectionCommand | CollModCommand, ): Promise { const controlAdapter = new MongoControlAdapterImpl(); @@ -294,6 +296,18 @@ describe('dropCollection via executeDdl', () => { }); }); +describe('renameCollection via executeDdl', () => { + it('renames an existing collection', async () => { + await db.createCollection('old_name'); + await execDdl(new RenameCollectionCommand('old_name', 'new_name')); + + const names = ( + await db.listCollections({ name: { $in: ['old_name', 'new_name'] } }).toArray() + ).map((c) => c['name']); + expect(names).toEqual(['new_name']); + }); +}); + describe('collMod via executeDdl', () => { it('applies validator to an existing collection', async () => { await db.createCollection('docs'); diff --git a/packages/3-mongo-target/2-mongo-adapter/test/ddl-lowering-oracle.test.ts b/packages/3-mongo-target/2-mongo-adapter/test/ddl-lowering-oracle.test.ts index 7cb4e03b3b4c..93b17d54d6a5 100644 --- a/packages/3-mongo-target/2-mongo-adapter/test/ddl-lowering-oracle.test.ts +++ b/packages/3-mongo-target/2-mongo-adapter/test/ddl-lowering-oracle.test.ts @@ -4,6 +4,7 @@ import { CreateIndexCommand, DropCollectionCommand, DropIndexCommand, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import { describe, expect, it } from 'vitest'; import { createMongoAdapter } from '../src/mongo-adapter'; @@ -16,6 +17,7 @@ async function lowerCmd( | CreateIndexCommand | DropCollectionCommand | DropIndexCommand + | RenameCollectionCommand | CollModCommand, ) { return adapter.lower({ command }, {}); @@ -232,6 +234,17 @@ describe('DDL lowering oracle — dropCollection', () => { }); }); +describe('DDL lowering oracle — renameCollection', () => { + it('produces the source collection and the new name', async () => { + const wire = await lowerCmd(new RenameCollectionCommand('profiles', 'users')); + expect({ ...wire }).toEqual({ + kind: 'renameCollection', + collection: 'profiles', + to: 'users', + }); + }); +}); + describe('DDL lowering oracle — dropIndex', () => { it('produces {dropIndexes: collection, index: name}', async () => { const wire = await lowerCmd(new DropIndexCommand('users', 'email_1')); diff --git a/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts b/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts index f30cc9be13cc..1a897eb404a8 100644 --- a/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts +++ b/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts @@ -18,6 +18,7 @@ import type { InsertManyWireCommand, InsertOneResult, InsertOneWireCommand, + RenameCollectionWireCommand, UpdateManyResult, UpdateManyWireCommand, UpdateOneResult, @@ -98,6 +99,8 @@ export class MongoDriverImpl implements MongoDriver { return this.executeCreateIndexCommand(wireCommand); case 'dropCollection': return this.executeDropCollectionCommand(wireCommand); + case 'renameCollection': + return this.executeRenameCollectionCommand(wireCommand); case 'dropIndex': return this.executeDropIndexCommand(wireCommand); case 'collMod': @@ -281,6 +284,10 @@ export class MongoDriverImpl implements MongoDriver { await this.db.collection(cmd.collection).drop(); } + protected async executeRenameCollectionCommand(cmd: RenameCollectionWireCommand): Promise { + await this.db.renameCollection(cmd.collection, cmd.to); + } + protected async executeDropIndexCommand(cmd: DropIndexWireCommand): Promise { await this.db.collection(cmd.collection).dropIndex(cmd.name); } diff --git a/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts b/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts index a25d78a1c54e..ea93359fc761 100644 --- a/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts +++ b/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts @@ -11,6 +11,7 @@ import { FindOneAndUpdateWireCommand, InsertManyWireCommand, InsertOneWireCommand, + RenameCollectionWireCommand, UpdateManyWireCommand, UpdateOneWireCommand, } from '@internal/mongo-wire'; @@ -457,6 +458,30 @@ describe('MongoDriver', () => { }); }); + describe('renameCollection', () => { + it('renames the collection and keeps its documents', async () => { + const db = seedClient.db(dbName); + const driver = MongoDriverImpl.fromDb(db); + const from = 'driver_rename_from'; + const to = 'driver_rename_to'; + + await db.dropCollection(from).catch(() => undefined); + await db.dropCollection(to).catch(() => undefined); + await db.collection(from).insertOne({ name: 'Ada' }); + + await driver.run(new RenameCollectionWireCommand(from, to)); + + const names = (await db.listCollections({ name: { $in: [from, to] } }).toArray()).map( + (c) => c.name, + ); + const docs = await db + .collection(to) + .find({}, { projection: { _id: 0 } }) + .toArray(); + expect({ names, docs }).toEqual({ names: [to], docs: [{ name: 'Ada' }] }); + }); + }); + describe('dropIndex', () => { it('drops a named index', async () => { const db = seedClient.db(dbName); diff --git a/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts b/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts new file mode 100644 index 000000000000..c35ae98d5229 --- /dev/null +++ b/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts @@ -0,0 +1,212 @@ +import { MongoControlAdapterImpl } from '@internal/adapter-mongo/control'; +import { MongoControlDriver } from '@internal/driver-mongo/control'; +import type { MigrationPlan } from '@internal/framework-components/control'; +import { buildFabricatedMigrationEdge } from '@internal/migration-tools/aggregate'; +import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; +import { MongoMigrationRunner, serializeMongoOps } from '@internal/target-mongo/control'; +import { renameCollection } from '@internal/target-mongo/migration'; +import { timeouts } from '@repo/test-utils'; +import { type Db, MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest'; + +const controlAdapter = new MongoControlAdapterImpl(); + +let replSet: MongoMemoryReplSet; +let client: MongoClient; +let db: Db; +const dbName = 'runner_rename_collection_test'; + +beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + db = client.db(dbName); +}, timeouts.spinUpMongoMemoryServer); + +afterAll(async () => { + await client?.close(); + await replSet?.stop(); +}, timeouts.spinUpMongoMemoryServer); + +beforeEach(async () => { + const collections = await db.listCollections().toArray(); + for (const col of collections) { + await db.dropCollection(col['name'] as string); + } +}); + +const CLOSED_PROFILE_SCHEMA = { + bsonType: 'object', + required: ['_id', 'email'], + properties: { + _id: { bsonType: 'objectId' }, + email: { bsonType: 'string' }, + }, + additionalProperties: false, +} as const; + +function usersContract(): MongoContract { + const users = new MongoCollection({ + indexes: [{ keys: [{ field: 'email', direction: 1 }], unique: true }], + validator: { + jsonSchema: { ...CLOSED_PROFILE_SCHEMA }, + validationLevel: 'strict', + validationAction: 'error', + }, + }); + return { + target: 'mongo', + targetFamily: 'mongo', + profileHash: 'p1', + capabilities: {}, + extensions: {}, + meta: {}, + roots: {}, + models: {}, + storage: { + storageHash: 'dest', + namespaces: { + __unbound__: { + id: '__unbound__', + kind: 'mongo-namespace', + entries: { collection: { users } }, + }, + }, + }, + } as unknown as MongoContract; +} + +const renamePlan: MigrationPlan = { + targetId: 'mongo', + operations: JSON.parse(serializeMongoOps([renameCollection('profiles', 'users')])), + origin: { storageHash: 'origin' }, + destination: { storageHash: 'dest' }, +}; + +async function createProfiles(): Promise { + await db.createCollection('profiles', { + validator: { $jsonSchema: CLOSED_PROFILE_SCHEMA }, + validationLevel: 'strict', + validationAction: 'error', + }); + await db.collection('profiles').createIndex({ email: 1 }, { unique: true, name: 'email_1' }); + await db + .collection('profiles') + .insertMany([{ email: 'ada@example.com' }, { email: 'bo@example.com' }]); + await controlAdapter.initMarker(new MongoControlDriver(db, client), 'app', { + storageHash: 'origin', + profileHash: 'p1', + }); +} + +async function runRename() { + const runner = new MongoMigrationRunner( + controlAdapter.createRunnerDependencies(new MongoControlDriver(db, client)), + ); + return runner.execute({ + plan: renamePlan, + migrationEdges: [ + buildFabricatedMigrationEdge({ + currentMarkerStorageHash: 'origin', + destinationStorageHash: 'dest', + operationCount: 1, + }), + ], + destinationContract: usersContract(), + policy: { allowedOperationClasses: ['additive', 'widening', 'destructive'] }, + frameworkComponents: [], + }); +} + +async function collectionNames(): Promise { + const collections = await db.listCollections({ name: { $in: ['profiles', 'users'] } }).toArray(); + return collections.map((c) => c.name).sort(); +} + +async function emails(collectionName: string): Promise { + const docs = await db.collection(collectionName).find().toArray(); + return docs.map((d) => String(d['email'])).sort(); +} + +describe('MongoMigrationRunner - renameCollection', () => { + it( + 'renames a collection and keeps its documents, unique index and closed validator', + async () => { + await createProfiles(); + + const result = await runRename(); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + const [users] = await db.listCollections({ name: 'users' }).toArray(); + const indexes = await db.collection('users').indexes(); + expect({ + names: await collectionNames(), + emails: await emails('users'), + validator: users?.['options']?.validator, + emailIndex: indexes.find((i) => i.name === 'email_1'), + }).toEqual({ + names: ['users'], + emails: ['ada@example.com', 'bo@example.com'], + validator: { $jsonSchema: CLOSED_PROFILE_SCHEMA }, + emailIndex: expect.objectContaining({ key: { email: 1 }, unique: true }), + }); + await expect(db.collection('users').insertOne({ email: 'ada@example.com' })).rejects.toThrow( + /duplicate key/, + ); + await expect( + db.collection('users').insertOne({ email: 'cy@example.com', undeclared: true }), + ).rejects.toThrow(/Document failed validation/); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'skips the rename through its after-check when run again', + async () => { + await createProfiles(); + const first = await runRename(); + expect(first.ok).toBe(true); + const reset = await controlAdapter.updateMarker( + new MongoControlDriver(db, client), + 'app', + 'dest', + { storageHash: 'origin', profileHash: 'p1' }, + ); + expect(reset).toBe(true); + + const second = await runRename(); + + expect(second.ok ? second.value.operationsExecuted : second.failure).toBe(0); + expect({ names: await collectionNames(), emails: await emails('users') }).toEqual({ + names: ['users'], + emails: ['ada@example.com', 'bo@example.com'], + }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'fails at the before-check and changes nothing when the target collection exists', + async () => { + await createProfiles(); + await db.collection('users').insertOne({ email: 'already@example.com' }); + + const result = await runRename(); + + expect(result.ok ? result.value : result.failure.code).toBe('MIGRATION.PRECHECK_FAILED'); + expect({ + names: await collectionNames(), + profiles: await emails('profiles'), + users: await emails('users'), + }).toEqual({ + names: ['profiles', 'users'], + profiles: ['ada@example.com', 'bo@example.com'], + users: ['already@example.com'], + }); + }, + timeouts.spinUpMongoMemoryServer, + ); +}); From 9b6df6e004508f4484f3edbc2c7c5d546babfb98 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:04:49 +0200 Subject: [PATCH 08/57] TML-3478: the rename test reads the validator with full collection info Signed-off-by: willbot Signed-off-by: Will Madden --- .../test/mongo/target-runner/runner-rename-collection.test.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts b/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts index c35ae98d5229..c3fe19453ace 100644 --- a/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts +++ b/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts @@ -140,12 +140,12 @@ describe('MongoMigrationRunner - renameCollection', () => { const result = await runRename(); expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); - const [users] = await db.listCollections({ name: 'users' }).toArray(); + const [users] = await db.listCollections({ name: 'users' }, { nameOnly: false }).toArray(); const indexes = await db.collection('users').indexes(); expect({ names: await collectionNames(), emails: await emails('users'), - validator: users?.['options']?.validator, + validator: users && 'options' in users ? users.options?.['validator'] : undefined, emailIndex: indexes.find((i) => i.name === 'email_1'), }).toEqual({ names: ['users'], From fc8d24ed10f1ee6ba71aa540cb60b5b3040f84ea Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:18:24 +0200 Subject: [PATCH 09/57] TML-3478: renameField and unsetField rewrite documents past the validator Both factories run one updateMany with bypassDocumentValidation, through the raw command, wire command, lowering and driver. renameField is widening and unsetField destructive, so db update runs them while a data transform stays refused. The runner and the operation serializer tell an operation that runs queries from one that executes DDL by its shape, not by the data class. A rewrite's storage name is ., and every MongoDB preview line names its collection with db.getCollection(""). Signed-off-by: willbot Signed-off-by: Will Madden --- .../4-query/query-ast/src/exports/control.ts | 1 + .../src/migration-operation-types.ts | 8 +- .../4-query/query-ast/src/raw-commands.ts | 14 +- .../query-ast/test/migration-helpers.test.ts | 17 + .../query-ast/test/raw-commands.test.ts | 11 + .../mongo-lowering/src/adapter-types.ts | 1 + .../mongo-wire/src/wire-commands.ts | 3 + .../7-runtime/src/param-ref-mutator.ts | 1 + .../7-runtime/test/param-ref-mutator.test.ts | 2 + .../9-family/src/core/operation-preview.ts | 54 +++- .../src/core/operation-storage-name.ts | 31 +- .../test/operation-preview-format.test.ts | 66 +++- .../9-family/test/operation-preview.test.ts | 4 +- .../test/operation-storage-name.test.ts | 19 ++ .../core/migrations/migration-factories.ts | 101 +++++- .../core/migrations/mongo-ops-serializer.ts | 14 +- .../src/core/migrations/mongo-runner.ts | 22 +- .../src/core/migrations/op-factory-call.ts | 66 +++- .../src/core/migrations/render-ops.ts | 4 +- .../1-mongo-target/src/exports/control.ts | 2 + .../1-mongo-target/src/exports/migration.ts | 2 + .../test/field-rewrites.test.ts | 192 +++++++++++ .../1-mongo-target/test/migration-e2e.test.ts | 29 ++ .../2-mongo-adapter/src/mongo-adapter.ts | 9 +- .../test/mongo-adapter.test.ts | 11 + .../3-mongo-driver/src/mongo-driver.ts | 5 +- .../3-mongo-driver/test/mongo-driver.test.ts | 32 ++ .../runner-field-rewrites.test.ts | 302 ++++++++++++++++++ 28 files changed, 970 insertions(+), 53 deletions(-) create mode 100644 packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts create mode 100644 test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts diff --git a/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts b/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts index 998135c7b6ea..19f63aff1080 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts @@ -34,4 +34,5 @@ export type { MongoMigrationPlanOperation, MongoMigrationStep, } from '../migration-operation-types'; +export { isMongoDataTransformOperation } from '../migration-operation-types'; export type { MongoFilterVisitor } from '../visitors'; diff --git a/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts b/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts index 485f6116878c..ade8ba24108f 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts @@ -33,7 +33,7 @@ export interface MongoDataTransformCheck { } export interface MongoDataTransformOperation extends MigrationPlanOperation { - readonly operationClass: 'data'; + readonly operationClass: MigrationOperationClass; /** * Human-readable label for this data transform. */ @@ -51,4 +51,10 @@ export interface MongoDataTransformOperation extends MigrationPlanOperation { export type AnyMongoMigrationOperation = MongoMigrationPlanOperation | MongoDataTransformOperation; +export function isMongoDataTransformOperation( + operation: AnyMongoMigrationOperation, +): operation is MongoDataTransformOperation { + return 'run' in operation; +} + export type { MigrationOperationClass, MigrationPlanOperation }; diff --git a/packages/2-mongo-family/4-query/query-ast/src/raw-commands.ts b/packages/2-mongo-family/4-query/query-ast/src/raw-commands.ts index 4d403103f0d7..050ac2139c5f 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/raw-commands.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/raw-commands.ts @@ -55,17 +55,29 @@ export class RawUpdateOneCommand extends MongoAstNode { } } +export interface RawUpdateManyOptions { + readonly bypassDocumentValidation?: boolean; +} + export class RawUpdateManyCommand extends MongoAstNode { readonly kind = 'rawUpdateMany' as const; readonly collection: string; readonly filter: Document; readonly update: Document | ReadonlyArray; + declare readonly bypassDocumentValidation?: boolean; - constructor(collection: string, filter: Document, update: Document | ReadonlyArray) { + constructor( + collection: string, + filter: Document, + update: Document | ReadonlyArray, + options?: RawUpdateManyOptions, + ) { super(); this.collection = collection; this.filter = filter; this.update = update; + if (options?.bypassDocumentValidation !== undefined) + this.bypassDocumentValidation = options.bypassDocumentValidation; this.freeze(); } } diff --git a/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts b/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts index 2562d5283488..af9db162a0aa 100644 --- a/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts +++ b/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from 'vitest'; import { buildIndexOpId, defaultMongoIndexName, keysToKeySpec } from '../src/migration-helpers'; +import { isMongoDataTransformOperation } from '../src/migration-operation-types'; describe('buildIndexOpId', () => { it('produces deterministic create ID for single key', () => { @@ -67,3 +68,19 @@ describe('keysToKeySpec', () => { expect(keysToKeySpec([{ field: 'content', direction: 'text' }])).toEqual({ content: 'text' }); }); }); + +describe('isMongoDataTransformOperation', () => { + it('tells an operation that runs queries from one that executes DDL, whatever its class', () => { + const common = { id: 'op', label: 'op', precheck: [], postcheck: [] }; + expect([ + isMongoDataTransformOperation({ ...common, operationClass: 'widening', execute: [] }), + isMongoDataTransformOperation({ + ...common, + operationClass: 'widening', + name: 'rewrite', + run: [], + }), + isMongoDataTransformOperation({ ...common, operationClass: 'data', name: 'data', run: [] }), + ]).toEqual([false, true, true]); + }); +}); diff --git a/packages/2-mongo-family/4-query/query-ast/test/raw-commands.test.ts b/packages/2-mongo-family/4-query/query-ast/test/raw-commands.test.ts index 7859f62c3bf9..e912739b368f 100644 --- a/packages/2-mongo-family/4-query/query-ast/test/raw-commands.test.ts +++ b/packages/2-mongo-family/4-query/query-ast/test/raw-commands.test.ts @@ -74,6 +74,17 @@ describe('RawUpdateOneCommand', () => { expect(cmd.update).toEqual(update); }); + it('carries bypassDocumentValidation only when given', () => { + const plain = new RawUpdateManyCommand('users', filter, update); + const bypassing = new RawUpdateManyCommand('users', filter, update, { + bypassDocumentValidation: true, + }); + expect({ + plainHasKey: 'bypassDocumentValidation' in plain, + bypassing: bypassing.bypassDocumentValidation, + }).toEqual({ plainHasKey: false, bypassing: true }); + }); + it('accepts pipeline-style update (array)', () => { const pipelineUpdate = [{ $set: { fullName: { $concat: ['$first', ' ', '$last'] } } }]; const cmd = new RawUpdateOneCommand('users', filter, pipelineUpdate); diff --git a/packages/2-mongo-family/6-transport/mongo-lowering/src/adapter-types.ts b/packages/2-mongo-family/6-transport/mongo-lowering/src/adapter-types.ts index afc676a7e306..8a9bb7a6a917 100644 --- a/packages/2-mongo-family/6-transport/mongo-lowering/src/adapter-types.ts +++ b/packages/2-mongo-family/6-transport/mongo-lowering/src/adapter-types.ts @@ -94,6 +94,7 @@ export type MongoLoweredDraft = readonly collection: string; readonly filter: Record; readonly update: Record | ReadonlyArray>; + readonly bypassDocumentValidation: boolean; } | { readonly kind: 'rawDeleteOne'; diff --git a/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts b/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts index 470f512d28ce..bc70571d2a2c 100644 --- a/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts +++ b/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts @@ -82,17 +82,20 @@ export class UpdateManyWireCommand extends MongoWireCommand { readonly filter: Document; readonly update: Document | ReadonlyArray; readonly upsert: boolean; + readonly bypassDocumentValidation: boolean; constructor( collection: string, filter: Document, update: Document | ReadonlyArray, upsert = false, + bypassDocumentValidation = false, ) { super(collection); this.filter = filter; this.update = update; this.upsert = upsert; + this.bypassDocumentValidation = bypassDocumentValidation; this.freeze(); } } diff --git a/packages/2-mongo-family/7-runtime/src/param-ref-mutator.ts b/packages/2-mongo-family/7-runtime/src/param-ref-mutator.ts index 9c84ef500826..468cc648ac69 100644 --- a/packages/2-mongo-family/7-runtime/src/param-ref-mutator.ts +++ b/packages/2-mongo-family/7-runtime/src/param-ref-mutator.ts @@ -285,6 +285,7 @@ function buildMutatedDraft( collection: draft.collection, filter: substituteDoc(draft.filter, overrides), update: substituteUpdate(draft.update, overrides), + bypassDocumentValidation: draft.bypassDocumentValidation, }; case 'deleteOne': return { diff --git a/packages/2-mongo-family/7-runtime/test/param-ref-mutator.test.ts b/packages/2-mongo-family/7-runtime/test/param-ref-mutator.test.ts index acd9b3ef381e..2d6686001db0 100644 --- a/packages/2-mongo-family/7-runtime/test/param-ref-mutator.test.ts +++ b/packages/2-mongo-family/7-runtime/test/param-ref-mutator.test.ts @@ -158,6 +158,7 @@ describe('flattenMongoParamRefs', () => { collection: 'users', filter: { id: new MongoParamRef(1) }, update: { $set: { v: new MongoParamRef(2) } }, + bypassDocumentValidation: false, } satisfies MongoLoweredDraft, expectedLength: 2, }, @@ -647,6 +648,7 @@ describe('createMongoParamRefMutator', () => { collection: 'users', filter: { id: new MongoParamRef(1) }, update: { $set: { v: new MongoParamRef(0) } }, + bypassDocumentValidation: false, } satisfies MongoLoweredDraft, read: (updated: MongoLoweredDraft) => { if (updated.kind !== 'rawUpdateMany') throw new Error('wrong kind'); diff --git a/packages/2-mongo-family/9-family/src/core/operation-preview.ts b/packages/2-mongo-family/9-family/src/core/operation-preview.ts index 1959fda7778e..6547f0d39305 100644 --- a/packages/2-mongo-family/9-family/src/core/operation-preview.ts +++ b/packages/2-mongo-family/9-family/src/core/operation-preview.ts @@ -3,6 +3,7 @@ import type { OperationPreview, } from '@internal/framework-components/control'; import type { + AnyMongoDdlCommand, CollModCommand, CreateCollectionCommand, CreateIndexCommand, @@ -12,6 +13,7 @@ import type { MongoIndexKey, RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; +import type { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; function formatKeySpec(keys: ReadonlyArray): string { const entries = keys.map((k) => `${JSON.stringify(k.field)}: ${JSON.stringify(k.direction)}`); @@ -36,17 +38,21 @@ function formatCreateCollectionOptions(cmd: CreateCollectionCommand): string | u return formatOptionEntries(cmd); } +function collectionRef(name: string): string { + return `db.getCollection(${JSON.stringify(name)})`; +} + class MongoDdlCommandFormatter implements MongoDdlCommandVisitor { createIndex(cmd: CreateIndexCommand): string { const keySpec = formatKeySpec(cmd.keys); const opts = formatOptions(cmd); return opts - ? `db.${cmd.collection}.createIndex(${keySpec}, ${opts})` - : `db.${cmd.collection}.createIndex(${keySpec})`; + ? `${collectionRef(cmd.collection)}.createIndex(${keySpec}, ${opts})` + : `${collectionRef(cmd.collection)}.createIndex(${keySpec})`; } dropIndex(cmd: DropIndexCommand): string { - return `db.${cmd.collection}.dropIndex(${JSON.stringify(cmd.name)})`; + return `${collectionRef(cmd.collection)}.dropIndex(${JSON.stringify(cmd.name)})`; } createCollection(cmd: CreateCollectionCommand): string { @@ -57,11 +63,11 @@ class MongoDdlCommandFormatter implements MongoDdlCommandVisitor { } dropCollection(cmd: DropCollectionCommand): string { - return `db.${cmd.collection}.drop()`; + return `${collectionRef(cmd.collection)}.drop()`; } renameCollection(cmd: RenameCollectionCommand): string { - return `db.${cmd.collection}.renameCollection(${JSON.stringify(cmd.to)})`; + return `${collectionRef(cmd.collection)}.renameCollection(${JSON.stringify(cmd.to)})`; } collMod(cmd: CollModCommand): string { @@ -76,21 +82,41 @@ class MongoDdlCommandFormatter implements MongoDdlCommandVisitor { const formatter = new MongoDdlCommandFormatter(); -interface MongoExecuteStep { - readonly command: { readonly accept: (visitor: MongoDdlCommandVisitor) => R }; +function formatUpdateMany(cmd: RawUpdateManyCommand): string { + const args = [JSON.stringify(cmd.filter), JSON.stringify(cmd.update)]; + if (cmd.bypassDocumentValidation) args.push('{ bypassDocumentValidation: true }'); + return `${collectionRef(cmd.collection)}.updateMany(${args.join(', ')})`; +} + +function arrayAt(value: object, key: string): readonly unknown[] { + const entry: unknown = Reflect.get(value, key); + return Array.isArray(entry) ? entry : []; +} + +function objectAt(value: unknown, key: string): object | undefined { + if (typeof value !== 'object' || value === null) return undefined; + const entry: unknown = Reflect.get(value, key); + return typeof entry === 'object' && entry !== null ? entry : undefined; +} + +function isDdlCommand(value: object | undefined): value is AnyMongoDdlCommand { + return value !== undefined && typeof Reflect.get(value, 'accept') === 'function'; +} + +function isRawUpdateMany(value: object | undefined): value is RawUpdateManyCommand { + return value !== undefined && Reflect.get(value, 'kind') === 'rawUpdateMany'; } export function formatMongoOperations(operations: readonly MigrationPlanOperation[]): string[] { const statements: string[] = []; for (const operation of operations) { - const candidate = operation as unknown as Record; - if (!('execute' in candidate) || !Array.isArray(candidate['execute'])) { - continue; + for (const step of arrayAt(operation, 'execute')) { + const command = objectAt(step, 'command'); + if (isDdlCommand(command)) statements.push(command.accept(formatter)); } - for (const step of candidate['execute'] as MongoExecuteStep[]) { - if (step.command && typeof step.command.accept === 'function') { - statements.push(step.command.accept(formatter)); - } + for (const plan of arrayAt(operation, 'run')) { + const command = objectAt(plan, 'command'); + if (isRawUpdateMany(command)) statements.push(formatUpdateMany(command)); } } return statements; diff --git a/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts b/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts index 2c2157a04e07..0fa961f55737 100644 --- a/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts +++ b/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts @@ -8,10 +8,32 @@ function collectionOfStep(step: unknown): string | undefined { return typeof collection === 'string' ? collection : undefined; } +const FIELD_REWRITE_OPERATORS = ['$unset', '$rename'] as const; + +function rewrittenFieldOf(command: object): string | undefined { + const update: unknown = Reflect.get(command, 'update'); + if (typeof update !== 'object' || update === null) return undefined; + for (const operator of FIELD_REWRITE_OPERATORS) { + const fields: unknown = Reflect.get(update, operator); + if (typeof fields === 'object' && fields !== null) return Object.keys(fields)[0]; + } + return undefined; +} + +function storageNameOfRunStep(step: unknown): string | undefined { + if (typeof step !== 'object' || step === null) return undefined; + const command: unknown = Reflect.get(step, 'command'); + if (typeof command !== 'object' || command === null) return undefined; + const collection: unknown = Reflect.get(command, 'collection'); + if (typeof collection !== 'string') return undefined; + const field = rewrittenFieldOf(command); + return field === undefined ? collection : `${collection}.${field}`; +} + /** * The collection an operation's command names, as the database knows it: the name the Mongo - * planner gives a dropped collection. An operation whose commands name no collection is named by - * its id. + * planner gives a dropped collection. A field rewrite is named `.`. An operation + * whose commands name no collection is named by its id. */ export function mongoStorageNameOf(operation: MigrationPlanOperation): string { const execute: unknown = Reflect.get(operation, 'execute'); @@ -20,5 +42,10 @@ export function mongoStorageNameOf(operation: MigrationPlanOperation): string { const collection = collectionOfStep(step); if (collection !== undefined) return collection; } + const run: unknown = Reflect.get(operation, 'run'); + for (const step of Array.isArray(run) ? run : []) { + const name = storageNameOfRunStep(step); + if (name !== undefined) return name; + } return operation.id; } diff --git a/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts b/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts index 88b66369601c..d1dc280f6142 100644 --- a/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts +++ b/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts @@ -7,6 +7,7 @@ import { type MongoMigrationPlanOperation, RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; +import { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; import { describe, expect, it } from 'vitest'; import { formatMongoOperations } from '../src/core/operation-preview'; @@ -30,7 +31,7 @@ describe('formatMongoOperations', () => { }; const result = formatMongoOperations([op]); expect(result).toEqual([ - 'db.users.createIndex({ "email": 1 }, { unique: true, name: "email_1" })', + 'db.getCollection("users").createIndex({ "email": 1 }, { unique: true, name: "email_1" })', ]); }); @@ -49,7 +50,7 @@ describe('formatMongoOperations', () => { postcheck: [], }; const result = formatMongoOperations([op]); - expect(result).toEqual(['db.posts.createIndex({ "title": 1 })']); + expect(result).toEqual(['db.getCollection("posts").createIndex({ "title": 1 })']); }); it('formats dropIndex', () => { @@ -67,7 +68,7 @@ describe('formatMongoOperations', () => { postcheck: [], }; const result = formatMongoOperations([op]); - expect(result).toEqual(['db.users.dropIndex("email_1")']); + expect(result).toEqual(['db.getCollection("users").dropIndex("email_1")']); }); it('formats compound index', () => { @@ -93,7 +94,7 @@ describe('formatMongoOperations', () => { }; const result = formatMongoOperations([op]); expect(result).toEqual([ - 'db.users.createIndex({ "email": 1, "tenantId": -1 }, { unique: true })', + 'db.getCollection("users").createIndex({ "email": 1, "tenantId": -1 }, { unique: true })', ]); }); @@ -132,6 +133,55 @@ describe('formatMongoOperations', () => { expect(result[1]).toContain('dropIndex'); }); + it('writes every collection name as an escaped string', () => { + const step = (command: MongoMigrationPlanOperation['execute'][number]['command']) => ({ + id: 'test', + label: 'test', + operationClass: 'widening' as const, + precheck: [], + execute: [{ description: 'step', command }], + postcheck: [], + }); + expect( + formatMongoOperations([ + step(new DropCollectionCommand('my-coll')), + step(new DropIndexCommand('a"b', 'x_1')), + step(new CreateIndexCommand('my-coll', [{ field: 'x', direction: 1 }])), + step(new RenameCollectionCommand('my-coll', 'a"b')), + ]), + ).toEqual([ + 'db.getCollection("my-coll").drop()', + 'db.getCollection("a\\"b").dropIndex("x_1")', + 'db.getCollection("my-coll").createIndex({ "x": 1 })', + 'db.getCollection("my-coll").renameCollection("a\\"b")', + ]); + }); + + it('formats a document rewrite as updateMany', () => { + const plan = { + collection: 'my-coll', + command: new RawUpdateManyCommand( + 'my-coll', + { name: { $exists: true } }, + { $rename: { name: 'fullName' } }, + { bypassDocumentValidation: true }, + ), + meta: { target: 'mongo', storageHash: '', lane: 'mongo-raw' }, + }; + const rewrite = { + id: 'field.my-coll.name.rename(fullName)', + label: 'Rename field', + operationClass: 'widening' as const, + name: 'rename', + precheck: [], + run: [plan], + postcheck: [], + }; + expect(formatMongoOperations([rewrite])).toEqual([ + 'db.getCollection("my-coll").updateMany({"name":{"$exists":true}}, {"$rename":{"name":"fullName"}}, { bypassDocumentValidation: true })', + ]); + }); + it('skips operations without execute steps', () => { const ops = [{ id: 'test', label: 'test', operationClass: 'additive' as const }]; const result = formatMongoOperations(ops); @@ -157,7 +207,7 @@ describe('formatMongoOperations', () => { }; const result = formatMongoOperations([op]); expect(result).toEqual([ - 'db.sessions.createIndex({ "expiresAt": 1 }, { sparse: true, expireAfterSeconds: 3600 })', + 'db.getCollection("sessions").createIndex({ "expiresAt": 1 }, { sparse: true, expireAfterSeconds: 3600 })', ]); }); @@ -244,7 +294,7 @@ describe('formatMongoOperations', () => { postcheck: [], }; const result = formatMongoOperations([op]); - expect(result).toEqual(['db.events.drop()']); + expect(result).toEqual(['db.getCollection("events").drop()']); }); it('formats renameCollection', () => { @@ -261,7 +311,9 @@ describe('formatMongoOperations', () => { ], postcheck: [], }; - expect(formatMongoOperations([op])).toEqual(['db.profiles.renameCollection("users")']); + expect(formatMongoOperations([op])).toEqual([ + 'db.getCollection("profiles").renameCollection("users")', + ]); }); it('formats collMod with validator', () => { diff --git a/packages/2-mongo-family/9-family/test/operation-preview.test.ts b/packages/2-mongo-family/9-family/test/operation-preview.test.ts index c94afb8b645e..0abeccf537b5 100644 --- a/packages/2-mongo-family/9-family/test/operation-preview.test.ts +++ b/packages/2-mongo-family/9-family/test/operation-preview.test.ts @@ -39,7 +39,7 @@ describe('MongoControlFamilyInstance OperationPreviewCapable', () => { const preview = instantiate().toOperationPreview([op]); expect(preview.statements).toEqual([ { - text: 'db.users.createIndex({ "email": 1 }, { unique: true, name: "email_1" })', + text: 'db.getCollection("users").createIndex({ "email": 1 }, { unique: true, name: "email_1" })', language: 'mongodb-shell', }, ]); @@ -61,7 +61,7 @@ describe('MongoControlFamilyInstance OperationPreviewCapable', () => { }; const preview = instantiate().toOperationPreview([op]); expect(preview.statements).toEqual([ - { text: 'db.users.dropIndex("email_1")', language: 'mongodb-shell' }, + { text: 'db.getCollection("users").dropIndex("email_1")', language: 'mongodb-shell' }, ]); }); diff --git a/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts b/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts index d92698fbea8b..d488f8310b63 100644 --- a/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts +++ b/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts @@ -1,5 +1,6 @@ import type { MigrationPlanOperation } from '@internal/framework-components/control'; import { DropCollectionCommand } from '@internal/mongo-query-ast/control'; +import { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; import { describe, expect, it } from 'vitest'; import { mongoStorageNameOf } from '../src/core/operation-storage-name'; @@ -22,6 +23,24 @@ describe('mongoStorageNameOf', () => { ]).toEqual(['events', 'logs']); }); + it('names a field rewrite by its collection and the field it removes or renames', () => { + const rewrite = (update: Record): MigrationPlanOperation => { + const command = new RawUpdateManyCommand('users', {}, update); + const run = [{ collection: 'users', command, meta: {} }]; + return { + id: 'rewrite', + label: 'rewrite', + operationClass: 'destructive', + run, + } as MigrationPlanOperation; + }; + expect([ + mongoStorageNameOf(rewrite({ $unset: { nickname: '' } })), + mongoStorageNameOf(rewrite({ $rename: { name: 'fullName' } })), + mongoStorageNameOf(rewrite({ $set: { status: 'active' } })), + ]).toEqual(['users.nickname', 'users.name', 'users']); + }); + it('names an operation whose commands name no collection by its id, never its label', () => { expect(mongoStorageNameOf(operation([]))).toBe('collection.events.drop'); }); diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts index 159014771199..840be8760717 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts @@ -22,10 +22,14 @@ import { type MongoMigrationPlanOperation, RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; -import type { MongoQueryPlan } from '@internal/mongo-query-ast/execution'; +import { + type MongoQueryPlan, + RawAggregateCommand, + RawUpdateManyCommand, +} from '@internal/mongo-query-ast/execution'; import { createFieldAccessor } from '@internal/mongo-query-builder'; import { collection } from '@internal/mongo-query-builder/contract-free'; -import type { MongoValue } from '@internal/mongo-value'; +import type { Document, MongoValue } from '@internal/mongo-value'; import { blindCast } from '@internal/utils/casts'; import { ifDefined } from '@internal/utils/defined'; import type { CollModMeta } from './op-factory-call'; @@ -108,6 +112,99 @@ export function dataTransform( }; } +export interface FieldRewriteOptions { + readonly filter?: Document; +} + +const REWRITE_PLAN_META = { target: 'mongo', storageHash: '', lane: 'mongo-raw' } as const; + +function documentsHolding(field: string, filter: Document | undefined): Document { + const holds = { [field]: { $exists: true } }; + return filter === undefined ? holds : { $and: [filter, holds] }; +} + +function rewritePlan(collectionName: string, command: RawAggregateCommand | RawUpdateManyCommand) { + return { collection: collectionName, command, meta: { ...REWRITE_PLAN_META } }; +} + +function noDocumentHolds( + collectionName: string, + field: string, + filter: Document | undefined, +): MongoDataTransformCheck { + return { + description: `no document in ${collectionName} holds ${field}`, + source: rewritePlan( + collectionName, + new RawAggregateCommand(collectionName, [ + { $match: documentsHolding(field, filter) }, + { $limit: 1 }, + ]), + ), + filter: MATCH_ALL_FILTER, + expect: 'notExists', + }; +} + +function rewriteUpdate( + collectionName: string, + field: string, + filter: Document | undefined, + update: Document, +): MongoQueryPlan { + return rewritePlan( + collectionName, + new RawUpdateManyCommand(collectionName, documentsHolding(field, filter), update, { + bypassDocumentValidation: true, + }), + ); +} + +function idSuffix(filter: Document | undefined): string { + return filter === undefined ? '' : `.where(${JSON.stringify(filter)})`; +} + +function labelSuffix(filter: Document | undefined): string { + return filter === undefined ? '' : ` where ${JSON.stringify(filter)}`; +} + +export function renameField( + collectionName: string, + from: string, + to: string, + options?: FieldRewriteOptions, +): MongoDataTransformOperation { + const filter = options?.filter; + const id = `field.${collectionName}.${from}.rename(${to})${idSuffix(filter)}`; + return { + id, + label: `Rename field ${from} to ${to} in collection ${collectionName}${labelSuffix(filter)}`, + operationClass: 'widening', + name: id, + precheck: [noDocumentHolds(collectionName, to, filter)], + run: [rewriteUpdate(collectionName, from, filter, { $rename: { [from]: to } })], + postcheck: [noDocumentHolds(collectionName, from, filter)], + }; +} + +export function unsetField( + collectionName: string, + field: string, + options?: FieldRewriteOptions, +): MongoDataTransformOperation { + const filter = options?.filter; + const id = `field.${collectionName}.${field}.unset${idSuffix(filter)}`; + return { + id, + label: `Remove field ${field} from collection ${collectionName}${labelSuffix(filter)}`, + operationClass: 'destructive', + name: id, + precheck: [], + run: [rewriteUpdate(collectionName, field, filter, { $unset: { [field]: '' } })], + postcheck: [noDocumentHolds(collectionName, field, filter)], + }; +} + function formatKeys(keys: ReadonlyArray): string { return keys.map((k) => `${k.field}:${k.direction}`).join(', '); } diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts index 6fcee0370f04..b792cf2d7e00 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts @@ -196,6 +196,7 @@ const RawUpdateManyJson = type({ collection: 'string', filter: 'Record', update: 'Record | Record[]', + 'bypassDocumentValidation?': 'boolean', }); const RawDeleteOneJson = type({ @@ -321,7 +322,7 @@ const DataTransformCheckJson = type({ const DataTransformOperationJson = type({ id: 'string', label: 'string', - operationClass: '"data"', + operationClass: '"additive" | "widening" | "destructive" | "data"', name: 'string', precheck: 'Record[]', run: 'Record[]', @@ -585,7 +586,12 @@ export function deserializeDmlCommand(json: unknown): AnyMongoCommand { } case 'rawUpdateMany': { const data = validate(RawUpdateManyJson, json, 'rawUpdateMany command'); - return new RawUpdateManyCommand(data.collection, data.filter, data.update); + return new RawUpdateManyCommand( + data.collection, + data.filter, + data.update, + ifDefined('bypassDocumentValidation', data.bypassDocumentValidation), + ); } case 'rawDeleteOne': { const data = validate(RawDeleteOneJson, json, 'rawDeleteOne command'); @@ -746,7 +752,7 @@ function deserializeStep(json: unknown): MongoMigrationStep { } function isDataTransformJson(json: unknown): boolean { - return recordValue(json, 'operationClass', 'migration operation') === 'data'; + return Array.isArray(recordValue(json, 'run', 'migration operation')); } function deserializeDdlOp(json: unknown): MongoMigrationPlanOperation { @@ -776,7 +782,7 @@ function deserializeDataTransformOp(json: unknown): MongoDataTransformOperation return { id: data.id, label: data.label, - operationClass: 'data', + operationClass: data.operationClass, name: data.name, precheck: data.precheck.map(deserializeDataTransformCheck), run: data.run.map(deserializeMongoQueryPlan), diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts index 4dc8d0632be5..c27b3e936b3b 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts @@ -26,8 +26,8 @@ import type { MongoDataTransformOperation, MongoInspectionCommandVisitor, MongoMigrationCheck, - MongoMigrationPlanOperation, } from '@internal/mongo-query-ast/control'; +import { isMongoDataTransformOperation } from '@internal/mongo-query-ast/control'; import { blindCast } from '@internal/utils/casts'; import { InternalError } from '@internal/utils/internal-error'; import { notOk, ok, type Result } from '@internal/utils/result'; @@ -116,12 +116,9 @@ export class MongoMigrationRunner { for (const operation of operations) { options.callbacks?.onOperationStart?.(operation); try { - if (operation.operationClass === 'data') { + if (isMongoDataTransformOperation(operation)) { const result = await this.executeDataTransform( - blindCast< - MongoDataTransformOperation, - 'operationClass data discriminates Mongo data transform operations' - >(operation), + operation, adapter, driver, filterEvaluator, @@ -136,14 +133,9 @@ export class MongoMigrationRunner { continue; } - const ddlOp = blindCast< - MongoMigrationPlanOperation, - 'non-data Mongo operations are DDL migration plan operations' - >(operation); - if (runPostchecks && runIdempotency) { const allSatisfied = await this.allChecksSatisfied( - ddlOp.postcheck, + operation.postcheck, inspectionExecutor, filterEvaluator, ); @@ -152,7 +144,7 @@ export class MongoMigrationRunner { if (runPrechecks) { const precheckResult = await this.evaluateChecks( - ddlOp.precheck, + operation.precheck, inspectionExecutor, filterEvaluator, ); @@ -165,13 +157,13 @@ export class MongoMigrationRunner { } } - for (const step of ddlOp.execute) { + for (const step of operation.execute) { await executeDdl(step.command); } if (runPostchecks) { const postcheckResult = await this.evaluateChecks( - ddlOp.postcheck, + operation.postcheck, inspectionExecutor, filterEvaluator, ); diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts index b867ae077d3f..b7c5e326794d 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts @@ -22,6 +22,7 @@ import type { MigrationOperationClass, } from '@internal/framework-components/control'; import type { + AnyMongoMigrationOperation, CollModOptions, CreateCollectionOptions, CreateIndexOptions, @@ -34,6 +35,7 @@ import type { MongoSchemaIndex, MongoSchemaValidator, } from '@internal/mongo-schema-ir'; +import type { Document } from '@internal/mongo-value'; import { type ImportRequirement, jsonToTsSource, TsExpression } from '@internal/ts-render'; import { ifDefined } from '@internal/utils/defined'; import { @@ -43,6 +45,8 @@ import { dropCollection, dropIndex, renameCollection, + renameField, + unsetField, } from './migration-factories'; export interface CollModMeta { @@ -64,7 +68,7 @@ abstract class OpFactoryCallNode extends TsExpression implements FrameworkOpFact abstract readonly factoryName: string; abstract readonly operationClass: MigrationOperationClass; abstract readonly label: string; - abstract toOp(): MongoMigrationPlanOperation; + abstract toOp(): AnyMongoMigrationOperation; importRequirements(): readonly ImportRequirement[] { return [{ moduleSpecifier: TARGET_MIGRATION_MODULE, symbol: this.factoryName }]; @@ -207,6 +211,64 @@ export class RenameCollectionCall extends OpFactoryCallNode { } } +function renderFilterOption(filter: Document | undefined): string { + return filter === undefined ? '' : `, ${jsonToTsSource({ filter })}`; +} + +export class RenameFieldCall extends OpFactoryCallNode { + readonly factoryName = 'renameField' as const; + readonly operationClass = 'widening' as const; + readonly collection: string; + readonly from: string; + readonly to: string; + readonly filter: Document | undefined; + readonly label: string; + + constructor(collection: string, from: string, to: string, filter?: Document) { + super(); + this.collection = collection; + this.from = from; + this.to = to; + this.filter = filter; + this.label = this.toOp().label; + this.freeze(); + } + + toOp(): AnyMongoMigrationOperation { + return renameField(this.collection, this.from, this.to, ifDefined('filter', this.filter)); + } + + renderTypeScript(): string { + return `renameField(${jsonToTsSource(this.collection)}, ${jsonToTsSource(this.from)}, ${jsonToTsSource(this.to)}${renderFilterOption(this.filter)})`; + } +} + +export class UnsetFieldCall extends OpFactoryCallNode { + readonly factoryName = 'unsetField' as const; + readonly operationClass = 'destructive' as const; + readonly collection: string; + readonly field: string; + readonly filter: Document | undefined; + readonly label: string; + + constructor(collection: string, field: string, filter?: Document) { + super(); + this.collection = collection; + this.field = field; + this.filter = filter; + this.label = this.toOp().label; + this.freeze(); + } + + toOp(): AnyMongoMigrationOperation { + return unsetField(this.collection, this.field, ifDefined('filter', this.filter)); + } + + renderTypeScript(): string { + return `unsetField(${jsonToTsSource(this.collection)}, ${jsonToTsSource(this.field)}${renderFilterOption(this.filter)})`; + } +} + export class CollModCall extends OpFactoryCallNode { readonly factoryName = 'collMod' as const; readonly collection: string; @@ -242,6 +304,8 @@ export type OpFactoryCall = | CreateCollectionCall | DropCollectionCall | RenameCollectionCall + | RenameFieldCall + | UnsetFieldCall | CollModCall; export function schemaIndexToCreateIndexOptions(index: MongoSchemaIndex): CreateIndexOptions { diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/render-ops.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/render-ops.ts index 49df5ff696f4..2a81affc22b0 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/render-ops.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/render-ops.ts @@ -1,6 +1,6 @@ -import type { MongoMigrationPlanOperation } from '@internal/mongo-query-ast/control'; +import type { AnyMongoMigrationOperation } from '@internal/mongo-query-ast/control'; import type { OpFactoryCall } from './op-factory-call'; -export function renderOps(calls: ReadonlyArray): MongoMigrationPlanOperation[] { +export function renderOps(calls: ReadonlyArray): AnyMongoMigrationOperation[] { return calls.map((call) => call.toOp()); } diff --git a/packages/3-mongo-target/1-mongo-target/src/exports/control.ts b/packages/3-mongo-target/1-mongo-target/src/exports/control.ts index 54609cfb607b..a3705b349bd3 100644 --- a/packages/3-mongo-target/1-mongo-target/src/exports/control.ts +++ b/packages/3-mongo-target/1-mongo-target/src/exports/control.ts @@ -20,8 +20,10 @@ export { DropCollectionCall, DropIndexCall, RenameCollectionCall, + RenameFieldCall, schemaCollectionToCreateCollectionOptions, schemaIndexToCreateIndexOptions, + UnsetFieldCall, } from '../core/migrations/op-factory-call'; export { PlannerProducedMongoMigration } from '../core/migrations/planner-produced-migration'; export { renderOps } from '../core/migrations/render-ops'; diff --git a/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts b/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts index fb18e090eff0..2550a3681c6d 100644 --- a/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts +++ b/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts @@ -20,6 +20,8 @@ export { dropCollection, dropIndex, renameCollection, + renameField, setValidation, + unsetField, validatedCollection, } from '../core/migrations/migration-factories'; diff --git a/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts b/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts new file mode 100644 index 000000000000..c2277ecc4d54 --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts @@ -0,0 +1,192 @@ +import { MongoExistsExpr } from '@internal/mongo-query-ast/control'; +import { RawAggregateCommand, RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; +import { describe, expect, it } from 'vitest'; +import { renameField, unsetField } from '../src/core/migrations/migration-factories'; +import { + deserializeMongoOps, + serializeMongoOps, +} from '../src/core/migrations/mongo-ops-serializer'; +import { RenameFieldCall, UnsetFieldCall } from '../src/core/migrations/op-factory-call'; + +const meta = { target: 'mongo', storageHash: '', lane: 'mongo-raw' }; + +function noDocumentHolds(collection: string, match: Record, description: string) { + return { + description, + source: { + collection, + command: new RawAggregateCommand(collection, [{ $match: match }, { $limit: 1 }]), + meta, + }, + filter: MongoExistsExpr.exists('_id'), + expect: 'notExists' as const, + }; +} + +describe('renameField', () => { + it('renames the field on every document that holds it, skipping validation', () => { + const op = renameField('users', 'name', 'fullName'); + + expect(op).toEqual({ + id: 'field.users.name.rename(fullName)', + label: 'Rename field name to fullName in collection users', + operationClass: 'widening', + name: 'field.users.name.rename(fullName)', + precheck: [ + noDocumentHolds( + 'users', + { fullName: { $exists: true } }, + 'no document in users holds fullName', + ), + ], + run: [ + { + collection: 'users', + command: new RawUpdateManyCommand( + 'users', + { name: { $exists: true } }, + { $rename: { name: 'fullName' } }, + { bypassDocumentValidation: true }, + ), + meta, + }, + ], + postcheck: [ + noDocumentHolds('users', { name: { $exists: true } }, 'no document in users holds name'), + ], + }); + }); + + it('limits the rename and its checks to documents that match the filter', () => { + const op = renameField('posts', 'title', 'headline', { filter: { type: 'bug' } }); + + expect({ + id: op.id, + label: op.label, + precheck: op.precheck[0]?.source.command, + run: op.run[0]?.command, + postcheck: op.postcheck[0]?.source.command, + }).toEqual({ + id: 'field.posts.title.rename(headline).where({"type":"bug"})', + label: 'Rename field title to headline in collection posts where {"type":"bug"}', + precheck: new RawAggregateCommand('posts', [ + { $match: { $and: [{ type: 'bug' }, { headline: { $exists: true } }] } }, + { $limit: 1 }, + ]), + run: new RawUpdateManyCommand( + 'posts', + { $and: [{ type: 'bug' }, { title: { $exists: true } }] }, + { $rename: { title: 'headline' } }, + { bypassDocumentValidation: true }, + ), + postcheck: new RawAggregateCommand('posts', [ + { $match: { $and: [{ type: 'bug' }, { title: { $exists: true } }] } }, + { $limit: 1 }, + ]), + }); + }); +}); + +describe('unsetField', () => { + it('removes the field from every document that holds it, skipping validation', () => { + expect(unsetField('users', 'nickname')).toEqual({ + id: 'field.users.nickname.unset', + label: 'Remove field nickname from collection users', + operationClass: 'destructive', + name: 'field.users.nickname.unset', + precheck: [], + run: [ + { + collection: 'users', + command: new RawUpdateManyCommand( + 'users', + { nickname: { $exists: true } }, + { $unset: { nickname: '' } }, + { bypassDocumentValidation: true }, + ), + meta, + }, + ], + postcheck: [ + noDocumentHolds( + 'users', + { nickname: { $exists: true } }, + 'no document in users holds nickname', + ), + ], + }); + }); + + it('limits the removal to documents that match the filter', () => { + const op = unsetField('posts', 'score', { filter: { type: 'bug' } }); + + expect({ id: op.id, run: op.run[0]?.command }).toEqual({ + id: 'field.posts.score.unset.where({"type":"bug"})', + run: new RawUpdateManyCommand( + 'posts', + { $and: [{ type: 'bug' }, { score: { $exists: true } }] }, + { $unset: { score: '' } }, + { bypassDocumentValidation: true }, + ), + }); + }); +}); + +describe('field rewrite calls', () => { + it('RenameFieldCall is widening, renders renameField(...) and lowers to renameField', () => { + const plain = new RenameFieldCall('users', 'name', 'fullName'); + const filtered = new RenameFieldCall('posts', 'title', 'headline', { type: 'bug' }); + + expect({ + frozen: Object.isFrozen(plain), + operationClass: plain.operationClass, + label: plain.label, + plain: plain.renderTypeScript(), + filtered: filtered.renderTypeScript(), + imports: plain.importRequirements(), + }).toEqual({ + frozen: true, + operationClass: 'widening', + label: 'Rename field name to fullName in collection users', + plain: 'renameField("users", "name", "fullName")', + filtered: 'renameField("posts", "title", "headline", { filter: { type: "bug" } })', + imports: [{ moduleSpecifier: '@internal/target-mongo/migration', symbol: 'renameField' }], + }); + expect(filtered.toOp()).toEqual( + renameField('posts', 'title', 'headline', { filter: { type: 'bug' } }), + ); + }); + + it('UnsetFieldCall is destructive, renders unsetField(...) and lowers to unsetField', () => { + const plain = new UnsetFieldCall('users', 'nickname'); + const filtered = new UnsetFieldCall('posts', 'score', { type: 'bug' }); + + expect({ + frozen: Object.isFrozen(plain), + operationClass: plain.operationClass, + label: plain.label, + plain: plain.renderTypeScript(), + filtered: filtered.renderTypeScript(), + imports: plain.importRequirements(), + }).toEqual({ + frozen: true, + operationClass: 'destructive', + label: 'Remove field nickname from collection users', + plain: 'unsetField("users", "nickname")', + filtered: 'unsetField("posts", "score", { filter: { type: "bug" } })', + imports: [{ moduleSpecifier: '@internal/target-mongo/migration', symbol: 'unsetField' }], + }); + expect(filtered.toOp()).toEqual(unsetField('posts', 'score', { filter: { type: 'bug' } })); + }); +}); + +describe('field rewrite serialization', () => { + it('round-trips both rewrites with their classes, told apart from DDL by their shape', () => { + const ops = [ + renameField('posts', 'title', 'headline', { filter: { type: 'bug' } }), + unsetField('users', 'nickname'), + ]; + + expect(deserializeMongoOps(JSON.parse(serializeMongoOps(ops)))).toEqual(ops); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts b/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts index da4d6c4c299b..1bb38a5dc574 100644 --- a/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts @@ -335,6 +335,35 @@ describe('migration file E2E', () => { expect(deserializeMongoOps(ops)).toEqual([call.toOp()]); }); + it('reproduces renameField and unsetField operations in ops.json', async () => { + const { renderCallsToTypeScript } = await import('../src/core/migrations/render-typescript'); + const { RenameFieldCall, UnsetFieldCall } = await import( + '../src/core/migrations/op-factory-call' + ); + const { deserializeMongoOps } = await import('../src/core/migrations/mongo-ops-serializer'); + const calls = [ + new RenameFieldCall('users', 'name', 'fullName'), + new UnsetFieldCall('posts', 'score', { type: 'bug' }), + ]; + + const tsSource = renderCallsToTypeScript(calls, defaultMeta); + const resolvedSource = tsSource.replace( + "'@internal/target-mongo/migration'", + `'${factoryExport}'`, + ); + await writeFile(join(tmpDir, 'migration.ts'), resolvedSource); + await writeContractFixtures(defaultMeta); + + const result = await runFile('migration.ts'); + expect({ exitCode: result.exitCode, stderr: result.stderr }).toEqual({ + exitCode: 0, + stderr: '', + }); + + const ops = JSON.parse(await readFile(join(tmpDir, 'ops.json'), 'utf-8')); + expect(deserializeMongoOps(ops)).toEqual(calls.map((call) => call.toOp())); + }); + it('round-trips describe() meta through TypeScript execution', async () => { const { renderCallsToTypeScript } = await import('../src/core/migrations/render-typescript'); const { DropCollectionCall } = await import('../src/core/migrations/op-factory-call'); diff --git a/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts b/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts index 1f458ad3e6ad..a9d5240e7144 100644 --- a/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts +++ b/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts @@ -186,6 +186,7 @@ class MongoAdapterImpl implements MongoAdapter { collection: command.collection, filter: command.filter, update: command.update, + bypassDocumentValidation: command.bypassDocumentValidation ?? false, }; case 'rawDeleteOne': return { kind: 'rawDeleteOne', collection: command.collection, filter: command.filter }; @@ -294,7 +295,13 @@ class MongoAdapterImpl implements MongoAdapter { case 'rawUpdateOne': return new UpdateOneWireCommand(draft.collection, draft.filter, draft.update); case 'rawUpdateMany': - return new UpdateManyWireCommand(draft.collection, draft.filter, draft.update); + return new UpdateManyWireCommand( + draft.collection, + draft.filter, + draft.update, + false, + draft.bypassDocumentValidation, + ); case 'rawDeleteOne': return new DeleteOneWireCommand(draft.collection, draft.filter); case 'rawDeleteMany': diff --git a/packages/3-mongo-target/2-mongo-adapter/test/mongo-adapter.test.ts b/packages/3-mongo-target/2-mongo-adapter/test/mongo-adapter.test.ts index 4f88a3376d68..ea59ee793344 100644 --- a/packages/3-mongo-target/2-mongo-adapter/test/mongo-adapter.test.ts +++ b/packages/3-mongo-target/2-mongo-adapter/test/mongo-adapter.test.ts @@ -297,6 +297,17 @@ describe('MongoAdapter', () => { expect(wire.update).toEqual(update); }); + it('passes bypassDocumentValidation through to the wire command', async () => { + const command = new RawUpdateManyCommand( + 'users', + {}, + { $rename: { name: 'fullName' } }, + { bypassDocumentValidation: true }, + ); + const wire = narrowWire(await adapter.lower(plan('users', command), {}), 'updateMany'); + expect(wire.bypassDocumentValidation).toBe(true); + }); + it('passes pipeline-style update (array) through unchanged', async () => { const filter = { firstName: { $exists: true } }; const update = [{ $set: { fullName: { $concat: ['$firstName', ' ', '$lastName'] } } }]; diff --git a/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts b/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts index 1a897eb404a8..762fc937b6e8 100644 --- a/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts +++ b/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts @@ -169,7 +169,10 @@ export class MongoDriverImpl implements MongoDriver { cmd: UpdateManyWireCommand, ): AsyncIterable { const collection = this.db.collection(cmd.collection); - const result = await collection.updateMany(cmd.filter, cmd.update, { upsert: cmd.upsert }); + const result = await collection.updateMany(cmd.filter, cmd.update, { + upsert: cmd.upsert, + ...(cmd.bypassDocumentValidation ? { bypassDocumentValidation: true } : {}), + }); yield { matchedCount: result.matchedCount, modifiedCount: result.modifiedCount, diff --git a/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts b/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts index ea93359fc761..c8dc6a39834f 100644 --- a/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts +++ b/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts @@ -173,6 +173,38 @@ describe('MongoDriver', () => { describe('updateMany', () => { const col = 'driver_update_many'; + it('skips validation when bypassDocumentValidation is set', async () => { + const db = seedClient.db(dbName); + const driver = MongoDriverImpl.fromDb(db); + const closed = 'driver_update_many_bypass'; + await db.dropCollection(closed).catch(() => undefined); + await db.createCollection(closed, { + validator: { + $jsonSchema: { + bsonType: 'object', + required: ['name'], + properties: { _id: { bsonType: 'objectId' }, name: { bsonType: 'string' } }, + additionalProperties: false, + }, + }, + validationLevel: 'strict', + validationAction: 'error', + }); + await db.collection(closed).insertOne({ name: 'Ada' }); + const rename = { $rename: { name: 'fullName' } }; + + await expect( + collect(driver.execute(new UpdateManyWireCommand(closed, {}, rename))), + ).rejects.toThrow(/Document failed validation/); + await collect(driver.execute(new UpdateManyWireCommand(closed, {}, rename, false, true))); + + const docs = await db + .collection(closed) + .find({}, { projection: { _id: 0 } }) + .toArray(); + expect(docs).toEqual([{ fullName: 'Ada' }]); + }); + it('updates multiple documents and returns counts', async () => { const driver = await createMongoDriver(connectionUri, dbName); try { diff --git a/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts b/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts new file mode 100644 index 000000000000..e3423a790f40 --- /dev/null +++ b/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts @@ -0,0 +1,302 @@ +import { MongoControlAdapterImpl } from '@internal/adapter-mongo/control'; +import { MongoControlDriver } from '@internal/driver-mongo/control'; +import type { + MigrationOperationPolicy, + MigrationPlan, +} from '@internal/framework-components/control'; +import { buildFabricatedMigrationEdge } from '@internal/migration-tools/aggregate'; +import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; +import type { AnyMongoMigrationOperation } from '@internal/mongo-query-ast/control'; +import { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; +import { MongoMigrationRunner, serializeMongoOps } from '@internal/target-mongo/control'; +import { + collMod, + createIndex, + dataTransform, + dropIndex, + renameField, + unsetField, +} from '@internal/target-mongo/migration'; +import { timeouts } from '@repo/test-utils'; +import { type Db, MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest'; + +const controlAdapter = new MongoControlAdapterImpl(); +const dbUpdatePolicy: MigrationOperationPolicy = { + allowedOperationClasses: ['additive', 'widening', 'destructive'], +}; + +let replSet: MongoMemoryReplSet; +let client: MongoClient; +let db: Db; +const dbName = 'runner_field_rewrites_test'; + +beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + db = client.db(dbName); +}, timeouts.spinUpMongoMemoryServer); + +afterAll(async () => { + await client?.close(); + await replSet?.stop(); +}, timeouts.spinUpMongoMemoryServer); + +beforeEach(async () => { + const collections = await db.listCollections().toArray(); + for (const col of collections) { + await db.dropCollection(col['name'] as string); + } + await controlAdapter.initMarker(new MongoControlDriver(db, client), 'app', { + storageHash: 'origin', + profileHash: 'p1', + }); +}); + +function closedSchema(field: string) { + return { + bsonType: 'object', + required: ['_id', field], + properties: { _id: { bsonType: 'objectId' }, [field]: { bsonType: 'string' } }, + additionalProperties: false, + }; +} + +function contractWith(collections: Record): MongoContract { + return { + target: 'mongo', + targetFamily: 'mongo', + profileHash: 'p1', + capabilities: {}, + extensions: {}, + meta: {}, + roots: {}, + models: {}, + storage: { + storageHash: 'dest', + namespaces: { + __unbound__: { + id: '__unbound__', + kind: 'mongo-namespace', + entries: { collection: collections }, + }, + }, + }, + } as unknown as MongoContract; +} + +const usersWithFullName = contractWith({ + users: new MongoCollection({ + indexes: [{ keys: [{ field: 'fullName', direction: 1 }], unique: true }], + validator: { + jsonSchema: closedSchema('fullName'), + validationLevel: 'strict', + validationAction: 'error', + }, + }), +}); + +const renameNamePlan: readonly AnyMongoMigrationOperation[] = [ + dropIndex('users', [{ field: 'name', direction: 1 }]), + renameField('users', 'name', 'fullName'), + createIndex('users', [{ field: 'fullName', direction: 1 }], { unique: true }), + collMod( + 'users', + { + validator: { $jsonSchema: closedSchema('fullName') }, + validationLevel: 'strict', + validationAction: 'error', + }, + { id: 'validator.users.update', label: 'Update validator on users' }, + ), +]; + +async function run( + operations: readonly AnyMongoMigrationOperation[], + destinationContract: MongoContract, +) { + const plan: MigrationPlan = { + targetId: 'mongo', + operations: JSON.parse(serializeMongoOps(operations)), + origin: { storageHash: 'origin' }, + destination: { storageHash: 'dest' }, + }; + const runner = new MongoMigrationRunner( + controlAdapter.createRunnerDependencies(new MongoControlDriver(db, client)), + ); + return runner.execute({ + plan, + migrationEdges: [ + buildFabricatedMigrationEdge({ + currentMarkerStorageHash: 'origin', + destinationStorageHash: 'dest', + operationCount: operations.length, + }), + ], + destinationContract, + policy: dbUpdatePolicy, + frameworkComponents: [], + }); +} + +async function createClosedUsers(): Promise { + await db.createCollection('users', { + validator: { $jsonSchema: closedSchema('name') }, + validationLevel: 'strict', + validationAction: 'error', + }); + await db.collection('users').createIndex({ name: 1 }, { unique: true, name: 'name_1' }); + await db.collection('users').insertMany([{ name: 'Ada' }, { name: 'Bo' }, { name: 'Cy' }]); +} + +async function documents(collectionName: string) { + return db + .collection(collectionName) + .find({}, { projection: { _id: 0 }, sort: { _id: 1 } }) + .toArray(); +} + +describe('MongoMigrationRunner - field rewrites', () => { + it( + 'renames a field required by a closed validator and indexed uniquely, then rebuilds the index on the new name', + async () => { + await createClosedUsers(); + + const result = await run(renameNamePlan, usersWithFullName); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(4); + const indexes = await db.collection('users').indexes(); + expect({ + documents: await documents('users'), + indexes: indexes.map(({ name, key, unique }) => ({ name, key, unique })), + }).toEqual({ + documents: [{ fullName: 'Ada' }, { fullName: 'Bo' }, { fullName: 'Cy' }], + indexes: [ + { name: '_id_', key: { _id: 1 }, unique: undefined }, + { name: 'fullName_1', key: { fullName: 1 }, unique: true }, + ], + }); + await expect( + db.collection('users').updateOne({ fullName: 'Ada' }, { $set: { fullName: 'Ada L.' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'skips every operation when run again after success', + async () => { + await createClosedUsers(); + const first = await run(renameNamePlan, usersWithFullName); + expect(first.ok).toBe(true); + await controlAdapter.updateMarker(new MongoControlDriver(db, client), 'app', 'dest', { + storageHash: 'origin', + profileHash: 'p1', + }); + + const second = await run(renameNamePlan, usersWithFullName); + + expect(second.ok ? second.value.operationsExecuted : second.failure).toBe(0); + expect(await documents('users')).toEqual([ + { fullName: 'Ada' }, + { fullName: 'Bo' }, + { fullName: 'Cy' }, + ]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'fails at the before-check and changes nothing when a document already holds the new name', + async () => { + await db.collection('users').insertMany([{ name: 'Ada' }, { name: 'Bo', fullName: 'Bo B.' }]); + + const result = await run([renameField('users', 'name', 'fullName')], usersWithFullName); + + expect(result.ok ? result.value : result.failure.code).toBe('MIGRATION.PRECHECK_FAILED'); + expect(await documents('users')).toEqual([ + { name: 'Ada' }, + { name: 'Bo', fullName: 'Bo B.' }, + ]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'removes a field from every document that holds it, past a closed validator', + async () => { + await db.createCollection('users', { + validator: { $jsonSchema: closedSchema('name') }, + validationLevel: 'strict', + validationAction: 'error', + }); + await db.collection('users').insertMany([{ name: 'Ada' }, { name: 'Bo' }]); + + const result = await run( + [unsetField('users', 'name')], + contractWith({ + users: new MongoCollection({ + validator: { + jsonSchema: closedSchema('name'), + validationLevel: 'strict', + validationAction: 'error', + }, + }), + }), + ); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect(await documents('users')).toEqual([{}, {}]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'limits both rewrites to documents that match the filter', + async () => { + await db.collection('posts').insertMany([ + { type: 'bug', title: 'Crash', score: 1 }, + { type: 'feature', title: 'Dark mode', score: 2 }, + ]); + + const result = await run( + [ + renameField('posts', 'title', 'headline', { filter: { type: 'bug' } }), + unsetField('posts', 'score', { filter: { type: 'bug' } }), + ], + contractWith({ posts: new MongoCollection({}) }), + ); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); + expect(await documents('posts')).toEqual([ + { type: 'bug', headline: 'Crash' }, + { type: 'feature', title: 'Dark mode', score: 2 }, + ]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'refuses a data transform under the same policy', + async () => { + await db.collection('posts').insertOne({ title: 'Crash' }); + const backfill = dataTransform('backfill-status', { + run: () => ({ + collection: 'posts', + command: new RawUpdateManyCommand('posts', {}, { $set: { status: 'open' } }), + meta: { target: 'mongo', storageHash: 'dest', lane: 'mongo-raw' }, + }), + }); + + const result = await run([backfill], contractWith({ posts: new MongoCollection({}) })); + + expect(result.ok ? result.value : result.failure.code).toBe('MIGRATION.POLICY_VIOLATION'); + expect(await documents('posts')).toEqual([{ title: 'Crash' }]); + }, + timeouts.spinUpMongoMemoryServer, + ); +}); From f1e5f1f85dbfab82375f746f0dce75e1225aefa8 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:31:05 +0200 Subject: [PATCH 10/57] TML-3478: field rewrites resume after a partial run and refuse names MongoDB reads as paths renameField's before-check now matches only documents holding both the old and the new name, which is what $rename would overwrite, so an interrupted rename can finish. Both factories refuse an empty field name, _id, a name containing a dot and a name starting with $ with MIGRATION.FIELD_NAME_UNSUPPORTED, and renameField refuses renaming a field to its own name. The preview shows run steps only for field rewrites, so a user's data transform previews as before. Signed-off-by: willbot Signed-off-by: Will Madden --- docs/reference/error-reference.md | 4 ++ .../9-family/src/core/operation-preview.ts | 19 ++++-- .../test/operation-preview-format.test.ts | 27 +++++++- .../core/migrations/migration-factories.ts | 61 ++++++++++++++++--- .../src/core/mongo-target-errors.ts | 5 +- .../test/field-rewrites.test.ts | 42 ++++++++++++- .../runner-field-rewrites.test.ts | 16 +++++ 7 files changed, 153 insertions(+), 21 deletions(-) diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index 9294ac260b88..531d16681ce8 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -1450,6 +1450,10 @@ The per-space migration planner received the same contract-space id more than on A migration operation's SQL step failed while being executed against the database during apply (`db init`, `db update`, `db migrate`). The envelope carries the database error detail so you can see which statement failed and why. Payload: `operationId`, `stepDescription`, `sql`, `sqlState`, `constraint`, `table`, `column`, `detail`. +### MIGRATION.FIELD_NAME_UNSUPPORTED + +A MongoDB `renameField` or `unsetField` operation names a field it cannot rewrite: an empty name, `_id` (MongoDB does not let a document change its `_id`), a name containing `.` (MongoDB reads it as a path into embedded documents), or a name starting with `$` (MongoDB reads it as an operator). `renameField` also refuses a new name equal to the old one. Raised when the operation is built, before anything runs. Payload: `collection`, `field`, `reason`. + ### MIGRATION.FILE_MISSING A required migration file is absent: either an on-disk package is missing `migration.json`/`ops.json` (migration-tools loader; re-emit via the package's `migration.ts`), or a `migration.ts` source file was expected at a package directory and not found (scaffold one with `migration new` or `migration plan`). Payload: `file`, `dir` (loader variant) or `dir` (source-file variant). diff --git a/packages/2-mongo-family/9-family/src/core/operation-preview.ts b/packages/2-mongo-family/9-family/src/core/operation-preview.ts index 6547f0d39305..0a8ec6c33339 100644 --- a/packages/2-mongo-family/9-family/src/core/operation-preview.ts +++ b/packages/2-mongo-family/9-family/src/core/operation-preview.ts @@ -103,8 +103,18 @@ function isDdlCommand(value: object | undefined): value is AnyMongoDdlCommand { return value !== undefined && typeof Reflect.get(value, 'accept') === 'function'; } -function isRawUpdateMany(value: object | undefined): value is RawUpdateManyCommand { - return value !== undefined && Reflect.get(value, 'kind') === 'rawUpdateMany'; +function isFieldRewrite(value: object | undefined): value is RawUpdateManyCommand { + return ( + value !== undefined && + Reflect.get(value, 'kind') === 'rawUpdateMany' && + Reflect.get(value, 'bypassDocumentValidation') === true + ); +} + +function fieldRewritesOf(operation: object): readonly RawUpdateManyCommand[] { + const commands = arrayAt(operation, 'run').map((plan) => objectAt(plan, 'command')); + const rewrites = commands.filter(isFieldRewrite); + return rewrites.length === commands.length ? rewrites : []; } export function formatMongoOperations(operations: readonly MigrationPlanOperation[]): string[] { @@ -114,10 +124,7 @@ export function formatMongoOperations(operations: readonly MigrationPlanOperatio const command = objectAt(step, 'command'); if (isDdlCommand(command)) statements.push(command.accept(formatter)); } - for (const plan of arrayAt(operation, 'run')) { - const command = objectAt(plan, 'command'); - if (isRawUpdateMany(command)) statements.push(formatUpdateMany(command)); - } + statements.push(...fieldRewritesOf(operation).map(formatUpdateMany)); } return statements; } diff --git a/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts b/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts index d1dc280f6142..8445e4e0159a 100644 --- a/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts +++ b/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts @@ -7,7 +7,7 @@ import { type MongoMigrationPlanOperation, RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; -import { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; +import { RawInsertOneCommand, RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; import { describe, expect, it } from 'vitest'; import { formatMongoOperations } from '../src/core/operation-preview'; @@ -182,6 +182,31 @@ describe('formatMongoOperations', () => { ]); }); + it('renders nothing for a user data transform, even one with an updateMany step', () => { + const meta = { target: 'mongo', storageHash: 'h', lane: 'mongo-raw' }; + const transform = { + id: 'data_transform.backfill', + label: 'Data transform: backfill', + operationClass: 'data' as const, + name: 'backfill', + precheck: [], + run: [ + { + collection: 'users', + command: new RawUpdateManyCommand('users', {}, { $set: { status: 'active' } }), + meta, + }, + { + collection: 'users', + command: new RawInsertOneCommand('users', { name: 'seed' }), + meta, + }, + ], + postcheck: [], + }; + expect(formatMongoOperations([transform])).toEqual([]); + }); + it('skips operations without execute steps', () => { const ops = [{ id: 'test', label: 'test', operationClass: 'additive' as const }]; const result = formatMongoOperations(ops); diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts index 840be8760717..5b2bfd6b664d 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts @@ -32,6 +32,7 @@ import { collection } from '@internal/mongo-query-builder/contract-free'; import type { Document, MongoValue } from '@internal/mongo-value'; import { blindCast } from '@internal/utils/casts'; import { ifDefined } from '@internal/utils/defined'; +import { mongoTargetError } from '../mongo-target-errors'; import type { CollModMeta } from './op-factory-call'; type StringField = { readonly codecId: 'mongo/string@1'; readonly nullable: false }; @@ -118,9 +119,39 @@ export interface FieldRewriteOptions { const REWRITE_PLAN_META = { target: 'mongo', storageHash: '', lane: 'mongo-raw' } as const; -function documentsHolding(field: string, filter: Document | undefined): Document { - const holds = { [field]: { $exists: true } }; - return filter === undefined ? holds : { $and: [filter, holds] }; +function documentsHolding(fields: readonly string[], filter: Document | undefined): Document { + const conditions: Document[] = [ + ...(filter === undefined ? [] : [filter]), + ...fields.map((field) => ({ [field]: { $exists: true } })), + ]; + const [only] = conditions; + return conditions.length === 1 && only !== undefined ? only : { $and: conditions }; +} + +const UNREWRITABLE_FIELD_NAMES: ReadonlyArray<{ + readonly matches: (field: string) => boolean; + readonly reason: string; +}> = [ + { matches: (field) => field === '', reason: 'it is empty' }, + { matches: (field) => field === '_id', reason: 'MongoDB does not let a document change its _id' }, + { + matches: (field) => field.includes('.'), + reason: 'MongoDB reads a name containing "." as a path into embedded documents', + }, + { + matches: (field) => field.startsWith('$'), + reason: 'MongoDB reads a name starting with "$" as an operator', + }, +]; + +function assertRewritableField(collectionName: string, field: string): void { + const reason = UNREWRITABLE_FIELD_NAMES.find((rule) => rule.matches(field))?.reason; + if (reason === undefined) return; + throw mongoTargetError( + 'MIGRATION.FIELD_NAME_UNSUPPORTED', + `Field "${field}" in collection ${collectionName} cannot be rewritten: ${reason}.`, + { meta: { collection: collectionName, field, reason } }, + ); } function rewritePlan(collectionName: string, command: RawAggregateCommand | RawUpdateManyCommand) { @@ -129,15 +160,15 @@ function rewritePlan(collectionName: string, command: RawAggregateCommand | RawU function noDocumentHolds( collectionName: string, - field: string, + fields: readonly string[], filter: Document | undefined, ): MongoDataTransformCheck { return { - description: `no document in ${collectionName} holds ${field}`, + description: `no document in ${collectionName} holds ${fields.length === 1 ? fields[0] : `both ${fields.join(' and ')}`}`, source: rewritePlan( collectionName, new RawAggregateCommand(collectionName, [ - { $match: documentsHolding(field, filter) }, + { $match: documentsHolding(fields, filter) }, { $limit: 1 }, ]), ), @@ -154,7 +185,7 @@ function rewriteUpdate( ): MongoQueryPlan { return rewritePlan( collectionName, - new RawUpdateManyCommand(collectionName, documentsHolding(field, filter), update, { + new RawUpdateManyCommand(collectionName, documentsHolding([field], filter), update, { bypassDocumentValidation: true, }), ); @@ -174,6 +205,15 @@ export function renameField( to: string, options?: FieldRewriteOptions, ): MongoDataTransformOperation { + assertRewritableField(collectionName, from); + assertRewritableField(collectionName, to); + if (from === to) { + throw mongoTargetError( + 'MIGRATION.FIELD_NAME_UNSUPPORTED', + `Field "${from}" in collection ${collectionName} cannot be renamed to its own name.`, + { meta: { collection: collectionName, field: from, reason: 'the new name is the old name' } }, + ); + } const filter = options?.filter; const id = `field.${collectionName}.${from}.rename(${to})${idSuffix(filter)}`; return { @@ -181,9 +221,9 @@ export function renameField( label: `Rename field ${from} to ${to} in collection ${collectionName}${labelSuffix(filter)}`, operationClass: 'widening', name: id, - precheck: [noDocumentHolds(collectionName, to, filter)], + precheck: [noDocumentHolds(collectionName, [from, to], filter)], run: [rewriteUpdate(collectionName, from, filter, { $rename: { [from]: to } })], - postcheck: [noDocumentHolds(collectionName, from, filter)], + postcheck: [noDocumentHolds(collectionName, [from], filter)], }; } @@ -192,6 +232,7 @@ export function unsetField( field: string, options?: FieldRewriteOptions, ): MongoDataTransformOperation { + assertRewritableField(collectionName, field); const filter = options?.filter; const id = `field.${collectionName}.${field}.unset${idSuffix(filter)}`; return { @@ -201,7 +242,7 @@ export function unsetField( name: id, precheck: [], run: [rewriteUpdate(collectionName, field, filter, { $unset: { [field]: '' } })], - postcheck: [noDocumentHolds(collectionName, field, filter)], + postcheck: [noDocumentHolds(collectionName, [field], filter)], }; } diff --git a/packages/3-mongo-target/1-mongo-target/src/core/mongo-target-errors.ts b/packages/3-mongo-target/1-mongo-target/src/core/mongo-target-errors.ts index 97c6ac4260a6..23fd6000ca22 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/mongo-target-errors.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/mongo-target-errors.ts @@ -7,7 +7,10 @@ export type MongoTargetErrorCode = | 'RUNTIME.ENCODE_FAILED' | 'RUNTIME.TYPE_PARAMS_INVALID'; -type MigrationSubcode = 'INVALID_OPERATION_ENTRY' | 'OPERATION_UNSUPPORTED'; +type MigrationSubcode = + | 'FIELD_NAME_UNSUPPORTED' + | 'INVALID_OPERATION_ENTRY' + | 'OPERATION_UNSUPPORTED'; export function mongoTargetError( code: MongoTargetErrorCode, diff --git a/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts b/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts index c2277ecc4d54..58bd239a7d7e 100644 --- a/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts @@ -35,8 +35,8 @@ describe('renameField', () => { precheck: [ noDocumentHolds( 'users', - { fullName: { $exists: true } }, - 'no document in users holds fullName', + { $and: [{ name: { $exists: true } }, { fullName: { $exists: true } }] }, + 'no document in users holds both name and fullName', ), ], run: [ @@ -70,7 +70,11 @@ describe('renameField', () => { id: 'field.posts.title.rename(headline).where({"type":"bug"})', label: 'Rename field title to headline in collection posts where {"type":"bug"}', precheck: new RawAggregateCommand('posts', [ - { $match: { $and: [{ type: 'bug' }, { headline: { $exists: true } }] } }, + { + $match: { + $and: [{ type: 'bug' }, { title: { $exists: true } }, { headline: { $exists: true } }], + }, + }, { $limit: 1 }, ]), run: new RawUpdateManyCommand( @@ -132,6 +136,38 @@ describe('unsetField', () => { }); }); +describe('field names a rewrite refuses', () => { + const refusals = [ + { field: 'a.b', reason: 'contains "."' }, + { field: '$a', reason: 'starts with "$"' }, + { field: '_id', reason: 'is _id' }, + { field: '', reason: 'is empty' }, + ]; + + for (const { field, reason } of refusals) { + it(`refuses a field name that ${reason}`, () => { + const expected = expect.objectContaining({ + code: 'MIGRATION.FIELD_NAME_UNSUPPORTED', + meta: expect.objectContaining({ collection: 'users', field }), + }); + expect(() => renameField('users', field, 'x')).toThrow(expected); + expect(() => renameField('users', 'x', field)).toThrow(expected); + expect(() => unsetField('users', field)).toThrow(expected); + expect(() => new RenameFieldCall('users', field, 'x')).toThrow(expected); + expect(() => new UnsetFieldCall('users', field)).toThrow(expected); + }); + } + + it('refuses renaming a field to its own name', () => { + expect(() => renameField('users', 'name', 'name')).toThrow( + expect.objectContaining({ + code: 'MIGRATION.FIELD_NAME_UNSUPPORTED', + meta: expect.objectContaining({ collection: 'users', field: 'name' }), + }), + ); + }); +}); + describe('field rewrite calls', () => { it('RenameFieldCall is widening, renders renameField(...) and lowers to renameField', () => { const plain = new RenameFieldCall('users', 'name', 'fullName'); diff --git a/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts b/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts index e3423a790f40..a27591d0a243 100644 --- a/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts +++ b/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts @@ -226,6 +226,22 @@ describe('MongoMigrationRunner - field rewrites', () => { timeouts.spinUpMongoMemoryServer, ); + it( + 'finishes a rename that stopped part way, where some documents already hold only the new name', + async () => { + await db.collection('users').insertMany([{ fullName: 'a' }, { name: 'b' }]); + + const result = await run( + [renameField('users', 'name', 'fullName')], + contractWith({ users: new MongoCollection({}) }), + ); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect(await documents('users')).toEqual([{ fullName: 'a' }, { fullName: 'b' }]); + }, + timeouts.spinUpMongoMemoryServer, + ); + it( 'removes a field from every document that holds it, past a closed validator', async () => { From 7f707e8b95f7ab7f1897e698da67aa71f4a31af1 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 10:50:41 +0200 Subject: [PATCH 11/57] TML-3478: the MongoDB planner carries out renames and asks about removed fields A model statement whose collection changes renames the collection in the schema the diff starts from and plans renameCollection. A field statement plans renameField, filtered to a variant's documents by its discriminator value. A field the destination no longer stores, in a collection the plan keeps, plans unsetField and is reported as lost, so the data-loss question offers rename and delete. The planner refuses _id, field names MongoDB reads as paths or operators, time-series collections, a missing source or taken target collection, a collection name MongoDB rejects, a field stored on one side only, and collections whose control policy is not managed. The temporary refusal (renameStatements, keepDataByHand) is deleted. Signed-off-by: willbot Signed-off-by: Will Madden --- .../src/control/control-migration-types.ts | 15 +- .../cli/src/control-api/operations/db-run.ts | 2 - .../control-api/operations/migration-plan.ts | 5 - .../control-api/statements/plan-questions.ts | 27 +- .../statements/ask-plan-questions.test.ts | 40 -- .../statements/data-loss-question.test.ts | 32 +- .../cli/test/orm/db-update-consent.test.ts | 1 - .../src/core/migrations/control-target.ts | 3 +- .../src/core/migrations/mongo-planner.ts | 165 ++--- .../migrations/mongo-statement-planning.ts | 481 +++++++++++++ .../test/db-update-statements.test.ts | 42 +- .../test/mongo-planner.data-loss.test.ts | 21 +- .../test/mongo-planner.statements.test.ts | 640 ++++++++++++++++-- .../1-mongo-target/test/mongo-planner.test.ts | 9 +- ...ete-statements-migration.mongo.e2e.test.ts | 23 +- .../mongo-db-update-consent.e2e.test.ts | 2 +- .../target-runner/runner-statements.test.ts | 360 ++++++++++ 17 files changed, 1547 insertions(+), 321 deletions(-) create mode 100644 packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts create mode 100644 test/integration/test/mongo/target-runner/runner-statements.test.ts diff --git a/packages/1-framework/1-core/framework-components/src/control/control-migration-types.ts b/packages/1-framework/1-core/framework-components/src/control/control-migration-types.ts index 8d1f0b85e152..21e778f3190a 100644 --- a/packages/1-framework/1-core/framework-components/src/control/control-migration-types.ts +++ b/packages/1-framework/1-core/framework-components/src/control/control-migration-types.ts @@ -9,7 +9,7 @@ * with additional fields for execution (precheck SQL, execute SQL, etc.). */ -import type { Contract, ContractWithDomain } from '@internal/contract/types'; +import type { Contract } from '@internal/contract/types'; import type { ImportRequirement } from '@internal/ts-render'; import type { Result } from '@internal/utils/result'; import type { TargetBoundComponentDescriptor } from '../shared/framework-components'; @@ -23,7 +23,6 @@ import type { OperationContext } from './control-operation-results'; import type { AppliedMigrationStatement, MigrationPlanSubjects, - MigrationSubject, ResolvedMigrationStatement, } from './migration-statements'; @@ -653,18 +652,6 @@ export interface TargetMigrationsCapability< adapter: ControlAdapterInstance, ): MigrationPlanner; createRunner(family: TFamilyInstance): MigrationRunner; - /** - * Set when the target's planner carries out no rename statement: a data-loss question then - * offers no rename, and says instead how to keep the data by hand. - */ - readonly renameStatements?: { - readonly refused: true; - /** How to keep the data of `subject` by hand, so that the plan made afterwards loses nothing. */ - readonly keepDataByHand: ( - subject: MigrationSubject, - fromContract: ContractWithDomain, - ) => string; - }; /** * Synthesizes a family-specific schema IR from a contract for offline planning. * The returned schema can be passed to `planner.plan({ schema })` as the "from" state. diff --git a/packages/1-framework/3-tooling/cli/src/control-api/operations/db-run.ts b/packages/1-framework/3-tooling/cli/src/control-api/operations/db-run.ts index c099b09e038c..ed173e3b0532 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/operations/db-run.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/operations/db-run.ts @@ -48,7 +48,6 @@ import { type AnswerPlanQuestions, askPlanQuestions, type ConsentedSubject, - keepDataByHandFor, type PlannedQuestions, refuseUnusedConsents, subjectText, @@ -326,7 +325,6 @@ export async function executeRun plan(resolveRenames(renames)), diff --git a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts index b58f25d10613..5f5afa4b6176 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts @@ -9,7 +9,6 @@ import { createControlStack, hasOperationPreview, type MigrationPlanOperation, - type MigrationSubject, migrationSubjectKey, type OperationPreview, planOriginOf, @@ -53,7 +52,6 @@ import { snapshotVerifierFor } from '../../utils/snapshot-content-verification'; import { type AnswerPlanQuestions, askPlanQuestions, - keepDataByHandFor, type PlannedSubject, subjectText, } from '../statements/plan-questions'; @@ -299,7 +297,6 @@ async function answerPlannedDataLoss(input: { readonly consents: readonly StatementText[]; readonly origin: ContractWithDomain; readonly originKnown: boolean; - readonly keepDataByHand: ((subject: MigrationSubject) => string) | undefined; readonly destination: ContractWithDomain; readonly answer: AnswerPlanQuestions; readonly planDelta: ( @@ -327,7 +324,6 @@ async function answerPlannedDataLoss(input: { consentAll: { delete: false, allow: false }, origin: input.origin, originKnown: input.originKnown, - keepDataByHand: input.keepDataByHand, destination: input.destination, answer: input.answer, replan: async (renames) => { @@ -752,7 +748,6 @@ async function executeMigrationPlanCommandInner( consents: statementTexts.filter((statement) => statement.verb !== 'rename'), origin: fromContract ?? EMPTY_ORIGIN, originKnown: true, - keepDataByHand: keepDataByHandFor(migrations, fromContract ?? EMPTY_ORIGIN), destination: toContract, answer: options.answerQuestions, planDelta: async (renames) => { diff --git a/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts b/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts index d25ef495ae6e..19b667346ae3 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts @@ -6,7 +6,6 @@ import { migrationSubjectKey, modelDisplayName, type ResolvedMigrationStatement, - type TargetMigrationsCapability, } from '@internal/framework-components/control'; import { InternalError } from '@internal/utils/internal-error'; import { notOk, ok, type Result } from '@internal/utils/result'; @@ -175,18 +174,6 @@ function namesSubject(verb: ConsentVerb, subject: string) { text === subject ? undefined : `${verb} names "${subject}": --${verb} ${subject}.`; } -/** - * How the target says to keep a subject's data by hand when its planner carries out no rename, or - * `undefined` when it carries them out. - */ -export function keepDataByHandFor( - migrations: { readonly renameStatements?: TargetMigrationsCapability['renameStatements'] }, - origin: ContractWithDomain, -): ((subject: MigrationSubject) => string) | undefined { - const refused = migrations.renameStatements; - return refused === undefined ? undefined : (subject) => refused.keepDataByHand(subject, origin); -} - /** A rename as a statement is written: a field's new name keeps the model, `User.nickname:User.`. */ function renameForm(subject: MigrationSubject, text: string): string { return subject.kind === 'field' @@ -226,29 +213,23 @@ function inDestination( * has, and no rename of the plan renamed, may be renamed instead of deleted; a rename is resolved * against the two contracts after the plan's other renames, and its old name must be the subject. * A subject the destination keeps, such as a field whose type changes, and data no model stores, - * can only be deleted. Where the target's planner carries out no rename, `keepDataByHand` says how - * to keep the data instead. + * can only be deleted. */ export function dataLossQuestion( loss: PlannedSubject, contracts: StatementContracts & { readonly destination: ContractWithDomain; readonly originKnown: boolean; - readonly keepDataByHand: ((subject: MigrationSubject) => string) | undefined; }, ): PlanQuestion { const subject = subjectText(loss.subject, contracts); const renamable = - contracts.keepDataByHand === undefined && loss.subject.kind !== 'storage' && !inDestination(loss.subject, contracts) && !renamedByPlan(loss.subject, contracts); const deleteNames = namesSubject('delete', subject); return { - question: [ - `${loss.label} ${lossText(loss.subject, subject, contracts.originKnown)}.`, - ...(contracts.keepDataByHand === undefined ? [] : [contracts.keepDataByHand(loss.subject)]), - ].join(' '), + question: `${loss.label} ${lossText(loss.subject, subject, contracts.originKnown)}.`, subject, verbs: renamable ? ['rename', 'delete'] : ['delete'], forms: renamable ? { rename: renameForm(loss.subject, subject) } : {}, @@ -383,7 +364,6 @@ export async function askPlanQuestions readonly consentAll: { readonly [verb in ConsentVerb]: boolean }; readonly origin: ContractWithDomain; readonly originKnown: boolean; - readonly keepDataByHand: ((subject: MigrationSubject) => string) | undefined; readonly destination: ContractWithDomain; readonly answer: AnswerPlanQuestions; readonly replan: (renames: readonly StatementText[]) => Promise>; @@ -405,7 +385,6 @@ export async function askPlanQuestions const contracts = { origin: input.origin, originKnown: input.originKnown, - keepDataByHand: input.keepDataByHand, destination: input.destination, renames, }; @@ -443,7 +422,7 @@ export async function askPlanQuestions const questions = [ ...withRecoveryOnFirstStorageQuestion( losses.map((loss) => ({ loss, question: dataLossQuestion(loss, contracts) })), - !input.originKnown && input.keepDataByHand === undefined, + !input.originKnown, ), ...widenings.map((widening) => accessWideningQuestion(widening, contracts)), ]; diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts index 46dda0df1d95..3bc6d5633d23 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts @@ -44,7 +44,6 @@ function run(replanned: PlannedQuestions, typed: readonly (readonly string[])[]) consentAll: { delete: false, allow: false }, origin, originKnown: true, - keepDataByHand: undefined, destination, answer: async (questions: readonly PlanQuestion[]) => { asked.push(questions.map(({ question }) => question)); @@ -101,7 +100,6 @@ describe('questions about storage when the origin contract is unknown', () => { consentAll: { delete: false, allow: false }, origin, originKnown: false, - keepDataByHand: undefined, destination, answer: async (questions) => { asked.push(...questions.map(({ question }) => question)); @@ -150,7 +148,6 @@ describe('questions about access', () => { consentAll: { delete: false, allow: false }, origin, originKnown: true, - keepDataByHand: undefined, destination, answer: async (questions) => { asked.push(...questions.map(({ question }) => question)); @@ -196,7 +193,6 @@ describe('allow statements given as pre-answers', () => { consentAll: { delete: false, allow: false }, origin, originKnown: true, - keepDataByHand: undefined, destination, answer: async (questions) => { asked.push(...questions.map(({ question }) => question)); @@ -256,7 +252,6 @@ describe('an access change consented before a typed rename', () => { consentAll: { delete: false, allow: false }, origin, originKnown: true, - keepDataByHand: undefined, destination, answer: async (questions) => { rounds += 1; @@ -277,38 +272,3 @@ describe('an access change consented before a typed rename', () => { ).toEqual([{ verb: 'allow', operationIndex: 3 }]); }); }); - -describe('questions about storage on a target that carries out no rename', () => { - it('say how to keep the data by hand, without the steps that make a rename possible', async () => { - const asked: string[] = []; - await askPlanQuestions({ - plan: { - dataLoss: [ - { - operationIndex: 0, - label: 'Drop collection events', - subject: { kind: 'storage', name: 'events' }, - }, - ], - accessWidening: [], - }, - askAccess: false, - renames: [], - preAnswers: [], - consentAll: { delete: false, allow: false }, - origin, - originKnown: false, - keepDataByHand: () => 'Rename the collection by hand first.', - destination, - answer: async (questions) => { - asked.push(...questions.map(({ question }) => question)); - return questions.map(({ subject }) => ({ verb: 'delete', text: subject })); - }, - replan: async () => ok({ dataLoss: [], accessWidening: [] }), - }); - - expect(asked).toEqual([ - 'Drop collection events would lose the data in "events", named by its storage name because the origin contract is unknown; --delete loses its rows. Rename the collection by hand first.', - ]); - }); -}); diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts index 5231178ba9a7..4487922e46ea 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts @@ -18,7 +18,6 @@ describe('dataLossQuestion for data no model names', () => { destination, renames: [], originKnown: true, - keepDataByHand: undefined, }); expect({ question: question.question, verbs: question.verbs }).toEqual({ @@ -34,7 +33,6 @@ describe('dataLossQuestion for data no model names', () => { destination, renames: [], originKnown: false, - keepDataByHand: undefined, }); expect(question.question).toBe( @@ -50,33 +48,27 @@ describe('dataLossQuestion verbs', () => { }); const withNickname = contractOf({ app: { models: { User: { fields: ['email', 'nickname'] } } } }); - const keepByHand = () => 'Rename it by hand first.'; - - function questionFor( - field: string, - destination: typeof origin, - keepDataByHand: (() => string) | undefined, - ) { + function questionFor(field: string, destination: typeof origin) { return dataLossQuestion( { operationIndex: 0, label: 'An operation', subject: { kind: 'field', namespaceId: unbound, model: 'User', field }, }, - { origin: withNickname, destination, renames: [], originKnown: true, keepDataByHand }, + { origin: withNickname, destination, renames: [], originKnown: true }, ); } - function verbsFor(field: string, destination: typeof origin, renamesPlannable: boolean) { - return questionFor(field, destination, renamesPlannable ? undefined : keepByHand).verbs; + function verbsFor(field: string, destination: typeof origin) { + return questionFor(field, destination).verbs; } it('offers rename for a field the destination no longer has', () => { - expect(verbsFor('nickname', destinationWithoutNickname, true)).toEqual(['rename', 'delete']); + expect(verbsFor('nickname', destinationWithoutNickname)).toEqual(['rename', 'delete']); }); it('offers only delete for a field that keeps its name, such as one whose type changes', () => { - expect(verbsFor('email', destinationWithoutNickname, true)).toEqual(['delete']); + expect(verbsFor('email', destinationWithoutNickname)).toEqual(['delete']); }); it('offers only delete for a field a rename statement of the plan already renamed', () => { @@ -92,23 +84,13 @@ describe('dataLossQuestion verbs', () => { destination, renames: [{ verb: 'rename', text: 'User.nickname:User.handle' }], originKnown: true, - keepDataByHand: undefined, }, ); expect(question.verbs).toEqual(['delete']); }); - it('offers only delete where the planner cannot carry out a rename, and says how to keep the data by hand', () => { - const question = questionFor('nickname', destinationWithoutNickname, keepByHand); - expect({ verbs: question.verbs, question: question.question }).toEqual({ - verbs: ['delete'], - question: - 'An operation would lose the values of field "User.nickname". Rename it by hand first.', - }); - }); - it('writes the rename form of a field with its model, as a field rename is written', () => { - expect(questionFor('nickname', destinationWithoutNickname, undefined).forms).toEqual({ + expect(questionFor('nickname', destinationWithoutNickname).forms).toEqual({ rename: 'User.nickname:User.', }); }); diff --git a/packages/1-framework/3-tooling/cli/test/orm/db-update-consent.test.ts b/packages/1-framework/3-tooling/cli/test/orm/db-update-consent.test.ts index 350daa383e85..f12e10bac12f 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/db-update-consent.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/db-update-consent.test.ts @@ -70,7 +70,6 @@ const contracts = { destination, renames: [], originKnown: true, - keepDataByHand: undefined, }; /** One question of each kind, across two spaces: an extension's recorded drop is a storage subject. */ diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/control-target.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/control-target.ts index 21479ec048f3..d8e1460cc605 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/control-target.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/control-target.ts @@ -20,7 +20,7 @@ import { notOk, ok } from '@internal/utils/result'; import { mongoTargetDescriptorMeta } from '../descriptor-meta'; import type { MongoTargetContract } from '../mongo-target-contract'; import { MongoTargetContractSerializer } from '../mongo-target-contract-serializer'; -import { keepDataByHand, MongoMigrationPlanner } from './mongo-planner'; +import { MongoMigrationPlanner } from './mongo-planner'; import { MongoMigrationRunner, type MongoMigrationRunnerExecuteOptions } from './mongo-runner'; import { MongoTargetSchemaVerifier } from './mongo-target-schema-verifier'; import { entityNamesDeclaredBy, scopeVerifyResultToSpace } from './scope-verify-result'; @@ -42,7 +42,6 @@ export const mongoTargetDescriptor: MongoControlTargetDescriptor) { return new MongoMigrationPlanner(); }, diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts index 308f72a53f45..a5749918c264 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts @@ -1,19 +1,18 @@ import { asNamespaceId, type Contract, type ContractWithDomain } from '@internal/contract/types'; import { contractToMongoSchemaIR } from '@internal/family-mongo/control'; import type { TargetBoundComponentDescriptor } from '@internal/framework-components/components'; -import { - describeMigrationStatement, - type MigrationOperationPolicy, - type MigrationOperationSubject, - type MigrationPlanner, - type MigrationPlannerConflict, - type MigrationPlannerResult, - type MigrationPlanWithAuthoringSurface, - type MigrationScaffoldContext, - type MigrationSubject, - type ModelCoordinate, - type PlanOrigin, - type ResolvedMigrationStatement, +import type { + MigrationOperationPolicy, + MigrationOperationSubject, + MigrationPlanner, + MigrationPlannerConflict, + MigrationPlannerResult, + MigrationPlanWithAuthoringSurface, + MigrationScaffoldContext, + MigrationSubject, + ModelCoordinate, + PlanOrigin, + ResolvedMigrationStatement, } from '@internal/framework-components/control'; import type { MongoContract } from '@internal/mongo-contract'; import type { @@ -25,6 +24,13 @@ import type { } from '@internal/mongo-schema-ir'; import { canonicalize, deepEqual } from '@internal/mongo-schema-ir'; import { blindCast } from '@internal/utils/casts'; +import { + collectionOf, + type FieldRemoval, + planFieldRemovals, + planStatements, + planValidatorFieldRemovals, +} from './mongo-statement-planning'; import type { OpFactoryCall } from './op-factory-call'; import { CollModCall, @@ -123,15 +129,6 @@ export type PlanCallsResult = | { readonly kind: 'success'; readonly calls: OpFactoryCall[] } | { readonly kind: 'failure'; readonly conflicts: MigrationPlannerConflict[] }; -/** The collection a model stores its documents in; a model without `@@map` names it verbatim. */ -function collectionOf(contract: ContractWithDomain | null, coordinate: ModelCoordinate): string { - const collection = - contract?.domain.namespaces[coordinate.namespaceId]?.models[coordinate.model]?.storage[ - 'collection' - ]; - return typeof collection === 'string' ? collection : coordinate.model; -} - /** * The model of `fromContract` that stores its documents in `collection`, if any: the root model * when variants share the collection with it. @@ -168,71 +165,16 @@ function collectionDrops( }); } -const NOT_IN_THIS_RELEASE = 'MongoDB cannot carry out rename statements in this release.'; - -function shellString(name: string): string { - return JSON.stringify(name); -} - -/** - * How to keep the documents of a subject a plan would drop, since the planner carries out no - * rename: rename the collection by hand before a plan that drops it is applied. Right for both - * `db update`, which then finds nothing to drop, and `migration plan`, whose written migration - * still drops it. - */ -export function keepDataByHand( - subject: MigrationSubject, - fromContract: ContractWithDomain, -): string { - const collection = - subject.kind === 'storage' ? subject.name : collectionOf(fromContract, subject); - return `If it was renamed, keep its documents instead: rename collection "${collection}" by hand on each database before a plan that drops it is applied there, for example with db.getCollection(${shellString(collection)}).renameCollection("") in mongosh. db update then drops nothing; a migration written by migration plan still drops "${collection}", so remove that operation from its migration.ts, or do not apply it where the collection was renamed.`; -} - -/** - * What a plan made without the statement does to the data, and how to keep it. Right for both - * `db update` and `migration plan`, since the planner does not know which command it serves. - * Mongo contracts key a model's fields by their stored names, so the statement's field names are - * the names a `$rename` needs. - */ -function keepTheData( - statement: ResolvedMigrationStatement, - fromContract: ContractWithDomain | null, - contract: ContractWithDomain, -): string { - const from = collectionOf(fromContract, statement.from); - if (statement.entity === 'field') { - const { field } = statement.from; - const newField = statement.to.field; - const collection = `db.getCollection(${shellString(from)})`; - return `${NOT_IN_THIS_RELEASE} Without the statement, the documents in collection "${from}" keep their values under "${field}", and nothing moves them to "${newField}". To move them, run these in mongosh on each database before a plan made without the statement is applied there, using the field names as they are stored. First turn off the collection's validator, which still requires "${field}": db.runCommand({ collMod: ${shellString(from)}, validationLevel: "off" }). Then drop each unique index that includes "${field}", or the move fails once two documents have lost it, for example ${collection}.dropIndex(${shellString(`${field}_1`)}). Then move the values: ${collection}.updateMany({}, { $rename: { ${shellString(field)}: ${shellString(newField)} } }). Applying the plan then creates the indexes on "${newField}" and turns the validator back on.`; - } - const to = collectionOf(contract, statement.to); - if (from === to) { - return `${NOT_IN_THIS_RELEASE} Both models store their documents in collection "${from}", so a plan made without the statement keeps them.`; - } - return `${NOT_IN_THIS_RELEASE} Without the statement, a plan drops collection "${from}" with its documents and creates collection "${to}". To keep the documents, rename the collection by hand on each database before a plan made without the statement is applied there, for example with db.getCollection(${shellString(from)}).renameCollection(${shellString(to)}) in mongosh. A migration written by migration plan without the statement still drops "${from}" wherever it is applied, so check its operations first.`; -} - -function statementNotApplied( - statement: ResolvedMigrationStatement, - fromContract: ContractWithDomain | null, - contract: ContractWithDomain, -): MigrationPlannerConflict { - return { - kind: 'statementRefused', - summary: `MongoDB does not apply rename statements in this release, so nothing was planned: ${describeMigrationStatement(statement, fromContract ?? contract, contract)}`, - why: keepTheData(statement, fromContract, contract), - refusedStatement: statement, - }; -} - export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> { planCalls(options: { readonly contract: unknown; readonly schema: unknown; readonly policy: MigrationOperationPolicy; readonly frameworkComponents: ReadonlyArray>; + /** Collection renames, which run first; `schema` already has each collection under its new name. */ + readonly collectionRenames?: readonly OpFactoryCall[]; + /** Document rewrites, which run after index drops and before index creates. */ + readonly rewrites?: readonly OpFactoryCall[]; }): PlanCallsResult { const contract = blindCast< MongoContract, @@ -342,8 +284,10 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> } const allCalls = [ + ...(options.collectionRenames ?? []), ...collCreates, ...drops, + ...(options.rewrites ?? []), ...creates, ...validatorOps, ...mutableOptionOps, @@ -376,7 +320,7 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> readonly fromContract: Contract | null; /** The origin the produced plan asserts: its `describe().from` and `origin`. */ readonly origin: PlanOrigin | null; - /** The `--rename` statements, resolved; MongoDB refuses any statement in this release. */ + /** The `--rename` statements, resolved. */ readonly statements: readonly ResolvedMigrationStatement[]; readonly frameworkComponents: ReadonlyArray>; /** @@ -390,27 +334,62 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> MongoContract, 'framework planner passes the Mongo contract selected for the mongo target' >(options.contract); - const [statement] = options.statements; - if (statement !== undefined) { - return { - kind: 'failure', - conflicts: [statementNotApplied(statement, options.fromContract, contract)], - }; - } - const result = this.planCalls(options); + const originIR = blindCast< + MongoSchemaIR, + 'framework planner passes the inspected Mongo schema IR selected for the mongo target' + >(options.schema); + const fromContract = options.fromContract; + const statementPlan = planStatements({ + statements: options.statements, + fromContract: fromContract ?? contract, + contract, + schema: originIR, + }); + if (!statementPlan.ok) return { kind: 'failure', conflicts: statementPlan.failure }; + const { schema, collectionRenames, fieldRenames, callOfStatement } = statementPlan.value; + const destinationIR = contractToMongoSchemaIR(contract); + const renamedTo = new Map(collectionRenames.map((call) => [call.from, call.to])); + const removals: readonly FieldRemoval[] = + fromContract === null + ? planValidatorFieldRemovals({ origin: schema, destination: destinationIR }) + : planFieldRemovals({ + statements: options.statements, + fromContract, + contract, + collectionAfterRenames: (collection) => renamedTo.get(collection) ?? collection, + keptCollections: new Set( + schema.collectionNames.filter((name) => destinationIR.collection(name) !== undefined), + ), + }); + const result = this.planCalls({ + ...options, + schema, + collectionRenames, + rewrites: [...fieldRenames, ...removals.map((removal) => removal.call)], + }); if (result.kind === 'failure') return result; + const { calls } = result; + const fieldLoss: readonly MigrationOperationSubject[] = removals.map((removal) => ({ + operationIndex: calls.indexOf(removal.call), + subject: removal.subject, + })); return { kind: 'success', plan: new PlannerProducedMongoMigration( - result.calls, + calls, { from: options.origin?.storageHash ?? null, to: contract.storage.storageHash, }, options.snapshotsImportPath, ), - appliedStatements: [], - dataLoss: collectionDrops(result.calls, options.fromContract), + appliedStatements: options.statements.map((statement, index) => { + const call = callOfStatement[index]; + return { statement, operationIndexes: call === undefined ? [] : [calls.indexOf(call)] }; + }), + dataLoss: [...collectionDrops(calls, fromContract), ...fieldLoss].sort( + (a, b) => a.operationIndex - b.operationIndex, + ), accessWidening: [], }; } diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts new file mode 100644 index 000000000000..9e1452afd9ab --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -0,0 +1,481 @@ +import { + asNamespaceId, + type ContractModelBase, + type ContractRelation, + type ContractWithDomain, + effectiveControlPolicy, +} from '@internal/contract/types'; +import { + describeMigrationStatement, + type FieldCoordinate, + type MigrationPlannerConflict, + type MigrationSubject, + type ModelCoordinate, + type ResolvedFieldRenameStatement, + type ResolvedMigrationStatement, + type ResolvedModelRenameStatement, +} from '@internal/framework-components/control'; +import type { MongoContract } from '@internal/mongo-contract'; +import { MongoSchemaCollection, MongoSchemaIR } from '@internal/mongo-schema-ir'; +import { notOk, ok, type Result } from '@internal/utils/result'; +import { + type OpFactoryCall, + RenameCollectionCall, + RenameFieldCall, + UnsetFieldCall, +} from './op-factory-call'; + +type Document = Record; + +/** The collection a model stores its documents in; a model without `@@map` names it verbatim. */ +export function collectionOf( + contract: ContractWithDomain | null, + coordinate: ModelCoordinate, +): string { + const collection = + contract?.domain.namespaces[coordinate.namespaceId]?.models[coordinate.model]?.storage[ + 'collection' + ]; + return typeof collection === 'string' ? collection : coordinate.model; +} + +function modelAt( + contract: ContractWithDomain, + coordinate: ModelCoordinate, +): ContractModelBase | undefined { + const namespace = contract.domain.namespaces[coordinate.namespaceId]; + return namespace !== undefined && Object.hasOwn(namespace.models, coordinate.model) + ? namespace.models[coordinate.model] + : undefined; +} + +/** + * The condition that selects a variant's documents in the collection it shares with its base: the + * base's discriminator field holding the variant's value. A model that is not a variant has none. + */ +function variantFilter( + contract: ContractWithDomain, + coordinate: ModelCoordinate, +): Document | undefined { + const base = modelAt(contract, coordinate)?.base; + if (base === undefined) return undefined; + const baseModel = contract.domain.namespaces[base.namespace]?.models[base.model]; + const field = baseModel?.discriminator?.field; + const value = baseModel?.variants?.[coordinate.model]?.value; + return field === undefined || value === undefined ? undefined : { [field]: value }; +} + +function collectionControlPolicy(contract: MongoContract, collection: string) { + const declared = Object.values(contract.storage.namespaces) + .map((namespace) => namespace?.entries.collection?.[collection]) + .find((entry) => entry !== undefined); + return effectiveControlPolicy(declared?.control, contract.defaultControlPolicy); +} + +/** Why MongoDB would read a stored field name as something other than a field, if it would. */ +function unsafeFieldNameReason(field: string): string | undefined { + if (field.includes('.')) + return 'MongoDB reads a name containing "." as a path into embedded documents'; + if (field.startsWith('$')) return 'MongoDB reads a name starting with "$" as an operator'; + return undefined; +} + +/** Why MongoDB would refuse a collection name, if it would. */ +function invalidCollectionNameReason(name: string): string | undefined { + if (name === '') return 'it is empty'; + if (name.includes('$')) return 'it contains "$"'; + if (name.includes('\0')) return 'it contains a null character'; + if (name.startsWith('system.')) return 'it starts with "system.", which MongoDB reserves'; + return undefined; +} + +function isEmbedRelation(relation: ContractRelation): boolean { + return !('on' in relation) && !('through' in relation); +} + +type StoredField = + | { readonly kind: 'field' } + | { readonly kind: 'reference' } + | { readonly kind: 'embed' } + | { readonly kind: 'absent' }; + +function storedField(model: ContractModelBase | undefined, field: string): StoredField { + if (model === undefined) return { kind: 'absent' }; + if (Object.hasOwn(model.fields, field)) return { kind: 'field' }; + const relation = Object.hasOwn(model.relations, field) ? model.relations[field] : undefined; + if (relation === undefined) return { kind: 'absent' }; + return isEmbedRelation(relation) ? { kind: 'embed' } : { kind: 'reference' }; +} + +export interface StatementPlan { + /** The origin schema with each renamed collection under its new name, for the diff to compare. */ + readonly schema: MongoSchemaIR; + readonly collectionRenames: readonly RenameCollectionCall[]; + readonly fieldRenames: readonly RenameFieldCall[]; + /** For each statement, in order, the call that carries it out, or `undefined` when storage does not change. */ + readonly callOfStatement: readonly (OpFactoryCall | undefined)[]; +} + +function refused( + statement: ResolvedMigrationStatement, + label: string, + reason: string, + why: string, +): MigrationPlannerConflict { + return { + kind: 'statementRefused', + summary: `${label}: ${reason}`, + why, + refusedStatement: statement, + }; +} + +const LEAVE_OUT = 'Leave out this statement.'; + +/** + * Plans the rename each statement states, in order: a model statement whose collection changes renames the + * collection in the schema the diff then starts from, and a field statement whose stored name + * changes renames the field in the documents that hold it. + */ +export function planStatements(input: { + readonly statements: readonly ResolvedMigrationStatement[]; + readonly fromContract: ContractWithDomain; + readonly contract: MongoContract; + readonly schema: MongoSchemaIR; +}): Result { + let schema = input.schema; + const collectionRenames: RenameCollectionCall[] = []; + const fieldRenames: RenameFieldCall[] = []; + const callOfStatement: (OpFactoryCall | undefined)[] = []; + const conflicts: MigrationPlannerConflict[] = []; + const renamedCollections = new Map(); + + for (const statement of input.statements) { + const label = `Cannot ${describeMigrationStatement(statement, input.fromContract, input.contract)}`; + const outcome = + statement.entity === 'model' + ? planModelStatement(statement, label, schema, input) + : planFieldStatement(statement, label, renamedCollections, input); + if (outcome.kind === 'refused') { + conflicts.push(outcome.conflict); + callOfStatement.push(undefined); + continue; + } + if (outcome.kind === 'renameCollection') { + schema = renameInSchema(schema, outcome.call.from, outcome.call.to); + renamedCollections.set(outcome.call.from, outcome.call.to); + collectionRenames.push(outcome.call); + } + if (outcome.kind === 'renameField') fieldRenames.push(outcome.call); + callOfStatement.push(outcome.kind === 'unchanged' ? undefined : outcome.call); + } + + return conflicts.length > 0 + ? notOk(conflicts) + : ok({ schema, collectionRenames, fieldRenames, callOfStatement }); +} + +type StatementOutcome = + | { readonly kind: 'unchanged' } + | { readonly kind: 'renameCollection'; readonly call: RenameCollectionCall } + | { readonly kind: 'renameField'; readonly call: RenameFieldCall } + | { readonly kind: 'refused'; readonly conflict: MigrationPlannerConflict }; + +function renameInSchema(schema: MongoSchemaIR, from: string, to: string): MongoSchemaIR { + return new MongoSchemaIR( + schema.collections.map((collection) => + collection.name === from + ? new MongoSchemaCollection({ + name: to, + indexes: collection.indexes, + ...(collection.validator === undefined ? {} : { validator: collection.validator }), + ...(collection.options === undefined ? {} : { options: collection.options }), + }) + : collection, + ), + ); +} + +function controlPolicyRefusal( + statement: ResolvedMigrationStatement, + label: string, + contract: MongoContract, + collection: string, +): StatementOutcome | undefined { + const policy = collectionControlPolicy(contract, collection); + if (policy === 'managed') return undefined; + return { + kind: 'refused', + conflict: refused( + statement, + label, + `collection "${collection}" has control policy "${policy}"`, + `Statements rename only collections, and fields of collections, whose control policy is "managed". Make the change in the database yourself and leave out this statement.`, + ), + }; +} + +function planModelStatement( + statement: ResolvedModelRenameStatement, + label: string, + schema: MongoSchemaIR, + input: { readonly fromContract: ContractWithDomain; readonly contract: MongoContract }, +): StatementOutcome { + const from = collectionOf(input.fromContract, statement.from); + const to = collectionOf(input.contract, statement.to); + if (from === to) return { kind: 'unchanged' }; + const policyRefusal = controlPolicyRefusal(statement, label, input.contract, to); + if (policyRefusal !== undefined) return policyRefusal; + const refuse = (reason: string, why: string): StatementOutcome => ({ + kind: 'refused', + conflict: refused(statement, label, reason, why), + }); + const source = schema.collection(from); + if (source === undefined) { + return refuse( + `the schema being planned from has no collection "${from}"`, + `The database has no collection "${from}", or "${from}" is a view, which MongoDB cannot rename. Inspect it with prisma db schema, or ${LEAVE_OUT.toLowerCase()}`, + ); + } + if (source.options?.timeseries !== undefined) { + return refuse( + `collection "${from}" is a time-series collection`, + `MongoDB cannot rename a time-series collection. Copy its documents into the new collection yourself, or ${LEAVE_OUT.toLowerCase()}`, + ); + } + const invalid = invalidCollectionNameReason(to); + if (invalid !== undefined) { + return refuse( + `MongoDB does not accept the collection name "${to}": ${invalid}`, + `Give the model a collection name MongoDB accepts with @@map, or ${LEAVE_OUT.toLowerCase()}`, + ); + } + if (schema.collection(to) !== undefined) { + return refuse( + `the schema being planned from already has a collection "${to}"`, + `A rename cannot replace a collection that already exists. Rename or drop collection "${to}" first, or ${LEAVE_OUT.toLowerCase()}`, + ); + } + return { kind: 'renameCollection', call: new RenameCollectionCall(from, to) }; +} + +function planFieldStatement( + statement: ResolvedFieldRenameStatement, + label: string, + renamedCollections: ReadonlyMap, + input: { readonly fromContract: ContractWithDomain; readonly contract: MongoContract }, +): StatementOutcome { + const refuse = (reason: string, why: string): StatementOutcome => ({ + kind: 'refused', + conflict: refused(statement, label, reason, why), + }); + const fromField = storedField(modelAt(input.fromContract, statement.from), statement.from.field); + const toField = storedField(modelAt(input.contract, statement.to), statement.to.field); + if (fromField.kind === 'embed' || toField.kind === 'embed') { + return refuse( + 'the field is an embedded relation', + `Renaming an embedded relation is not supported in this release. ${LEAVE_OUT}`, + ); + } + if (fromField.kind !== 'field' && toField.kind !== 'field') return { kind: 'unchanged' }; + if (fromField.kind !== 'field' || toField.kind !== 'field') { + return refuse( + 'the field is stored in documents on one side of the statement only', + `A relation is not stored in documents, so a statement cannot rename a field to or from one. ${LEAVE_OUT}`, + ); + } + for (const field of [statement.from.field, statement.to.field]) { + if (field === '_id') { + return refuse( + 'it renames _id', + `MongoDB does not let a document change its _id. ${LEAVE_OUT}`, + ); + } + const unsafe = unsafeFieldNameReason(field); + if (unsafe !== undefined) { + return refuse( + `the stored field name "${field}" cannot be rewritten`, + `${unsafe}. Give the field a stored name without "." and not starting with "$", or ${LEAVE_OUT.toLowerCase()}`, + ); + } + } + const originCollection = collectionOf(input.fromContract, statement.from); + const collection = renamedCollections.get(originCollection) ?? originCollection; + if (collection !== collectionOf(input.contract, statement.to)) { + return refuse( + `the model's collection changes from "${originCollection}" to "${collectionOf(input.contract, statement.to)}", and no statement renames it`, + `Add a statement that renames the model, so its collection is renamed with its documents, or ${LEAVE_OUT.toLowerCase()}`, + ); + } + if (statement.from.field === statement.to.field) return { kind: 'unchanged' }; + const policyRefusal = controlPolicyRefusal(statement, label, input.contract, collection); + if (policyRefusal !== undefined) return policyRefusal; + return { + kind: 'renameField', + call: new RenameFieldCall( + collection, + statement.from.field, + statement.to.field, + variantFilter(input.fromContract, statement.from), + ), + }; +} + +export interface FieldRemoval { + readonly call: UnsetFieldCall; + readonly subject: MigrationSubject; +} + +function modelCoordinates(contract: ContractWithDomain): readonly ModelCoordinate[] { + return Object.entries(contract.domain?.namespaces ?? {}).flatMap(([namespaceId, namespace]) => + Object.keys(namespace.models).map((model) => ({ + namespaceId: asNamespaceId(namespaceId), + model, + })), + ); +} + +function sameModel(a: ModelCoordinate, b: ModelCoordinate): boolean { + return a.namespaceId === b.namespaceId && a.model === b.model; +} + +/** + * The destination model an origin model became: the one a statement renamed it to, else the one of + * the same name, else the only one stored the same way (same collection, same variant value). + */ +function destinationModelOf( + origin: ModelCoordinate, + input: FieldRemovalInput, +): ModelCoordinate | undefined { + const renamed = input.statements.find( + (statement): statement is ResolvedModelRenameStatement => + statement.entity === 'model' && sameModel(statement.from, origin), + ); + if (renamed !== undefined) return renamed.to; + if (modelAt(input.contract, origin) !== undefined) return origin; + const collection = input.collectionAfterRenames(collectionOf(input.fromContract, origin)); + const filter = JSON.stringify(variantFilter(input.fromContract, origin) ?? null); + const stored = modelCoordinates(input.contract).filter( + (candidate) => + collectionOf(input.contract, candidate) === collection && + JSON.stringify(variantFilter(input.contract, candidate) ?? null) === filter, + ); + const [only] = stored; + return stored.length === 1 ? only : undefined; +} + +interface FieldRemovalInput { + readonly statements: readonly ResolvedMigrationStatement[]; + readonly fromContract: ContractWithDomain; + readonly contract: MongoContract; + readonly collectionAfterRenames: (collection: string) => string; + readonly keptCollections: ReadonlySet; +} + +function renamedByStatement(input: FieldRemovalInput, field: FieldCoordinate): boolean { + return input.statements.some( + (statement) => + statement.entity === 'field' && + sameModel(statement.from, field) && + statement.from.field === field.field, + ); +} + +function storedFieldsOf(contract: ContractWithDomain, coordinate: ModelCoordinate): Set { + const model = modelAt(contract, coordinate); + if (model === undefined) return new Set(); + const base = + model.base === undefined + ? undefined + : modelAt(contract, { + namespaceId: asNamespaceId(model.base.namespace), + model: model.base.model, + }); + return new Set([...Object.keys(model.fields), ...Object.keys(base?.fields ?? {})]); +} + +function fieldsDeclaredIn(contract: ContractWithDomain, collection: string): Set { + return new Set( + modelCoordinates(contract) + .filter((coordinate) => collectionOf(contract, coordinate) === collection) + .flatMap((coordinate) => Object.keys(modelAt(contract, coordinate)?.fields ?? {})), + ); +} + +/** + * Each field an origin model stores that its destination model no longer stores, in a collection + * the plan keeps: removing it is data loss, carried out by unsetting it from the documents that + * hold it. A variant's field is unset only from that variant's documents. A field the base no + * longer stores is unset only when no model of the collection stores it any more, since a variant + * may now store it. + */ +export function planFieldRemovals(input: FieldRemovalInput): readonly FieldRemoval[] { + return modelCoordinates(input.fromContract).flatMap((origin) => { + const collection = input.collectionAfterRenames(collectionOf(input.fromContract, origin)); + if (!input.keptCollections.has(collection)) return []; + const destination = destinationModelOf(origin, input); + if (destination === undefined) return []; + const model = modelAt(input.fromContract, origin); + if (model === undefined) return []; + const filter = variantFilter(input.fromContract, origin); + const stillStored = + filter === undefined + ? fieldsDeclaredIn(input.contract, collection) + : storedFieldsOf(input.contract, destination); + return Object.keys(model.fields) + .filter( + (field) => + field !== '_id' && + unsafeFieldNameReason(field) === undefined && + !stillStored.has(field) && + !renamedByStatement(input, { ...origin, field }), + ) + .sort() + .map((field) => ({ + call: new UnsetFieldCall(collection, field, filter), + subject: { kind: 'field' as const, ...origin, field }, + })); + }); +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function propertyNames(schema: unknown): readonly string[] { + const properties = isRecord(schema) ? schema['properties'] : undefined; + return isRecord(properties) ? Object.keys(properties) : []; +} + +function branchPropertyNames(schema: unknown): readonly string[] { + const branches = isRecord(schema) ? schema['oneOf'] : undefined; + return Array.isArray(branches) ? branches.flatMap(propertyNames) : []; +} + +/** + * Without an origin contract, a field is known removed when the origin validator declares it at the + * top level and the destination validator declares it nowhere. It is named by its storage name. + */ +export function planValidatorFieldRemovals(input: { + readonly origin: MongoSchemaIR; + readonly destination: MongoSchemaIR; +}): readonly FieldRemoval[] { + return input.origin.collections.flatMap((originCollection) => { + const destination = input.destination.collection(originCollection.name); + if (destination === undefined || originCollection.validator === undefined) return []; + const kept = new Set([ + ...propertyNames(destination.validator?.jsonSchema), + ...branchPropertyNames(destination.validator?.jsonSchema), + ]); + return propertyNames(originCollection.validator.jsonSchema) + .filter( + (field) => + field !== '_id' && unsafeFieldNameReason(field) === undefined && !kept.has(field), + ) + .sort() + .map((field) => ({ + call: new UnsetFieldCall(originCollection.name, field), + subject: { kind: 'storage' as const, name: `${originCollection.name}.${field}` }, + })); + }); +} diff --git a/packages/3-mongo-target/1-mongo-target/test/db-update-statements.test.ts b/packages/3-mongo-target/1-mongo-target/test/db-update-statements.test.ts index 06e6bbbd8c9f..db695cd0ac91 100644 --- a/packages/3-mongo-target/1-mongo-target/test/db-update-statements.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/db-update-statements.test.ts @@ -9,6 +9,7 @@ import type { TargetMigrationsCapability, } from '@internal/framework-components/control'; import { writeContractSnapshot } from '@internal/migration-tools/contract-snapshot-store'; +import { MongoCollection } from '@internal/mongo-contract'; import { MongoSchemaCollection, MongoSchemaIR } from '@internal/mongo-schema-ir'; import { join } from 'pathe'; import { afterEach, describe, expect, it } from 'vitest'; @@ -21,7 +22,16 @@ function contractWithModel(storageHash: string, model: string): Contract { return { target: 'mongo', targetFamily: 'mongo', - storage: { storageHash, namespaces: {} }, + storage: { + storageHash, + namespaces: { + app: { + id: 'app', + kind: 'mongo-namespace', + entries: { collection: { [model]: new MongoCollection({}) } }, + }, + }, + }, domain: { namespaces: { app: { models: { [model]: { fields: {}, relations: {}, storage: {} } } } }, }, @@ -56,7 +66,7 @@ afterEach(async () => { }); describe('db update with statements on MongoDB', () => { - it('fails planning with the statement the Mongo planner refuses', async () => { + it('plans the collection rename the statement states, and nothing that loses data', async () => { const dir = await mkdtemp(join(tmpdir(), 'mongo-db-update-statements-')); tempDirs.push(dir); const migrationsDir = join(dir, 'migrations'); @@ -91,21 +101,21 @@ describe('db update with statements on MongoDB', () => { }); expect(result).toMatchObject({ - ok: false, - failure: { - code: 'PLANNING_FAILED', - conflicts: [ - { - kind: 'statementRefused', - summary: expect.stringContaining('MongoDB does not apply rename statements'), - refusedStatement: { - kind: 'rename', - entity: 'model', - from: { namespaceId: 'app', model: 'Profile' }, - to: { namespaceId: 'app', model: 'User' }, - }, - }, + ok: true, + value: { + appliedStatements: [ + { statement: { entity: 'model', from: { model: 'Profile' } }, operationIndexes: [0] }, ], + dataLoss: [], + plan: { + operations: [ + { + id: 'collection.Profile.rename(User)', + label: 'Rename collection Profile to User', + operationClass: 'widening', + }, + ], + }, }, }); }); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts index 46c548a6ad51..6100bb55cc34 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts @@ -4,7 +4,7 @@ import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; import { MongoSchemaCollection, MongoSchemaIndex, MongoSchemaIR } from '@internal/mongo-schema-ir'; import { expectDataLossMatchesDestructive } from '@repo/test-utils/data-loss-expectations'; import { describe, expect, it } from 'vitest'; -import { keepDataByHand, MongoMigrationPlanner } from '../src/core/migrations/mongo-planner'; +import { MongoMigrationPlanner } from '../src/core/migrations/mongo-planner'; import type { PlannerProducedMongoMigration } from '../src/core/migrations/planner-produced-migration'; const ALL_CLASSES_POLICY: MigrationOperationPolicy = { @@ -143,22 +143,3 @@ describe('MongoDB planner, data loss', () => { ]); }); }); - -describe('keepDataByHand', () => { - it('says to rename the collection a dropped model stores its documents in', () => { - expect( - keepDataByHand( - { kind: 'model', namespaceId: asNamespaceId('__unbound__'), model: 'Event' }, - contractWith({ Event: 'events' }, 'from'), - ), - ).toBe( - 'If it was renamed, keep its documents instead: rename collection "events" by hand on each database before a plan that drops it is applied there, for example with db.getCollection("events").renameCollection("") in mongosh. db update then drops nothing; a migration written by migration plan still drops "events", so remove that operation from its migration.ts, or do not apply it where the collection was renamed.', - ); - }); - - it('names a collection no model stores by its name', () => { - expect(keepDataByHand({ kind: 'storage', name: 'audit' }, contractWith({}, 'from'))).toBe( - 'If it was renamed, keep its documents instead: rename collection "audit" by hand on each database before a plan that drops it is applied there, for example with db.getCollection("audit").renameCollection("") in mongosh. db update then drops nothing; a migration written by migration plan still drops "audit", so remove that operation from its migration.ts, or do not apply it where the collection was renamed.', - ); - }); -}); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts index 387c18e25a46..8b745cd26764 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts @@ -1,20 +1,58 @@ -import { asNamespaceId } from '@internal/contract/types'; +import { asNamespaceId, type ControlPolicy } from '@internal/contract/types'; import type { MigrationOperationPolicy, ResolvedMigrationStatement, } from '@internal/framework-components/control'; import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; -import { MongoSchemaCollection, MongoSchemaIR } from '@internal/mongo-schema-ir'; +import { + MongoSchemaCollection, + MongoSchemaCollectionOptions, + MongoSchemaIndex, + MongoSchemaIR, + MongoSchemaValidator, +} from '@internal/mongo-schema-ir'; import { describe, expect, it } from 'vitest'; import { MongoMigrationPlanner } from '../src/core/migrations/mongo-planner'; +import type { PlannerProducedMongoMigration } from '../src/core/migrations/planner-produced-migration'; const ALL_CLASSES_POLICY: MigrationOperationPolicy = { allowedOperationClasses: ['additive', 'widening', 'destructive', 'data'], }; -const NAMESPACE = asNamespaceId('__unbound__'); +const NS = asNamespaceId('__unbound__'); -function contractWithModel(model: string, collection: string): MongoContract { +interface ModelSpec { + readonly collection: string; + readonly fields: readonly string[]; + readonly relations?: readonly string[]; + readonly base?: string; + readonly discriminator?: string; + readonly variants?: Readonly>; +} + +interface CollectionSpec { + readonly unique?: readonly string[]; + readonly required?: readonly string[]; + readonly control?: ControlPolicy; +} + +function jsonSchema(fields: readonly string[]) { + return { + bsonType: 'object', + required: ['_id', ...fields], + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ...fields.map((field) => [field, { bsonType: 'string' }]), + ]), + additionalProperties: false, + }; +} + +function contract( + seed: string, + models: Readonly>, + collections: Readonly>, +): MongoContract { return { target: 'mongo', targetFamily: 'mongo', @@ -26,103 +64,573 @@ function contractWithModel(model: string, collection: string): MongoContract { domain: { namespaces: { __unbound__: { - models: { [model]: { fields: {}, relations: {}, storage: { collection } } }, + models: Object.fromEntries( + Object.entries(models).map(([name, spec]) => [ + name, + { + fields: Object.fromEntries(spec.fields.map((field) => [field, {}])), + relations: Object.fromEntries( + (spec.relations ?? []).map((relation) => [ + relation, + { + to: { model: 'Other', namespace: '__unbound__' }, + cardinality: 'N:1', + on: { localFields: ['otherId'], targetFields: ['_id'] }, + }, + ]), + ), + storage: { collection: spec.collection }, + ...(spec.base === undefined + ? {} + : { base: { namespace: '__unbound__', model: spec.base } }), + ...(spec.discriminator === undefined + ? {} + : { + discriminator: { field: spec.discriminator }, + variants: Object.fromEntries( + Object.entries(spec.variants ?? {}).map(([variant, value]) => [ + variant, + { value }, + ]), + ), + }), + }, + ]), + ), }, }, }, storage: { - storageHash: `storage-${collection}`, + storageHash: `storage-${seed}`, namespaces: { __unbound__: { id: '__unbound__', kind: 'mongo-namespace', - entries: { collection: { [collection]: new MongoCollection({}) } }, + entries: { + collection: Object.fromEntries( + Object.entries(collections).map(([name, spec]) => [ + name, + new MongoCollection({ + indexes: (spec.unique ?? []).map((field) => ({ + keys: [{ field, direction: 1 as const }], + unique: true, + })), + ...(spec.required === undefined + ? {} + : { + validator: { + jsonSchema: jsonSchema(spec.required), + validationLevel: 'strict' as const, + validationAction: 'error' as const, + }, + }), + ...(spec.control === undefined ? {} : { control: spec.control }), + }), + ]), + ), + }, }, }, }, } as unknown as MongoContract; } -const modelRename: ResolvedMigrationStatement = { - kind: 'rename', - entity: 'model', - from: { namespaceId: NAMESPACE, model: 'Profile' }, - to: { namespaceId: NAMESPACE, model: 'User' }, -}; - -const fieldRename: ResolvedMigrationStatement = { - kind: 'rename', - entity: 'field', - from: { namespaceId: NAMESPACE, model: 'Profile', field: 'name' }, - to: { namespaceId: NAMESPACE, model: 'User', field: 'fullName' }, -}; +function schemaCollection( + name: string, + spec: CollectionSpec & { readonly timeseries?: boolean } = {}, +): MongoSchemaCollection { + return new MongoSchemaCollection({ + name, + indexes: (spec.unique ?? []).map( + (field) => new MongoSchemaIndex({ keys: [{ field, direction: 1 }], unique: true }), + ), + ...(spec.required === undefined + ? {} + : { + validator: new MongoSchemaValidator({ + jsonSchema: jsonSchema(spec.required), + validationLevel: 'strict', + validationAction: 'error', + }), + }), + ...(spec.timeseries + ? { options: new MongoSchemaCollectionOptions({ timeseries: { timeField: 'at' } }) } + : {}), + }); +} -function plan( - statements: readonly ResolvedMigrationStatement[], - collections: { readonly from: string; readonly to: string } = { from: 'profiles', to: 'users' }, -) { +function plan(input: { + readonly from: MongoContract | null; + readonly to: MongoContract; + readonly schema: readonly MongoSchemaCollection[]; + readonly statements?: readonly ResolvedMigrationStatement[]; + readonly policy?: MigrationOperationPolicy; +}) { return new MongoMigrationPlanner().plan({ - contract: contractWithModel('User', collections.to), - schema: new MongoSchemaIR([new MongoSchemaCollection({ name: collections.from })]), - policy: ALL_CLASSES_POLICY, - fromContract: contractWithModel('Profile', collections.from), + contract: input.to, + schema: new MongoSchemaIR(input.schema), + policy: input.policy ?? ALL_CLASSES_POLICY, + fromContract: input.from, origin: null, - statements, + statements: input.statements ?? [], frameworkComponents: [], snapshotsImportPath: '../../snapshots', }); } -describe('MongoMigrationPlanner with statements', () => { - it('plans the rename as a collection drop and create when no statement is given', async () => { - const result = plan([]); - expect(result.kind).toBe('success'); - if (result.kind !== 'success') throw new Error('Expected success'); - const operations = await Promise.all(result.plan.operations); - expect(operations.map((operation) => operation.id)).toEqual([ - 'collection.users.create', - 'collection.profiles.drop', +function planned(result: ReturnType) { + if (result.kind !== 'success') throw new Error(JSON.stringify(result.conflicts, null, 2)); + const operations = (result.plan as PlannerProducedMongoMigration).operations; + return { + labels: operations.map((operation) => operation.label), + classes: operations.map((operation) => operation.operationClass), + applied: result.appliedStatements.map((applied) => applied.operationIndexes), + dataLoss: result.dataLoss, + }; +} + +function refusal(result: ReturnType) { + if (result.kind !== 'failure') throw new Error('Expected the plan to be refused'); + return result.conflicts.map(({ kind, summary, why }) => ({ kind, summary, why })); +} + +const renameModel = (from: string, to: string): ResolvedMigrationStatement => ({ + kind: 'rename', + entity: 'model', + from: { namespaceId: NS, model: from }, + to: { namespaceId: NS, model: to }, +}); + +const renameField = ( + fromModel: string, + from: string, + toModel: string, + to: string, +): ResolvedMigrationStatement => ({ + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: fromModel, field: from }, + to: { namespaceId: NS, model: toModel, field: to }, +}); + +const profiles = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, +); +const users = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'name'] } }, + { users: { unique: ['name'], required: ['name'] } }, +); +const profilesSchema = [schemaCollection('profiles', { unique: ['name'], required: ['name'] })]; + +describe('MongoMigrationPlanner, model statements', () => { + it('without a statement, drops the old collection and creates the new one', () => { + expect(planned(plan({ from: profiles, to: users, schema: profilesSchema })).labels).toEqual([ + 'Create collection users', + 'Drop index on profiles (name:1)', + 'Create index on users (name:1)', + 'Drop collection profiles', ]); - expect(result.appliedStatements).toEqual([]); }); - it('refuses the first statement, plans nothing, and says how to keep the documents', () => { - expect(plan([modelRename, fieldRename])).toEqual({ - kind: 'failure', - conflicts: [ - { - kind: 'statementRefused', - summary: - 'MongoDB does not apply rename statements in this release, so nothing was planned: rename model "Profile" to "User"', - why: 'MongoDB cannot carry out rename statements in this release. Without the statement, a plan drops collection "profiles" with its documents and creates collection "users". To keep the documents, rename the collection by hand on each database before a plan made without the statement is applied there, for example with db.getCollection("profiles").renameCollection("users") in mongosh. A migration written by migration plan without the statement still drops "profiles" wherever it is applied, so check its operations first.', - refusedStatement: modelRename, - }, + it('renames the collection, which keeps its indexes and validator, and reports the rename applied', () => { + expect( + planned( + plan({ + from: profiles, + to: users, + schema: profilesSchema, + statements: [renameModel('Profile', 'User')], + }), + ), + ).toEqual({ + labels: ['Rename collection profiles to users'], + classes: ['widening'], + applied: [[0]], + dataLoss: [], + }); + }); + + it('plans nothing for a model whose collection keeps its name', () => { + const to = contract( + 'to', + { User: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + planned( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameModel('Profile', 'User')], + }), + ), + ).toEqual({ labels: [], classes: [], applied: [[]], dataLoss: [] }); + }); +}); + +describe('MongoMigrationPlanner, field statements', () => { + const fullName = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'fullName'] } }, + { profiles: { unique: ['fullName'], required: ['fullName'] } }, + ); + + it('renames the field between the index drop and the index create, before the validator update', () => { + expect( + planned( + plan({ + from: profiles, + to: fullName, + schema: profilesSchema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ), + ).toEqual({ + labels: [ + 'Drop index on profiles (name:1)', + 'Rename field name to fullName in collection profiles', + 'Create index on profiles (fullName:1)', + 'Update validator on profiles (added: fullName; removed: name)', ], + classes: ['widening', 'widening', 'additive', 'widening'], + applied: [[1]], + dataLoss: [], }); }); - it('names a field through the model the destination contract names, and says how to move its values', () => { - const result = plan([fieldRename]); - if (result.kind !== 'failure') throw new Error('Expected failure'); - expect(result.conflicts).toMatchObject([ + it('renames the collection first, then the field in the renamed collection', () => { + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'fullName'] } }, + { users: { unique: ['fullName'], required: ['fullName'] } }, + ); + expect( + planned( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [ + renameModel('Profile', 'User'), + renameField('Profile', 'name', 'User', 'fullName'), + ], + }), + ), + ).toMatchObject({ + labels: [ + 'Rename collection profiles to users', + 'Drop index on users (name:1)', + 'Rename field name to fullName in collection users', + 'Create index on users (fullName:1)', + 'Update validator on users (added: fullName; removed: name)', + ], + applied: [[0], [2]], + dataLoss: [], + }); + }); + + it("renames a variant's field only in that variant's documents", () => { + const tasks = (bugField: string) => + contract( + bugField, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type'], + discriminator: 'type', + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: [bugField], base: 'Task' }, + Feature: { collection: 'tasks', fields: ['score'], base: 'Task' }, + }, + { tasks: {} }, + ); + expect( + planned( + plan({ + from: tasks('score'), + to: tasks('severity'), + schema: [schemaCollection('tasks')], + statements: [renameField('Bug', 'score', 'Bug', 'severity')], + }), + ).labels, + ).toEqual(['Rename field score to severity in collection tasks where {"type":"bug"}']); + }); + + it('plans nothing for a relation renamed to another relation, which documents do not store', () => { + const withRelation = (relation: string) => + contract( + relation, + { Profile: { collection: 'profiles', fields: ['_id', 'name'], relations: [relation] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + planned( + plan({ + from: withRelation('owner'), + to: withRelation('author'), + schema: profilesSchema, + statements: [renameField('Profile', 'owner', 'Profile', 'author')], + }), + ), + ).toEqual({ labels: [], classes: [], applied: [[]], dataLoss: [] }); + }); +}); + +describe('MongoMigrationPlanner, statements it refuses', () => { + it('refuses renaming _id', () => { + const to = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['key', 'name'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameField('Profile', '_id', 'Profile', 'key')], + }), + ), + ).toEqual([ { kind: 'statementRefused', - summary: expect.stringContaining('rename field "User.name" to "User.fullName"'), - why: 'MongoDB cannot carry out rename statements in this release. Without the statement, the documents in collection "profiles" keep their values under "name", and nothing moves them to "fullName". To move them, run these in mongosh on each database before a plan made without the statement is applied there, using the field names as they are stored. First turn off the collection\'s validator, which still requires "name": db.runCommand({ collMod: "profiles", validationLevel: "off" }). Then drop each unique index that includes "name", or the move fails once two documents have lost it, for example db.getCollection("profiles").dropIndex("name_1"). Then move the values: db.getCollection("profiles").updateMany({}, { $rename: { "name": "fullName" } }). Applying the plan then creates the indexes on "fullName" and turns the validator back on.', - refusedStatement: fieldRename, + summary: 'Cannot rename field "Profile._id" to "Profile.key": it renames _id', + why: 'MongoDB does not let a document change its _id. Leave out this statement.', }, ]); }); - it('says a plan without the statement keeps the documents when both models use one collection', () => { - const result = plan([modelRename], { from: 'posts', to: 'posts' }); - if (result.kind !== 'failure') throw new Error('Expected failure'); - expect(result.conflicts).toMatchObject([ - { - kind: 'statementRefused', - why: 'MongoDB cannot carry out rename statements in this release. Both models store their documents in collection "posts", so a plan made without the statement keeps them.', - refusedStatement: modelRename, - }, + it('refuses a stored field name MongoDB reads as a path or an operator', () => { + const to = (field: string) => + contract( + field, + { Profile: { collection: 'profiles', fields: ['_id', field] } }, + { profiles: {} }, + ); + expect( + ['a.b', '$a'].flatMap((field) => + refusal( + plan({ + from: to('name'), + to: to(field), + schema: [schemaCollection('profiles')], + statements: [renameField('Profile', 'name', 'Profile', field)], + }), + ).map(({ summary }) => summary), + ), + ).toEqual([ + 'Cannot rename field "Profile.name" to "Profile.a.b": the stored field name "a.b" cannot be rewritten', + 'Cannot rename field "Profile.name" to "Profile.$a": the stored field name "$a" cannot be rewritten', + ]); + }); + + it('refuses renaming a time-series collection, a collection the schema does not have, and onto a name it already has', () => { + const statements = [renameModel('Profile', 'User')]; + expect( + [ + [schemaCollection('profiles', { unique: ['name'], required: ['name'], timeseries: true })], + [], + [...profilesSchema, schemaCollection('users')], + ].flatMap((schema) => + refusal(plan({ from: profiles, to: users, schema, statements })).map( + ({ summary }) => summary, + ), + ), + ).toEqual([ + 'Cannot rename model "Profile" to "User": collection "profiles" is a time-series collection', + 'Cannot rename model "Profile" to "User": the schema being planned from has no collection "profiles"', + 'Cannot rename model "Profile" to "User": the schema being planned from already has a collection "users"', + ]); + }); + + it('refuses a collection name MongoDB rejects', () => { + const to = (collection: string) => + contract('to', { User: { collection, fields: ['_id', 'name'] } }, { [collection]: {} }); + expect( + ['a$b', 'system.users', 'a\0b'].flatMap((collection) => + refusal( + plan({ + from: profiles, + to: to(collection), + schema: profilesSchema, + statements: [renameModel('Profile', 'User')], + }), + ).map(({ summary }) => summary), + ), + ).toEqual([ + 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "a$b": it contains "$"', + 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "system.users": it starts with "system.", which MongoDB reserves', + 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "a\0b": it contains a null character', + ]); + }); + + it('refuses a field stored on one side of the statement only', () => { + const to = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'name'], relations: ['owner'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + const from = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'name', 'ownerName'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + refusal( + plan({ + from, + to, + schema: profilesSchema, + statements: [renameField('Profile', 'ownerName', 'Profile', 'owner')], + }), + ).map(({ summary }) => summary), + ).toEqual([ + 'Cannot rename field "Profile.ownerName" to "Profile.owner": the field is stored in documents on one side of the statement only', + ]); + }); + + it('refuses a statement on a collection whose control policy is not managed', () => { + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'name'] } }, + { users: { unique: ['name'], required: ['name'], control: 'external' } }, + ); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameModel('Profile', 'User')], + }), + ).map(({ summary }) => summary), + ).toEqual([ + 'Cannot rename model "Profile" to "User": collection "users" has control policy "external"', + ]); + }); + + it("refuses a field rename when the model's collection changes and no statement renames it", () => { + const to = contract( + 'to', + { Profile: { collection: 'users', fields: ['_id', 'fullName'] } }, + { users: {} }, + ); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ).map(({ summary }) => summary), + ).toEqual([ + 'Cannot rename field "Profile.name" to "Profile.fullName": the model\'s collection changes from "profiles" to "users", and no statement renames it', + ]); + }); +}); + +describe('MongoMigrationPlanner, removed fields', () => { + const withoutName = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'email'] } }, + { profiles: { required: ['email'] } }, + ); + const from = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'email', 'name'] } }, + { profiles: { required: ['email', 'name'] } }, + ); + const schema = [schemaCollection('profiles', { required: ['email', 'name'] })]; + + it('unsets a removed field from every document and reports the field as lost', () => { + expect(planned(plan({ from, to: withoutName, schema }))).toEqual({ + labels: [ + 'Remove field name from collection profiles', + 'Update validator on profiles (removed: name)', + ], + classes: ['destructive', 'widening'], + applied: [], + dataLoss: [ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Profile', field: 'name' }, + }, + ], + }); + }); + + it('names the field by its storage name when the origin contract is unknown', () => { + expect(planned(plan({ from: null, to: withoutName, schema })).dataLoss).toEqual([ + { operationIndex: 0, subject: { kind: 'storage', name: 'profiles.name' } }, ]); }); + + it('unsets nothing when the collection is dropped', () => { + const none = contract('to', {}, {}); + expect(planned(plan({ from, to: none, schema })).labels).toEqual(['Drop collection profiles']); + }); + + it("unsets a variant's removed field only from that variant's documents, and keeps a field another variant now stores", () => { + const tasks = ( + bugFields: readonly string[], + featureFields: readonly string[], + baseFields: readonly string[], + ) => + contract( + bugFields.join(), + { + Task: { + collection: 'tasks', + fields: ['_id', 'type', ...baseFields], + discriminator: 'type', + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: bugFields, base: 'Task' }, + Feature: { collection: 'tasks', fields: featureFields, base: 'Task' }, + }, + { tasks: {} }, + ); + expect( + planned( + plan({ + from: tasks(['score', 'severity'], ['score'], ['title']), + to: tasks(['severity'], ['score', 'title'], []), + schema: [schemaCollection('tasks')], + }), + ), + ).toMatchObject({ + labels: ['Remove field score from collection tasks where {"type":"bug"}'], + dataLoss: [ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Bug', field: 'score' }, + }, + ], + }); + }); + + it('unsets nothing for a field a statement renames', () => { + const fullName = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'email', 'fullName'] } }, + { profiles: { required: ['email', 'fullName'] } }, + ); + expect( + planned( + plan({ + from, + to: fullName, + schema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ).dataLoss, + ).toEqual([]); + }); }); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts index 6f2c1c167d05..23589dc23340 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts @@ -782,10 +782,13 @@ describe('MongoMigrationPlanner', () => { }), }), ]); - const [operation] = planSuccess(planner, contract, origin).operations; - expect(operation?.label).toBe( - 'Update validator on items (changed: name; added: note; removed: legacy)', + const labels = planSuccess(planner, contract, origin).operations.map( + (operation) => operation.label, ); + expect(labels).toEqual([ + 'Remove field legacy from collection items', + 'Update validator on items (changed: name; added: note; removed: legacy)', + ]); }); }); diff --git a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts index 202cc2113ef4..0a9f32922949 100644 --- a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts +++ b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts @@ -3,7 +3,7 @@ * * Dropping a collection loses its documents, so `migration plan` and `db update` ask what the drop means; `--delete ` answers. The MongoDB planner refuses rename statements in this release. * - * Journey S1 removes `Event` through `migration plan`: refused without `--delete Event`, planned with it, and applied by `migrate`. Journey S2 does the same through `db update`, where `--confirm ` no longer consents. Journey S3 adds a required field with no refusal. Journey S4 gives a rename, which the planner refuses with `statementRefused`. + * Journey S1 removes `Event` through `migration plan`: refused without `--delete Event`, planned with it, and applied by `migrate`. Journey S2 does the same through `db update`, where `--confirm ` no longer consents. Journey S3 adds a required field with no refusal. Journey S4 gives a rename, which renames the collection and keeps its documents. */ import { copyFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; @@ -278,13 +278,14 @@ describe('Journeys (MongoDB): statements that consent to data loss', { ]); }); - it('S4: the planner refuses a rename statement', async () => { + it('S4: a rename statement renames the collection and keeps its documents', async () => { const dbName = 'mongo_delete_rename'; const ctx = setupProject(withDatabase(replSet.getUri(), dbName)); created.add(ctx.testDir); await emit(ctx, 'S4.01'); const create = await runDbUpdate(ctx, ['--no-interactive', '--json']); expect(create.exitCode, `S4.02: db update creates the collections: ${create.stderr}`).toBe(0); + await client.db(dbName).collection('events').insertOne({ name: 'launch' }); useFixture(ctx, 'contract-events-renamed.ts'); await emit(ctx, 'S4.03'); @@ -294,12 +295,16 @@ describe('Journeys (MongoDB): statements that consent to data loss', { 'Event:Occasion', '--json', ]); - expect(renamed.exitCode, 'S4.04: the rename is refused').not.toBe(0); - const refusal = engineError(renamed); - expect(refusal?.code, 'S4.04: planning failed').toBe('MIGRATION.PLANNING_FAILED'); - expect(JSON.stringify(refusal?.meta), 'S4.04: the conflict is the refused statement').toContain( - 'statementRefused', - ); - expect(await collections(dbName), 'S4.04: nothing changed').toEqual(['events', 'users']); + expect(renamed.exitCode, `S4.04: the rename runs: ${renamed.stderr}`).toBe(0); + expect(await collections(dbName), 'S4.04: the collection is renamed').toEqual([ + 'occasions', + 'users', + ]); + const documents = await client + .db(dbName) + .collection('occasions') + .find({}, { projection: { _id: 0 } }) + .toArray(); + expect(documents, 'S4.04: its documents are kept').toEqual([{ name: 'launch' }]); }); }); diff --git a/test/integration/test/cli-journeys/mongo-db-update-consent.e2e.test.ts b/test/integration/test/cli-journeys/mongo-db-update-consent.e2e.test.ts index 3fd52b2730c4..5fada18d872c 100644 --- a/test/integration/test/cli-journeys/mongo-db-update-consent.e2e.test.ts +++ b/test/integration/test/cli-journeys/mongo-db-update-consent.e2e.test.ts @@ -163,7 +163,7 @@ describe('Journey: Mongo db update asks for consent only to lose data', { expect(unconfirmed.exitCode).toBe(2); expect(parseJsonOutput(unconfirmed)).toMatchObject({ code: 'CLI.CONSENT_REQUIRED', - meta: { unanswered: [{ subject: 'Event', verbs: ['delete'] }] }, + meta: { unanswered: [{ subject: 'Event', verbs: ['rename', 'delete'] }] }, }); const kept = await client.db(dbName).listCollections({ name: 'events' }).toArray(); expect(kept.map(({ name }) => name)).toEqual(['events']); diff --git a/test/integration/test/mongo/target-runner/runner-statements.test.ts b/test/integration/test/mongo/target-runner/runner-statements.test.ts new file mode 100644 index 000000000000..5c4b3feda32f --- /dev/null +++ b/test/integration/test/mongo/target-runner/runner-statements.test.ts @@ -0,0 +1,360 @@ +import { introspectSchema, MongoControlAdapterImpl } from '@internal/adapter-mongo/control'; +import { asNamespaceId } from '@internal/contract/types'; +import { MongoControlDriver } from '@internal/driver-mongo/control'; +import type { + MigrationOperationPolicy, + MigrationPlan, + ResolvedMigrationStatement, +} from '@internal/framework-components/control'; +import { buildFabricatedMigrationEdge } from '@internal/migration-tools/aggregate'; +import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; +import type { AnyMongoMigrationOperation } from '@internal/mongo-query-ast/control'; +import { + MongoMigrationPlanner, + MongoMigrationRunner, + serializeMongoOps, +} from '@internal/target-mongo/control'; +import { timeouts } from '@repo/test-utils'; +import { type Db, MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest'; + +const controlAdapter = new MongoControlAdapterImpl(); +const NS = asNamespaceId('__unbound__'); +const policy: MigrationOperationPolicy = { + allowedOperationClasses: ['additive', 'widening', 'destructive'], +}; + +let replSet: MongoMemoryReplSet; +let client: MongoClient; +let db: Db; +const dbName = 'runner_statements_test'; + +beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + db = client.db(dbName); +}, timeouts.spinUpMongoMemoryServer); + +afterAll(async () => { + await client?.close(); + await replSet?.stop(); +}, timeouts.spinUpMongoMemoryServer); + +beforeEach(async () => { + const collections = await db.listCollections().toArray(); + for (const col of collections) { + await db.dropCollection(col['name'] as string); + } +}); + +function closedSchema(fields: readonly string[]) { + return { + bsonType: 'object', + required: ['_id', ...fields], + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ...fields.map((field) => [field, { bsonType: 'string' }]), + ]), + additionalProperties: false, + }; +} + +interface ModelSpec { + readonly collection: string; + readonly fields: readonly string[]; + readonly base?: string; + readonly variants?: Readonly>; +} + +function contract( + storageHash: string, + models: Readonly>, + collections: Readonly>, +): MongoContract { + return { + target: 'mongo', + targetFamily: 'mongo', + profileHash: 'p1', + capabilities: {}, + extensions: {}, + meta: {}, + roots: {}, + domain: { + namespaces: { + __unbound__: { + models: Object.fromEntries( + Object.entries(models).map(([name, spec]) => [ + name, + { + fields: Object.fromEntries(spec.fields.map((field) => [field, {}])), + relations: {}, + storage: { collection: spec.collection }, + ...(spec.base === undefined + ? {} + : { base: { namespace: '__unbound__', model: spec.base } }), + ...(spec.variants === undefined + ? {} + : { + discriminator: { field: 'type' }, + variants: Object.fromEntries( + Object.entries(spec.variants).map(([variant, value]) => [ + variant, + { value }, + ]), + ), + }), + }, + ]), + ), + }, + }, + }, + storage: { + storageHash, + namespaces: { + __unbound__: { + id: '__unbound__', + kind: 'mongo-namespace', + entries: { collection: collections }, + }, + }, + }, + } as unknown as MongoContract; +} + +function closedCollection(fields: readonly string[], unique: readonly string[] = []) { + return new MongoCollection({ + indexes: unique.map((field) => ({ keys: [{ field, direction: 1 as const }], unique: true })), + validator: { + jsonSchema: closedSchema(fields), + validationLevel: 'strict', + validationAction: 'error', + }, + }); +} + +async function createClosed( + name: string, + fields: readonly string[], + unique: readonly string[], + documents: readonly Record[], +) { + await db.createCollection(name, { + validator: { $jsonSchema: closedSchema(fields) }, + validationLevel: 'strict', + validationAction: 'error', + }); + for (const field of unique) { + await db.collection(name).createIndex({ [field]: 1 }, { unique: true, name: `${field}_1` }); + } + await db.collection(name).insertMany(documents.map((document) => ({ ...document }))); +} + +async function planAndRun( + from: MongoContract, + to: MongoContract, + statements: readonly ResolvedMigrationStatement[] = [], +) { + await controlAdapter.initMarker(new MongoControlDriver(db, client), 'app', { + storageHash: from.storage.storageHash, + profileHash: 'p1', + }); + const planned = new MongoMigrationPlanner().plan({ + contract: to, + schema: await introspectSchema(db), + policy, + fromContract: from, + origin: { storageHash: from.storage.storageHash }, + statements, + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + if (planned.kind !== 'success') throw new Error(JSON.stringify(planned.conflicts, null, 2)); + const operations = planned.plan.operations as readonly AnyMongoMigrationOperation[]; + const plan: MigrationPlan = { + targetId: 'mongo', + operations: JSON.parse(serializeMongoOps(operations)), + origin: { storageHash: from.storage.storageHash }, + destination: { storageHash: to.storage.storageHash }, + }; + const result = await new MongoMigrationRunner( + controlAdapter.createRunnerDependencies(new MongoControlDriver(db, client)), + ).execute({ + plan, + migrationEdges: [ + buildFabricatedMigrationEdge({ + currentMarkerStorageHash: from.storage.storageHash, + destinationStorageHash: to.storage.storageHash, + operationCount: operations.length, + }), + ], + destinationContract: to, + policy, + frameworkComponents: [], + }); + return { planned, result }; +} + +async function furtherPlanLabels(to: MongoContract): Promise { + const planned = new MongoMigrationPlanner().plan({ + contract: to, + schema: await introspectSchema(db), + policy, + fromContract: to, + origin: { storageHash: to.storage.storageHash }, + statements: [], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + if (planned.kind !== 'success') throw new Error(JSON.stringify(planned.conflicts)); + return planned.plan.operations.map((operation) => operation.label); +} + +async function documents(name: string) { + return db + .collection(name) + .find({}, { projection: { _id: 0 }, sort: { _id: 1 } }) + .toArray(); +} + +describe('MongoMigrationPlanner and runner - statements', () => { + it( + 'renames a model and a uniquely indexed field behind a closed validator, keeping every document', + async () => { + await createClosed('profiles', ['name'], ['name'], [{ name: 'Ada' }, { name: 'Bo' }]); + const from = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: closedCollection(['name'], ['name']) }, + ); + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'fullName'] } }, + { users: closedCollection(['fullName'], ['fullName']) }, + ); + + const { planned, result } = await planAndRun(from, to, [ + { + kind: 'rename', + entity: 'model', + from: { namespaceId: NS, model: 'Profile' }, + to: { namespaceId: NS, model: 'User' }, + }, + { + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: 'Profile', field: 'name' }, + to: { namespaceId: NS, model: 'User', field: 'fullName' }, + }, + ]); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(5); + expect(planned.kind === 'success' ? planned.dataLoss : undefined).toEqual([]); + expect({ + collections: ( + await db.listCollections({ name: { $in: ['profiles', 'users'] } }).toArray() + ).map((c) => c.name), + users: await documents('users'), + further: await furtherPlanLabels(to), + }).toEqual({ + collections: ['users'], + users: [{ fullName: 'Ada' }, { fullName: 'Bo' }], + further: [], + }); + await expect( + db.collection('users').updateOne({ fullName: 'Ada' }, { $set: { fullName: 'Ada L.' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect(db.collection('users').insertOne({ fullName: 'Bo' })).rejects.toThrow( + /duplicate key/, + ); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'removes a field the destination no longer has from every document', + async () => { + await createClosed( + 'users', + ['email', 'nickname'], + [], + [ + { email: 'a@x', nickname: 'a' }, + { email: 'b@x', nickname: 'b' }, + ], + ); + const from = contract( + 'from', + { User: { collection: 'users', fields: ['_id', 'email', 'nickname'] } }, + { users: closedCollection(['email', 'nickname']) }, + ); + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'email'] } }, + { users: closedCollection(['email']) }, + ); + + const { planned, result } = await planAndRun(from, to); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); + expect(planned.kind === 'success' ? planned.dataLoss : undefined).toEqual([ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'User', field: 'nickname' }, + }, + ]); + expect({ users: await documents('users'), further: await furtherPlanLabels(to) }).toEqual({ + users: [{ email: 'a@x' }, { email: 'b@x' }], + further: [], + }); + await expect( + db.collection('users').updateOne({ email: 'a@x' }, { $set: { email: 'a@y' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + "renames a variant's field only in its own documents, so another variant's field of that name keeps its values", + async () => { + await db.collection('tasks').insertMany([ + { type: 'bug', score: 3 }, + { type: 'feature', score: 5 }, + ]); + const tasks = (storageHash: string, bugField: string) => + contract( + storageHash, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type'], + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: [bugField], base: 'Task' }, + Feature: { collection: 'tasks', fields: ['score'], base: 'Task' }, + }, + { tasks: new MongoCollection({}) }, + ); + + const { result } = await planAndRun(tasks('from', 'score'), tasks('to', 'severity'), [ + { + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: 'Bug', field: 'score' }, + to: { namespaceId: NS, model: 'Bug', field: 'severity' }, + }, + ]); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect(await documents('tasks')).toEqual([ + { type: 'bug', severity: 3 }, + { type: 'feature', score: 5 }, + ]); + }, + timeouts.spinUpMongoMemoryServer, + ); +}); From 50e0e32f7a4a8c4791ebbba636d2ebd6f0c94165 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 11:14:54 +0200 Subject: [PATCH 12/57] TML-3478: the statements runner test reads operation labels with their type Signed-off-by: willbot Signed-off-by: Will Madden --- .../test/mongo/target-runner/runner-statements.test.ts | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/test/integration/test/mongo/target-runner/runner-statements.test.ts b/test/integration/test/mongo/target-runner/runner-statements.test.ts index 5c4b3feda32f..3bd4862d5ff2 100644 --- a/test/integration/test/mongo/target-runner/runner-statements.test.ts +++ b/test/integration/test/mongo/target-runner/runner-statements.test.ts @@ -211,7 +211,9 @@ async function furtherPlanLabels(to: MongoContract): Promise snapshotsImportPath: '../../snapshots', }); if (planned.kind !== 'success') throw new Error(JSON.stringify(planned.conflicts)); - return planned.plan.operations.map((operation) => operation.label); + return (planned.plan.operations as readonly AnyMongoMigrationOperation[]).map( + (operation) => operation.label, + ); } async function documents(name: string) { From 7f47aa52720ec654c15c84cceafb67ff5e55f8aa Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 11:29:58 +0200 Subject: [PATCH 13/57] TML-3478: removed MongoDB fields leave no document the validator rejects, and every printed advice works A base field moved to one variant is now unset from the documents of every other variant, filtered by discriminator $nin, and asked about as a field. Without an origin contract, a destination collection with no validator says nothing about removed fields and is skipped. A field question whose model the destination names differently offers only delete and says to pass the model rename first. The storage question says --delete loses that data on both families. The resolver reads Model.a.b as the field "a.b" when no namespace is named Model, so the planner names the real reason. Refusal advice that failed when followed is reworded. Signed-off-by: willbot Signed-off-by: Will Madden --- .../control-api/statements/plan-questions.ts | 36 ++++- .../statements/resolve-statements.ts | 8 +- .../cli/test/control-api/db-update.test.ts | 2 +- .../statements/ask-plan-questions.test.ts | 4 +- .../statements/data-loss-question.test.ts | 47 ++++++- .../resolve-field-statements.test.ts | 18 +++ .../migrations/mongo-statement-planning.ts | 118 ++++++++++------ .../test/mongo-planner.statements.test.ts | 129 ++++++++++++++---- projects/migration-statements/deferred.md | 12 ++ .../target-runner/runner-statements.test.ts | 78 +++++++++++ 10 files changed, 381 insertions(+), 71 deletions(-) diff --git a/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts b/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts index 19b667346ae3..94db5cbbee63 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts @@ -145,12 +145,12 @@ function lossText(subject: MigrationSubject, text: string, originKnown: boolean) case 'storage': return originKnown ? `would lose the data in "${text}", which no model of the origin contract stores` - : `would lose the data in "${text}", named by its storage name because the origin contract is unknown; --delete loses its rows`; + : `would lose the data in "${text}", named by its storage name because the origin contract is unknown; --delete loses that data`; } } /** - * How to keep the rows of a storage subject that was renamed, when the origin contract is unknown: + * How to keep the data of a storage subject that was renamed, when the origin contract is unknown: * store the snapshot of the contract the database is at, as `MIGRATION.STATEMENT_ORIGIN_UNKNOWN` * advises, then answer with a rename. The refusal gives it once, on its first storage question. The * engine does not put the CLI's name in a question's text, so the steps name `prisma`. @@ -174,6 +174,11 @@ function namesSubject(verb: ConsentVerb, subject: string) { text === subject ? undefined : `${verb} names "${subject}": --${verb} ${subject}.`; } +function modelRenameAdvice(model: ModelCoordinate, contracts: StatementContracts): string { + const name = modelText(contracts.origin, model); + return `The destination contract has no model "${name}"; to keep the values instead, also pass --rename ${name}:, naming the model that stores them now, and the question then offers to rename the field.`; +} + /** A rename as a statement is written: a field's new name keeps the model, `User.nickname:User.`. */ function renameForm(subject: MigrationSubject, text: string): string { return subject.kind === 'field' @@ -208,12 +213,30 @@ function inDestination( return subject.kind === 'model' || Object.hasOwn(destinationModel.fields, subject.field); } +/** + * The model of a field subject, under the name a rename of the plan gives it, when the destination + * has no model of that name: a rename of the field cannot be written until a statement renames the + * model. + */ +function missingFieldModel( + subject: MigrationSubject, + contracts: StatementContracts & { readonly destination: ContractWithDomain }, +): ModelCoordinate | undefined { + if (subject.kind !== 'field') return undefined; + const model = renamedModel(subject, contracts) ?? subject; + return contracts.destination.domain.namespaces[model.namespaceId]?.models[model.model] === + undefined + ? model + : undefined; +} + /** * The question for one operation that would lose data. A model or field the destination no longer * has, and no rename of the plan renamed, may be renamed instead of deleted; a rename is resolved * against the two contracts after the plan's other renames, and its old name must be the subject. * A subject the destination keeps, such as a field whose type changes, and data no model stores, - * can only be deleted. + * can only be deleted. A field whose model the destination names differently can be renamed only + * after a statement renames the model, so the question says to pass one. */ export function dataLossQuestion( loss: PlannedSubject, @@ -223,13 +246,18 @@ export function dataLossQuestion( }, ): PlanQuestion { const subject = subjectText(loss.subject, contracts); + const missingModel = missingFieldModel(loss.subject, contracts); const renamable = + missingModel === undefined && loss.subject.kind !== 'storage' && !inDestination(loss.subject, contracts) && !renamedByPlan(loss.subject, contracts); const deleteNames = namesSubject('delete', subject); return { - question: `${loss.label} ${lossText(loss.subject, subject, contracts.originKnown)}.`, + question: [ + `${loss.label} ${lossText(loss.subject, subject, contracts.originKnown)}.`, + ...(missingModel === undefined ? [] : [modelRenameAdvice(missingModel, contracts)]), + ].join(' '), subject, verbs: renamable ? ['rename', 'delete'] : ['delete'], forms: renamable ? { rename: renameForm(loss.subject, subject) } : {}, diff --git a/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts b/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts index 8105d8d9cc3b..339e539685cc 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts @@ -356,7 +356,13 @@ function readSide( segments: StatementSide, ): Result { if (segments.length === 1) return readModel(state, side, undefined, segments[0]); - if (segments.length === 3) return readField(state, side, ...segments); + if (segments.length === 3) { + const [first, second, third] = segments; + const qualified = readField(state, side, first, second, third); + if (qualified.ok) return qualified; + const dotted = readField(state, side, undefined, first, `${second}.${third}`); + return dotted.ok ? dotted : qualified; + } const [first, second] = segments; const asModel = readModel(state, side, first, second); const asField = readField(state, side, undefined, first, second); diff --git a/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts index b82f805b80e7..9dd6a7de515e 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts @@ -579,7 +579,7 @@ describe('executeDbUpdate', () => { expect(asked).toEqual([ { - question: `Drop column nickname from user would lose the data in "user.nickname", named by its storage name because the origin contract is unknown; --delete loses its rows. ${ORIGIN_SNAPSHOT_RECOVERY}`, + question: `Drop column nickname from user would lose the data in "user.nickname", named by its storage name because the origin contract is unknown; --delete loses that data. ${ORIGIN_SNAPSHOT_RECOVERY}`, subject: 'user.nickname', verbs: ['delete'], }, diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts index 3bc6d5633d23..87ec3a153bff 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts @@ -109,8 +109,8 @@ describe('questions about storage when the origin contract is unknown', () => { }); expect(asked).toEqual([ - `Drop table Legacy would lose the data in "Legacy", named by its storage name because the origin contract is unknown; --delete loses its rows. ${ORIGIN_SNAPSHOT_RECOVERY}`, - 'Drop table Profile would lose the data in "Profile", named by its storage name because the origin contract is unknown; --delete loses its rows.', + `Drop table Legacy would lose the data in "Legacy", named by its storage name because the origin contract is unknown; --delete loses that data. ${ORIGIN_SNAPSHOT_RECOVERY}`, + 'Drop table Profile would lose the data in "Profile", named by its storage name because the origin contract is unknown; --delete loses that data.', ]); expect(ORIGIN_SNAPSHOT_RECOVERY).toContain('--dry-run'); expect(ORIGIN_SNAPSHOT_RECOVERY).toContain('`prisma db update --advance-ref --dry-run`'); diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts index 4487922e46ea..bdf3b335ad5d 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts @@ -36,7 +36,7 @@ describe('dataLossQuestion for data no model names', () => { }); expect(question.question).toBe( - 'Drop column age from user would lose the data in "public.user.age", named by its storage name because the origin contract is unknown; --delete loses its rows.', + 'Drop column age from user would lose the data in "public.user.age", named by its storage name because the origin contract is unknown; --delete loses that data.', ); }); }); @@ -95,3 +95,48 @@ describe('dataLossQuestion verbs', () => { }); }); }); + +describe('dataLossQuestion for a field whose model the destination names differently', () => { + const app = asNamespaceId('app'); + const profile = contractOf({ app: { models: { Profile: { fields: ['email', 'nickname'] } } } }); + const person = contractOf({ app: { models: { Person: { fields: ['email', 'alias'] } } } }); + const loss = { + operationIndex: 0, + label: 'Remove field nickname from collection profiles', + subject: { kind: 'field', namespaceId: app, model: 'Profile', field: 'nickname' }, + } as const; + + it('offers only delete, and says to rename the model first', () => { + const question = dataLossQuestion(loss, { + origin: profile, + destination: person, + renames: [], + originKnown: true, + }); + expect({ verbs: question.verbs, question: question.question }).toEqual({ + verbs: ['delete'], + question: + 'Remove field nickname from collection profiles would lose the values of field "Profile.nickname". The destination contract has no model "Profile"; to keep the values instead, also pass --rename Profile:, naming the model that stores them now, and the question then offers to rename the field.', + }); + }); + + it('offers the field rename once the printed model rename is passed, and accepts it', () => { + const question = dataLossQuestion(loss, { + origin: profile, + destination: person, + renames: [{ verb: 'rename', text: 'Profile:Person' }], + originKnown: true, + }); + expect({ + subject: question.subject, + verbs: question.verbs, + forms: question.forms, + accepted: question.validate('rename', 'Person.nickname:Person.alias'), + }).toEqual({ + subject: 'Person.nickname', + verbs: ['rename', 'delete'], + forms: { rename: 'Person.nickname:Person.' }, + accepted: undefined, + }); + }); +}); diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts index e5df619d2571..a084eff32c3e 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts @@ -274,3 +274,21 @@ describe('resolveStatements, field renames', () => { }); }); }); + +describe('a field whose stored name holds a dot', () => { + it('reads Model.a.b as the field "a.b" when no namespace is named "Model"', () => { + const result = resolve( + ['User.o.d:User.od'], + contractOf({ __unbound__: { models: { User: { fields: ['_id', 'o.d'] } } } }), + contractOf({ __unbound__: { models: { User: { fields: ['_id', 'od'] } } } }), + ); + expect(expectValue(result)).toEqual([ + { + kind: 'rename', + entity: 'field', + from: { namespaceId: '__unbound__', model: 'User', field: 'o.d' }, + to: { namespaceId: '__unbound__', model: 'User', field: 'od' }, + }, + ]); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index 9e1452afd9ab..252802b37a59 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -11,6 +11,7 @@ import { type MigrationPlannerConflict, type MigrationSubject, type ModelCoordinate, + modelDisplayName, type ResolvedFieldRenameStatement, type ResolvedMigrationStatement, type ResolvedModelRenameStatement, @@ -240,14 +241,14 @@ function planModelStatement( if (source.options?.timeseries !== undefined) { return refuse( `collection "${from}" is a time-series collection`, - `MongoDB cannot rename a time-series collection. Copy its documents into the new collection yourself, or ${LEAVE_OUT.toLowerCase()}`, + `MongoDB cannot rename a time-series collection. Leave out this statement. To keep the documents, first create collection "${to}" yourself with the same time-series options and copy them into it; the plan then drops "${from}" once --delete names the model.`, ); } const invalid = invalidCollectionNameReason(to); if (invalid !== undefined) { return refuse( `MongoDB does not accept the collection name "${to}": ${invalid}`, - `Give the model a collection name MongoDB accepts with @@map, or ${LEAVE_OUT.toLowerCase()}`, + 'Give the model a collection name MongoDB accepts with @@map, then run this command again.', ); } if (schema.collection(to) !== undefined) { @@ -284,27 +285,30 @@ function planFieldStatement( `A relation is not stored in documents, so a statement cannot rename a field to or from one. ${LEAVE_OUT}`, ); } - for (const field of [statement.from.field, statement.to.field]) { - if (field === '_id') { - return refuse( - 'it renames _id', - `MongoDB does not let a document change its _id. ${LEAVE_OUT}`, - ); - } - const unsafe = unsafeFieldNameReason(field); - if (unsafe !== undefined) { - return refuse( - `the stored field name "${field}" cannot be rewritten`, - `${unsafe}. Give the field a stored name without "." and not starting with "$", or ${LEAVE_OUT.toLowerCase()}`, - ); - } + if (statement.from.field === '_id' || statement.to.field === '_id') { + return refuse('it renames _id', `MongoDB does not let a document change its _id. ${LEAVE_OUT}`); + } + const unsafeFrom = unsafeFieldNameReason(statement.from.field); + if (unsafeFrom !== undefined) { + return refuse( + `the stored field name "${statement.from.field}" cannot be rewritten`, + `${unsafeFrom}, so a statement cannot move its values. Rename it in the database yourself, or ${LEAVE_OUT.toLowerCase()}`, + ); + } + const unsafeTo = unsafeFieldNameReason(statement.to.field); + if (unsafeTo !== undefined) { + return refuse( + `the stored field name "${statement.to.field}" cannot be written`, + `${unsafeTo}. Give the field a stored name without "." that does not start with "$" (with @map), or ${LEAVE_OUT.toLowerCase()}`, + ); } const originCollection = collectionOf(input.fromContract, statement.from); const collection = renamedCollections.get(originCollection) ?? originCollection; - if (collection !== collectionOf(input.contract, statement.to)) { + const destinationCollection = collectionOf(input.contract, statement.to); + if (collection !== destinationCollection) { return refuse( - `the model's collection changes from "${originCollection}" to "${collectionOf(input.contract, statement.to)}", and no statement renames it`, - `Add a statement that renames the model, so its collection is renamed with its documents, or ${LEAVE_OUT.toLowerCase()}`, + `the model's collection changes from "${originCollection}" to "${destinationCollection}", and no statement renames it`, + collectionChangeAdvice(statement, originCollection, destinationCollection), ); } if (statement.from.field === statement.to.field) return { kind: 'unchanged' }; @@ -321,6 +325,17 @@ function planFieldStatement( }; } +function collectionChangeAdvice( + statement: ResolvedFieldRenameStatement, + from: string, + to: string, +): string { + if (!sameModel(statement.from, statement.to)) { + return `Add --rename ${modelDisplayName(statement.from)}:${modelDisplayName(statement.to)} before this statement, so the collection is renamed with its documents.`; + } + return `A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(statement.from)} to answer its question, then in its migration.ts replace the drop of "${from}" and the create of "${to}" with renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}), imported with the other operations. Then change the field and run this statement again.`; +} + export interface FieldRemoval { readonly call: UnsetFieldCall; readonly subject: MigrationSubject; @@ -394,20 +409,35 @@ function storedFieldsOf(contract: ContractWithDomain, coordinate: ModelCoordinat return new Set([...Object.keys(model.fields), ...Object.keys(base?.fields ?? {})]); } -function fieldsDeclaredIn(contract: ContractWithDomain, collection: string): Set { - return new Set( - modelCoordinates(contract) - .filter((coordinate) => collectionOf(contract, coordinate) === collection) - .flatMap((coordinate) => Object.keys(modelAt(contract, coordinate)?.fields ?? {})), - ); +/** + * The documents of a root model that lose `field`: all of them, or, when some of its destination + * variants now store the field themselves, those whose discriminator value is none of theirs. + * `undefined` when the root model itself still stores it. + */ +function rootDocumentsLosing( + contract: ContractWithDomain, + root: ModelCoordinate, + field: string, +): { readonly filter: Document | undefined } | undefined { + const model = modelAt(contract, root); + if (model === undefined || Object.hasOwn(model.fields, field)) return undefined; + const discriminator = model.discriminator?.field; + const keeping = Object.entries(model.variants ?? {}) + .filter(([variant]) => + Object.hasOwn(modelAt(contract, { ...root, model: variant })?.fields ?? {}, field), + ) + .map(([, entry]) => entry.value) + .sort(); + if (discriminator === undefined || keeping.length === 0) return { filter: undefined }; + return { filter: { [discriminator]: { $nin: keeping } } }; } /** * Each field an origin model stores that its destination model no longer stores, in a collection * the plan keeps: removing it is data loss, carried out by unsetting it from the documents that - * hold it. A variant's field is unset only from that variant's documents. A field the base no - * longer stores is unset only when no model of the collection stores it any more, since a variant - * may now store it. + * hold it. A variant's field is unset only from that variant's documents, unless its destination + * base now stores it. A base field is unset from the documents of every variant that does not now + * store it itself. */ export function planFieldRemovals(input: FieldRemovalInput): readonly FieldRemoval[] { return modelCoordinates(input.fromContract).flatMap((origin) => { @@ -417,24 +447,34 @@ export function planFieldRemovals(input: FieldRemovalInput): readonly FieldRemov if (destination === undefined) return []; const model = modelAt(input.fromContract, origin); if (model === undefined) return []; - const filter = variantFilter(input.fromContract, origin); - const stillStored = - filter === undefined - ? fieldsDeclaredIn(input.contract, collection) - : storedFieldsOf(input.contract, destination); + const variant = variantFilter(input.fromContract, origin); + const losing = (field: string): { readonly filter: Document | undefined } | undefined => { + if (variant !== undefined) { + return storedFieldsOf(input.contract, destination).has(field) + ? undefined + : { filter: variant }; + } + return rootDocumentsLosing(input.contract, destination, field); + }; return Object.keys(model.fields) .filter( (field) => field !== '_id' && unsafeFieldNameReason(field) === undefined && - !stillStored.has(field) && !renamedByStatement(input, { ...origin, field }), ) .sort() - .map((field) => ({ - call: new UnsetFieldCall(collection, field, filter), - subject: { kind: 'field' as const, ...origin, field }, - })); + .flatMap((field) => { + const lost = losing(field); + return lost === undefined + ? [] + : [ + { + call: new UnsetFieldCall(collection, field, lost.filter), + subject: { kind: 'field' as const, ...origin, field }, + }, + ]; + }); }); } @@ -462,7 +502,7 @@ export function planValidatorFieldRemovals(input: { }): readonly FieldRemoval[] { return input.origin.collections.flatMap((originCollection) => { const destination = input.destination.collection(originCollection.name); - if (destination === undefined || originCollection.validator === undefined) return []; + if (destination?.validator === undefined || originCollection.validator === undefined) return []; const kept = new Set([ ...propertyNames(destination.validator?.jsonSchema), ...branchPropertyNames(destination.validator?.jsonSchema), diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts index 8b745cd26764..4d915cc2a278 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts @@ -406,6 +406,32 @@ describe('MongoMigrationPlanner, statements it refuses', () => { ]); }); + it('refuses renaming a stored field whose name MongoDB reads as a path, and says to rename it by hand', () => { + const named = (field: string) => + contract( + field, + { Profile: { collection: 'profiles', fields: ['_id', field] } }, + { profiles: {} }, + ); + expect( + refusal( + plan({ + from: named('o.d'), + to: named('od'), + schema: [schemaCollection('profiles')], + statements: [renameField('Profile', 'o.d', 'Profile', 'od')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: + 'Cannot rename field "Profile.o.d" to "Profile.od": the stored field name "o.d" cannot be rewritten', + why: 'MongoDB reads a name containing "." as a path into embedded documents, so a statement cannot move its values. Rename it in the database yourself, or leave out this statement.', + }, + ]); + }); + it('refuses a stored field name MongoDB reads as a path or an operator', () => { const to = (field: string) => contract( @@ -425,8 +451,8 @@ describe('MongoMigrationPlanner, statements it refuses', () => { ).map(({ summary }) => summary), ), ).toEqual([ - 'Cannot rename field "Profile.name" to "Profile.a.b": the stored field name "a.b" cannot be rewritten', - 'Cannot rename field "Profile.name" to "Profile.$a": the stored field name "$a" cannot be rewritten', + 'Cannot rename field "Profile.name" to "Profile.a.b": the stored field name "a.b" cannot be written', + 'Cannot rename field "Profile.name" to "Profile.$a": the stored field name "$a" cannot be written', ]); }); @@ -533,6 +559,18 @@ describe('MongoMigrationPlanner, statements it refuses', () => { ).toEqual([ 'Cannot rename field "Profile.name" to "Profile.fullName": the model\'s collection changes from "profiles" to "users", and no statement renames it', ]); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + )[0]?.why, + ).toBe( + 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question, then in its migration.ts replace the drop of "profiles" and the create of "users" with renameCollection("profiles", "users"), imported with the other operations. Then change the field and run this statement again.', + ); }); }); @@ -577,31 +615,33 @@ describe('MongoMigrationPlanner, removed fields', () => { expect(planned(plan({ from, to: none, schema })).labels).toEqual(['Drop collection profiles']); }); - it("unsets a variant's removed field only from that variant's documents, and keeps a field another variant now stores", () => { - const tasks = ( - bugFields: readonly string[], - featureFields: readonly string[], - baseFields: readonly string[], - ) => - contract( - bugFields.join(), - { - Task: { - collection: 'tasks', - fields: ['_id', 'type', ...baseFields], - discriminator: 'type', - variants: { Bug: 'bug', Feature: 'feature' }, - }, - Bug: { collection: 'tasks', fields: bugFields, base: 'Task' }, - Feature: { collection: 'tasks', fields: featureFields, base: 'Task' }, + const tasks = ( + seed: string, + baseFields: readonly string[], + bugFields: readonly string[], + featureFields: readonly string[], + ) => + contract( + seed, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type', ...baseFields], + discriminator: 'type', + variants: { Bug: 'bug', Feature: 'feature' }, }, - { tasks: {} }, - ); + Bug: { collection: 'tasks', fields: bugFields, base: 'Task' }, + Feature: { collection: 'tasks', fields: featureFields, base: 'Task' }, + }, + { tasks: {} }, + ); + + it("unsets a variant's removed field only from that variant's documents", () => { expect( planned( plan({ - from: tasks(['score', 'severity'], ['score'], ['title']), - to: tasks(['severity'], ['score', 'title'], []), + from: tasks('from', [], ['score', 'severity'], ['score']), + to: tasks('to', [], ['severity'], ['score']), schema: [schemaCollection('tasks')], }), ), @@ -616,6 +656,49 @@ describe('MongoMigrationPlanner, removed fields', () => { }); }); + it('unsets a base field moved to one variant from the documents of every other variant', () => { + expect( + planned( + plan({ + from: tasks('from', ['note'], [], []), + to: tasks('to', [], ['note'], []), + schema: [schemaCollection('tasks')], + }), + ), + ).toMatchObject({ + labels: ['Remove field note from collection tasks where {"type":{"$nin":["bug"]}}'], + dataLoss: [ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Task', field: 'note' }, + }, + ], + }); + }); + + it('keeps a variant field its base now stores', () => { + expect( + planned( + plan({ + from: tasks('from', [], ['note'], []), + to: tasks('to', ['note'], [], []), + schema: [schemaCollection('tasks')], + }), + ).labels, + ).toEqual([]); + }); + + it('unsets nothing without an origin contract when the destination collection has no validator', () => { + const unvalidated = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'email'] } }, + { profiles: {} }, + ); + expect(planned(plan({ from: null, to: unvalidated, schema })).labels).toEqual([ + 'Remove validator on profiles', + ]); + }); + it('unsets nothing for a field a statement renames', () => { const fullName = contract( 'to', diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index 5f6948efc015..048a9419fd15 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -71,3 +71,15 @@ Adding a required field to a populated collection, or making an optional field r On MongoDB a variant's field rename touches only that variant's documents. On SQL today a variant's field is a column in the shared table, and a rename applies to every row. When SQL supports variants stored in their own tables, a variant's field statements must act on that variant's table only. + +## `@map` accepts a MongoDB field name MongoDB reads as a path or an operator + +**Found:** slice 4a review, dispatch 3 (2026-10-09). Authoring side. + +A MongoDB field can be given a stored name containing `.` or starting with `$` through `@map`, and nothing rejects it. MongoDB reads such a name as a path into embedded documents or as an operator, so the derived validator rejects documents that hold it, and the planner cannot rename or unset it: a statement naming it is refused, and a removed field with such a name is left as it is. Authoring should refuse these names. + +## A data-loss question can offer one flag value per verb + +**Found:** slice 4a review, dispatch 3 (2026-10-09). Engine side. + +The statement prompt writes one flag value per verb, so a question about a field whose model the destination names differently cannot print the two statements that keep its values (the model rename, then the field rename). The question offers only `--delete` and says to pass the model rename first, after which the next question offers the field rename. Printing both statements at once would need the engine to accept several flag values for one verb. diff --git a/test/integration/test/mongo/target-runner/runner-statements.test.ts b/test/integration/test/mongo/target-runner/runner-statements.test.ts index 3bd4862d5ff2..170743abe758 100644 --- a/test/integration/test/mongo/target-runner/runner-statements.test.ts +++ b/test/integration/test/mongo/target-runner/runner-statements.test.ts @@ -359,4 +359,82 @@ describe('MongoMigrationPlanner and runner - statements', () => { }, timeouts.spinUpMongoMemoryServer, ); + + it( + 'removes a base field moved to one variant from the documents of every other variant', + async () => { + const branch = (value: string, fields: readonly string[]) => ({ + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ['type', { enum: [value] }], + ...fields.map((field) => [field, { bsonType: 'string' }]), + ]), + required: ['type'], + additionalProperties: false, + }); + const tasksValidator = (base: readonly string[], bug: readonly string[]) => ({ + bsonType: 'object', + required: ['_id', 'type'], + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ['type', { bsonType: 'string' }], + ...base.map((field) => [field, { bsonType: 'string' }]), + ]), + oneOf: [branch('bug', [...base, ...bug]), branch('feature', [...base, 'priority'])], + }); + const tasks = (storageHash: string, base: readonly string[], bug: readonly string[]) => + contract( + storageHash, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type', ...base], + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: bug, base: 'Task' }, + Feature: { collection: 'tasks', fields: ['priority'], base: 'Task' }, + }, + { + tasks: new MongoCollection({ + validator: { + jsonSchema: tasksValidator(base, bug), + validationLevel: 'strict', + validationAction: 'error', + }, + }), + }, + ); + await db.createCollection('tasks', { + validator: { $jsonSchema: tasksValidator(['note'], []) }, + validationLevel: 'strict', + validationAction: 'error', + }); + await db.collection('tasks').insertMany([ + { type: 'bug', note: 'b' }, + { type: 'feature', note: 'f' }, + ]); + const to = tasks('to', [], ['note']); + + const { planned, result } = await planAndRun(tasks('from', ['note'], []), to); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); + expect(planned.kind === 'success' ? planned.dataLoss : undefined).toEqual([ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Task', field: 'note' }, + }, + ]); + expect({ tasks: await documents('tasks'), further: await furtherPlanLabels(to) }).toEqual({ + tasks: [{ type: 'bug', note: 'b' }, { type: 'feature' }], + further: [], + }); + await expect( + db.collection('tasks').updateOne({ type: 'feature' }, { $set: { priority: 'p1' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + db.collection('tasks').updateOne({ type: 'bug' }, { $set: { note: 'b2' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + }, + timeouts.spinUpMongoMemoryServer, + ); }); From a438e88b88df3214822ffabeb039904b5006f89c Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 11:51:56 +0200 Subject: [PATCH 14/57] TML-3478: a base field renamed after moving to a variant stays with that variant, and printed advice for collection changes and time-series collections works A field statement on a root model renames only the documents of variants that do not now store the old name, with the same $nin filter the removal uses. The advice for a field rename whose collection changes now says to run node migration.ts again before migrate. The planner compares a time-series or clustered collection's options only on the keys the contract declares, as verify does, so the server's own defaults no longer read as an immutable change. Signed-off-by: willbot Signed-off-by: Will Madden --- .../src/core/migrations/mongo-planner.ts | 21 +- .../migrations/mongo-statement-planning.ts | 12 +- .../test/mongo-planner.statements.test.ts | 43 +++- ...ete-statements-migration.mongo.e2e.test.ts | 90 ++++++- .../contract-events-remapped-title.ts | 24 ++ .../contract-events-remapped.ts | 23 ++ .../target-runner/runner-statements.test.ts | 225 ++++++++++++++---- 7 files changed, 376 insertions(+), 62 deletions(-) create mode 100644 test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped-title.ts create mode 100644 test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped.ts diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts index a5749918c264..6b80b70fa81e 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts @@ -109,14 +109,31 @@ function validatorUpdateLabel( : `Update validator on ${collName} (${parts.join('; ')})`; } +/** + * The origin's value of an option the server fills in, keeping only the keys the destination + * names: MongoDB adds defaults such as a time-series collection's `bucketMaxSpanSeconds`, which a + * contract that leaves them out does not change. + */ +function asDeclared(origin: unknown, dest: unknown): unknown { + if (!isPlainObject(origin) || !isPlainObject(dest)) return origin; + return Object.fromEntries(Object.entries(origin).filter(([key]) => Object.hasOwn(dest, key))); +} + function hasImmutableOptionChange( origin: MongoSchemaCollectionOptions | undefined, dest: MongoSchemaCollectionOptions | undefined, ): string | undefined { if (canonicalize(origin?.capped) !== canonicalize(dest?.capped)) return 'capped'; - if (canonicalize(origin?.timeseries) !== canonicalize(dest?.timeseries)) return 'timeseries'; + if ( + canonicalize(asDeclared(origin?.timeseries, dest?.timeseries)) !== + canonicalize(dest?.timeseries) + ) + return 'timeseries'; if (canonicalize(origin?.collation) !== canonicalize(dest?.collation)) return 'collation'; - if (canonicalize(origin?.clusteredIndex) !== canonicalize(dest?.clusteredIndex)) + if ( + canonicalize(asDeclared(origin?.clusteredIndex, dest?.clusteredIndex)) !== + canonicalize(dest?.clusteredIndex) + ) return 'clusteredIndex'; return undefined; } diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index 252802b37a59..0e9ea5ef1073 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -314,14 +314,12 @@ function planFieldStatement( if (statement.from.field === statement.to.field) return { kind: 'unchanged' }; const policyRefusal = controlPolicyRefusal(statement, label, input.contract, collection); if (policyRefusal !== undefined) return policyRefusal; + const filter = + variantFilter(input.fromContract, statement.from) ?? + rootDocumentsLosing(input.contract, statement.to, statement.from.field)?.filter; return { kind: 'renameField', - call: new RenameFieldCall( - collection, - statement.from.field, - statement.to.field, - variantFilter(input.fromContract, statement.from), - ), + call: new RenameFieldCall(collection, statement.from.field, statement.to.field, filter), }; } @@ -333,7 +331,7 @@ function collectionChangeAdvice( if (!sameModel(statement.from, statement.to)) { return `Add --rename ${modelDisplayName(statement.from)}:${modelDisplayName(statement.to)} before this statement, so the collection is renamed with its documents.`; } - return `A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(statement.from)} to answer its question, then in its migration.ts replace the drop of "${from}" and the create of "${to}" with renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}), imported with the other operations. Then change the field and run this statement again.`; + return `A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(statement.from)} to answer its question; in its migration.ts replace the drop of "${from}" and the create of "${to}" with renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}), imported with the other operations; run node migration.ts in the migration's directory, so it writes ops.json again; then apply it with migrate. Then change the field and run this statement again.`; } export interface FieldRemoval { diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts index 4d915cc2a278..4130dc12d34c 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts @@ -34,6 +34,7 @@ interface CollectionSpec { readonly unique?: readonly string[]; readonly required?: readonly string[]; readonly control?: ControlPolicy; + readonly timeField?: string; } function jsonSchema(fields: readonly string[]) { @@ -125,6 +126,9 @@ function contract( }, }), ...(spec.control === undefined ? {} : { control: spec.control }), + ...(spec.timeField === undefined + ? {} + : { options: { timeseries: { timeField: spec.timeField } } }), }), ]), ), @@ -569,7 +573,7 @@ describe('MongoMigrationPlanner, statements it refuses', () => { }), )[0]?.why, ).toBe( - 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question, then in its migration.ts replace the drop of "profiles" and the create of "users" with renameCollection("profiles", "users"), imported with the other operations. Then change the field and run this statement again.', + 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question; in its migration.ts replace the drop of "profiles" and the create of "users" with renameCollection("profiles", "users"), imported with the other operations; run node migration.ts in the migration\'s directory, so it writes ops.json again; then apply it with migrate. Then change the field and run this statement again.', ); }); }); @@ -676,6 +680,23 @@ describe('MongoMigrationPlanner, removed fields', () => { }); }); + it('renames a base field moved to one variant only in the documents of the other variants', () => { + expect( + planned( + plan({ + from: tasks('from', ['note'], [], []), + to: tasks('to', ['remark'], ['note'], []), + schema: [schemaCollection('tasks')], + statements: [renameField('Task', 'note', 'Task', 'remark')], + }), + ), + ).toMatchObject({ + labels: ['Rename field note to remark in collection tasks where {"type":{"$nin":["bug"]}}'], + applied: [[0]], + dataLoss: [], + }); + }); + it('keeps a variant field its base now stores', () => { expect( planned( @@ -717,3 +738,23 @@ describe('MongoMigrationPlanner, removed fields', () => { ).toEqual([]); }); }); + +describe('MongoMigrationPlanner, time-series options the server fills in', () => { + it('plans nothing when the live collection holds only server defaults the contract leaves out', () => { + const readings = contract( + 'to', + { Reading: { collection: 'readings', fields: ['_id', 'at'] } }, + { readings: { timeField: 'at' } }, + ); + const serverFilled = { + timeField: 'at', + granularity: 'seconds' as const, + bucketMaxSpanSeconds: 3600, + }; + const live = new MongoSchemaCollection({ + name: 'readings', + options: new MongoSchemaCollectionOptions({ timeseries: serverFilled }), + }); + expect(planned(plan({ from: readings, to: readings, schema: [live] })).labels).toEqual([]); + }); +}); diff --git a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts index 0a9f32922949..8038185e6dd1 100644 --- a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts +++ b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts @@ -3,7 +3,7 @@ * * Dropping a collection loses its documents, so `migration plan` and `db update` ask what the drop means; `--delete ` answers. The MongoDB planner refuses rename statements in this release. * - * Journey S1 removes `Event` through `migration plan`: refused without `--delete Event`, planned with it, and applied by `migrate`. Journey S2 does the same through `db update`, where `--confirm ` no longer consents. Journey S3 adds a required field with no refusal. Journey S4 gives a rename, which renames the collection and keeps its documents. + * Journey S1 removes `Event` through `migration plan`: refused without `--delete Event`, planned with it, and applied by `migrate`. Journey S2 does the same through `db update`, where `--confirm ` no longer consents. Journey S3 adds a required field with no refusal. Journey S4 gives a rename, which renames the collection and keeps its documents. Journey S5 follows, as printed, the refusal of a field rename whose model keeps its name while its collection changes. */ import { copyFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; @@ -23,6 +23,7 @@ import { runDbUpdate, runMigrate, runMigrationPlan, + selfEmitMigration, } from '../utils/journey-test-helpers'; const FIXTURES_DIR = join(fixtureAppDir, 'fixtures/mongo-cli-journeys'); @@ -307,4 +308,91 @@ describe('Journeys (MongoDB): statements that consent to data loss', { .toArray(); expect(documents, 'S4.04: its documents are kept').toEqual([{ name: 'launch' }]); }); + + it('S5: the advice for a field rename whose collection changes keeps the documents when followed as printed', async () => { + const dbName = 'mongo_remap_then_rename'; + const ctx = setupProject(withDatabase(replSet.getUri(), dbName)); + created.add(ctx.testDir); + await emit(ctx, 'S5.01'); + const initial = await runMigrationPlan(ctx, ['--name', 'initial']); + expect(initial.exitCode, `S5.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(ctx)).exitCode, 'S5.03: migrate initial').toBe(0); + await client.db(dbName).collection('events').insertOne({ name: 'launch' }); + const origin = latestMigrationDirName(ctx); + + useFixture(ctx, 'contract-events-remapped-title.ts'); + await emit(ctx, 'S5.04'); + const refused = await runMigrationPlan(ctx, [ + '--from', + origin, + '--rename', + 'Event.name:Event.title', + '--json', + ]); + expect(refused.exitCode, 'S5.05: the field rename is refused').not.toBe(0); + expect(JSON.stringify(engineError(refused)?.meta), 'S5.05: the advice').toContain( + 'run node migration.ts in the migration', + ); + + useFixture(ctx, 'contract-events-remapped.ts'); + await emit(ctx, 'S5.06'); + const remap = await runMigrationPlan(ctx, [ + '--name', + 'remap', + '--from', + origin, + '--delete', + 'Event', + ]); + expect( + remap.exitCode, + `S5.07: plan the collection change with --delete Event: ${remap.stderr}`, + ).toBe(0); + const remapDir = latestMigrationDirName(ctx); + const migrationTs = join(ctx.testDir, 'migrations/app', remapDir, 'migration.ts'); + const planned = readFileSync(migrationTs, 'utf-8'); + const edited = planned + .replace( + /createCollection\(['"]occasions['"][^)]*\)/, + 'renameCollection("events", "occasions")', + ) + .replace(/,?\s*dropCollection\(['"]events['"]\)/, '') + .replace(/\bdropCollection,/, 'dropCollection,\n renameCollection,'); + expect(edited, `S5.08: migration.ts is edited as the advice says:\n${planned}`).toContain( + 'renameCollection("events", "occasions")', + ); + expect(edited, `S5.08: the drop is gone:\n${edited}`).not.toMatch(/dropCollection\(['"]events/); + writeFileSync(migrationTs, edited, 'utf-8'); + const emitted = await selfEmitMigration(ctx, ['--dir', `migrations/app/${remapDir}`]); + expect(emitted.exitCode, `S5.09: node migration.ts: ${emitted.stderr}`).toBe(0); + const applyRemap = await runMigrate(ctx); + expect(applyRemap.exitCode, `S5.10: migrate: ${applyRemap.stderr}`).toBe(0); + + useFixture(ctx, 'contract-events-remapped-title.ts'); + await emit(ctx, 'S5.11'); + const rename = await runMigrationPlan(ctx, [ + '--name', + 'title', + '--from', + remapDir, + '--rename', + 'Event.name:Event.title', + ]); + expect(rename.exitCode, `S5.12: plan the field rename: ${rename.stderr}`).toBe(0); + const applyRename = await runMigrate(ctx); + expect(applyRename.exitCode, `S5.13: migrate: ${applyRename.stderr}`).toBe(0); + + expect(await collections(dbName), 'S5.14: the collection is renamed').toEqual([ + 'occasions', + 'users', + ]); + const documents = await client + .db(dbName) + .collection('occasions') + .find({}, { projection: { _id: 0 } }) + .toArray(); + expect(documents, 'S5.14: its document is kept under the new field').toEqual([ + { title: 'launch' }, + ]); + }); }); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped-title.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped-title.ts new file mode 100644 index 000000000000..fbb0781c9cca --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped-title.ts @@ -0,0 +1,24 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'occasions', + fields: { + _id: field.objectId(), + title: field.string(), + }, + indexes: [index({ title: 1 })], +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped.ts new file mode 100644 index 000000000000..a5dfff876c3a --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped.ts @@ -0,0 +1,23 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'occasions', + fields: { + _id: field.objectId(), + name: field.string(), + }, +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/mongo/target-runner/runner-statements.test.ts b/test/integration/test/mongo/target-runner/runner-statements.test.ts index 170743abe758..9b78dbce016d 100644 --- a/test/integration/test/mongo/target-runner/runner-statements.test.ts +++ b/test/integration/test/mongo/target-runner/runner-statements.test.ts @@ -47,7 +47,8 @@ afterAll(async () => { beforeEach(async () => { const collections = await db.listCollections().toArray(); for (const col of collections) { - await db.dropCollection(col['name'] as string); + const name = col['name'] as string; + if (!name.startsWith('system.')) await db.dropCollection(name); } }); @@ -223,6 +224,67 @@ async function documents(name: string) { .toArray(); } +function branchSchema(value: string, fields: readonly string[]) { + return { + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ['type', { enum: [value] }], + ...fields.map((field) => [field, { bsonType: 'string' }]), + ]), + required: ['type'], + additionalProperties: false, + }; +} + +function tasksValidator(base: readonly string[], bug: readonly string[]) { + return { + bsonType: 'object', + required: ['_id', 'type'], + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ['type', { bsonType: 'string' }], + ...base.map((field) => [field, { bsonType: 'string' }]), + ]), + oneOf: [branchSchema('bug', [...base, ...bug]), branchSchema('feature', [...base, 'priority'])], + }; +} + +function tasks(storageHash: string, base: readonly string[], bug: readonly string[]) { + return contract( + storageHash, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type', ...base], + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: bug, base: 'Task' }, + Feature: { collection: 'tasks', fields: ['priority'], base: 'Task' }, + }, + { + tasks: new MongoCollection({ + validator: { + jsonSchema: tasksValidator(base, bug), + validationLevel: 'strict', + validationAction: 'error', + }, + }), + }, + ); +} + +async function createTasksWithNote(): Promise { + await db.createCollection('tasks', { + validator: { $jsonSchema: tasksValidator(['note'], []) }, + validationLevel: 'strict', + validationAction: 'error', + }); + await db.collection('tasks').insertMany([ + { type: 'bug', note: 'b' }, + { type: 'feature', note: 'f' }, + ]); +} + describe('MongoMigrationPlanner and runner - statements', () => { it( 'renames a model and a uniquely indexed field behind a closed validator, keeping every document', @@ -363,56 +425,7 @@ describe('MongoMigrationPlanner and runner - statements', () => { it( 'removes a base field moved to one variant from the documents of every other variant', async () => { - const branch = (value: string, fields: readonly string[]) => ({ - properties: Object.fromEntries([ - ['_id', { bsonType: 'objectId' }], - ['type', { enum: [value] }], - ...fields.map((field) => [field, { bsonType: 'string' }]), - ]), - required: ['type'], - additionalProperties: false, - }); - const tasksValidator = (base: readonly string[], bug: readonly string[]) => ({ - bsonType: 'object', - required: ['_id', 'type'], - properties: Object.fromEntries([ - ['_id', { bsonType: 'objectId' }], - ['type', { bsonType: 'string' }], - ...base.map((field) => [field, { bsonType: 'string' }]), - ]), - oneOf: [branch('bug', [...base, ...bug]), branch('feature', [...base, 'priority'])], - }); - const tasks = (storageHash: string, base: readonly string[], bug: readonly string[]) => - contract( - storageHash, - { - Task: { - collection: 'tasks', - fields: ['_id', 'type', ...base], - variants: { Bug: 'bug', Feature: 'feature' }, - }, - Bug: { collection: 'tasks', fields: bug, base: 'Task' }, - Feature: { collection: 'tasks', fields: ['priority'], base: 'Task' }, - }, - { - tasks: new MongoCollection({ - validator: { - jsonSchema: tasksValidator(base, bug), - validationLevel: 'strict', - validationAction: 'error', - }, - }), - }, - ); - await db.createCollection('tasks', { - validator: { $jsonSchema: tasksValidator(['note'], []) }, - validationLevel: 'strict', - validationAction: 'error', - }); - await db.collection('tasks').insertMany([ - { type: 'bug', note: 'b' }, - { type: 'feature', note: 'f' }, - ]); + await createTasksWithNote(); const to = tasks('to', [], ['note']); const { planned, result } = await planAndRun(tasks('from', ['note'], []), to); @@ -437,4 +450,114 @@ describe('MongoMigrationPlanner and runner - statements', () => { }, timeouts.spinUpMongoMemoryServer, ); + + it( + 'renames a base field moved to one variant only in the documents of the other variants', + async () => { + await createTasksWithNote(); + const to = tasks('to', ['remark'], ['note']); + + const { result } = await planAndRun(tasks('from', ['note'], []), to, [ + { + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: 'Task', field: 'note' }, + to: { namespaceId: NS, model: 'Task', field: 'remark' }, + }, + ]); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); + expect({ tasks: await documents('tasks'), further: await furtherPlanLabels(to) }).toEqual({ + tasks: [ + { type: 'bug', note: 'b' }, + { type: 'feature', remark: 'f' }, + ], + further: [], + }); + await expect( + db.collection('tasks').updateOne({ type: 'feature' }, { $set: { priority: 'p1' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + db.collection('tasks').updateOne({ type: 'bug' }, { $set: { note: 'b2' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + function readings(storageHash: string, model: string, collection: string) { + return contract( + storageHash, + { [model]: { collection, fields: ['_id', 'at'] } }, + { [collection]: new MongoCollection({ options: { timeseries: { timeField: 'at' } } }) }, + ); + } + + it( + 'plans nothing for a time-series collection that has not changed', + async () => { + const to = readings('to', 'Reading', 'readings'); + + const { result } = await planAndRun(contract('from', {}, {}), to); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect(await furtherPlanLabels(to)).toEqual([]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'keeps the documents of a time-series collection when its refusal is followed as printed', + async () => { + const from = readings('from', 'Reading', 'readings'); + const to = readings('to', 'Measurement', 'measurements'); + await db.createCollection('readings', { timeseries: { timeField: 'at' } }); + await db.collection('readings').insertOne({ at: new Date('2026-01-01T00:00:00Z') }); + const refused = new MongoMigrationPlanner().plan({ + contract: to, + schema: await introspectSchema(db), + policy, + fromContract: from, + origin: { storageHash: 'from' }, + statements: [ + { + kind: 'rename', + entity: 'model', + from: { namespaceId: NS, model: 'Reading' }, + to: { namespaceId: NS, model: 'Measurement' }, + }, + ], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + expect(refused.kind === 'failure' ? refused.conflicts[0]?.why : refused.kind).toBe( + 'MongoDB cannot rename a time-series collection. Leave out this statement. To keep the documents, first create collection "measurements" yourself with the same time-series options and copy them into it; the plan then drops "readings" once --delete names the model.', + ); + + await db.createCollection('measurements', { timeseries: { timeField: 'at' } }); + const copied = await db + .collection('readings') + .find({}, { projection: { _id: 0 } }) + .toArray(); + await db.collection('measurements').insertMany(copied); + const { planned, result } = await planAndRun(from, to); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect( + planned.kind === 'success' ? planned.dataLoss.map(({ subject }) => subject) : [], + ).toEqual([{ kind: 'model', namespaceId: NS, model: 'Reading' }]); + expect({ + measurements: await db + .collection('measurements') + .find({}, { projection: { _id: 0 } }) + .toArray(), + readings: await db.listCollections({ name: 'readings' }).toArray(), + further: await furtherPlanLabels(to), + }).toEqual({ + measurements: [{ at: new Date('2026-01-01T00:00:00Z') }], + readings: [], + further: [], + }); + }, + timeouts.spinUpMongoMemoryServer, + ); }); From dee0274e226e06048be69f93fc209b214060524a Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:06:55 +0200 Subject: [PATCH 15/57] TML-3478: journeys, docs and upgrade fragments for MongoDB renames and deletes Journeys on PSL contracts with documents rename a model and a uniquely indexed field, rename a variant's field and remove a field, through migration plan and migrate and through db update, following every printed flag. The Migration System and MongoDB Family docs, the error reference, the CLI README and the prisma-8 skill describe what MongoDB now does, and the app and extension upgrade fragments record the changes. Signed-off-by: willbot Signed-off-by: Will Madden --- .../subsystems/10. MongoDB Family.md | 15 +- .../subsystems/7. Migration System.md | 20 +- docs/reference/error-reference.md | 4 +- packages/1-framework/3-tooling/cli/README.md | 8 +- skills/prisma-8/references/migrations.md | 9 +- ...rename-delete-statements.mongo.e2e.test.ts | 403 ++++++++++++++++++ .../app/instructions.md | 45 ++ .../extension/instructions.md | 77 ++++ 8 files changed, 556 insertions(+), 25 deletions(-) create mode 100644 test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts create mode 100644 upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md create mode 100644 upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md diff --git a/docs/architecture docs/subsystems/10. MongoDB Family.md b/docs/architecture docs/subsystems/10. MongoDB Family.md index a62db1c22326..e5a2e95157f2 100644 --- a/docs/architecture docs/subsystems/10. MongoDB Family.md +++ b/docs/architecture docs/subsystems/10. MongoDB Family.md @@ -435,12 +435,14 @@ Both commands write into `migrations/_/`. The destination contr The Mongo *target* (not the framework) owns the scaffolding pipeline end-to-end. The framework defines the shape of a migration package and the runner contract; everything family-specific — what factories exist, how the planner IR renders into a runnable file, what the diff produces — lives under `packages/3-mongo-target/` and `packages/2-mongo-family/`. -Concretely, the Mongo planner produces an `OpFactoryCall[]` IR (a discriminated union of frozen AST classes — `CreateIndexCall`, `DropIndexCall`, `CreateCollectionCall`, `DropCollectionCall`, `CollModCall`) and feeds it to two renderers: an operation renderer that calls the matching factory functions to materialize `MongoMigrationPlanOperation[]` for `ops.json`, and a TypeScript renderer that emits a `migration.ts` calling the same factories. Both outputs derive from the same IR, so re-running `./migration.ts` after editing it produces an `ops.json` that matches the planner's output for the unedited regions. See [ADR 195 — Planner IR with two renderers](../adrs/ADR%20195%20-%20Planner%20IR%20with%20two%20renderers.md) and [ADR 194 — Plans carry their own authoring surface](../adrs/ADR%20194%20-%20Plans%20carry%20their%20own%20authoring%20surface.md). +Concretely, the Mongo planner produces an `OpFactoryCall[]` IR (a discriminated union of frozen AST classes — `CreateIndexCall`, `DropIndexCall`, `CreateCollectionCall`, `DropCollectionCall`, `RenameCollectionCall`, `RenameFieldCall`, `UnsetFieldCall`, `CollModCall`) and feeds it to two renderers: an operation renderer that calls the matching factory functions to materialize `MongoMigrationPlanOperation[]` for `ops.json`, and a TypeScript renderer that emits a `migration.ts` calling the same factories. Both outputs derive from the same IR, so re-running `./migration.ts` after editing it produces an `ops.json` that matches the planner's output for the unedited regions. See [ADR 195 — Planner IR with two renderers](../adrs/ADR%20195%20-%20Planner%20IR%20with%20two%20renderers.md) and [ADR 194 — Plans carry their own authoring surface](../adrs/ADR%20194%20-%20Plans%20carry%20their%20own%20authoring%20surface.md). The factories the scaffold imports live in two layered modules: - `@internal/family-mongo/migration` — the `Migration` base class (a `MongoMigration` subclass that `implements MigrationPlan`) plus `Migration.run(...)` for the shebang self-emit path ([ADR 196](../adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md)). -- `@internal/target-mongo/migration` — the operation factories (`createCollection`, `dropCollection`, `createIndex`, `dropIndex`, `collMod`, `setValidation`, `validatedCollection`, `dataTransform`) and the `placeholder(slot)` utility for unfilled scaffold slots. +- `@internal/target-mongo/migration` — the operation factories (`createCollection`, `dropCollection`, `renameCollection`, `createIndex`, `dropIndex`, `collMod`, `setValidation`, `validatedCollection`, `renameField`, `unsetField`, `dataTransform`) and the `placeholder(slot)` utility for unfilled scaffold slots. + +`renameCollection(from, to)` renames a collection, keeping its documents, indexes and validator; it checks that the source exists and the target does not. `renameField(collection, from, to, { filter? })` and `unsetField(collection, field, { filter? })` rewrite the documents that hold the field with one `updateMany` (`$rename` or `$unset`) that skips validation; `filter` limits them to one variant's documents. They refuse an empty field name, `_id`, and a name containing `.` or starting with `$`. The planner emits all three for `--rename` and `--delete` statements; see [Subsystem 7 — Rename statements](7.%20Migration%20System.md#rename-statements). A scaffolded `migration.ts` therefore looks like: @@ -472,14 +474,7 @@ Non-nullable fields are added to `required`. Nullable fields are still listed in ### Polymorphic variants -For polymorphic models (`@@discriminator` on the base, `@@base` on each variant) using single-collection inheritance, the *intent* is that variants share the base model's collection and the validator becomes a discriminated `oneOf` over variant shapes (one branch per discriminator value, base fields hoisted to the top-level `properties`/`required`). This pattern is described in [Schema validation](#schema-validation-jsonschema) and reflects how the runtime interpreter already handles polymorphic indexes ([ADR 173](../adrs/ADR%20173%20-%20Polymorphism%20via%20discriminator%20and%20variants.md)). - -**Known gaps in the migration planner today** (tracked as [TML-2447](https://linear.app/prisma-company/issue/TML-2447)): - -- The planner emits standalone `createCollection` ops for variant models that should share the base's collection. Applying these would create unwanted empty collections; in example apps the migration artifacts are committed as-is and the variant ops have to be removed by hand before apply. -- Variant validators include only variant-specific fields and miss every base-model field (including `_id` and the discriminator). They are structurally wrong even when the variant collection is intentional. - -Until these are addressed, polymorphic schemas may need post-scaffold edits to `migration.ts` (or to `ops.json` after self-emit) before the migration is safe to apply. +For polymorphic models (`@@discriminator` on the base, `@@base` on each variant) using single-collection inheritance, variants share the base model's collection, and the planner creates that one collection. Its validator is a discriminated `oneOf` over variant shapes: base fields at the top-level `properties`/`required`, and one branch per discriminator value that repeats the base fields, adds the variant's own fields, pins the discriminator value and closes with `additionalProperties: false`. This pattern is described in [Schema validation](#schema-validation-jsonschema) and matches how the runtime handles polymorphic indexes ([ADR 173](../adrs/ADR%20173%20-%20Polymorphism%20via%20discriminator%20and%20variants.md)). Every variant document holds its discriminator value, so a variant's field rename or removal touches only that variant's documents. ### Author-editable seams diff --git a/docs/architecture docs/subsystems/7. Migration System.md b/docs/architecture docs/subsystems/7. Migration System.md index 231d9e9dbffd..18c8ba2de419 100644 --- a/docs/architecture docs/subsystems/7. Migration System.md +++ b/docs/architecture docs/subsystems/7. Migration System.md @@ -191,7 +191,7 @@ prisma migration plan --name tidy-users --rename Profile:User --rename User.name prisma db update --rename Profile:User --rename User.name:User.fullName --delete Legacy --allow User ``` -**The questions.** The planner's success result lists what the plan would lose (`dataLoss`) and whose access it would widen (`accessWidening`), each entry an operation's position and its subject (a `MigrationSubject`: a model, a field, or a storage name no model of the origin contract stores). Before `migration plan` writes anything, and before `db update` applies anything, the command asks one question per data-loss subject, answered by `delete` or, for a model or field the destination no longer has, `rename`. Where the target's migrations capability sets `renameStatements: { refused: true, keepDataByHand }`, as MongoDB's does, the question offers no rename and ends with the target's text on keeping the data by hand (renaming the collection in `mongosh` on each database before a plan that drops it is applied there; `db update` then drops nothing, and a migration written by `migration plan` still drops it, so remove that operation or do not apply it there). `db update` also asks one question per operation that changes who can read or write a model's rows, answered by `allow`: the question says the operation widens access when it does, such as disabling row-level security, and that it changes access otherwise, such as dropping a policy, since a permissive policy grants access. Two such operations on one model are two questions, each answered by its own `--allow`. The questions go through the CLI engine's statement prompt (`ctx.prompt.statements(questions, { last: true })`), which answers them from the flags first. In a terminal it asks the rest in turn, and the user types `delete`, `allow` or `rename :`; a typed rename is planned again with the other renames, and a plan that still has the operation asked about fails with `MIGRATION.STATEMENT_DID_NOT_RESOLVE_LOSS`; a loss the new plan has on the same subject, such as a type change on the renamed field, is asked next. Where nobody can answer (not interactive, or `--yes`), the command fails with `CLI.CONSENT_REQUIRED`, listing every unanswered question with the flags that answer it, and a `--delete` or `--allow` that answers no question fails with `CLI.CONSENT_UNUSED`; both happen before anything is written or applied. A `db update --dry-run` asks nothing; it checks the same flags against the subjects an apply would ask about and fails an unmatched one with `MIGRATION.STATEMENT_ANSWERS_NO_QUESTION`, so a script that checks the code handles both. `--confirm` answers none of these questions. The `--rename` values are taken from the run first (`ctx.statements.take('rename')`) and planned from the start, so the drop a rename replaces is never asked about. The shared question loop is `askPlanQuestions` in [`plan-questions.ts`](../../../packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts); the control API takes the answers through a callback, so it never depends on the engine's prompt. +**The questions.** The planner's success result lists what the plan would lose (`dataLoss`) and whose access it would widen (`accessWidening`), each entry an operation's position and its subject (a `MigrationSubject`: a model, a field, or a storage name no model of the origin contract stores). Before `migration plan` writes anything, and before `db update` applies anything, the command asks one question per data-loss subject, answered by `delete` or, for a model or field the destination no longer has, `rename`. A field whose model the destination names differently, with no statement renaming the model, is asked about with `delete` only, and the question says to pass the model rename first; the next question then offers the field rename. `db update` also asks one question per operation that changes who can read or write a model's rows, answered by `allow`: the question says the operation widens access when it does, such as disabling row-level security, and that it changes access otherwise, such as dropping a policy, since a permissive policy grants access. Two such operations on one model are two questions, each answered by its own `--allow`. The questions go through the CLI engine's statement prompt (`ctx.prompt.statements(questions, { last: true })`), which answers them from the flags first. In a terminal it asks the rest in turn, and the user types `delete`, `allow` or `rename :`; a typed rename is planned again with the other renames, and a plan that still has the operation asked about fails with `MIGRATION.STATEMENT_DID_NOT_RESOLVE_LOSS`; a loss the new plan has on the same subject, such as a type change on the renamed field, is asked next. Where nobody can answer (not interactive, or `--yes`), the command fails with `CLI.CONSENT_REQUIRED`, listing every unanswered question with the flags that answer it, and a `--delete` or `--allow` that answers no question fails with `CLI.CONSENT_UNUSED`; both happen before anything is written or applied. A `db update --dry-run` asks nothing; it checks the same flags against the subjects an apply would ask about and fails an unmatched one with `MIGRATION.STATEMENT_ANSWERS_NO_QUESTION`, so a script that checks the code handles both. `--confirm` answers none of these questions. The `--rename` values are taken from the run first (`ctx.statements.take('rename')`) and planned from the start, so the drop a rename replaces is never asked about. The shared question loop is `askPlanQuestions` in [`plan-questions.ts`](../../../packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts); the control API takes the answers through a callback, so it never depends on the engine's prompt. #### Rename statements @@ -207,6 +207,15 @@ prisma db update --rename Profile:User --rename User.name:User.fullName --delete Rename operations are `widening`, so neither command asks about them. The `migration.ts` that `migration plan` writes contains `...this.renameTable(...)` and `...this.renameColumn(...)` calls, as a user would write them by hand, and running it again writes the same `ops.json` and `migration.json`. +The MongoDB planner does the same with collections and documents ([`mongo-statement-planning.ts`](../../../packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts)). A statement names a field by its contract key, as on SQL; MongoDB's contract keys a field by its stored name, so `id @map("_id")` is `User._id` in a statement. + +- A model statement whose collection name changes renames the collection in the working copy and plans `renameCollection`. MongoDB keeps the documents, indexes and validator, so the diff then plans only what really changed. +- A field statement whose stored name changes plans `renameField`: one `updateMany` with `$rename` on the documents that hold the old name, skipping validation, since the old validator requires the old field. A variant's field is renamed only in that variant's documents, filtered by its discriminator value, so another variant that stores a field with the same name keeps it. A base field that a variant now stores itself is renamed only in the documents of the other variants. +- A field that an origin model stores and its destination model no longer stores, in a collection the plan keeps, is data loss. The plan unsets it with `unsetField` (`$unset`, skipping validation) from the documents that hold it, with the same variant filters, and asks about it as a `field` subject, so `--delete User.nickname` or a `--rename` answers it. Without an origin contract, a removed field is found from the origin validator's top-level properties and named `.`; a collection whose destination has no validator is skipped, and a variant's removed field is not found. +- Operations run in this order: collection renames, collection creates, index drops, document rewrites, index creates, validator changes, option changes, collection drops. Rewrites run after the index drops, so a unique index on the old field does not fail once documents lose it, and before the index creates, so a unique index on the new field sees the moved values. + +`renameCollection` and `renameField` are `widening` and `unsetField` is `destructive`, so `db update` runs them. The runner and the operation serializer tell a document rewrite from a DDL operation by its shape (`run` against `execute`), not by its class; a hand-written `dataTransform` keeps the class `data`. Each rewrite's after-check, that no matching document holds the old name, lets a run that stopped part way finish when run again. The `migration.ts` that `migration plan` writes calls `renameCollection(...)`, `renameField(...)` and `unsetField(...)` from the target's migration module. + **Result.** The planner's success result carries `appliedStatements` (`AppliedMigrationStatement`): one entry per statement, in order, with `operationIndexes`: the positions, in the plan's operations, of the operations the statement accounts for, empty when its storage did not change. Statements point at operations by position because operation ids are not unique within a plan (on Postgres, same-named tables in two schemas share them), and ids stay as they are so that re-emitting an existing migration does not change its hash. The framework's `describeMigrationStatement` writes the statement in domain names, such as `rename model "Profile" to "User"`, for every family's messages and for the CLI. `migration plan` and `db update` print each statement with its description and number of operations, and their JSON output carries the description and the operation positions, which index the output's `operations`. Messages name a model of the unbound namespace by its name alone, and JSON output leaves `namespaceId` out of a coordinate in the unbound namespace (`migrationStatementJson`), because that namespace's id is internal. **Refusals.** When the planner cannot carry out a resolved statement, planning fails with `MIGRATION.PLANNING_FAILED` and a conflict of kind `statementRefused` that carries the statement in its `refusedStatement` field. The [error reference](../../reference/error-reference.md#migrationplanning_failed) lists the cases. @@ -216,7 +225,8 @@ Rename operations are `widening`, so neither command asks about them. The `migra - A model move to another namespace resolves, but the planner refuses it with `statementRefused`. - A model that keeps its name but changes its table cannot be renamed by a statement, because the model has the same name on both sides. Rename the table by hand with `this.renameTable(...)` in its own `migration.ts`. A field statement on such a model is refused with the same advice. - Only models and fields can be renamed. Value objects, their fields, enum values and namespaces cannot. -- The MongoDB planner refuses every statement with `statementRefused`, so `migration plan` and `db update` with `--rename` fail with `MIGRATION.PLANNING_FAILED` and plan nothing, rather than dropping and creating the collection. +- On MongoDB, a statement cannot rename `_id`, a field whose stored name MongoDB reads as a path or an operator (it contains `.` or starts with `$`), a time-series collection, or a field inside a value object. A collection rename onto a name the database already has, or onto a name MongoDB rejects, is refused too. +- **[Placeholder: decision pending]** On a MongoDB contract without a validator, such as one written with the TypeScript contract builder, renaming or removing a field does not change the storage hash. What `migration plan` and `migrate` do with such a change is described here once it is decided. #### Delete and allow statements @@ -226,9 +236,9 @@ Rename operations are `widening`, so neither command asks about them. The `migra #### The origin of `db update` -`migration plan` already has an origin contract: the one `--from` resolves to. `db update` introspects the database instead, and on every run reads the contract for the marker's hash from the [snapshot store](#contract-resolution-through-the-snapshot-store). The planner gets it as `fromContract`, and names each subject of `dataLoss` and `accessWidening` through it. A missing or unreadable snapshot fails only a run with `--rename`; otherwise each subject is named by its storage name. The plan it hands the runner keeps `origin: null` either way (see [`db update`](#db-update-live-reconciliation)). The planner's operations do not depend on whether the snapshot was read: every strategy that emits a data transform needs the `data` operation class, which `db update` does not allow, and codec field-event hooks get the prior contract only when the plan has an origin, which a `db update` plan never has. `db update` stores a snapshot only when it advances a ref: the `db` ref by default, or with `--db ` only the ref named by `--advance-ref `. So a database last updated with `--db ` and no `--advance-ref` has no snapshot, and renames fail there with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`, which names the hash and the directory it looked in. Running `db update` again with the same statements fails with `MIGRATION.STATEMENT_UNRESOLVED`, because the origin is then a contract without the old names. +`migration plan` already has an origin contract: the one `--from` resolves to. `db update` introspects the database instead, and on every run reads the contract for the marker's hash from the [snapshot store](#contract-resolution-through-the-snapshot-store). The planner gets it as `fromContract`, and names each subject of `dataLoss` and `accessWidening` through it. A missing or unreadable snapshot fails only a run with `--rename`; otherwise each subject is named by its storage name. The plan it hands the runner keeps `origin: null` either way (see [`db update`](#db-update-live-reconciliation)). On SQL, the planner's operations do not depend on whether the snapshot was read: every strategy that emits a data transform needs the `data` operation class, which `db update` does not allow, and codec field-event hooks get the prior contract only when the plan has an origin, which a `db update` plan never has. On MongoDB they can: without the snapshot, a removed field is found only from the origin validator, as described under [Rename statements](#rename-statements). `db update` stores a snapshot only when it advances a ref: the `db` ref by default, or with `--db ` only the ref named by `--advance-ref `. So a database last updated with `--db ` and no `--advance-ref` has no snapshot, and renames fail there with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`, which names the hash and the directory it looked in. Running `db update` again with the same statements fails with `MIGRATION.STATEMENT_UNRESOLVED`, because the origin is then a contract without the old names. -The rest of the migration-statements work adds `--convert`, `--backfill`, the remaining renames and model moves across namespaces; and brings statements to MongoDB. +The rest of the migration-statements work adds `--convert`, `--backfill`, the remaining renames and model moves across namespaces. ## Authoring Surface @@ -462,7 +472,7 @@ DDL operations carry commands like `CreateIndexCommand`, `CreateCollectionComman Data transforms carry `operationClass: 'data'` and use the **same three-phase envelope** as DDL operations. The runner has a single execution loop; there is no DT-specific dispatch path. The two families fill the envelope with their own query shapes: - **Postgres.** The `dataTransform` factory lowers the user-authored `check` / `run` query plans into the standard envelope at factory time. `precheck` is `[{ sql: 'SELECT EXISTS () AS ok', params: , … }]` (asserts "work to do"); `execute` carries the run plans as `SqlMigrationPlanOperationStep` entries with `params` threaded through `driver.query`; `postcheck` is `[{ sql: 'SELECT NOT EXISTS () AS ok', params: , … }]` (asserts "work done"). Idempotent re-apply uses the same pre-satisfied-skip path that DDL ops already use — when the postcheck is already satisfied, the runner records a `postcheck_pre_satisfied` skip without re-running execute. Routing the run-plan parameters through the driver's parameter binder (rather than inlining values into SQL text) is strictly safer than rolling a per-driver literal serializer. -- **Mongo.** The `run` array carries `MongoQueryPlan` objects. Checks use the same `{ description, source, filter, expect }` shape as DDL checks, with `MongoQueryPlan` as the source type. Both are evaluated by the same `FilterEvaluator` path. DML commands execute through `Adapter.lower()` → `Driver.execute()` — the same transport as runtime queries. +- **Mongo.** The `run` array carries `MongoQueryPlan` objects. Checks use the same `{ description, source, filter, expect }` shape as DDL checks, with `MongoQueryPlan` as the source type. Both are evaluated by the same `FilterEvaluator` path. DML commands execute through `Adapter.lower()` → `Driver.execute()` — the same transport as runtime queries. The runner and the serializer recognise such an operation by its `run` array, not by its class: the planner's own document rewrites (`renameField`, `unsetField`) have the same shape but carry the class of their effect, `widening` or `destructive`, so `db update` runs them. The shared envelope means policy gating on `operationClass: 'data'` (CI checks, audit, "this run may not perform data work") is uniform across families and decoupled from how each family describes its check/run payload. diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index 531d16681ce8..77202b1818f2 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -81,7 +81,7 @@ The migration-file CLI (`prisma migration`) received `--config` without a path a ### CLI.CONSENT_REQUIRED -Raised by the CLI engine, not by Prisma ORM's own code; the engine's entry is the reference: [CLI.CONSENT_REQUIRED](https://docs.prisma.io/docs/cli/error-reference#CLI.CONSENT_REQUIRED). `migration plan` and `db update` raise it when the plan would lose data, or for `db update` widen who can read or write rows, and the run has nobody to answer: it is not interactive, or `--yes` is set. The command asks one question per model, field or storage name whose data an operation would lose, and `db update` also one per model whose rows an operation would open up, such as disabling row-level security. The error lists every question still unanswered, with the flags that answer each: `--delete ` lets the plan lose that data, for a model or field `--rename :` keeps it under a new name, and `--allow ` lets `db update` widen access to that model's rows. Data no model of the origin contract stores can only be deleted. A field of a model the plan renames is named through the model's new name: after `--rename Profile:User`, the question about the dropped `nickname` field is answered with `--delete User.nickname` or `--rename User.nickname:User.handle`. Nothing is written or applied. `--confirm` does not answer these questions. Payload: `subject`, `verbs`, `unanswered` (each with its `subject` and `verbs`). +Raised by the CLI engine, not by Prisma ORM's own code; the engine's entry is the reference: [CLI.CONSENT_REQUIRED](https://docs.prisma.io/docs/cli/error-reference#CLI.CONSENT_REQUIRED). `migration plan` and `db update` raise it when the plan would lose data, or for `db update` widen who can read or write rows, and the run has nobody to answer: it is not interactive, or `--yes` is set. The command asks one question per model, field or storage name whose data an operation would lose, and `db update` also one per model whose rows an operation would open up, such as disabling row-level security. The error lists every question still unanswered, with the flags that answer each: `--delete ` lets the plan lose that data, for a model or field `--rename :` keeps it under a new name, and `--allow ` lets `db update` widen access to that model's rows. Data no model of the origin contract stores can only be deleted. A field of a model the plan renames is named through the model's new name: after `--rename Profile:User`, the question about the dropped `nickname` field is answered with `--delete User.nickname` or `--rename User.nickname:User.handle`. A field whose model the destination names differently, with no statement renaming the model, is offered only `--delete`, and the question says to pass `--rename :` first; the next question then offers the field rename. On MongoDB, removing a field from a model is asked about too, and `--delete` removes it from every document that holds it; with no snapshot of the origin contract the field is named `.`. Nothing is written or applied. `--confirm` does not answer these questions. Payload: `subject`, `verbs`, `unanswered` (each with its `subject` and `verbs`). ### CLI.CONSENT_UNUSED @@ -1576,7 +1576,7 @@ An apply command (`db migrate`/`db update`) cannot find a path through the on-di ### MIGRATION.PLANNING_FAILED -Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); a rename that produces an operation the command's operation policy does not allow; or any statement on MongoDB, whose planner refuses statements in this release and plans nothing. `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. +Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, replace the drop and create in its `migration.ts` with `renameCollection(...)`, run `node migration.ts` again, and apply it with `migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. `migration plan` raises it when the planner produces no operations. The conflict's kind names the case, and its `why` is the next action. With an earlier contract (`noDatabaseChange`), the contract changed in a way the database does not see, for example after a Prisma upgrade that changes how `contract.json` records values. If you deploy with migrations, run `prisma migration new --from `, with the earlier contract's hash from the message, to write a migration with no operations, then `prisma db migrate`. If you manage the database with `db init` or `db update`, run `prisma db sign` on each database instead. With no earlier contract (`nothingToCreate`), the contract describes nothing to create; if the database needs something the contract does not describe, run `prisma migration new` and write the operations in `migration.ts` by hand. `migration new` writes a migration only to the emitted contract, so with `--to` the message says to emit the target first, and for a database managed with `db update` it names `prisma db sign `. When the migrations directory is empty and the `db` ref points at a contract that describes nothing to create (`nothingToBaseline`), there is no baseline migration to plan, and no command writes one with no operations yet; keep using `db update`. diff --git a/packages/1-framework/3-tooling/cli/README.md b/packages/1-framework/3-tooling/cli/README.md index c6ef0fd78a92..63004ea18ae6 100644 --- a/packages/1-framework/3-tooling/cli/README.md +++ b/packages/1-framework/3-tooling/cli/README.md @@ -950,11 +950,11 @@ Update your database schema to match the currently emitted contract. prisma db update [--db ] [--to ] [--advance-ref ] [--config ] [--dry-run] [--rename ]... [--delete ]... [--allow ]... [--interactive|--no-interactive] [--json] [-v] [-q] [--color/--no-color] ``` -**Rename statements (`--rename :`, repeatable):** tell `db update` that a model or field was renamed, so it renames the table or column instead of dropping and creating it. The statements work exactly as they do for `migration plan` (see below), and produce the same operations. The origin contract they resolve against is the contract the database's marker names, read from the local snapshot store (`migrations/snapshots//`). The snapshot is read on every run, so the planner sees the prior contract whenever one is stored; a missing or unreadable snapshot fails only a run with `--rename`. Either way the plan applies from whatever state the database is in, as `db update` always has. `db update` keeps a snapshot of the contract it applies whenever it advances a ref: the `db` ref by default, or with `--db ` only the ref you name with `--advance-ref `. So to use renames against a database you update with `--db `, give that earlier run `--advance-ref `. Without a readable snapshot, a run with statements fails with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`, naming the hash and the directory it looked in, and gives the steps that store the missing snapshot: emit the contract the database is at, run `db update --advance-ref --dry-run` against it and check that it plans no operations, then run it without `--dry-run`, which changes nothing in the database and stores the snapshot, then emit the new contract and run the rename. A database with no marker has nothing to rename, and fails the same way. Running the same statements a second time fails with `MIGRATION.STATEMENT_UNRESOLVED`, because the database is now at a contract that no longer has the old names. The statements the plan applied are listed under `Statements applied` after the operations (dry run and apply), and as `appliedStatements` in `--json` output, each with its `description` and `operationIndexes`, the positions in `operations` of the operations it accounts for. +**Rename statements (`--rename :`, repeatable):** tell `db update` that a model or field was renamed, so it renames the table or column, or on MongoDB the collection or the field in every document, instead of dropping and creating it. The statements work exactly as they do for `migration plan` (see below), and produce the same operations. The origin contract they resolve against is the contract the database's marker names, read from the local snapshot store (`migrations/snapshots//`). The snapshot is read on every run, so the planner sees the prior contract whenever one is stored; a missing or unreadable snapshot fails only a run with `--rename`. Either way the plan applies from whatever state the database is in, as `db update` always has. `db update` keeps a snapshot of the contract it applies whenever it advances a ref: the `db` ref by default, or with `--db ` only the ref you name with `--advance-ref `. So to use renames against a database you update with `--db `, give that earlier run `--advance-ref `. Without a readable snapshot, a run with statements fails with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`, naming the hash and the directory it looked in, and gives the steps that store the missing snapshot: emit the contract the database is at, run `db update --advance-ref --dry-run` against it and check that it plans no operations, then run it without `--dry-run`, which changes nothing in the database and stores the snapshot, then emit the new contract and run the rename. A database with no marker has nothing to rename, and fails the same way. Running the same statements a second time fails with `MIGRATION.STATEMENT_UNRESOLVED`, because the database is now at a contract that no longer has the old names. The statements the plan applied are listed under `Statements applied` after the operations (dry run and apply), and as `appliedStatements` in `--json` output, each with its `description` and `operationIndexes`, the positions in `operations` of the operations it accounts for. **Questions before an apply:** an apply asks one question per model, field or storage name whose data an operation would lose (dropping a table or a column, or a type change that can change values), and one per model whose rows an operation would let more people read or write (disabling row-level security, for example). It asks before it applies anything. Subjects are named through the origin contract read from the snapshot store, as `migration plan` names them; without a readable snapshot each subject is its storage name, which can only be deleted or allowed, and the question says the origin contract is unknown. Recorded migrations of extension spaces are asked about for the data they would lose, never for widened access. Each question is answered with a statement: -- `--delete ` lets the update lose that data, for example `--delete Legacy` or `--delete User.nickname`. A field of a model the update renames is named through the model's new name. +- `--delete ` lets the update lose that data, for example `--delete Legacy` or `--delete User.nickname`. A field of a model the update renames is named through the model's new name. On MongoDB, a field removed from a model is asked about, and `--delete` removes it from every document that holds it; without a snapshot of the origin contract it is named `.`, for example `--delete profiles.nickname`. - `--rename :` keeps the data of a model or field under a new name instead. A field's new name keeps its model: `--rename User.nickname:User.handle`, which the question writes as `User.nickname:User.`. - `--allow ` lets the update change who can read or write that model's rows, for example `--allow User`. Each such operation is its own question: dropping a policy and disabling row-level security on one model take two `--allow User` flags. The question says disabling row-level security widens access, and that dropping a policy changes it, since a permissive policy grants access. @@ -1040,12 +1040,12 @@ prisma migration plan [--config ] [--name ] [--from ] [--t - `migrations//ops.json` — planned operations (empty list `[]` if placeholders blocked the planner) - `migrations/snapshots//contract.{json,d.ts}` — bookend contracts, written write-if-absent, keyed by each contract's storage hash (one entry for `from` when applicable, one for `to`) -**Rename statements:** each `--rename` names the old and the new name in contract vocabulary, never a table or a column. Each side is one of `Model`, `namespace.Model`, `Model.field` or `namespace.Model.field`; both sides name a model, or both name a field of the same model. Names match exactly, including case. A name without a namespace resolves when exactly one namespace declares the model. A field's model is named as the destination contract names it, so a model rename followed by a field rename on it is `--rename Profile:User --rename User.name:User.fullName`. The old name must exist in the starting contract and not in the destination; the new name must exist in the destination and not in the starting contract. Statements resolve before anything is written; a statement that is malformed fails with `MIGRATION.STATEMENT_INVALID`, and one that does not resolve fails with `MIGRATION.STATEMENT_UNRESOLVED`, naming what it searched and what it found. Each error's next step is the statement to type instead: the accepted forms, the statement written the right way round, the statements a model-and-field mix may have meant, or the order that works. A statement that has already been applied (the origin already has the new name and not the old one) says to leave it out. Statements cannot swap two names in one plan, and the error gives the three plans that do, through a temporary name. A plan from an empty database (`--from @empty`, or greenfield) has no models to rename, so every statement is unresolved. Moving a model to another namespace and renaming value objects are not supported in this release. On MongoDB the planner refuses every statement in this release: planning fails with `MIGRATION.PLANNING_FAILED`, carrying the statement, and nothing is planned. The planner renames the table or column, and the constraints and indexes named after it, ahead of the rest of the plan. On Postgres, a check, a row-level-security policy, or an expression or partial index whose SQL names a renamed column is dropped and created again, and adding a check back reads every row of the table under an exclusive lock, so on a large table such a rename takes time; `migration.ts` contains the same `...this.renameTable(...)` and `...this.renameColumn(...)` calls you could have written by hand. The statements are listed under `Statements applied` after the operations, each with its number of operations, and as `appliedStatements` in `--json` output, each with its `statement`, `description` and `operationIndexes`, the positions in `operations` of the operations it accounts for. A coordinate in a contract with no namespaces has no `namespaceId` in the JSON. A statement whose storage does not change, such as a model whose table name is kept with `@@map`, is listed with no operations. When the storage did not change at all and no statement needs an operation, no migration package is written and the output says so; when the storage changed but the planner planned nothing, planning fails with `MIGRATION.PLANNING_FAILED`, statements or not. A statement the planner cannot carry out, for example on a table whose control policy is not `managed`, fails planning with `MIGRATION.PLANNING_FAILED`, and the refused statement is carried in the error's conflicts. +**Rename statements:** each `--rename` names the old and the new name in contract vocabulary, never a table or a column. Each side is one of `Model`, `namespace.Model`, `Model.field` or `namespace.Model.field`; both sides name a model, or both name a field of the same model. Names match exactly, including case. A name without a namespace resolves when exactly one namespace declares the model. A field's model is named as the destination contract names it, so a model rename followed by a field rename on it is `--rename Profile:User --rename User.name:User.fullName`. The old name must exist in the starting contract and not in the destination; the new name must exist in the destination and not in the starting contract. Statements resolve before anything is written; a statement that is malformed fails with `MIGRATION.STATEMENT_INVALID`, and one that does not resolve fails with `MIGRATION.STATEMENT_UNRESOLVED`, naming what it searched and what it found. Each error's next step is the statement to type instead: the accepted forms, the statement written the right way round, the statements a model-and-field mix may have meant, or the order that works. A statement that has already been applied (the origin already has the new name and not the old one) says to leave it out. Statements cannot swap two names in one plan, and the error gives the three plans that do, through a temporary name. A plan from an empty database (`--from @empty`, or greenfield) has no models to rename, so every statement is unresolved. Moving a model to another namespace and renaming value objects are not supported in this release. The planner renames the table or column, and the constraints and indexes named after it, ahead of the rest of the plan. On Postgres, a check, a row-level-security policy, or an expression or partial index whose SQL names a renamed column is dropped and created again, and adding a check back reads every row of the table under an exclusive lock, so on a large table such a rename takes time; `migration.ts` contains the same `...this.renameTable(...)` and `...this.renameColumn(...)` calls you could have written by hand. On MongoDB a field is named by its stored name (`id @map("_id")` is `User._id`). A model rename renames the collection, which keeps its documents, indexes and validator (`renameCollection`); a field rename renames the field in every document that holds it (`renameField`), and only in that variant's documents for a variant's field; `migration.ts` calls these factories. The statements are listed under `Statements applied` after the operations, each with its number of operations, and as `appliedStatements` in `--json` output, each with its `statement`, `description` and `operationIndexes`, the positions in `operations` of the operations it accounts for. A coordinate in a contract with no namespaces has no `namespaceId` in the JSON. A statement whose storage does not change, such as a model whose table name is kept with `@@map`, is listed with no operations. When the storage did not change at all and no statement needs an operation, no migration package is written and the output says so; when the storage changed but the planner planned nothing, planning fails with `MIGRATION.PLANNING_FAILED`, statements or not. A statement the planner cannot carry out, for example on a table whose control policy is not `managed`, fails planning with `MIGRATION.PLANNING_FAILED`, and the refused statement is carried in the error's conflicts. **Data loss:** a plan that would lose data is written only once you say what each such operation means. The command asks one question per model, field or storage name whose data an operation would lose (dropping a table or a column, or a type change that can change values), before it writes anything, including the baseline package of an auto-baseline. Each question is answered with a statement: - `--delete ` lets the plan lose that data, for example `--delete Legacy` for a model or `--delete User.nickname` for a field. Data that no model of the starting contract stores is named by its storage name, as the question gives it, and can only be deleted. A field of a model the plan renames is named through the model's new name, as a field rename is written: after `--rename Profile:User`, a dropped `nickname` field is answered with `--delete User.nickname` or `--rename User.nickname:User.handle`. -- `--rename :` keeps the data of a model or field under a new name; a field's new name keeps its model (`--rename User.nickname:User.handle`). A rename on the command line is planned from the start, so the drop it replaces is never asked about. The question offers it only for a model or field the destination no longer has, so a field whose type changes is answered with `--delete`, and not on MongoDB, whose planner refuses renames in this release. There the question says how to keep the documents instead: rename the collection in `mongosh` on each database before a plan that drops it is applied; `db update` then drops nothing, and a migration written by `migration plan` still drops it. +- `--rename :` keeps the data of a model or field under a new name; a field's new name keeps its model (`--rename User.nickname:User.handle`). A rename on the command line is planned from the start, so the drop it replaces is never asked about. The question offers it only for a model or field the destination no longer has, so a field whose type changes is answered with `--delete`, and for a field whose model the destination names differently only after a `--rename` of the model, which the question then says to pass. On MongoDB, a field removed from a model is asked about too, and `--delete` removes it from every document that holds it (`unsetField`). Where nobody can answer (the run is not interactive, or `--yes` is set), the command fails with `CLI.CONSENT_REQUIRED` and lists every unanswered question with the flags that answer it. In a terminal it asks each question in turn: type `delete` to let the data go, or `rename :`; a rejected answer is asked again. A typed rename is planned again with the other renames, and when the plan still loses the subject's data the command fails with `MIGRATION.STATEMENT_DID_NOT_RESOLVE_LOSS`. A `--delete` that answers no question fails with `CLI.CONSENT_UNUSED` before anything is written. `--confirm` no longer consents to anything here. The deletes are listed under `Statements applied` after the renames, for example `delete model "Legacy" (1 operation)`, and in `--json` output each `appliedStatements` entry carries its `verb`, `rename` or `delete`; a delete's `statement` is `{ kind: 'delete', subject }`. The `⚠` marker stays on each destructive operation. diff --git a/skills/prisma-8/references/migrations.md b/skills/prisma-8/references/migrations.md index 0ff2871ffbb6..158d917aff20 100644 --- a/skills/prisma-8/references/migrations.md +++ b/skills/prisma-8/references/migrations.md @@ -375,8 +375,9 @@ On Postgres the operations are **methods on the `Migration` base class**, each t **Mongo** factories (from `@prisma/orm-mongo/target/migration`): -- Collections: `createCollection`, `dropCollection`, `validatedCollection`, `setValidation`. +- Collections: `createCollection`, `dropCollection`, `renameCollection(from, to)` (keeps the documents, indexes and validator), `validatedCollection`, `setValidation`. - Indexes: `createIndex`, `dropIndex`. +- Document rewrites: `renameField(collection, from, to, { filter? })` and `unsetField(collection, field, { filter? })`, each one `updateMany` that skips validation; `filter` limits them to one variant's documents, such as `{ filter: { type: 'bug' } }`. - Collection options: `collMod`. - Data transforms: `dataTransform(name, { check, run })` (free factory; `check`/`run` use Mongo query-plan shapes). @@ -476,7 +477,7 @@ If self-emit itself fails (e.g. the contract has moved on and the operations no The concept: the planner cannot tell whether a dropped model or field was meant to go, or was renamed. So before `migration plan` writes a package, or `db update` applies, the command asks one question per model, field or storage name an operation would lose data from. `db update` also asks one per model whose rows an operation would open up, such as dropping a row-level-security policy. Each question is answered by a statement: -- `--delete ` lets the data go: `--delete Legacy`, `--delete User.nickname`. A subject no model stores is named by its storage name (`public.audit_log` on Postgres, `audit_log` on SQLite) and can only be deleted. +- `--delete ` lets the data go: `--delete Legacy`, `--delete User.nickname`. A subject no model stores is named by its storage name (`public.audit_log` on Postgres, `audit_log` on SQLite, `profiles.nickname` for a MongoDB field when `db update` has no snapshot of the origin contract) and can only be deleted. On MongoDB, removing a field from a model is asked about, and `--delete` removes it from every document that holds it. - `--rename :` keeps it under a new name: `--rename Profile:User`, or for a field `--rename User.nickname:User.handle` (the new name keeps its model). A field of a renamed model is named through the new name (`--delete User.nickname` after `--rename Profile:User`). - `--allow ` (`db update` only) lets an operation that changes who can read or write the model's rows run: one flag per operation, so a policy drop and disabling row-level security on one model take `--allow User --allow User`. @@ -499,7 +500,7 @@ In a terminal the command asks each question in turn; type `delete`, `allow`, or 6. **Aggregate `check` closure in Postgres `this.dataTransform`.** Returning `count(*)` or `bool_and(...)` breaks the precheck/postcheck contract — both sides resolve to constants. Use a rowset shape: `select('id').where().limit(1)`. 7. **Two contract references in one migration.** Building a query plan against a different contract than the one passed to `this.dataTransform(endContract, ...)` raises `MIGRATION.DATA_TRANSFORM_CONTRACT_MISMATCH`. Always import `endContract` once at module scope and use the same reference. 11. **Calling Postgres operations as free functions.** `addColumn('public', 'user', {...})` does not exist as an import; the operations are `this.addColumn({ schema, table, column })` and friends on the `Migration` base class, with `col(...)` building the column. Only `col`, `rawSql`, `placeholder`, `Migration`, and `MigrationCLI` are imported. -8. **Renaming a model or field without a statement (Postgres, SQLite).** The planner cannot tell a rename from a drop and a create, so without a statement it plans a destructive drop+add. State the rename on the command line: `migration plan --name --rename User:Person --rename Person.email:Person.emailAddress`, or the same `--rename` flags on `db update`. A field is named through its model as the new contract names it. The planner then renames the table or column and the constraints and indexes named after it. For a rename statements do not cover, such as a model that keeps its name but changes its `@@map`, hand-edit `migration.ts` with `...this.renameTable(...)` or `...this.renameColumn(...)`, then self-emit. +8. **Renaming a model or field without a statement.** The planner cannot tell a rename from a drop and a create, so without a statement it plans a destructive drop+add. State the rename on the command line: `migration plan --name --rename User:Person --rename Person.email:Person.emailAddress`, or the same `--rename` flags on `db update`. A field is named through its model as the new contract names it. The planner then renames the table or column and the constraints and indexes named after it; on MongoDB it renames the collection, and the field in every document that holds it (only in that variant's documents for a variant's field). On MongoDB a field is named by its stored name: `id @map("_id")` is `User._id`. For a rename statements do not cover, such as a model that keeps its name but changes its `@@map`, hand-edit `migration.ts` with `...this.renameTable(...)` or `...this.renameColumn(...)` (Postgres, SQLite) or `renameCollection(...)` (MongoDB), then self-emit. 9. **Planning with no `db` ref and no `--from` in a project that already has migrations.** The origin falls through to the empty database, which would make the plan a full-create migration; `migration plan` refuses with `MIGRATION.PLAN_ORIGIN_UNKNOWN` rather than writing it. Pick the exit that matches your intent — the error lists them, and `references/migration-model.md` § *The trap* explains which to choose. 10. **Hand-authoring `migration.ts` from a blank file, or rewriting the rendered import line.** Migration files are framework-rendered — let `prisma migration plan` (or `migration new`) render the package, then edit only the holes the framework leaves for you. On Postgres leave the rendered `@internal/postgres/migration` (or `@internal/sqlite/migration`) import path alone; on Mongo leave `@prisma/orm-mongo/target/migration` as rendered. Add symbols to the existing import line rather than introducing new import paths. @@ -508,7 +509,7 @@ In a terminal the command asks each question in turn; type `delete`, `allow`, or - **Runtime-apply migrations.** Prisma 8 doesn't apply pending migrations from your app's startup code (the "Drizzle pattern" for serverless / edge). Workaround: run `prisma db migrate` from your deploy pipeline before the app starts. If you need runtime-apply built-in, file a feature request via the `references/feedback.md` skill. - **Seeds-as-first-class.** Prisma 8 doesn't ship a `prisma db seed` equivalent. Workaround: write a TypeScript script that imports your `db` instance and runs your setup queries; invoke it from `package.json`'s scripts. If you need first-class seeding, file a feature request via the `references/feedback.md` skill. - **Migration squashing.** Prisma 8 doesn't squash older migrations into a baseline. They accumulate; for very large histories, manual baseline-and-truncate is the path. If you need built-in squashing, file a feature request via the `references/feedback.md` skill. -- **Renames beyond models and fields, and renames on MongoDB.** `--rename` covers models and fields on Postgres and SQLite. It does not rename enum values, namespaces or value objects, and does not move a model to another namespace. On MongoDB, `--rename` fails with `MIGRATION.PLANNING_FAILED` and plans nothing. A data-loss question about a dropped collection offers only `--delete` and says how to keep the documents by hand; a migration written by `migration plan` still drops the collection, so remove that operation or do not apply it where the collection was renamed. Workaround: follow the error's advice and make the rename by hand in `mongosh` on each database, before a plan made without `--rename` is applied there: `renameCollection` for a model, and for a field turn off the collection's validator, drop each unique index that includes the field, then `updateMany({}, { $rename: ... })`. Applying the plan then creates the new indexes and turns the validator back on. There is no Mongo rename operation to write in `migration.ts`. If you need one of these, file a feature request via the `references/feedback.md` skill. +- **Renames beyond models and fields.** `--rename` covers models and fields on Postgres, SQLite and MongoDB. It does not rename enum values, namespaces or value objects (on MongoDB, a field inside a value object), and does not move a model to another namespace. On MongoDB it also refuses `_id`, a field whose stored name contains `.` or starts with `$`, and a time-series collection; each refusal (`MIGRATION.PLANNING_FAILED`) says what to do instead. If you need one of these, file a feature request via the `references/feedback.md` skill. ## Graph and history commands diff --git a/test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts b/test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts new file mode 100644 index 000000000000..768791ef0e44 --- /dev/null +++ b/test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts @@ -0,0 +1,403 @@ +/** + * Journeys (MongoDB) for `--rename` and `--delete`, on PSL contracts with documents. + * + * R1 plans and migrates a model rename, a field rename with a unique index, a variant's field + * rename and a field removal. R2 does the same through `db update`, then runs it again. R3 removes + * a field through `db update` with no snapshot of the origin contract. R4 follows the question + * about a field whose model the destination names differently. Every refusal is followed as + * printed. + */ + +import { mkdirSync, rmSync, writeFileSync } from 'node:fs'; +import { rm } from 'node:fs/promises'; +import { join } from 'node:path'; +import { timeouts } from '@repo/test-utils'; +import { type Db, MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, afterEach, beforeAll, describe, expect, it } from 'vitest'; +import { fixtureAppDir } from '../utils/cli-test-helpers'; +import { + type EngineCommandResult, + engineError, + type JourneyContext, + latestMigrationDirName, + parseJsonOutput, + runContractEmit, + runDbUpdate, + runDbVerify, + runMigrate, + runMigrationPlan, +} from '../utils/journey-test-helpers'; + +const BEFORE = `// use prisma-8 + +model Profile { + id ObjectId @id @map("_id") + name String + nickname String + + @@unique([name]) + @@map("profiles") +} + +model Task { + id ObjectId @id @map("_id") + title String + type String + + @@discriminator(type) + @@map("tasks") +} + +model Bug { + id ObjectId @id @map("_id") + score String + + @@base(Task, "bug") +} + +model Feature { + id ObjectId @id @map("_id") + score String + + @@base(Task, "feature") +} +`; + +const AFTER = `// use prisma-8 + +model User { + id ObjectId @id @map("_id") + fullName String + + @@unique([fullName]) + @@map("users") +} + +model Task { + id ObjectId @id @map("_id") + title String + type String + + @@discriminator(type) + @@map("tasks") +} + +model Bug { + id ObjectId @id @map("_id") + severity String + + @@base(Task, "bug") +} + +model Feature { + id ObjectId @id @map("_id") + score String + + @@base(Task, "feature") +} +`; + +const RENAMES = [ + '--rename', + 'Profile:User', + '--rename', + 'User.name:User.fullName', + '--rename', + 'Bug.score:Bug.severity', +]; + +function withDatabase(baseUri: string, dbName: string): string { + const url = new URL(baseUri); + url.pathname = `/${dbName}`; + return url.toString(); +} + +function setupProject(connectionString: string): JourneyContext { + const testDir = join( + fixtureAppDir, + `test-mongo-renames-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + const outputDir = join(testDir, 'output'); + mkdirSync(outputDir, { recursive: true }); + mkdirSync(join(testDir, 'migrations'), { recursive: true }); + writeFileSync( + join(testDir, 'package.json'), + JSON.stringify({ + name: 'mongo-renames-journey', + private: true, + type: 'module', + dependencies: { '@prisma/orm-mongo': 'workspace:0.16.0' }, + }), + ); + const configPath = join(testDir, 'prisma.config.ts'); + writeFileSync( + configPath, + [ + "import { defineConfig as ormConfig } from '@prisma/orm-mongo/config';", + "import { definePrismaConfig } from '@prisma/cli-engine';", + '', + 'export default definePrismaConfig({', + ' orm: ormConfig({', + " contract: './contract.prisma',", + " output: 'output',", + ` db: { connection: ${JSON.stringify(connectionString)} },`, + ' }),', + '});', + '', + ].join('\n'), + ); + return { testDir, configPath, outputDir }; +} + +async function useSchema(ctx: JourneyContext, schema: string, label: string): Promise { + writeFileSync(join(ctx.testDir, 'contract.prisma'), schema); + const emitted = await runContractEmit(ctx); + expect(emitted.exitCode, `${label}: emit: ${emitted.stderr}`).toBe(0); +} + +/** The flags a refusal's next actions print for `verb`, as they would be typed. */ +function printedFlags(result: EngineCommandResult, verb: string): readonly string[] { + const actions = JSON.stringify(engineError(result)?.nextActions ?? []); + const pattern = new RegExp(`--${verb} (?:'([^']+)'|([^\\s"\\\\]+))`, 'g'); + return [...actions.matchAll(pattern)].flatMap((match) => [ + `--${verb}`, + match[1] ?? match[2] ?? '', + ]); +} + +async function seed(db: Db): Promise { + await db.collection('profiles').insertMany([ + { name: 'Ada', nickname: 'a' }, + { name: 'Bo', nickname: 'b' }, + ]); + await db.collection('tasks').insertMany([ + { title: 'crash', type: 'bug', score: 'high' }, + { title: 'dark mode', type: 'feature', score: 'p1' }, + ]); +} + +async function documents(db: Db, collection: string) { + return db + .collection(collection) + .find({}, { projection: { _id: 0 }, sort: { _id: 1 } }) + .toArray(); +} + +async function expectEndState(ctx: JourneyContext, db: Db, label: string): Promise { + const collections = (await db.listCollections().toArray()) + .map(({ name }) => name) + .filter((name) => !name.startsWith('_prisma')) + .sort(); + expect( + { + collections, + users: await documents(db, 'users'), + tasks: await documents(db, 'tasks'), + }, + `${label}: documents under the new names`, + ).toEqual({ + collections: ['tasks', 'users'], + users: [{ fullName: 'Ada' }, { fullName: 'Bo' }], + tasks: [ + { title: 'crash', type: 'bug', severity: 'high' }, + { title: 'dark mode', type: 'feature', score: 'p1' }, + ], + }); + await expect( + db.collection('users').updateOne({ fullName: 'Ada' }, { $set: { fullName: 'Ada L.' } }), + `${label}: a user accepts an update`, + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + db.collection('tasks').updateOne({ type: 'bug' }, { $set: { severity: 'low' } }), + `${label}: a bug accepts an update`, + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + db.collection('tasks').updateOne({ type: 'feature' }, { $set: { score: 'p2' } }), + `${label}: a feature accepts an update`, + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + db.collection('users').insertOne({ fullName: 'Bo' }), + `${label}: the unique index is on the new field`, + ).rejects.toThrow(/duplicate key/); + + const verify = await runDbVerify(ctx, ['--schema-only']); + expect(verify.exitCode, `${label}: db verify --schema-only: ${verify.stderr}`).toBe(0); + const further = await runDbUpdate(ctx, ['--dry-run', '--no-interactive', '--json']); + expect(further.exitCode, `${label}: further plan: ${further.stderr}`).toBe(0); + expect( + parseJsonOutput<{ plan: { operations: readonly unknown[] } }>(further).plan.operations, + `${label}: a further plan is empty`, + ).toEqual([]); +} + +describe('Journeys (MongoDB): --rename and --delete', { + timeout: timeouts.spinUpMongoMemoryServer, +}, () => { + let replSet: MongoMemoryReplSet; + let client: MongoClient; + const created = new Set(); + + beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + instanceOpts: [ + { launchTimeout: timeouts.spinUpMongoMemoryServer, storageEngine: 'wiredTiger' }, + ], + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + }, timeouts.spinUpMongoMemoryServer); + + afterEach(async () => { + for (const dir of created) { + await rm(dir, { recursive: true, force: true }).catch(() => {}); + } + created.clear(); + }); + + afterAll(async () => { + await client?.close().catch(() => {}); + await replSet?.stop().catch(() => {}); + }, timeouts.spinUpMongoMemoryServer); + + function project(dbName: string) { + const ctx = setupProject(withDatabase(replSet.getUri(), dbName)); + created.add(ctx.testDir); + return { ctx, db: client.db(dbName) }; + } + + it('R1: migration plan refuses until the printed --delete is given, then migrate renames and removes', async () => { + const { ctx, db } = project('mongo_renames_plan'); + await useSchema(ctx, BEFORE, 'R1.01'); + const initial = await runMigrationPlan(ctx, ['--name', 'initial']); + expect(initial.exitCode, `R1.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(ctx)).exitCode, 'R1.03: migrate initial').toBe(0); + await seed(db); + const origin = latestMigrationDirName(ctx); + await useSchema(ctx, AFTER, 'R1.04'); + + const refused = await runMigrationPlan(ctx, ['--from', origin, ...RENAMES, '--json']); + expect(refused.exitCode, 'R1.05: the removed field needs a statement').toBe(2); + expect(engineError(refused)?.code, 'R1.05').toBe('CLI.CONSENT_REQUIRED'); + const printed = printedFlags(refused, 'delete'); + expect(printed, 'R1.05: the printed flag').toEqual(['--delete', 'User.nickname']); + + const plan = await runMigrationPlan(ctx, [ + '--name', + 'tidy', + '--from', + origin, + ...RENAMES, + ...printed, + '--json', + ]); + expect(plan.exitCode, `R1.06: plan with the printed flag: ${plan.stderr}`).toBe(0); + const applied = await runMigrate(ctx); + expect(applied.exitCode, `R1.07: migrate: ${applied.stderr}`).toBe(0); + + await expectEndState(ctx, db, 'R1.08'); + }); + + it('R2: db update refuses until the printed --delete is given, then a second run fails on its first statement', async () => { + const { ctx, db } = project('mongo_renames_update'); + await useSchema(ctx, BEFORE, 'R2.01'); + const create = await runDbUpdate(ctx, ['--no-interactive']); + expect(create.exitCode, `R2.02: db update creates: ${create.stderr}`).toBe(0); + await seed(db); + await useSchema(ctx, AFTER, 'R2.03'); + + const refused = await runDbUpdate(ctx, ['--no-interactive', ...RENAMES, '--json']); + expect(refused.exitCode, 'R2.04: the removed field needs a statement').toBe(2); + expect(engineError(refused)?.code, 'R2.04').toBe('CLI.CONSENT_REQUIRED'); + expect(await documents(db, 'profiles'), 'R2.04: nothing changed').toHaveLength(2); + const printed = printedFlags(refused, 'delete'); + expect(printed, 'R2.04: the printed flag').toEqual(['--delete', 'User.nickname']); + + const updated = await runDbUpdate(ctx, ['--no-interactive', ...RENAMES, ...printed]); + expect(updated.exitCode, `R2.05: db update with the printed flag: ${updated.stderr}`).toBe(0); + + await expectEndState(ctx, db, 'R2.06'); + + const again = await runDbUpdate(ctx, ['--no-interactive', ...RENAMES, ...printed, '--json']); + expect(again.exitCode, 'R2.07: the same statements again').not.toBe(0); + expect( + { code: engineError(again)?.code, summary: engineError(again)?.summary }, + 'R2.07: it fails on the first statement', + ).toMatchObject({ + code: 'MIGRATION.STATEMENT_UNRESOLVED', + summary: expect.stringContaining('Profile:User'), + }); + }); + + it('R3: db update with no snapshot names a removed field by its collection, and --delete . removes it', async () => { + const { ctx, db } = project('mongo_renames_no_snapshot'); + await useSchema(ctx, BEFORE, 'R3.01'); + const create = await runDbUpdate(ctx, ['--no-interactive']); + expect(create.exitCode, `R3.02: db update creates: ${create.stderr}`).toBe(0); + await seed(db); + rmSync(join(ctx.testDir, 'migrations'), { recursive: true, force: true }); + await useSchema(ctx, BEFORE.replace(' nickname String\n', ''), 'R3.03'); + + const refused = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(refused.exitCode, 'R3.04: the removed field needs a statement').toBe(2); + const printed = printedFlags(refused, 'delete'); + expect(printed, 'R3.04: named by its collection').toEqual(['--delete', 'profiles.nickname']); + + const updated = await runDbUpdate(ctx, ['--no-interactive', ...printed]); + expect(updated.exitCode, `R3.05: db update with the printed flag: ${updated.stderr}`).toBe(0); + expect(await documents(db, 'profiles'), 'R3.05: the field is gone').toEqual([ + { name: 'Ada' }, + { name: 'Bo' }, + ]); + }); + + it('R4: a field whose model the destination names differently is renamed once the printed model rename is given', async () => { + const { ctx, db } = project('mongo_renames_model_first'); + await useSchema(ctx, BEFORE, 'R4.01'); + const create = await runDbUpdate(ctx, ['--no-interactive']); + expect(create.exitCode, `R4.02: db update creates: ${create.stderr}`).toBe(0); + await seed(db); + await useSchema( + ctx, + BEFORE.replace('model Profile {', 'model Person {').replace( + ' nickname String', + ' alias String', + ), + 'R4.03', + ); + + const refused = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(refused.exitCode, 'R4.04: the removed field needs a statement').toBe(2); + const question = JSON.stringify(engineError(refused)); + expect(question, 'R4.04: says to rename the model first').toContain( + 'also pass --rename Profile:', + ); + + const modelFirst = await runDbUpdate(ctx, [ + '--no-interactive', + '--rename', + 'Profile:Person', + '--json', + ]); + expect(modelFirst.exitCode, 'R4.05: the field question now offers a rename').toBe(2); + const renameForm = printedFlags(modelFirst, 'rename'); + expect(renameForm, 'R4.05: the printed rename form').toEqual([ + '--rename', + 'Person.nickname:Person.', + ]); + + const renamed = await runDbUpdate(ctx, [ + '--no-interactive', + '--rename', + 'Profile:Person', + '--rename', + 'Person.nickname:Person.alias', + ]); + expect(renamed.exitCode, `R4.06: db update with both renames: ${renamed.stderr}`).toBe(0); + expect(await documents(db, 'profiles'), 'R4.06: the values are kept').toEqual([ + { name: 'Ada', alias: 'a' }, + { name: 'Bo', alias: 'b' }, + ]); + }); +}); diff --git a/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md b/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md new file mode 100644 index 000000000000..7ea062ffca72 --- /dev/null +++ b/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md @@ -0,0 +1,45 @@ +--- +changes: + - id: mongo-rename-statements-carried-out + summary: | + On MongoDB, `migration plan` and `db update` carry out `--rename`: a model rename renames its collection, keeping the documents, indexes and validator, and a field rename renames the field in every document that holds it. They used to refuse every statement with `MIGRATION.PLANNING_FAILED`. A question about a dropped collection now offers `--rename` as well as `--delete`. + - id: mongo-removed-field-asks + summary: | + On MongoDB, removing a field from a model is now data loss. `migration plan` and `db update` ask about it, and fail with `CLI.CONSENT_REQUIRED` where nobody can answer, until `--delete .` or a `--rename` answers it. `--delete` removes the field from every document that holds it. + - id: storage-question-wording + summary: | + A data-loss question about data named by its storage name, because the origin contract is unknown, ends with `--delete loses that data` instead of `--delete loses its rows`. + detection: + glob: "**/*.{ts,mts,cts,js,mjs,cjs}" + matches: + - '--delete loses its rows' +--- + +# MongoDB carries out `--rename` and `--delete` + +## `mongo-rename-statements-carried-out` + +`--rename` now works on MongoDB, with the same syntax as on SQL. A field is named by its contract key, which on MongoDB is its stored name: `id @map("_id")` is `User._id`. + +```sh +prisma migration plan --name tidy --rename Profile:User --rename User.name:User.fullName +prisma db update --rename Profile:User --rename User.name:User.fullName +``` + +- A model rename whose collection name changes renames the collection (`renameCollection`), keeping its documents, indexes and validator. +- A field rename renames the field in every document that holds it (`renameField`), skipping validation. A variant's field is renamed only in that variant's documents. +- The planner refuses, with the reason and what to do instead: `_id`, a field whose stored name contains `.` or starts with `$`, a time-series collection, a target collection that already exists or that MongoDB rejects, and a collection whose control policy is not `managed`. + +If you renamed a collection or a field by hand in `mongosh` because `--rename` was refused, nothing changes for migrations you already applied. From now on, give the `--rename` statement instead. + +A script or test that expects `--rename` on MongoDB to fail with `MIGRATION.PLANNING_FAILED` now gets a plan. A script that reads `meta.unanswered` of `CLI.CONSENT_REQUIRED` for a dropped MongoDB collection now finds `verbs: ['rename', 'delete']` instead of `['delete']`. + +## `mongo-removed-field-asks` + +Removing a field from a MongoDB model used to update the collection's validator and ask nothing. Documents that still held the field then failed every update with `Document failed validation`. Now the plan removes the field from those documents (`unsetField`, a `destructive` operation) and asks first. Where nobody can answer, the command fails with `CLI.CONSENT_REQUIRED`, whose next actions name `--delete User.nickname` and `--rename 'User.nickname:User.'`. + +In a script or CI job that runs `prisma db update --no-interactive` or `prisma migration plan` after a field was removed, add `--delete .` for each removed field, or `--rename` if the field was renamed. When `db update` has no snapshot of the contract the database is at, the field is named by its collection: `--delete profiles.nickname`. + +## `storage-question-wording` + +A test that asserts the text of a data-loss question about data named by its storage name expects `--delete loses that data` where it expected `--delete loses its rows`. diff --git a/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md b/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md new file mode 100644 index 000000000000..84dbfdab6401 --- /dev/null +++ b/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md @@ -0,0 +1,77 @@ +--- +changes: + - id: rename-statements-capability-removed + summary: | + `TargetMigrationsCapability.renameStatements` (with `refused` and `keepDataByHand`), added in 8.0.0-rc.17, is removed: every target's planner carries out rename statements. The CLI no longer appends a target's text to a data-loss question. + detection: + glob: "**/*.{ts,mts,cts}" + matches: + - '(?` implementation adds `renameCollection(command)`. A Mongo driver's `run(wireCommand)` handles `kind: 'renameCollection'`, as the shipped driver does with `db.renameCollection(collection, to)`, without `dropTarget`. An adapter's DDL lowering maps the command to `new RenameCollectionWireCommand(command.collection, command.to)`. + +## `mongo-run-shaped-operations` + +Where code narrows `AnyMongoMigrationOperation` by `operation.operationClass === 'data'`, use the guard instead: + +```diff +- if (operation.operationClass === 'data') { ++ if (isMongoDataTransformOperation(operation)) { +``` + +`isMongoDataTransformOperation` comes from `@prisma/orm-family-mongo/query-ast/control`. An operation without `run` is a DDL operation. A hand-written `dataTransform` still has class `data`, so a policy that leaves out `data` (as `db update`'s does) still refuses it, and runs `renameField` and `unsetField`. + +## `mongo-update-many-bypass-validation` + +A hand-built lowered draft of kind `rawUpdateMany` adds `bypassDocumentValidation: false`. A driver's `updateMany` passes `bypassDocumentValidation: true` to MongoDB when the wire command carries it. + +## `mongo-op-factory-call-to-op` + +Code typed against the result of `toOp()` or `renderOps(...)` as `MongoMigrationPlanOperation` widens it to `AnyMongoMigrationOperation`, and narrows with `isMongoDataTransformOperation` where it needs `execute`. From cbb753480e4c270f259fb6436eea7b35c67d0989 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:18:23 +0200 Subject: [PATCH 16/57] TML-3478: ADR 188 classes planner rewrites by their effect; ADR 264 gains the collection rename Signed-off-by: willbot Signed-off-by: Will Madden --- .../adrs/ADR 188 - MongoDB migration operation model.md | 6 ++++-- ...es its storage verbatim, and a rename is an operation.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md index 17457d3dba3d..cb5cbad321b0 100644 --- a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md +++ b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md @@ -1,5 +1,7 @@ # ADR 188 — MongoDB migration operation model: data-driven commands and checks +> **Update — 2026-10:** document rewrites the planner writes for `--rename` and `--delete` statements are classed by their effect, not as `data`. See § Data transform operations. + ## Grounding example The migration planner has compared two contracts and determined that a unique ascending index on `users.email` needs to be created. It needs to produce an operation that the runner can execute. @@ -125,7 +127,7 @@ The envelope described above covers DDL operations — creating indexes, modifyi ```ts interface MongoDataTransformOperation extends MigrationPlanOperation { - readonly operationClass: 'data'; + readonly operationClass: MigrationOperationClass; readonly name: string; readonly precheck: readonly MongoDataTransformCheck[]; readonly run: readonly MongoQueryPlan[]; @@ -133,7 +135,7 @@ interface MongoDataTransformOperation extends MigrationPlanOperation { } ``` -The `operationClass` is always `'data'`. The `run` array contains `MongoQueryPlan` objects — aggregation pipelines or raw commands (`rawUpdateMany`, `rawInsertOne`, etc.) — rather than DDL commands. The `precheck` and `postcheck` arrays contain `MongoDataTransformCheck` objects instead of `MongoMigrationCheck`. +A transform a user writes, or one the planner scaffolds with a placeholder, has `operationClass: 'data'`. A rewrite the planner writes itself carries the class of its effect, as every other planned operation does: `renameField` is `widening` and `unsetField` is `destructive`. So `db update`, which does not allow `data`, can run a rename. The runner and the serializer tell a transform from a DDL operation by its shape (`run` against `execute`), not by its class. The `run` array contains `MongoQueryPlan` objects — aggregation pipelines or raw commands (`rawUpdateMany`, `rawInsertOne`, etc.) — rather than DDL commands. The `precheck` and `postcheck` arrays contain `MongoDataTransformCheck` objects instead of `MongoMigrationCheck`. `MongoDataTransformCheck` has the same four-field shape as `MongoMigrationCheck`: diff --git a/docs/architecture docs/adrs/ADR 264 - A model names its storage verbatim, and a rename is an operation.md b/docs/architecture docs/adrs/ADR 264 - A model names its storage verbatim, and a rename is an operation.md index c749ad742877..0c276e9f5b0b 100644 --- a/docs/architecture docs/adrs/ADR 264 - A model names its storage verbatim, and a rename is an operation.md +++ b/docs/architecture docs/adrs/ADR 264 - A model names its storage verbatim, and a rename is an operation.md @@ -67,7 +67,7 @@ A schema that declares no `@@map` and whose storage was created under the loweri - A model with no `@@map` names storage that can be quoted exactly as written, which is what a database created by an earlier Prisma version holds. - `contract infer` round-trips: a table whose name already equals the model name infers with no `@@map` and verifies clean, and a snake_case table still infers with `@@map`. - A rename needs a hand-written migration. The user states one line per renamed table and the operations follow from the contracts. -- MongoDB has no rename operation. A collection is renamed outside Prisma, with `renameCollection`. +- MongoDB renames a collection with the `renameCollection` operation, which a `--rename` statement plans (see [Migration System § Statements](../subsystems/7.%20Migration%20System.md#statements)). ## Alternatives considered From 79c531939ab50b4b28b116147f275b14a32b68f0 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:20:51 +0200 Subject: [PATCH 17/57] TML-3478: ADR 188 says which transforms are data, whoever writes them Signed-off-by: willbot Signed-off-by: Will Madden --- .../adrs/ADR 188 - MongoDB migration operation model.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md index cb5cbad321b0..6fe746dfeccd 100644 --- a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md +++ b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md @@ -135,7 +135,7 @@ interface MongoDataTransformOperation extends MigrationPlanOperation { } ``` -A transform a user writes, or one the planner scaffolds with a placeholder, has `operationClass: 'data'`. A rewrite the planner writes itself carries the class of its effect, as every other planned operation does: `renameField` is `widening` and `unsetField` is `destructive`. So `db update`, which does not allow `data`, can run a rename. The runner and the serializer tell a transform from a DDL operation by its shape (`run` against `execute`), not by its class. The `run` array contains `MongoQueryPlan` objects — aggregation pipelines or raw commands (`rawUpdateMany`, `rawInsertOne`, etc.) — rather than DDL commands. The `precheck` and `postcheck` arrays contain `MongoDataTransformCheck` objects instead of `MongoMigrationCheck`. +A transform built with `dataTransform`, whether a user writes it or the planner scaffolds it with a placeholder, has `operationClass: 'data'`. The `renameField` and `unsetField` rewrites carry the class of their effect, whoever writes them, as every other operation does: `renameField` is `widening` and `unsetField` is `destructive`. So `db update`, which does not allow `data`, can run a rename. The runner and the serializer tell a transform from a DDL operation by its shape (`run` against `execute`), not by its class. The `run` array contains `MongoQueryPlan` objects — aggregation pipelines or raw commands (`rawUpdateMany`, `rawInsertOne`, etc.) — rather than DDL commands. The `precheck` and `postcheck` arrays contain `MongoDataTransformCheck` objects instead of `MongoMigrationCheck`. `MongoDataTransformCheck` has the same four-field shape as `MongoMigrationCheck`: From 6a204a1ee050bc65be0e3f0f0f562fde516e8498 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:21:32 +0200 Subject: [PATCH 18/57] TML-3478: the extension fragment detects AnyMongoDdlCommand and only MongoDB toOp and renderOps calls Signed-off-by: willbot Signed-off-by: Will Madden --- .../mongo-rename-delete-statements/extension/instructions.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md b/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md index 84dbfdab6401..bda1a3dfd611 100644 --- a/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md +++ b/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md @@ -13,7 +13,7 @@ changes: detection: glob: "**/*.{ts,mts,cts}" matches: - - '(? Date: Fri, 9 Oct 2026 12:37:51 +0200 Subject: [PATCH 19/57] TML-3478: MongoDB operations that run queries are DML operations MongoDataTransformOperation becomes MongoDmlOperation, MongoDataTransformCheck MongoDmlCheck, and isMongoDataTransformOperation isMongoDmlOperation, since the planner's document rewrites have that shape with class widening or destructive. The dataTransform factory keeps its name. The serializer and runner helpers, the docs and the extension fragment use the same term. Signed-off-by: willbot Signed-off-by: Will Madden --- .../subsystems/7. Migration System.md | 4 +-- .../4-query/query-ast/src/exports/control.ts | 6 ++-- .../src/migration-operation-types.ts | 28 ++++++++------- .../query-ast/test/migration-helpers.test.ts | 10 +++--- .../9-family/src/core/mongo-migration.ts | 4 +-- .../core/migrations/migration-factories.ts | 16 ++++----- .../core/migrations/mongo-ops-serializer.ts | 26 +++++++------- .../src/core/migrations/mongo-runner.ts | 34 +++++++------------ .../test/data-transform.test.ts | 6 ++-- .../1-mongo-target/test/mongo-runner.test.ts | 2 +- .../extension/instructions.md | 23 ++++++++----- 11 files changed, 79 insertions(+), 80 deletions(-) diff --git a/docs/architecture docs/subsystems/7. Migration System.md b/docs/architecture docs/subsystems/7. Migration System.md index 18c8ba2de419..9d9a616171d4 100644 --- a/docs/architecture docs/subsystems/7. Migration System.md +++ b/docs/architecture docs/subsystems/7. Migration System.md @@ -214,7 +214,7 @@ The MongoDB planner does the same with collections and documents ([`mongo-statem - A field that an origin model stores and its destination model no longer stores, in a collection the plan keeps, is data loss. The plan unsets it with `unsetField` (`$unset`, skipping validation) from the documents that hold it, with the same variant filters, and asks about it as a `field` subject, so `--delete User.nickname` or a `--rename` answers it. Without an origin contract, a removed field is found from the origin validator's top-level properties and named `.`; a collection whose destination has no validator is skipped, and a variant's removed field is not found. - Operations run in this order: collection renames, collection creates, index drops, document rewrites, index creates, validator changes, option changes, collection drops. Rewrites run after the index drops, so a unique index on the old field does not fail once documents lose it, and before the index creates, so a unique index on the new field sees the moved values. -`renameCollection` and `renameField` are `widening` and `unsetField` is `destructive`, so `db update` runs them. The runner and the operation serializer tell a document rewrite from a DDL operation by its shape (`run` against `execute`), not by its class; a hand-written `dataTransform` keeps the class `data`. Each rewrite's after-check, that no matching document holds the old name, lets a run that stopped part way finish when run again. The `migration.ts` that `migration plan` writes calls `renameCollection(...)`, `renameField(...)` and `unsetField(...)` from the target's migration module. +`renameCollection` and `renameField` are `widening` and `unsetField` is `destructive`, so `db update` runs them. Document rewrites are DML operations (`MongoDmlOperation`): the runner and the operation serializer tell a DML operation from a DDL operation by its shape (`run` against `execute`), with `isMongoDmlOperation`, not by its class; a hand-written `dataTransform` is a DML operation that keeps the class `data`. Each rewrite's after-check, that no matching document holds the old name, lets a run that stopped part way finish when run again. The `migration.ts` that `migration plan` writes calls `renameCollection(...)`, `renameField(...)` and `unsetField(...)` from the target's migration module. **Result.** The planner's success result carries `appliedStatements` (`AppliedMigrationStatement`): one entry per statement, in order, with `operationIndexes`: the positions, in the plan's operations, of the operations the statement accounts for, empty when its storage did not change. Statements point at operations by position because operation ids are not unique within a plan (on Postgres, same-named tables in two schemas share them), and ids stay as they are so that re-emitting an existing migration does not change its hash. The framework's `describeMigrationStatement` writes the statement in domain names, such as `rename model "Profile" to "User"`, for every family's messages and for the CLI. `migration plan` and `db update` print each statement with its description and number of operations, and their JSON output carries the description and the operation positions, which index the output's `operations`. Messages name a model of the unbound namespace by its name alone, and JSON output leaves `namespaceId` out of a coordinate in the unbound namespace (`migrationStatementJson`), because that namespace's id is internal. @@ -472,7 +472,7 @@ DDL operations carry commands like `CreateIndexCommand`, `CreateCollectionComman Data transforms carry `operationClass: 'data'` and use the **same three-phase envelope** as DDL operations. The runner has a single execution loop; there is no DT-specific dispatch path. The two families fill the envelope with their own query shapes: - **Postgres.** The `dataTransform` factory lowers the user-authored `check` / `run` query plans into the standard envelope at factory time. `precheck` is `[{ sql: 'SELECT EXISTS () AS ok', params: , … }]` (asserts "work to do"); `execute` carries the run plans as `SqlMigrationPlanOperationStep` entries with `params` threaded through `driver.query`; `postcheck` is `[{ sql: 'SELECT NOT EXISTS () AS ok', params: , … }]` (asserts "work done"). Idempotent re-apply uses the same pre-satisfied-skip path that DDL ops already use — when the postcheck is already satisfied, the runner records a `postcheck_pre_satisfied` skip without re-running execute. Routing the run-plan parameters through the driver's parameter binder (rather than inlining values into SQL text) is strictly safer than rolling a per-driver literal serializer. -- **Mongo.** The `run` array carries `MongoQueryPlan` objects. Checks use the same `{ description, source, filter, expect }` shape as DDL checks, with `MongoQueryPlan` as the source type. Both are evaluated by the same `FilterEvaluator` path. DML commands execute through `Adapter.lower()` → `Driver.execute()` — the same transport as runtime queries. The runner and the serializer recognise such an operation by its `run` array, not by its class: the planner's own document rewrites (`renameField`, `unsetField`) have the same shape but carry the class of their effect, `widening` or `destructive`, so `db update` runs them. +- **Mongo.** The `run` array carries `MongoQueryPlan` objects. Checks use the same `{ description, source, filter, expect }` shape as DDL checks, with `MongoQueryPlan` as the source type. Both are evaluated by the same `FilterEvaluator` path. DML commands execute through `Adapter.lower()` → `Driver.execute()` — the same transport as runtime queries. Such an operation is a DML operation (`MongoDmlOperation`), and the runner and the serializer recognise it by its `run` array, with `isMongoDmlOperation`, not by its class: the planner's own document rewrites (`renameField`, `unsetField`) have the same shape but carry the class of their effect, `widening` or `destructive`, so `db update` runs them. The shared envelope means policy gating on `operationClass: 'data'` (CI checks, audit, "this run may not perform data work") is uniform across families and decoupled from how each family describes its check/run payload. diff --git a/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts b/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts index 19f63aff1080..f29e84e2b188 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts @@ -28,11 +28,11 @@ export { ListCollectionsCommand, ListIndexesCommand } from '../inspection-comman export { buildIndexOpId, defaultMongoIndexName, keysToKeySpec } from '../migration-helpers'; export type { AnyMongoMigrationOperation, - MongoDataTransformCheck, - MongoDataTransformOperation, + MongoDmlCheck, + MongoDmlOperation, MongoMigrationCheck, MongoMigrationPlanOperation, MongoMigrationStep, } from '../migration-operation-types'; -export { isMongoDataTransformOperation } from '../migration-operation-types'; +export { isMongoDmlOperation } from '../migration-operation-types'; export type { MongoFilterVisitor } from '../visitors'; diff --git a/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts b/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts index ade8ba24108f..90c368e23903 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts @@ -25,35 +25,37 @@ export interface MongoMigrationPlanOperation extends MigrationPlanOperation { readonly postcheck: readonly MongoMigrationCheck[]; } -export interface MongoDataTransformCheck { +export interface MongoDmlCheck { readonly description: string; readonly source: MongoQueryPlan; readonly filter: MongoFilterExpr; readonly expect: 'exists' | 'notExists'; } -export interface MongoDataTransformOperation extends MigrationPlanOperation { +/** + * An operation that runs queries (`run`) where a DDL operation executes commands (`execute`): a + * `dataTransform`, with class `data`, or a document rewrite the planner plans, with the class of + * its effect. + */ +export interface MongoDmlOperation extends MigrationPlanOperation { readonly operationClass: MigrationOperationClass; - /** - * Human-readable label for this data transform. - */ readonly name: string; /** - * Optional opt-in routing identity. Presence opts the transform into - * invariant-aware routing; absence means the transform is - * path-dependent and not referenceable from refs. + * Optional opt-in routing identity. Presence opts the operation into + * invariant-aware routing; absence means it is path-dependent and not + * referenceable from refs. */ readonly invariantId?: string; - readonly precheck: readonly MongoDataTransformCheck[]; + readonly precheck: readonly MongoDmlCheck[]; readonly run: readonly MongoQueryPlan[]; - readonly postcheck: readonly MongoDataTransformCheck[]; + readonly postcheck: readonly MongoDmlCheck[]; } -export type AnyMongoMigrationOperation = MongoMigrationPlanOperation | MongoDataTransformOperation; +export type AnyMongoMigrationOperation = MongoMigrationPlanOperation | MongoDmlOperation; -export function isMongoDataTransformOperation( +export function isMongoDmlOperation( operation: AnyMongoMigrationOperation, -): operation is MongoDataTransformOperation { +): operation is MongoDmlOperation { return 'run' in operation; } diff --git a/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts b/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts index af9db162a0aa..329bfaf6aa7c 100644 --- a/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts +++ b/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest'; import { buildIndexOpId, defaultMongoIndexName, keysToKeySpec } from '../src/migration-helpers'; -import { isMongoDataTransformOperation } from '../src/migration-operation-types'; +import { isMongoDmlOperation } from '../src/migration-operation-types'; describe('buildIndexOpId', () => { it('produces deterministic create ID for single key', () => { @@ -69,18 +69,18 @@ describe('keysToKeySpec', () => { }); }); -describe('isMongoDataTransformOperation', () => { +describe('isMongoDmlOperation', () => { it('tells an operation that runs queries from one that executes DDL, whatever its class', () => { const common = { id: 'op', label: 'op', precheck: [], postcheck: [] }; expect([ - isMongoDataTransformOperation({ ...common, operationClass: 'widening', execute: [] }), - isMongoDataTransformOperation({ + isMongoDmlOperation({ ...common, operationClass: 'widening', execute: [] }), + isMongoDmlOperation({ ...common, operationClass: 'widening', name: 'rewrite', run: [], }), - isMongoDataTransformOperation({ ...common, operationClass: 'data', name: 'data', run: [] }), + isMongoDmlOperation({ ...common, operationClass: 'data', name: 'data', run: [] }), ]).toEqual([false, true, true]); }); }); diff --git a/packages/2-mongo-family/9-family/src/core/mongo-migration.ts b/packages/2-mongo-family/9-family/src/core/mongo-migration.ts index f774663b0b4d..fa7ed2aaa0f9 100644 --- a/packages/2-mongo-family/9-family/src/core/mongo-migration.ts +++ b/packages/2-mongo-family/9-family/src/core/mongo-migration.ts @@ -12,9 +12,9 @@ import { MongoContractView } from './ir/mongo-contract-view'; * the abstract `targetId` member from `Migration`. * * The operation type parameter is `AnyMongoMigrationOperation` — the union - * of DDL-shaped `MongoMigrationPlanOperation` and `MongoDataTransformOperation` — + * of DDL-shaped `MongoMigrationPlanOperation` and `MongoDmlOperation` — * so subclasses can return a mix of schema operations (e.g. `createIndex`, - * `setValidation`) and data-transform operations (e.g. `dataTransform`). + * `setValidation`) and DML operations (e.g. `dataTransform`, `renameField`). * Mirrors the generic parameter used by `PlannerProducedMongoMigration`. * * Binds the framework base's `Start` / `End` contract generics so a subclass diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts index 5b2bfd6b664d..f294f1ec3571 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts @@ -1,6 +1,6 @@ import type { - MongoDataTransformCheck, - MongoDataTransformOperation, + MongoDmlCheck, + MongoDmlOperation, MongoFilterExpr, MongoIndexKey, } from '@internal/mongo-query-ast/control'; @@ -83,9 +83,9 @@ export function dataTransform( }; run: () => MongoQueryPlan | Buildable; }, -): MongoDataTransformOperation { - let precheck: readonly MongoDataTransformCheck[] = []; - let postcheck: readonly MongoDataTransformCheck[] = []; +): MongoDmlOperation { + let precheck: readonly MongoDmlCheck[] = []; + let postcheck: readonly MongoDmlCheck[] = []; if (options.check) { const source = resolveQuery(options.check.source()); @@ -162,7 +162,7 @@ function noDocumentHolds( collectionName: string, fields: readonly string[], filter: Document | undefined, -): MongoDataTransformCheck { +): MongoDmlCheck { return { description: `no document in ${collectionName} holds ${fields.length === 1 ? fields[0] : `both ${fields.join(' and ')}`}`, source: rewritePlan( @@ -204,7 +204,7 @@ export function renameField( from: string, to: string, options?: FieldRewriteOptions, -): MongoDataTransformOperation { +): MongoDmlOperation { assertRewritableField(collectionName, from); assertRewritableField(collectionName, to); if (from === to) { @@ -231,7 +231,7 @@ export function unsetField( collectionName: string, field: string, options?: FieldRewriteOptions, -): MongoDataTransformOperation { +): MongoDmlOperation { assertRewritableField(collectionName, field); const filter = options?.filter; const id = `field.${collectionName}.${field}.unset${idSuffix(filter)}`; diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts index b792cf2d7e00..f2a3f2c9922d 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts @@ -11,8 +11,8 @@ import { ListCollectionsCommand, ListIndexesCommand, MongoAndExpr, - type MongoDataTransformCheck, - type MongoDataTransformOperation, + type MongoDmlCheck, + type MongoDmlOperation, MongoExistsExpr, MongoFieldFilter, type MongoFilterExpr, @@ -312,14 +312,14 @@ const DdlOperationJson = type({ postcheck: 'Record[]', }); -const DataTransformCheckJson = type({ +const DmlCheckJson = type({ description: 'string', source: 'Record', filter: 'Record', expect: '"exists" | "notExists"', }); -const DataTransformOperationJson = type({ +const DmlOperationJson = type({ id: 'string', label: 'string', operationClass: '"additive" | "widening" | "destructive" | "data"', @@ -751,7 +751,7 @@ function deserializeStep(json: unknown): MongoMigrationStep { }; } -function isDataTransformJson(json: unknown): boolean { +function isDmlOperationJson(json: unknown): boolean { return Array.isArray(recordValue(json, 'run', 'migration operation')); } @@ -767,8 +767,8 @@ function deserializeDdlOp(json: unknown): MongoMigrationPlanOperation { }; } -function deserializeDataTransformCheck(json: unknown): MongoDataTransformCheck { - const data = validate(DataTransformCheckJson, json, 'data transform check'); +function deserializeDmlCheck(json: unknown): MongoDmlCheck { + const data = validate(DmlCheckJson, json, 'DML operation check'); return { description: data.description, source: deserializeMongoQueryPlan(data.source), @@ -777,22 +777,22 @@ function deserializeDataTransformCheck(json: unknown): MongoDataTransformCheck { }; } -function deserializeDataTransformOp(json: unknown): MongoDataTransformOperation { - const data = validate(DataTransformOperationJson, json, 'data transform operation'); +function deserializeDmlOp(json: unknown): MongoDmlOperation { + const data = validate(DmlOperationJson, json, 'DML operation'); return { id: data.id, label: data.label, operationClass: data.operationClass, name: data.name, - precheck: data.precheck.map(deserializeDataTransformCheck), + precheck: data.precheck.map(deserializeDmlCheck), run: data.run.map(deserializeMongoQueryPlan), - postcheck: data.postcheck.map(deserializeDataTransformCheck), + postcheck: data.postcheck.map(deserializeDmlCheck), }; } export function deserializeMongoOp(json: unknown): AnyMongoMigrationOperation { - if (isDataTransformJson(json)) { - return deserializeDataTransformOp(json); + if (isDmlOperationJson(json)) { + return deserializeDmlOp(json); } return deserializeDdlOp(json); } diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts index c27b3e936b3b..8c2534759200 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts @@ -22,12 +22,12 @@ import type { MongoContract } from '@internal/mongo-contract'; import type { MongoAdapter, MongoDriver } from '@internal/mongo-lowering'; import type { AnyMongoMigrationOperation, - MongoDataTransformCheck, - MongoDataTransformOperation, + MongoDmlCheck, + MongoDmlOperation, MongoInspectionCommandVisitor, MongoMigrationCheck, } from '@internal/mongo-query-ast/control'; -import { isMongoDataTransformOperation } from '@internal/mongo-query-ast/control'; +import { isMongoDmlOperation } from '@internal/mongo-query-ast/control'; import { blindCast } from '@internal/utils/casts'; import { InternalError } from '@internal/utils/internal-error'; import { notOk, ok, type Result } from '@internal/utils/result'; @@ -116,8 +116,8 @@ export class MongoMigrationRunner { for (const operation of operations) { options.callbacks?.onOperationStart?.(operation); try { - if (isMongoDataTransformOperation(operation)) { - const result = await this.executeDataTransform( + if (isMongoDmlOperation(operation)) { + const result = await this.executeDmlOperation( operation, adapter, driver, @@ -298,8 +298,8 @@ export class MongoMigrationRunner { } } - private async executeDataTransform( - op: MongoDataTransformOperation, + private async executeDmlOperation( + op: MongoDmlOperation, adapter: MongoAdapter, driver: MongoDriver, filterEvaluator: FilterEvaluator, @@ -308,7 +308,7 @@ export class MongoMigrationRunner { runPostchecks: boolean, ): Promise<{ executed: boolean; failure?: MongoMigrationRunnerResult }> { if (runPostchecks && runIdempotency && op.postcheck.length > 0) { - const allSatisfied = await this.evaluateDataTransformChecks( + const allSatisfied = await this.evaluateDmlChecks( op.postcheck, adapter, driver, @@ -318,12 +318,7 @@ export class MongoMigrationRunner { } if (runPrechecks && op.precheck.length > 0) { - const passed = await this.evaluateDataTransformChecks( - op.precheck, - adapter, - driver, - filterEvaluator, - ); + const passed = await this.evaluateDmlChecks(op.precheck, adapter, driver, filterEvaluator); if (!passed) { return { executed: false, @@ -346,12 +341,7 @@ export class MongoMigrationRunner { } if (runPostchecks && op.postcheck.length > 0) { - const passed = await this.evaluateDataTransformChecks( - op.postcheck, - adapter, - driver, - filterEvaluator, - ); + const passed = await this.evaluateDmlChecks(op.postcheck, adapter, driver, filterEvaluator); if (!passed) { return { executed: false, @@ -369,8 +359,8 @@ export class MongoMigrationRunner { return { executed: true }; } - private async evaluateDataTransformChecks( - checks: readonly MongoDataTransformCheck[], + private async evaluateDmlChecks( + checks: readonly MongoDmlCheck[], adapter: MongoAdapter, driver: MongoDriver, filterEvaluator: FilterEvaluator, diff --git a/packages/3-mongo-target/1-mongo-target/test/data-transform.test.ts b/packages/3-mongo-target/1-mongo-target/test/data-transform.test.ts index 39d5e093a8a9..5a9514385c8f 100644 --- a/packages/3-mongo-target/1-mongo-target/test/data-transform.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/data-transform.test.ts @@ -2,7 +2,7 @@ import { CliStructuredError } from '@internal/errors/control'; import { placeholder } from '@internal/errors/migration'; import type { AnyMongoMigrationOperation, - MongoDataTransformOperation, + MongoDmlOperation, } from '@internal/mongo-query-ast/control'; import { AggregateCommand, @@ -20,11 +20,11 @@ import { serializeMongoOps, } from '../src/core/migrations/mongo-ops-serializer'; -function asDataTransformOp(op: AnyMongoMigrationOperation): MongoDataTransformOperation { +function asDataTransformOp(op: AnyMongoMigrationOperation): MongoDmlOperation { if (op.operationClass !== 'data') { throw new Error('Expected data transform operation'); } - return op as MongoDataTransformOperation; + return op as MongoDmlOperation; } function makePlanMeta() { diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts index d1a6473f3223..0b08446305a4 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts @@ -280,7 +280,7 @@ async function execute( }); } -describe('MongoMigrationRunner.executeDataTransform', () => { +describe('MongoMigrationRunner.executeDmlOperation', () => { it('runs the DML wire command once when there are no checks to gate it', async () => { const harness = makeHarness(); const op = dataTransform('backfill-status', { run: () => makeRunPlan() }); diff --git a/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md b/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md index bda1a3dfd611..5101fde03895 100644 --- a/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md +++ b/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md @@ -14,14 +14,14 @@ changes: glob: "**/*.{ts,mts,cts}" matches: - '(?` implementation adds `renameCollection(command)`. A Mongo driver's `run(wireCommand)` handles `kind: 'renameCollection'`, as the shipped driver does with `db.renameCollection(collection, to)`, without `dropTarget`. An adapter's DDL lowering maps the command to `new RenameCollectionWireCommand(command.collection, command.to)`. -## `mongo-run-shaped-operations` +## `mongo-dml-operations` + +Rename the types where code names them: + +```diff +- import type { MongoDataTransformCheck, MongoDataTransformOperation } from '@prisma/orm-family-mongo/query-ast/control'; ++ import type { MongoDmlCheck, MongoDmlOperation } from '@prisma/orm-family-mongo/query-ast/control'; +``` Where code narrows `AnyMongoMigrationOperation` by `operation.operationClass === 'data'`, use the guard instead: ```diff - if (operation.operationClass === 'data') { -+ if (isMongoDataTransformOperation(operation)) { ++ if (isMongoDmlOperation(operation)) { ``` -`isMongoDataTransformOperation` comes from `@prisma/orm-family-mongo/query-ast/control`. An operation without `run` is a DDL operation. A hand-written `dataTransform` still has class `data`, so a policy that leaves out `data` (as `db update`'s does) still refuses it, and runs `renameField` and `unsetField`. +`isMongoDmlOperation` comes from `@prisma/orm-family-mongo/query-ast/control`. An operation without `run` is a DDL operation. The `dataTransform` factory keeps its name: it builds a DML operation with class `data`. A hand-written `dataTransform` still has class `data`, so a policy that leaves out `data` (as `db update`'s does) still refuses it, and runs `renameField` and `unsetField`. ## `mongo-update-many-bypass-validation` @@ -73,4 +80,4 @@ A hand-built lowered draft of kind `rawUpdateMany` adds `bypassDocumentValidatio ## `mongo-op-factory-call-to-op` -Code typed against the result of `toOp()` or `renderOps(...)` as `MongoMigrationPlanOperation` widens it to `AnyMongoMigrationOperation`, and narrows with `isMongoDataTransformOperation` where it needs `execute`. +Code typed against the result of `toOp()` or `renderOps(...)` as `MongoMigrationPlanOperation` widens it to `AnyMongoMigrationOperation`, and narrows with `isMongoDmlOperation` where it needs `execute` or `run`. From f36ab4a4deb73816f9c2d9221b51ed98ad133b65 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:38:43 +0200 Subject: [PATCH 20/57] TML-3478: one family rule recognises the planner's field rewrites fieldRewriteOf reads an operation as a renameField or unsetField rewrite when it is one updateMany that skips validation and renames or unsets one field. The preview and the storage-name helper both use it, so a hand-written transform is neither previewed nor named as a field. Signed-off-by: willbot Signed-off-by: Will Madden --- .../9-family/src/core/field-rewrite.ts | 38 ++++++++++++++++++ .../9-family/src/core/operation-preview.ts | 18 ++------- .../src/core/operation-storage-name.ts | 31 +++----------- .../9-family/test/field-rewrite.test.ts | 40 +++++++++++++++++++ .../test/operation-storage-name.test.ts | 4 +- 5 files changed, 90 insertions(+), 41 deletions(-) create mode 100644 packages/2-mongo-family/9-family/src/core/field-rewrite.ts create mode 100644 packages/2-mongo-family/9-family/test/field-rewrite.test.ts diff --git a/packages/2-mongo-family/9-family/src/core/field-rewrite.ts b/packages/2-mongo-family/9-family/src/core/field-rewrite.ts new file mode 100644 index 000000000000..a831f56dfb51 --- /dev/null +++ b/packages/2-mongo-family/9-family/src/core/field-rewrite.ts @@ -0,0 +1,38 @@ +import type { MigrationPlanOperation } from '@internal/framework-components/control'; +import { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; + +/** A document rewrite the planner plans: `renameField` or `unsetField`. */ +export interface FieldRewrite { + readonly collection: string; + /** The field the rewrite renames or removes, by its old name. */ + readonly field: string; + readonly command: RawUpdateManyCommand; +} + +const FIELD_OPERATORS: ReadonlySet = new Set(['$rename', '$unset']); + +function onlyKey(value: unknown): string | undefined { + if (typeof value !== 'object' || value === null || Array.isArray(value)) return undefined; + const keys = Object.keys(value); + const [key] = keys; + return keys.length === 1 ? key : undefined; +} + +/** + * The planner's field rewrite an operation is, if it is one: a single `updateMany` that skips + * validation and renames or unsets one field. + */ +export function fieldRewriteOf(operation: MigrationPlanOperation): FieldRewrite | undefined { + const run: unknown = Reflect.get(operation, 'run'); + if (!Array.isArray(run) || run.length !== 1) return undefined; + const [step]: readonly unknown[] = run; + const command: unknown = + typeof step === 'object' && step !== null ? Reflect.get(step, 'command') : undefined; + if (!(command instanceof RawUpdateManyCommand) || command.bypassDocumentValidation !== true) { + return undefined; + } + const operator = onlyKey(command.update); + if (operator === undefined || !FIELD_OPERATORS.has(operator)) return undefined; + const field = onlyKey(Reflect.get(command.update, operator)); + return field === undefined ? undefined : { collection: command.collection, field, command }; +} diff --git a/packages/2-mongo-family/9-family/src/core/operation-preview.ts b/packages/2-mongo-family/9-family/src/core/operation-preview.ts index 0a8ec6c33339..fbd3ee5b508b 100644 --- a/packages/2-mongo-family/9-family/src/core/operation-preview.ts +++ b/packages/2-mongo-family/9-family/src/core/operation-preview.ts @@ -14,6 +14,7 @@ import type { RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import type { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; +import { fieldRewriteOf } from './field-rewrite'; function formatKeySpec(keys: ReadonlyArray): string { const entries = keys.map((k) => `${JSON.stringify(k.field)}: ${JSON.stringify(k.direction)}`); @@ -103,20 +104,6 @@ function isDdlCommand(value: object | undefined): value is AnyMongoDdlCommand { return value !== undefined && typeof Reflect.get(value, 'accept') === 'function'; } -function isFieldRewrite(value: object | undefined): value is RawUpdateManyCommand { - return ( - value !== undefined && - Reflect.get(value, 'kind') === 'rawUpdateMany' && - Reflect.get(value, 'bypassDocumentValidation') === true - ); -} - -function fieldRewritesOf(operation: object): readonly RawUpdateManyCommand[] { - const commands = arrayAt(operation, 'run').map((plan) => objectAt(plan, 'command')); - const rewrites = commands.filter(isFieldRewrite); - return rewrites.length === commands.length ? rewrites : []; -} - export function formatMongoOperations(operations: readonly MigrationPlanOperation[]): string[] { const statements: string[] = []; for (const operation of operations) { @@ -124,7 +111,8 @@ export function formatMongoOperations(operations: readonly MigrationPlanOperatio const command = objectAt(step, 'command'); if (isDdlCommand(command)) statements.push(command.accept(formatter)); } - statements.push(...fieldRewritesOf(operation).map(formatUpdateMany)); + const rewrite = fieldRewriteOf(operation); + if (rewrite !== undefined) statements.push(formatUpdateMany(rewrite.command)); } return statements; } diff --git a/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts b/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts index 0fa961f55737..36025cbd9121 100644 --- a/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts +++ b/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts @@ -1,4 +1,5 @@ import type { MigrationPlanOperation } from '@internal/framework-components/control'; +import { fieldRewriteOf } from './field-rewrite'; function collectionOfStep(step: unknown): string | undefined { if (typeof step !== 'object' || step === null) return undefined; @@ -8,31 +9,9 @@ function collectionOfStep(step: unknown): string | undefined { return typeof collection === 'string' ? collection : undefined; } -const FIELD_REWRITE_OPERATORS = ['$unset', '$rename'] as const; - -function rewrittenFieldOf(command: object): string | undefined { - const update: unknown = Reflect.get(command, 'update'); - if (typeof update !== 'object' || update === null) return undefined; - for (const operator of FIELD_REWRITE_OPERATORS) { - const fields: unknown = Reflect.get(update, operator); - if (typeof fields === 'object' && fields !== null) return Object.keys(fields)[0]; - } - return undefined; -} - -function storageNameOfRunStep(step: unknown): string | undefined { - if (typeof step !== 'object' || step === null) return undefined; - const command: unknown = Reflect.get(step, 'command'); - if (typeof command !== 'object' || command === null) return undefined; - const collection: unknown = Reflect.get(command, 'collection'); - if (typeof collection !== 'string') return undefined; - const field = rewrittenFieldOf(command); - return field === undefined ? collection : `${collection}.${field}`; -} - /** * The collection an operation's command names, as the database knows it: the name the Mongo - * planner gives a dropped collection. A field rewrite is named `.`. An operation + * planner gives a dropped collection. A planner field rewrite is named `.`. An operation * whose commands name no collection is named by its id. */ export function mongoStorageNameOf(operation: MigrationPlanOperation): string { @@ -42,10 +21,12 @@ export function mongoStorageNameOf(operation: MigrationPlanOperation): string { const collection = collectionOfStep(step); if (collection !== undefined) return collection; } + const rewrite = fieldRewriteOf(operation); + if (rewrite !== undefined) return `${rewrite.collection}.${rewrite.field}`; const run: unknown = Reflect.get(operation, 'run'); for (const step of Array.isArray(run) ? run : []) { - const name = storageNameOfRunStep(step); - if (name !== undefined) return name; + const collection = collectionOfStep(step); + if (collection !== undefined) return collection; } return operation.id; } diff --git a/packages/2-mongo-family/9-family/test/field-rewrite.test.ts b/packages/2-mongo-family/9-family/test/field-rewrite.test.ts new file mode 100644 index 000000000000..2a9d3aa22662 --- /dev/null +++ b/packages/2-mongo-family/9-family/test/field-rewrite.test.ts @@ -0,0 +1,40 @@ +import type { MigrationPlanOperation } from '@internal/framework-components/control'; +import { RawInsertOneCommand, RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; +import { describe, expect, it } from 'vitest'; +import { fieldRewriteOf } from '../src/core/field-rewrite'; + +const meta = { target: 'mongo', storageHash: '', lane: 'mongo-raw' }; + +function dmlOperation(commands: readonly object[]): MigrationPlanOperation { + const run = commands.map((command) => ({ collection: 'users', command, meta })); + return { id: 'op', label: 'op', operationClass: 'destructive', run } as MigrationPlanOperation; +} + +const bypass = { bypassDocumentValidation: true }; + +describe('fieldRewriteOf', () => { + it('reads the collection and field of the planner rewrites', () => { + const unset = new RawUpdateManyCommand('users', {}, { $unset: { nickname: '' } }, bypass); + const rename = new RawUpdateManyCommand('users', {}, { $rename: { name: 'fullName' } }, bypass); + expect([fieldRewriteOf(dmlOperation([unset])), fieldRewriteOf(dmlOperation([rename]))]).toEqual( + [ + { collection: 'users', field: 'nickname', command: unset }, + { collection: 'users', field: 'name', command: rename }, + ], + ); + }); + + it('does not read a hand-written transform that does not skip validation, or that does more', () => { + const unset = new RawUpdateManyCommand('users', {}, { $unset: { nickname: '' } }); + const set = new RawUpdateManyCommand('users', {}, { $set: { status: 'active' } }, bypass); + const two = new RawUpdateManyCommand('users', {}, { $unset: { a: '', b: '' } }, bypass); + const rewrite = new RawUpdateManyCommand('users', {}, { $unset: { nickname: '' } }, bypass); + expect([ + fieldRewriteOf(dmlOperation([unset])), + fieldRewriteOf(dmlOperation([set])), + fieldRewriteOf(dmlOperation([two])), + fieldRewriteOf(dmlOperation([rewrite, new RawInsertOneCommand('users', { a: 1 })])), + fieldRewriteOf({ id: 'ddl', label: 'ddl', operationClass: 'additive' }), + ]).toEqual([undefined, undefined, undefined, undefined, undefined]); + }); +}); diff --git a/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts b/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts index d488f8310b63..f8fe6665c110 100644 --- a/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts +++ b/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts @@ -25,7 +25,9 @@ describe('mongoStorageNameOf', () => { it('names a field rewrite by its collection and the field it removes or renames', () => { const rewrite = (update: Record): MigrationPlanOperation => { - const command = new RawUpdateManyCommand('users', {}, update); + const command = new RawUpdateManyCommand('users', {}, update, { + bypassDocumentValidation: true, + }); const run = [{ collection: 'users', command, meta: {} }]; return { id: 'rewrite', From e8862a066ee0cf3114215993be1bbebd8c4e2328 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:40:08 +0200 Subject: [PATCH 21/57] TML-3478: one rule for field names MongoDB cannot rewrite, and a same-name rename is an invalid operation The planner uses the factories' unrewritableFieldNameReason instead of its own copy. renameField to the same name raises MIGRATION.INVALID_OPERATION_ENTRY, since nothing is wrong with the name. Signed-off-by: willbot Signed-off-by: Will Madden --- docs/reference/error-reference.md | 4 ++-- .../core/migrations/migration-factories.ts | 20 ++++++++++++++---- .../migrations/mongo-statement-planning.ts | 21 +++++-------------- .../test/field-rewrites.test.ts | 11 +++++++--- 4 files changed, 31 insertions(+), 25 deletions(-) diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index 77202b1818f2..be5a3a85fe77 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -1452,7 +1452,7 @@ A migration operation's SQL step failed while being executed against the databas ### MIGRATION.FIELD_NAME_UNSUPPORTED -A MongoDB `renameField` or `unsetField` operation names a field it cannot rewrite: an empty name, `_id` (MongoDB does not let a document change its `_id`), a name containing `.` (MongoDB reads it as a path into embedded documents), or a name starting with `$` (MongoDB reads it as an operator). `renameField` also refuses a new name equal to the old one. Raised when the operation is built, before anything runs. Payload: `collection`, `field`, `reason`. +A MongoDB `renameField` or `unsetField` operation names a field it cannot rewrite: an empty name, `_id` (MongoDB does not let a document change its `_id`), a name containing `.` (MongoDB reads it as a path into embedded documents), or a name starting with `$` (MongoDB reads it as an operator). Raised when the operation is built, before anything runs. Payload: `collection`, `field`, `reason`. ### MIGRATION.FILE_MISSING @@ -1496,7 +1496,7 @@ The migration name given to `migration new`/`migration plan --name` contains no ### MIGRATION.INVALID_OPERATION_ENTRY -An operation returned by an authored migration class failed schema validation during emit: each entry of `operations` must carry `id`, `label`, and an `operationClass` of `additive`, `widening`, `destructive`, or `data`. Also raised when deserializing a persisted Mongo migration plan hits a malformed or unknown entry (filter, pipeline stage, DML/DDL/inspection command). Payload: `index`, `reason` (emit validation) or `context`, `kind` (plan deserialization). +An operation returned by an authored migration class failed schema validation during emit: each entry of `operations` must carry `id`, `label`, and an `operationClass` of `additive`, `widening`, `destructive`, or `data`. Also raised when deserializing a persisted Mongo migration plan hits a malformed or unknown entry (filter, pipeline stage, DML/DDL/inspection command). A MongoDB `renameField` whose new name is its old name raises it when the operation is built, with `context: 'renameField'`, `collection`, `field` and `reason`. Payload: `index`, `reason` (emit validation) or `context`, `kind` (plan deserialization). ### MIGRATION.INVALID_REF_FILE diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts index f294f1ec3571..0c9d0165b7ef 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts @@ -144,8 +144,13 @@ const UNREWRITABLE_FIELD_NAMES: ReadonlyArray<{ }, ]; +/** Why `renameField` and `unsetField` cannot rewrite a field of this name, if they cannot. */ +export function unrewritableFieldNameReason(field: string): string | undefined { + return UNREWRITABLE_FIELD_NAMES.find((rule) => rule.matches(field))?.reason; +} + function assertRewritableField(collectionName: string, field: string): void { - const reason = UNREWRITABLE_FIELD_NAMES.find((rule) => rule.matches(field))?.reason; + const reason = unrewritableFieldNameReason(field); if (reason === undefined) return; throw mongoTargetError( 'MIGRATION.FIELD_NAME_UNSUPPORTED', @@ -209,9 +214,16 @@ export function renameField( assertRewritableField(collectionName, to); if (from === to) { throw mongoTargetError( - 'MIGRATION.FIELD_NAME_UNSUPPORTED', - `Field "${from}" in collection ${collectionName} cannot be renamed to its own name.`, - { meta: { collection: collectionName, field: from, reason: 'the new name is the old name' } }, + 'MIGRATION.INVALID_OPERATION_ENTRY', + `renameField renames field "${from}" in collection ${collectionName} to its own name.`, + { + meta: { + context: 'renameField', + collection: collectionName, + field: from, + reason: 'the new name is the old name', + }, + }, ); } const filter = options?.filter; diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index 0e9ea5ef1073..6c630882dcbe 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -19,6 +19,7 @@ import { import type { MongoContract } from '@internal/mongo-contract'; import { MongoSchemaCollection, MongoSchemaIR } from '@internal/mongo-schema-ir'; import { notOk, ok, type Result } from '@internal/utils/result'; +import { unrewritableFieldNameReason } from './migration-factories'; import { type OpFactoryCall, RenameCollectionCall, @@ -73,14 +74,6 @@ function collectionControlPolicy(contract: MongoContract, collection: string) { return effectiveControlPolicy(declared?.control, contract.defaultControlPolicy); } -/** Why MongoDB would read a stored field name as something other than a field, if it would. */ -function unsafeFieldNameReason(field: string): string | undefined { - if (field.includes('.')) - return 'MongoDB reads a name containing "." as a path into embedded documents'; - if (field.startsWith('$')) return 'MongoDB reads a name starting with "$" as an operator'; - return undefined; -} - /** Why MongoDB would refuse a collection name, if it would. */ function invalidCollectionNameReason(name: string): string | undefined { if (name === '') return 'it is empty'; @@ -288,14 +281,14 @@ function planFieldStatement( if (statement.from.field === '_id' || statement.to.field === '_id') { return refuse('it renames _id', `MongoDB does not let a document change its _id. ${LEAVE_OUT}`); } - const unsafeFrom = unsafeFieldNameReason(statement.from.field); + const unsafeFrom = unrewritableFieldNameReason(statement.from.field); if (unsafeFrom !== undefined) { return refuse( `the stored field name "${statement.from.field}" cannot be rewritten`, `${unsafeFrom}, so a statement cannot move its values. Rename it in the database yourself, or ${LEAVE_OUT.toLowerCase()}`, ); } - const unsafeTo = unsafeFieldNameReason(statement.to.field); + const unsafeTo = unrewritableFieldNameReason(statement.to.field); if (unsafeTo !== undefined) { return refuse( `the stored field name "${statement.to.field}" cannot be written`, @@ -457,8 +450,7 @@ export function planFieldRemovals(input: FieldRemovalInput): readonly FieldRemov return Object.keys(model.fields) .filter( (field) => - field !== '_id' && - unsafeFieldNameReason(field) === undefined && + unrewritableFieldNameReason(field) === undefined && !renamedByStatement(input, { ...origin, field }), ) .sort() @@ -506,10 +498,7 @@ export function planValidatorFieldRemovals(input: { ...branchPropertyNames(destination.validator?.jsonSchema), ]); return propertyNames(originCollection.validator.jsonSchema) - .filter( - (field) => - field !== '_id' && unsafeFieldNameReason(field) === undefined && !kept.has(field), - ) + .filter((field) => unrewritableFieldNameReason(field) === undefined && !kept.has(field)) .sort() .map((field) => ({ call: new UnsetFieldCall(originCollection.name, field), diff --git a/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts b/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts index 58bd239a7d7e..80555333c583 100644 --- a/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts @@ -158,11 +158,16 @@ describe('field names a rewrite refuses', () => { }); } - it('refuses renaming a field to its own name', () => { + it('refuses renaming a field to its own name as an invalid operation, not an unsupported name', () => { expect(() => renameField('users', 'name', 'name')).toThrow( expect.objectContaining({ - code: 'MIGRATION.FIELD_NAME_UNSUPPORTED', - meta: expect.objectContaining({ collection: 'users', field: 'name' }), + code: 'MIGRATION.INVALID_OPERATION_ENTRY', + meta: expect.objectContaining({ + context: 'renameField', + collection: 'users', + field: 'name', + reason: 'the new name is the old name', + }), }), ); }); From 7777ed26a59455f5a40d6b86835e1b2308877047 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:41:00 +0200 Subject: [PATCH 22/57] TML-3478: planCalls takes the statement calls and their schema as one typed value StatementCalls holds the renamed schema, the collection renames and the document rewrites, typed as their call classes, so a caller cannot pass renames with a schema that does not have them. Signed-off-by: willbot Signed-off-by: Will Madden --- .../src/core/migrations/mongo-planner.ts | 29 ++++++++++--------- .../migrations/mongo-statement-planning.ts | 13 +++++++++ 2 files changed, 29 insertions(+), 13 deletions(-) diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts index 6b80b70fa81e..320caec7d50b 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts @@ -30,6 +30,7 @@ import { planFieldRemovals, planStatements, planValidatorFieldRemovals, + type StatementCalls, } from './mongo-statement-planning'; import type { OpFactoryCall } from './op-factory-call'; import { @@ -188,19 +189,19 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> readonly schema: unknown; readonly policy: MigrationOperationPolicy; readonly frameworkComponents: ReadonlyArray>; - /** Collection renames, which run first; `schema` already has each collection under its new name. */ - readonly collectionRenames?: readonly OpFactoryCall[]; - /** Document rewrites, which run after index drops and before index creates. */ - readonly rewrites?: readonly OpFactoryCall[]; + /** What the statements and removed fields plan; its `schema` is diffed in place of `schema`. */ + readonly statementCalls?: StatementCalls; }): PlanCallsResult { const contract = blindCast< MongoContract, 'framework planner passes the Mongo contract selected for the mongo target' >(options.contract); - const originIR = blindCast< - MongoSchemaIR, - 'framework planner passes the inspected Mongo schema IR selected for the mongo target' - >(options.schema); + const originIR = + options.statementCalls?.schema ?? + blindCast< + MongoSchemaIR, + 'framework planner passes the inspected Mongo schema IR selected for the mongo target' + >(options.schema); const destinationIR = contractToMongoSchemaIR(contract); const collCreates: OpFactoryCall[] = []; @@ -301,10 +302,10 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> } const allCalls = [ - ...(options.collectionRenames ?? []), + ...(options.statementCalls?.collectionRenames ?? []), ...collCreates, ...drops, - ...(options.rewrites ?? []), + ...(options.statementCalls?.rewrites ?? []), ...creates, ...validatorOps, ...mutableOptionOps, @@ -380,9 +381,11 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> }); const result = this.planCalls({ ...options, - schema, - collectionRenames, - rewrites: [...fieldRenames, ...removals.map((removal) => removal.call)], + statementCalls: { + schema, + collectionRenames, + rewrites: [...fieldRenames, ...removals.map((removal) => removal.call)], + }, }); if (result.kind === 'failure') return result; const { calls } = result; diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index 6c630882dcbe..04bf8fdffebd 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -101,6 +101,19 @@ function storedField(model: ContractModelBase | undefined, field: string): Store return isEmbedRelation(relation) ? { kind: 'embed' } : { kind: 'reference' }; } +/** + * The calls statements and removed fields add to a plan, with the schema the diff starts from: + * the origin with each renamed collection under its new name, so the diff does not drop and + * create it again. + */ +export interface StatementCalls { + readonly schema: MongoSchemaIR; + /** Run first. */ + readonly collectionRenames: readonly RenameCollectionCall[]; + /** Run after index drops and before index creates. */ + readonly rewrites: readonly (RenameFieldCall | UnsetFieldCall)[]; +} + export interface StatementPlan { /** The origin schema with each renamed collection under its new name, for the diff to compare. */ readonly schema: MongoSchemaIR; From e0676e4ce256cefc22cc3a1649b50c454b6961dc Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:43:39 +0200 Subject: [PATCH 23/57] TML-3478: split the MongoDB statement tests by what they test The planner statement tests split into model statements, field statements and removed fields, and refusals, with shared builders in mongo-statement-fixtures.ts, as SQL's statement-planning tests are. The runner statement tests split off the variant cases, with the database setup and helpers in runner-statements-fixtures.ts. Signed-off-by: willbot Signed-off-by: Will Madden --- .../mongo-planner.statements.fields.test.ts | 282 +++++++ .../mongo-planner.statements.refusals.test.ts | 205 +++++ .../test/mongo-planner.statements.test.ts | 702 +----------------- .../test/mongo-statement-fixtures.ts | 231 ++++++ .../runner-statements-fixtures.ts | 244 ++++++ .../target-runner/runner-statements.test.ts | 459 ++---------- .../runner-statements.variants.test.ts | 195 +++++ 7 files changed, 1220 insertions(+), 1098 deletions(-) create mode 100644 packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts create mode 100644 packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts create mode 100644 packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts create mode 100644 test/integration/test/mongo/target-runner/runner-statements-fixtures.ts create mode 100644 test/integration/test/mongo/target-runner/runner-statements.variants.test.ts diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts new file mode 100644 index 000000000000..8164e477343a --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts @@ -0,0 +1,282 @@ +import { describe, expect, it } from 'vitest'; +import { + contract, + NS, + plan, + planned, + profiles, + profilesSchema, + renameField, + renameModel, + schemaCollection, +} from './mongo-statement-fixtures'; + +describe('MongoMigrationPlanner, field statements', () => { + const fullName = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'fullName'] } }, + { profiles: { unique: ['fullName'], required: ['fullName'] } }, + ); + + it('renames the field between the index drop and the index create, before the validator update', () => { + expect( + planned( + plan({ + from: profiles, + to: fullName, + schema: profilesSchema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ), + ).toEqual({ + labels: [ + 'Drop index on profiles (name:1)', + 'Rename field name to fullName in collection profiles', + 'Create index on profiles (fullName:1)', + 'Update validator on profiles (added: fullName; removed: name)', + ], + classes: ['widening', 'widening', 'additive', 'widening'], + applied: [[1]], + dataLoss: [], + }); + }); + + it('renames the collection first, then the field in the renamed collection', () => { + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'fullName'] } }, + { users: { unique: ['fullName'], required: ['fullName'] } }, + ); + expect( + planned( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [ + renameModel('Profile', 'User'), + renameField('Profile', 'name', 'User', 'fullName'), + ], + }), + ), + ).toMatchObject({ + labels: [ + 'Rename collection profiles to users', + 'Drop index on users (name:1)', + 'Rename field name to fullName in collection users', + 'Create index on users (fullName:1)', + 'Update validator on users (added: fullName; removed: name)', + ], + applied: [[0], [2]], + dataLoss: [], + }); + }); + + it("renames a variant's field only in that variant's documents", () => { + const tasks = (bugField: string) => + contract( + bugField, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type'], + discriminator: 'type', + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: [bugField], base: 'Task' }, + Feature: { collection: 'tasks', fields: ['score'], base: 'Task' }, + }, + { tasks: {} }, + ); + expect( + planned( + plan({ + from: tasks('score'), + to: tasks('severity'), + schema: [schemaCollection('tasks')], + statements: [renameField('Bug', 'score', 'Bug', 'severity')], + }), + ).labels, + ).toEqual(['Rename field score to severity in collection tasks where {"type":"bug"}']); + }); + + it('plans nothing for a relation renamed to another relation, which documents do not store', () => { + const withRelation = (relation: string) => + contract( + relation, + { Profile: { collection: 'profiles', fields: ['_id', 'name'], relations: [relation] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + planned( + plan({ + from: withRelation('owner'), + to: withRelation('author'), + schema: profilesSchema, + statements: [renameField('Profile', 'owner', 'Profile', 'author')], + }), + ), + ).toEqual({ labels: [], classes: [], applied: [[]], dataLoss: [] }); + }); +}); + +describe('MongoMigrationPlanner, removed fields', () => { + const withoutName = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'email'] } }, + { profiles: { required: ['email'] } }, + ); + const from = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'email', 'name'] } }, + { profiles: { required: ['email', 'name'] } }, + ); + const schema = [schemaCollection('profiles', { required: ['email', 'name'] })]; + + it('unsets a removed field from every document and reports the field as lost', () => { + expect(planned(plan({ from, to: withoutName, schema }))).toEqual({ + labels: [ + 'Remove field name from collection profiles', + 'Update validator on profiles (removed: name)', + ], + classes: ['destructive', 'widening'], + applied: [], + dataLoss: [ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Profile', field: 'name' }, + }, + ], + }); + }); + + it('names the field by its storage name when the origin contract is unknown', () => { + expect(planned(plan({ from: null, to: withoutName, schema })).dataLoss).toEqual([ + { operationIndex: 0, subject: { kind: 'storage', name: 'profiles.name' } }, + ]); + }); + + it('unsets nothing when the collection is dropped', () => { + const none = contract('to', {}, {}); + expect(planned(plan({ from, to: none, schema })).labels).toEqual(['Drop collection profiles']); + }); + + const tasks = ( + seed: string, + baseFields: readonly string[], + bugFields: readonly string[], + featureFields: readonly string[], + ) => + contract( + seed, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type', ...baseFields], + discriminator: 'type', + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: bugFields, base: 'Task' }, + Feature: { collection: 'tasks', fields: featureFields, base: 'Task' }, + }, + { tasks: {} }, + ); + + it("unsets a variant's removed field only from that variant's documents", () => { + expect( + planned( + plan({ + from: tasks('from', [], ['score', 'severity'], ['score']), + to: tasks('to', [], ['severity'], ['score']), + schema: [schemaCollection('tasks')], + }), + ), + ).toMatchObject({ + labels: ['Remove field score from collection tasks where {"type":"bug"}'], + dataLoss: [ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Bug', field: 'score' }, + }, + ], + }); + }); + + it('unsets a base field moved to one variant from the documents of every other variant', () => { + expect( + planned( + plan({ + from: tasks('from', ['note'], [], []), + to: tasks('to', [], ['note'], []), + schema: [schemaCollection('tasks')], + }), + ), + ).toMatchObject({ + labels: ['Remove field note from collection tasks where {"type":{"$nin":["bug"]}}'], + dataLoss: [ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Task', field: 'note' }, + }, + ], + }); + }); + + it('renames a base field moved to one variant only in the documents of the other variants', () => { + expect( + planned( + plan({ + from: tasks('from', ['note'], [], []), + to: tasks('to', ['remark'], ['note'], []), + schema: [schemaCollection('tasks')], + statements: [renameField('Task', 'note', 'Task', 'remark')], + }), + ), + ).toMatchObject({ + labels: ['Rename field note to remark in collection tasks where {"type":{"$nin":["bug"]}}'], + applied: [[0]], + dataLoss: [], + }); + }); + + it('keeps a variant field its base now stores', () => { + expect( + planned( + plan({ + from: tasks('from', [], ['note'], []), + to: tasks('to', ['note'], [], []), + schema: [schemaCollection('tasks')], + }), + ).labels, + ).toEqual([]); + }); + + it('unsets nothing without an origin contract when the destination collection has no validator', () => { + const unvalidated = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'email'] } }, + { profiles: {} }, + ); + expect(planned(plan({ from: null, to: unvalidated, schema })).labels).toEqual([ + 'Remove validator on profiles', + ]); + }); + + it('unsets nothing for a field a statement renames', () => { + const fullName = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'email', 'fullName'] } }, + { profiles: { required: ['email', 'fullName'] } }, + ); + expect( + planned( + plan({ + from, + to: fullName, + schema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ).dataLoss, + ).toEqual([]); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts new file mode 100644 index 000000000000..c6a1409a69df --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts @@ -0,0 +1,205 @@ +import { describe, expect, it } from 'vitest'; +import { + contract, + plan, + profiles, + profilesSchema, + refusal, + renameField, + renameModel, + schemaCollection, + users, +} from './mongo-statement-fixtures'; + +describe('MongoMigrationPlanner, statements it refuses', () => { + it('refuses renaming _id', () => { + const to = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['key', 'name'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameField('Profile', '_id', 'Profile', 'key')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: 'Cannot rename field "Profile._id" to "Profile.key": it renames _id', + why: 'MongoDB does not let a document change its _id. Leave out this statement.', + }, + ]); + }); + + it('refuses renaming a stored field whose name MongoDB reads as a path, and says to rename it by hand', () => { + const named = (field: string) => + contract( + field, + { Profile: { collection: 'profiles', fields: ['_id', field] } }, + { profiles: {} }, + ); + expect( + refusal( + plan({ + from: named('o.d'), + to: named('od'), + schema: [schemaCollection('profiles')], + statements: [renameField('Profile', 'o.d', 'Profile', 'od')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: + 'Cannot rename field "Profile.o.d" to "Profile.od": the stored field name "o.d" cannot be rewritten', + why: 'MongoDB reads a name containing "." as a path into embedded documents, so a statement cannot move its values. Rename it in the database yourself, or leave out this statement.', + }, + ]); + }); + + it('refuses a stored field name MongoDB reads as a path or an operator', () => { + const to = (field: string) => + contract( + field, + { Profile: { collection: 'profiles', fields: ['_id', field] } }, + { profiles: {} }, + ); + expect( + ['a.b', '$a'].flatMap((field) => + refusal( + plan({ + from: to('name'), + to: to(field), + schema: [schemaCollection('profiles')], + statements: [renameField('Profile', 'name', 'Profile', field)], + }), + ).map(({ summary }) => summary), + ), + ).toEqual([ + 'Cannot rename field "Profile.name" to "Profile.a.b": the stored field name "a.b" cannot be written', + 'Cannot rename field "Profile.name" to "Profile.$a": the stored field name "$a" cannot be written', + ]); + }); + + it('refuses renaming a time-series collection, a collection the schema does not have, and onto a name it already has', () => { + const statements = [renameModel('Profile', 'User')]; + expect( + [ + [schemaCollection('profiles', { unique: ['name'], required: ['name'], timeseries: true })], + [], + [...profilesSchema, schemaCollection('users')], + ].flatMap((schema) => + refusal(plan({ from: profiles, to: users, schema, statements })).map( + ({ summary }) => summary, + ), + ), + ).toEqual([ + 'Cannot rename model "Profile" to "User": collection "profiles" is a time-series collection', + 'Cannot rename model "Profile" to "User": the schema being planned from has no collection "profiles"', + 'Cannot rename model "Profile" to "User": the schema being planned from already has a collection "users"', + ]); + }); + + it('refuses a collection name MongoDB rejects', () => { + const to = (collection: string) => + contract('to', { User: { collection, fields: ['_id', 'name'] } }, { [collection]: {} }); + expect( + ['a$b', 'system.users', 'a\0b'].flatMap((collection) => + refusal( + plan({ + from: profiles, + to: to(collection), + schema: profilesSchema, + statements: [renameModel('Profile', 'User')], + }), + ).map(({ summary }) => summary), + ), + ).toEqual([ + 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "a$b": it contains "$"', + 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "system.users": it starts with "system.", which MongoDB reserves', + 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "a\0b": it contains a null character', + ]); + }); + + it('refuses a field stored on one side of the statement only', () => { + const to = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'name'], relations: ['owner'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + const from = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'name', 'ownerName'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + refusal( + plan({ + from, + to, + schema: profilesSchema, + statements: [renameField('Profile', 'ownerName', 'Profile', 'owner')], + }), + ).map(({ summary }) => summary), + ).toEqual([ + 'Cannot rename field "Profile.ownerName" to "Profile.owner": the field is stored in documents on one side of the statement only', + ]); + }); + + it('refuses a statement on a collection whose control policy is not managed', () => { + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'name'] } }, + { users: { unique: ['name'], required: ['name'], control: 'external' } }, + ); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameModel('Profile', 'User')], + }), + ).map(({ summary }) => summary), + ).toEqual([ + 'Cannot rename model "Profile" to "User": collection "users" has control policy "external"', + ]); + }); + + it("refuses a field rename when the model's collection changes and no statement renames it", () => { + const to = contract( + 'to', + { Profile: { collection: 'users', fields: ['_id', 'fullName'] } }, + { users: {} }, + ); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ).map(({ summary }) => summary), + ).toEqual([ + 'Cannot rename field "Profile.name" to "Profile.fullName": the model\'s collection changes from "profiles" to "users", and no statement renames it', + ]); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + )[0]?.why, + ).toBe( + 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question; in its migration.ts replace the drop of "profiles" and the create of "users" with renameCollection("profiles", "users"), imported with the other operations; run node migration.ts in the migration\'s directory, so it writes ops.json again; then apply it with migrate. Then change the field and run this statement again.', + ); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts index 4130dc12d34c..741013bf8e24 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts @@ -1,233 +1,14 @@ -import { asNamespaceId, type ControlPolicy } from '@internal/contract/types'; -import type { - MigrationOperationPolicy, - ResolvedMigrationStatement, -} from '@internal/framework-components/control'; -import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; -import { - MongoSchemaCollection, - MongoSchemaCollectionOptions, - MongoSchemaIndex, - MongoSchemaIR, - MongoSchemaValidator, -} from '@internal/mongo-schema-ir'; +import { MongoSchemaCollection, MongoSchemaCollectionOptions } from '@internal/mongo-schema-ir'; import { describe, expect, it } from 'vitest'; -import { MongoMigrationPlanner } from '../src/core/migrations/mongo-planner'; -import type { PlannerProducedMongoMigration } from '../src/core/migrations/planner-produced-migration'; - -const ALL_CLASSES_POLICY: MigrationOperationPolicy = { - allowedOperationClasses: ['additive', 'widening', 'destructive', 'data'], -}; - -const NS = asNamespaceId('__unbound__'); - -interface ModelSpec { - readonly collection: string; - readonly fields: readonly string[]; - readonly relations?: readonly string[]; - readonly base?: string; - readonly discriminator?: string; - readonly variants?: Readonly>; -} - -interface CollectionSpec { - readonly unique?: readonly string[]; - readonly required?: readonly string[]; - readonly control?: ControlPolicy; - readonly timeField?: string; -} - -function jsonSchema(fields: readonly string[]) { - return { - bsonType: 'object', - required: ['_id', ...fields], - properties: Object.fromEntries([ - ['_id', { bsonType: 'objectId' }], - ...fields.map((field) => [field, { bsonType: 'string' }]), - ]), - additionalProperties: false, - }; -} - -function contract( - seed: string, - models: Readonly>, - collections: Readonly>, -): MongoContract { - return { - target: 'mongo', - targetFamily: 'mongo', - profileHash: 'test-profile', - capabilities: {}, - extensions: {}, - meta: {}, - roots: {}, - domain: { - namespaces: { - __unbound__: { - models: Object.fromEntries( - Object.entries(models).map(([name, spec]) => [ - name, - { - fields: Object.fromEntries(spec.fields.map((field) => [field, {}])), - relations: Object.fromEntries( - (spec.relations ?? []).map((relation) => [ - relation, - { - to: { model: 'Other', namespace: '__unbound__' }, - cardinality: 'N:1', - on: { localFields: ['otherId'], targetFields: ['_id'] }, - }, - ]), - ), - storage: { collection: spec.collection }, - ...(spec.base === undefined - ? {} - : { base: { namespace: '__unbound__', model: spec.base } }), - ...(spec.discriminator === undefined - ? {} - : { - discriminator: { field: spec.discriminator }, - variants: Object.fromEntries( - Object.entries(spec.variants ?? {}).map(([variant, value]) => [ - variant, - { value }, - ]), - ), - }), - }, - ]), - ), - }, - }, - }, - storage: { - storageHash: `storage-${seed}`, - namespaces: { - __unbound__: { - id: '__unbound__', - kind: 'mongo-namespace', - entries: { - collection: Object.fromEntries( - Object.entries(collections).map(([name, spec]) => [ - name, - new MongoCollection({ - indexes: (spec.unique ?? []).map((field) => ({ - keys: [{ field, direction: 1 as const }], - unique: true, - })), - ...(spec.required === undefined - ? {} - : { - validator: { - jsonSchema: jsonSchema(spec.required), - validationLevel: 'strict' as const, - validationAction: 'error' as const, - }, - }), - ...(spec.control === undefined ? {} : { control: spec.control }), - ...(spec.timeField === undefined - ? {} - : { options: { timeseries: { timeField: spec.timeField } } }), - }), - ]), - ), - }, - }, - }, - }, - } as unknown as MongoContract; -} - -function schemaCollection( - name: string, - spec: CollectionSpec & { readonly timeseries?: boolean } = {}, -): MongoSchemaCollection { - return new MongoSchemaCollection({ - name, - indexes: (spec.unique ?? []).map( - (field) => new MongoSchemaIndex({ keys: [{ field, direction: 1 }], unique: true }), - ), - ...(spec.required === undefined - ? {} - : { - validator: new MongoSchemaValidator({ - jsonSchema: jsonSchema(spec.required), - validationLevel: 'strict', - validationAction: 'error', - }), - }), - ...(spec.timeseries - ? { options: new MongoSchemaCollectionOptions({ timeseries: { timeField: 'at' } }) } - : {}), - }); -} - -function plan(input: { - readonly from: MongoContract | null; - readonly to: MongoContract; - readonly schema: readonly MongoSchemaCollection[]; - readonly statements?: readonly ResolvedMigrationStatement[]; - readonly policy?: MigrationOperationPolicy; -}) { - return new MongoMigrationPlanner().plan({ - contract: input.to, - schema: new MongoSchemaIR(input.schema), - policy: input.policy ?? ALL_CLASSES_POLICY, - fromContract: input.from, - origin: null, - statements: input.statements ?? [], - frameworkComponents: [], - snapshotsImportPath: '../../snapshots', - }); -} - -function planned(result: ReturnType) { - if (result.kind !== 'success') throw new Error(JSON.stringify(result.conflicts, null, 2)); - const operations = (result.plan as PlannerProducedMongoMigration).operations; - return { - labels: operations.map((operation) => operation.label), - classes: operations.map((operation) => operation.operationClass), - applied: result.appliedStatements.map((applied) => applied.operationIndexes), - dataLoss: result.dataLoss, - }; -} - -function refusal(result: ReturnType) { - if (result.kind !== 'failure') throw new Error('Expected the plan to be refused'); - return result.conflicts.map(({ kind, summary, why }) => ({ kind, summary, why })); -} - -const renameModel = (from: string, to: string): ResolvedMigrationStatement => ({ - kind: 'rename', - entity: 'model', - from: { namespaceId: NS, model: from }, - to: { namespaceId: NS, model: to }, -}); - -const renameField = ( - fromModel: string, - from: string, - toModel: string, - to: string, -): ResolvedMigrationStatement => ({ - kind: 'rename', - entity: 'field', - from: { namespaceId: NS, model: fromModel, field: from }, - to: { namespaceId: NS, model: toModel, field: to }, -}); - -const profiles = contract( - 'from', - { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, - { profiles: { unique: ['name'], required: ['name'] } }, -); -const users = contract( - 'to', - { User: { collection: 'users', fields: ['_id', 'name'] } }, - { users: { unique: ['name'], required: ['name'] } }, -); -const profilesSchema = [schemaCollection('profiles', { unique: ['name'], required: ['name'] })]; +import { + contract, + plan, + planned, + profiles, + profilesSchema, + renameModel, + users, +} from './mongo-statement-fixtures'; describe('MongoMigrationPlanner, model statements', () => { it('without a statement, drops the old collection and creates the new one', () => { @@ -276,469 +57,6 @@ describe('MongoMigrationPlanner, model statements', () => { }); }); -describe('MongoMigrationPlanner, field statements', () => { - const fullName = contract( - 'to', - { Profile: { collection: 'profiles', fields: ['_id', 'fullName'] } }, - { profiles: { unique: ['fullName'], required: ['fullName'] } }, - ); - - it('renames the field between the index drop and the index create, before the validator update', () => { - expect( - planned( - plan({ - from: profiles, - to: fullName, - schema: profilesSchema, - statements: [renameField('Profile', 'name', 'Profile', 'fullName')], - }), - ), - ).toEqual({ - labels: [ - 'Drop index on profiles (name:1)', - 'Rename field name to fullName in collection profiles', - 'Create index on profiles (fullName:1)', - 'Update validator on profiles (added: fullName; removed: name)', - ], - classes: ['widening', 'widening', 'additive', 'widening'], - applied: [[1]], - dataLoss: [], - }); - }); - - it('renames the collection first, then the field in the renamed collection', () => { - const to = contract( - 'to', - { User: { collection: 'users', fields: ['_id', 'fullName'] } }, - { users: { unique: ['fullName'], required: ['fullName'] } }, - ); - expect( - planned( - plan({ - from: profiles, - to, - schema: profilesSchema, - statements: [ - renameModel('Profile', 'User'), - renameField('Profile', 'name', 'User', 'fullName'), - ], - }), - ), - ).toMatchObject({ - labels: [ - 'Rename collection profiles to users', - 'Drop index on users (name:1)', - 'Rename field name to fullName in collection users', - 'Create index on users (fullName:1)', - 'Update validator on users (added: fullName; removed: name)', - ], - applied: [[0], [2]], - dataLoss: [], - }); - }); - - it("renames a variant's field only in that variant's documents", () => { - const tasks = (bugField: string) => - contract( - bugField, - { - Task: { - collection: 'tasks', - fields: ['_id', 'type'], - discriminator: 'type', - variants: { Bug: 'bug', Feature: 'feature' }, - }, - Bug: { collection: 'tasks', fields: [bugField], base: 'Task' }, - Feature: { collection: 'tasks', fields: ['score'], base: 'Task' }, - }, - { tasks: {} }, - ); - expect( - planned( - plan({ - from: tasks('score'), - to: tasks('severity'), - schema: [schemaCollection('tasks')], - statements: [renameField('Bug', 'score', 'Bug', 'severity')], - }), - ).labels, - ).toEqual(['Rename field score to severity in collection tasks where {"type":"bug"}']); - }); - - it('plans nothing for a relation renamed to another relation, which documents do not store', () => { - const withRelation = (relation: string) => - contract( - relation, - { Profile: { collection: 'profiles', fields: ['_id', 'name'], relations: [relation] } }, - { profiles: { unique: ['name'], required: ['name'] } }, - ); - expect( - planned( - plan({ - from: withRelation('owner'), - to: withRelation('author'), - schema: profilesSchema, - statements: [renameField('Profile', 'owner', 'Profile', 'author')], - }), - ), - ).toEqual({ labels: [], classes: [], applied: [[]], dataLoss: [] }); - }); -}); - -describe('MongoMigrationPlanner, statements it refuses', () => { - it('refuses renaming _id', () => { - const to = contract( - 'to', - { Profile: { collection: 'profiles', fields: ['key', 'name'] } }, - { profiles: { unique: ['name'], required: ['name'] } }, - ); - expect( - refusal( - plan({ - from: profiles, - to, - schema: profilesSchema, - statements: [renameField('Profile', '_id', 'Profile', 'key')], - }), - ), - ).toEqual([ - { - kind: 'statementRefused', - summary: 'Cannot rename field "Profile._id" to "Profile.key": it renames _id', - why: 'MongoDB does not let a document change its _id. Leave out this statement.', - }, - ]); - }); - - it('refuses renaming a stored field whose name MongoDB reads as a path, and says to rename it by hand', () => { - const named = (field: string) => - contract( - field, - { Profile: { collection: 'profiles', fields: ['_id', field] } }, - { profiles: {} }, - ); - expect( - refusal( - plan({ - from: named('o.d'), - to: named('od'), - schema: [schemaCollection('profiles')], - statements: [renameField('Profile', 'o.d', 'Profile', 'od')], - }), - ), - ).toEqual([ - { - kind: 'statementRefused', - summary: - 'Cannot rename field "Profile.o.d" to "Profile.od": the stored field name "o.d" cannot be rewritten', - why: 'MongoDB reads a name containing "." as a path into embedded documents, so a statement cannot move its values. Rename it in the database yourself, or leave out this statement.', - }, - ]); - }); - - it('refuses a stored field name MongoDB reads as a path or an operator', () => { - const to = (field: string) => - contract( - field, - { Profile: { collection: 'profiles', fields: ['_id', field] } }, - { profiles: {} }, - ); - expect( - ['a.b', '$a'].flatMap((field) => - refusal( - plan({ - from: to('name'), - to: to(field), - schema: [schemaCollection('profiles')], - statements: [renameField('Profile', 'name', 'Profile', field)], - }), - ).map(({ summary }) => summary), - ), - ).toEqual([ - 'Cannot rename field "Profile.name" to "Profile.a.b": the stored field name "a.b" cannot be written', - 'Cannot rename field "Profile.name" to "Profile.$a": the stored field name "$a" cannot be written', - ]); - }); - - it('refuses renaming a time-series collection, a collection the schema does not have, and onto a name it already has', () => { - const statements = [renameModel('Profile', 'User')]; - expect( - [ - [schemaCollection('profiles', { unique: ['name'], required: ['name'], timeseries: true })], - [], - [...profilesSchema, schemaCollection('users')], - ].flatMap((schema) => - refusal(plan({ from: profiles, to: users, schema, statements })).map( - ({ summary }) => summary, - ), - ), - ).toEqual([ - 'Cannot rename model "Profile" to "User": collection "profiles" is a time-series collection', - 'Cannot rename model "Profile" to "User": the schema being planned from has no collection "profiles"', - 'Cannot rename model "Profile" to "User": the schema being planned from already has a collection "users"', - ]); - }); - - it('refuses a collection name MongoDB rejects', () => { - const to = (collection: string) => - contract('to', { User: { collection, fields: ['_id', 'name'] } }, { [collection]: {} }); - expect( - ['a$b', 'system.users', 'a\0b'].flatMap((collection) => - refusal( - plan({ - from: profiles, - to: to(collection), - schema: profilesSchema, - statements: [renameModel('Profile', 'User')], - }), - ).map(({ summary }) => summary), - ), - ).toEqual([ - 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "a$b": it contains "$"', - 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "system.users": it starts with "system.", which MongoDB reserves', - 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "a\0b": it contains a null character', - ]); - }); - - it('refuses a field stored on one side of the statement only', () => { - const to = contract( - 'to', - { Profile: { collection: 'profiles', fields: ['_id', 'name'], relations: ['owner'] } }, - { profiles: { unique: ['name'], required: ['name'] } }, - ); - const from = contract( - 'from', - { Profile: { collection: 'profiles', fields: ['_id', 'name', 'ownerName'] } }, - { profiles: { unique: ['name'], required: ['name'] } }, - ); - expect( - refusal( - plan({ - from, - to, - schema: profilesSchema, - statements: [renameField('Profile', 'ownerName', 'Profile', 'owner')], - }), - ).map(({ summary }) => summary), - ).toEqual([ - 'Cannot rename field "Profile.ownerName" to "Profile.owner": the field is stored in documents on one side of the statement only', - ]); - }); - - it('refuses a statement on a collection whose control policy is not managed', () => { - const to = contract( - 'to', - { User: { collection: 'users', fields: ['_id', 'name'] } }, - { users: { unique: ['name'], required: ['name'], control: 'external' } }, - ); - expect( - refusal( - plan({ - from: profiles, - to, - schema: profilesSchema, - statements: [renameModel('Profile', 'User')], - }), - ).map(({ summary }) => summary), - ).toEqual([ - 'Cannot rename model "Profile" to "User": collection "users" has control policy "external"', - ]); - }); - - it("refuses a field rename when the model's collection changes and no statement renames it", () => { - const to = contract( - 'to', - { Profile: { collection: 'users', fields: ['_id', 'fullName'] } }, - { users: {} }, - ); - expect( - refusal( - plan({ - from: profiles, - to, - schema: profilesSchema, - statements: [renameField('Profile', 'name', 'Profile', 'fullName')], - }), - ).map(({ summary }) => summary), - ).toEqual([ - 'Cannot rename field "Profile.name" to "Profile.fullName": the model\'s collection changes from "profiles" to "users", and no statement renames it', - ]); - expect( - refusal( - plan({ - from: profiles, - to, - schema: profilesSchema, - statements: [renameField('Profile', 'name', 'Profile', 'fullName')], - }), - )[0]?.why, - ).toBe( - 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question; in its migration.ts replace the drop of "profiles" and the create of "users" with renameCollection("profiles", "users"), imported with the other operations; run node migration.ts in the migration\'s directory, so it writes ops.json again; then apply it with migrate. Then change the field and run this statement again.', - ); - }); -}); - -describe('MongoMigrationPlanner, removed fields', () => { - const withoutName = contract( - 'to', - { Profile: { collection: 'profiles', fields: ['_id', 'email'] } }, - { profiles: { required: ['email'] } }, - ); - const from = contract( - 'from', - { Profile: { collection: 'profiles', fields: ['_id', 'email', 'name'] } }, - { profiles: { required: ['email', 'name'] } }, - ); - const schema = [schemaCollection('profiles', { required: ['email', 'name'] })]; - - it('unsets a removed field from every document and reports the field as lost', () => { - expect(planned(plan({ from, to: withoutName, schema }))).toEqual({ - labels: [ - 'Remove field name from collection profiles', - 'Update validator on profiles (removed: name)', - ], - classes: ['destructive', 'widening'], - applied: [], - dataLoss: [ - { - operationIndex: 0, - subject: { kind: 'field', namespaceId: NS, model: 'Profile', field: 'name' }, - }, - ], - }); - }); - - it('names the field by its storage name when the origin contract is unknown', () => { - expect(planned(plan({ from: null, to: withoutName, schema })).dataLoss).toEqual([ - { operationIndex: 0, subject: { kind: 'storage', name: 'profiles.name' } }, - ]); - }); - - it('unsets nothing when the collection is dropped', () => { - const none = contract('to', {}, {}); - expect(planned(plan({ from, to: none, schema })).labels).toEqual(['Drop collection profiles']); - }); - - const tasks = ( - seed: string, - baseFields: readonly string[], - bugFields: readonly string[], - featureFields: readonly string[], - ) => - contract( - seed, - { - Task: { - collection: 'tasks', - fields: ['_id', 'type', ...baseFields], - discriminator: 'type', - variants: { Bug: 'bug', Feature: 'feature' }, - }, - Bug: { collection: 'tasks', fields: bugFields, base: 'Task' }, - Feature: { collection: 'tasks', fields: featureFields, base: 'Task' }, - }, - { tasks: {} }, - ); - - it("unsets a variant's removed field only from that variant's documents", () => { - expect( - planned( - plan({ - from: tasks('from', [], ['score', 'severity'], ['score']), - to: tasks('to', [], ['severity'], ['score']), - schema: [schemaCollection('tasks')], - }), - ), - ).toMatchObject({ - labels: ['Remove field score from collection tasks where {"type":"bug"}'], - dataLoss: [ - { - operationIndex: 0, - subject: { kind: 'field', namespaceId: NS, model: 'Bug', field: 'score' }, - }, - ], - }); - }); - - it('unsets a base field moved to one variant from the documents of every other variant', () => { - expect( - planned( - plan({ - from: tasks('from', ['note'], [], []), - to: tasks('to', [], ['note'], []), - schema: [schemaCollection('tasks')], - }), - ), - ).toMatchObject({ - labels: ['Remove field note from collection tasks where {"type":{"$nin":["bug"]}}'], - dataLoss: [ - { - operationIndex: 0, - subject: { kind: 'field', namespaceId: NS, model: 'Task', field: 'note' }, - }, - ], - }); - }); - - it('renames a base field moved to one variant only in the documents of the other variants', () => { - expect( - planned( - plan({ - from: tasks('from', ['note'], [], []), - to: tasks('to', ['remark'], ['note'], []), - schema: [schemaCollection('tasks')], - statements: [renameField('Task', 'note', 'Task', 'remark')], - }), - ), - ).toMatchObject({ - labels: ['Rename field note to remark in collection tasks where {"type":{"$nin":["bug"]}}'], - applied: [[0]], - dataLoss: [], - }); - }); - - it('keeps a variant field its base now stores', () => { - expect( - planned( - plan({ - from: tasks('from', [], ['note'], []), - to: tasks('to', ['note'], [], []), - schema: [schemaCollection('tasks')], - }), - ).labels, - ).toEqual([]); - }); - - it('unsets nothing without an origin contract when the destination collection has no validator', () => { - const unvalidated = contract( - 'to', - { Profile: { collection: 'profiles', fields: ['_id', 'email'] } }, - { profiles: {} }, - ); - expect(planned(plan({ from: null, to: unvalidated, schema })).labels).toEqual([ - 'Remove validator on profiles', - ]); - }); - - it('unsets nothing for a field a statement renames', () => { - const fullName = contract( - 'to', - { Profile: { collection: 'profiles', fields: ['_id', 'email', 'fullName'] } }, - { profiles: { required: ['email', 'fullName'] } }, - ); - expect( - planned( - plan({ - from, - to: fullName, - schema, - statements: [renameField('Profile', 'name', 'Profile', 'fullName')], - }), - ).dataLoss, - ).toEqual([]); - }); -}); - describe('MongoMigrationPlanner, time-series options the server fills in', () => { it('plans nothing when the live collection holds only server defaults the contract leaves out', () => { const readings = contract( diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts new file mode 100644 index 000000000000..b78398411db1 --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts @@ -0,0 +1,231 @@ +import { asNamespaceId, type ControlPolicy } from '@internal/contract/types'; +import type { + MigrationOperationPolicy, + ResolvedMigrationStatement, +} from '@internal/framework-components/control'; +import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; +import { + MongoSchemaCollection, + MongoSchemaCollectionOptions, + MongoSchemaIndex, + MongoSchemaIR, + MongoSchemaValidator, +} from '@internal/mongo-schema-ir'; +import { MongoMigrationPlanner } from '../src/core/migrations/mongo-planner'; +import type { PlannerProducedMongoMigration } from '../src/core/migrations/planner-produced-migration'; + +export const ALL_CLASSES_POLICY: MigrationOperationPolicy = { + allowedOperationClasses: ['additive', 'widening', 'destructive', 'data'], +}; + +export const NS = asNamespaceId('__unbound__'); + +export interface ModelSpec { + readonly collection: string; + readonly fields: readonly string[]; + readonly relations?: readonly string[]; + readonly base?: string; + readonly discriminator?: string; + readonly variants?: Readonly>; +} + +export interface CollectionSpec { + readonly unique?: readonly string[]; + readonly required?: readonly string[]; + readonly control?: ControlPolicy; + readonly timeField?: string; +} + +export function jsonSchema(fields: readonly string[]) { + return { + bsonType: 'object', + required: ['_id', ...fields], + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ...fields.map((field) => [field, { bsonType: 'string' }]), + ]), + additionalProperties: false, + }; +} + +export function contract( + seed: string, + models: Readonly>, + collections: Readonly>, +): MongoContract { + return { + target: 'mongo', + targetFamily: 'mongo', + profileHash: 'test-profile', + capabilities: {}, + extensions: {}, + meta: {}, + roots: {}, + domain: { + namespaces: { + __unbound__: { + models: Object.fromEntries( + Object.entries(models).map(([name, spec]) => [ + name, + { + fields: Object.fromEntries(spec.fields.map((field) => [field, {}])), + relations: Object.fromEntries( + (spec.relations ?? []).map((relation) => [ + relation, + { + to: { model: 'Other', namespace: '__unbound__' }, + cardinality: 'N:1', + on: { localFields: ['otherId'], targetFields: ['_id'] }, + }, + ]), + ), + storage: { collection: spec.collection }, + ...(spec.base === undefined + ? {} + : { base: { namespace: '__unbound__', model: spec.base } }), + ...(spec.discriminator === undefined + ? {} + : { + discriminator: { field: spec.discriminator }, + variants: Object.fromEntries( + Object.entries(spec.variants ?? {}).map(([variant, value]) => [ + variant, + { value }, + ]), + ), + }), + }, + ]), + ), + }, + }, + }, + storage: { + storageHash: `storage-${seed}`, + namespaces: { + __unbound__: { + id: '__unbound__', + kind: 'mongo-namespace', + entries: { + collection: Object.fromEntries( + Object.entries(collections).map(([name, spec]) => [ + name, + new MongoCollection({ + indexes: (spec.unique ?? []).map((field) => ({ + keys: [{ field, direction: 1 as const }], + unique: true, + })), + ...(spec.required === undefined + ? {} + : { + validator: { + jsonSchema: jsonSchema(spec.required), + validationLevel: 'strict' as const, + validationAction: 'error' as const, + }, + }), + ...(spec.control === undefined ? {} : { control: spec.control }), + ...(spec.timeField === undefined + ? {} + : { options: { timeseries: { timeField: spec.timeField } } }), + }), + ]), + ), + }, + }, + }, + }, + } as unknown as MongoContract; +} + +export function schemaCollection( + name: string, + spec: CollectionSpec & { readonly timeseries?: boolean } = {}, +): MongoSchemaCollection { + return new MongoSchemaCollection({ + name, + indexes: (spec.unique ?? []).map( + (field) => new MongoSchemaIndex({ keys: [{ field, direction: 1 }], unique: true }), + ), + ...(spec.required === undefined + ? {} + : { + validator: new MongoSchemaValidator({ + jsonSchema: jsonSchema(spec.required), + validationLevel: 'strict', + validationAction: 'error', + }), + }), + ...(spec.timeseries + ? { options: new MongoSchemaCollectionOptions({ timeseries: { timeField: 'at' } }) } + : {}), + }); +} + +export function plan(input: { + readonly from: MongoContract | null; + readonly to: MongoContract; + readonly schema: readonly MongoSchemaCollection[]; + readonly statements?: readonly ResolvedMigrationStatement[]; + readonly policy?: MigrationOperationPolicy; +}) { + return new MongoMigrationPlanner().plan({ + contract: input.to, + schema: new MongoSchemaIR(input.schema), + policy: input.policy ?? ALL_CLASSES_POLICY, + fromContract: input.from, + origin: null, + statements: input.statements ?? [], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); +} + +export function planned(result: ReturnType) { + if (result.kind !== 'success') throw new Error(JSON.stringify(result.conflicts, null, 2)); + const operations = (result.plan as PlannerProducedMongoMigration).operations; + return { + labels: operations.map((operation) => operation.label), + classes: operations.map((operation) => operation.operationClass), + applied: result.appliedStatements.map((applied) => applied.operationIndexes), + dataLoss: result.dataLoss, + }; +} + +export function refusal(result: ReturnType) { + if (result.kind !== 'failure') throw new Error('Expected the plan to be refused'); + return result.conflicts.map(({ kind, summary, why }) => ({ kind, summary, why })); +} + +export const renameModel = (from: string, to: string): ResolvedMigrationStatement => ({ + kind: 'rename', + entity: 'model', + from: { namespaceId: NS, model: from }, + to: { namespaceId: NS, model: to }, +}); + +export const renameField = ( + fromModel: string, + from: string, + toModel: string, + to: string, +): ResolvedMigrationStatement => ({ + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: fromModel, field: from }, + to: { namespaceId: NS, model: toModel, field: to }, +}); + +export const profiles = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, +); +export const users = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'name'] } }, + { users: { unique: ['name'], required: ['name'] } }, +); +export const profilesSchema = [ + schemaCollection('profiles', { unique: ['name'], required: ['name'] }), +]; diff --git a/test/integration/test/mongo/target-runner/runner-statements-fixtures.ts b/test/integration/test/mongo/target-runner/runner-statements-fixtures.ts new file mode 100644 index 000000000000..290aa45ccf3c --- /dev/null +++ b/test/integration/test/mongo/target-runner/runner-statements-fixtures.ts @@ -0,0 +1,244 @@ +import { introspectSchema, MongoControlAdapterImpl } from '@internal/adapter-mongo/control'; +import { asNamespaceId } from '@internal/contract/types'; +import { MongoControlDriver } from '@internal/driver-mongo/control'; +import type { + MigrationOperationPolicy, + MigrationPlan, + ResolvedMigrationStatement, +} from '@internal/framework-components/control'; +import { buildFabricatedMigrationEdge } from '@internal/migration-tools/aggregate'; +import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; +import type { AnyMongoMigrationOperation } from '@internal/mongo-query-ast/control'; +import { + MongoMigrationPlanner, + MongoMigrationRunner, + serializeMongoOps, +} from '@internal/target-mongo/control'; +import { timeouts } from '@repo/test-utils'; +import { type Db, MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, beforeAll, beforeEach } from 'vitest'; + +const controlAdapter = new MongoControlAdapterImpl(); +export const NS = asNamespaceId('__unbound__'); +const policy: MigrationOperationPolicy = { + allowedOperationClasses: ['additive', 'widening', 'destructive'], +}; + +export function closedSchema(fields: readonly string[]) { + return { + bsonType: 'object', + required: ['_id', ...fields], + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ...fields.map((field) => [field, { bsonType: 'string' }]), + ]), + additionalProperties: false, + }; +} + +export interface ModelSpec { + readonly collection: string; + readonly fields: readonly string[]; + readonly base?: string; + readonly variants?: Readonly>; +} + +export function contract( + storageHash: string, + models: Readonly>, + collections: Readonly>, +): MongoContract { + return { + target: 'mongo', + targetFamily: 'mongo', + profileHash: 'p1', + capabilities: {}, + extensions: {}, + meta: {}, + roots: {}, + domain: { + namespaces: { + __unbound__: { + models: Object.fromEntries( + Object.entries(models).map(([name, spec]) => [ + name, + { + fields: Object.fromEntries(spec.fields.map((field) => [field, {}])), + relations: {}, + storage: { collection: spec.collection }, + ...(spec.base === undefined + ? {} + : { base: { namespace: '__unbound__', model: spec.base } }), + ...(spec.variants === undefined + ? {} + : { + discriminator: { field: 'type' }, + variants: Object.fromEntries( + Object.entries(spec.variants).map(([variant, value]) => [ + variant, + { value }, + ]), + ), + }), + }, + ]), + ), + }, + }, + }, + storage: { + storageHash, + namespaces: { + __unbound__: { + id: '__unbound__', + kind: 'mongo-namespace', + entries: { collection: collections }, + }, + }, + }, + } as unknown as MongoContract; +} + +export function closedCollection(fields: readonly string[], unique: readonly string[] = []) { + return new MongoCollection({ + indexes: unique.map((field) => ({ keys: [{ field, direction: 1 as const }], unique: true })), + validator: { + jsonSchema: closedSchema(fields), + validationLevel: 'strict', + validationAction: 'error', + }, + }); +} + +/** + * Starts a MongoDB replica set for the file, empties the database before each test, and returns + * the helpers that plan and run against it. + */ +export function useStatementsDatabase(dbName: string) { + let replSet: MongoMemoryReplSet | undefined; + let client: MongoClient | undefined; + let database: Db | undefined; + const live = (): Db => { + if (database === undefined) throw new Error('The database is not started'); + return database; + }; + const connected = (): MongoClient => { + if (client === undefined) throw new Error('The database is not started'); + return client; + }; + + beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + database = client.db(dbName); + }, timeouts.spinUpMongoMemoryServer); + + afterAll(async () => { + await client?.close(); + await replSet?.stop(); + }, timeouts.spinUpMongoMemoryServer); + + beforeEach(async () => { + const collections = await live().listCollections().toArray(); + for (const col of collections) { + const name = col['name'] as string; + if (!name.startsWith('system.')) await live().dropCollection(name); + } + }); + + async function createClosed( + name: string, + fields: readonly string[], + unique: readonly string[], + documents: readonly Record[], + ) { + await live().createCollection(name, { + validator: { $jsonSchema: closedSchema(fields) }, + validationLevel: 'strict', + validationAction: 'error', + }); + for (const field of unique) { + await live() + .collection(name) + .createIndex({ [field]: 1 }, { unique: true, name: `${field}_1` }); + } + await live() + .collection(name) + .insertMany(documents.map((document) => ({ ...document }))); + } + + async function planAndRun( + from: MongoContract, + to: MongoContract, + statements: readonly ResolvedMigrationStatement[] = [], + ) { + await controlAdapter.initMarker(new MongoControlDriver(live(), connected()), 'app', { + storageHash: from.storage.storageHash, + profileHash: 'p1', + }); + const planned = new MongoMigrationPlanner().plan({ + contract: to, + schema: await introspectSchema(live()), + policy, + fromContract: from, + origin: { storageHash: from.storage.storageHash }, + statements, + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + if (planned.kind !== 'success') throw new Error(JSON.stringify(planned.conflicts, null, 2)); + const operations = planned.plan.operations as readonly AnyMongoMigrationOperation[]; + const plan: MigrationPlan = { + targetId: 'mongo', + operations: JSON.parse(serializeMongoOps(operations)), + origin: { storageHash: from.storage.storageHash }, + destination: { storageHash: to.storage.storageHash }, + }; + const result = await new MongoMigrationRunner( + controlAdapter.createRunnerDependencies(new MongoControlDriver(live(), connected())), + ).execute({ + plan, + migrationEdges: [ + buildFabricatedMigrationEdge({ + currentMarkerStorageHash: from.storage.storageHash, + destinationStorageHash: to.storage.storageHash, + operationCount: operations.length, + }), + ], + destinationContract: to, + policy, + frameworkComponents: [], + }); + return { planned, result }; + } + + async function furtherPlanLabels(to: MongoContract): Promise { + const planned = new MongoMigrationPlanner().plan({ + contract: to, + schema: await introspectSchema(live()), + policy, + fromContract: to, + origin: { storageHash: to.storage.storageHash }, + statements: [], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + if (planned.kind !== 'success') throw new Error(JSON.stringify(planned.conflicts)); + return (planned.plan.operations as readonly AnyMongoMigrationOperation[]).map( + (operation) => operation.label, + ); + } + + async function documents(name: string) { + return live() + .collection(name) + .find({}, { projection: { _id: 0 }, sort: { _id: 1 } }) + .toArray(); + } + + return { db: live, createClosed, planAndRun, furtherPlanLabels, documents }; +} diff --git a/test/integration/test/mongo/target-runner/runner-statements.test.ts b/test/integration/test/mongo/target-runner/runner-statements.test.ts index 9b78dbce016d..82fc33f8213e 100644 --- a/test/integration/test/mongo/target-runner/runner-statements.test.ts +++ b/test/integration/test/mongo/target-runner/runner-statements.test.ts @@ -1,295 +1,27 @@ -import { introspectSchema, MongoControlAdapterImpl } from '@internal/adapter-mongo/control'; -import { asNamespaceId } from '@internal/contract/types'; -import { MongoControlDriver } from '@internal/driver-mongo/control'; -import type { - MigrationOperationPolicy, - MigrationPlan, - ResolvedMigrationStatement, -} from '@internal/framework-components/control'; -import { buildFabricatedMigrationEdge } from '@internal/migration-tools/aggregate'; -import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; -import type { AnyMongoMigrationOperation } from '@internal/mongo-query-ast/control'; -import { - MongoMigrationPlanner, - MongoMigrationRunner, - serializeMongoOps, -} from '@internal/target-mongo/control'; +import { introspectSchema } from '@internal/adapter-mongo/control'; +import type { MigrationOperationPolicy } from '@internal/framework-components/control'; +import { MongoCollection } from '@internal/mongo-contract'; +import { MongoMigrationPlanner } from '@internal/target-mongo/control'; import { timeouts } from '@repo/test-utils'; -import { type Db, MongoClient } from 'mongodb'; -import { MongoMemoryReplSet } from 'mongodb-memory-server'; -import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest'; +import { describe, expect, it } from 'vitest'; +import { + closedCollection, + contract, + NS, + useStatementsDatabase, +} from './runner-statements-fixtures'; + +const live = useStatementsDatabase('runner_statements_test'); -const controlAdapter = new MongoControlAdapterImpl(); -const NS = asNamespaceId('__unbound__'); const policy: MigrationOperationPolicy = { allowedOperationClasses: ['additive', 'widening', 'destructive'], }; -let replSet: MongoMemoryReplSet; -let client: MongoClient; -let db: Db; -const dbName = 'runner_statements_test'; - -beforeAll(async () => { - replSet = await MongoMemoryReplSet.create({ - replSet: { count: 1, storageEngine: 'wiredTiger' }, - }); - client = new MongoClient(replSet.getUri()); - await client.connect(); - db = client.db(dbName); -}, timeouts.spinUpMongoMemoryServer); - -afterAll(async () => { - await client?.close(); - await replSet?.stop(); -}, timeouts.spinUpMongoMemoryServer); - -beforeEach(async () => { - const collections = await db.listCollections().toArray(); - for (const col of collections) { - const name = col['name'] as string; - if (!name.startsWith('system.')) await db.dropCollection(name); - } -}); - -function closedSchema(fields: readonly string[]) { - return { - bsonType: 'object', - required: ['_id', ...fields], - properties: Object.fromEntries([ - ['_id', { bsonType: 'objectId' }], - ...fields.map((field) => [field, { bsonType: 'string' }]), - ]), - additionalProperties: false, - }; -} - -interface ModelSpec { - readonly collection: string; - readonly fields: readonly string[]; - readonly base?: string; - readonly variants?: Readonly>; -} - -function contract( - storageHash: string, - models: Readonly>, - collections: Readonly>, -): MongoContract { - return { - target: 'mongo', - targetFamily: 'mongo', - profileHash: 'p1', - capabilities: {}, - extensions: {}, - meta: {}, - roots: {}, - domain: { - namespaces: { - __unbound__: { - models: Object.fromEntries( - Object.entries(models).map(([name, spec]) => [ - name, - { - fields: Object.fromEntries(spec.fields.map((field) => [field, {}])), - relations: {}, - storage: { collection: spec.collection }, - ...(spec.base === undefined - ? {} - : { base: { namespace: '__unbound__', model: spec.base } }), - ...(spec.variants === undefined - ? {} - : { - discriminator: { field: 'type' }, - variants: Object.fromEntries( - Object.entries(spec.variants).map(([variant, value]) => [ - variant, - { value }, - ]), - ), - }), - }, - ]), - ), - }, - }, - }, - storage: { - storageHash, - namespaces: { - __unbound__: { - id: '__unbound__', - kind: 'mongo-namespace', - entries: { collection: collections }, - }, - }, - }, - } as unknown as MongoContract; -} - -function closedCollection(fields: readonly string[], unique: readonly string[] = []) { - return new MongoCollection({ - indexes: unique.map((field) => ({ keys: [{ field, direction: 1 as const }], unique: true })), - validator: { - jsonSchema: closedSchema(fields), - validationLevel: 'strict', - validationAction: 'error', - }, - }); -} - -async function createClosed( - name: string, - fields: readonly string[], - unique: readonly string[], - documents: readonly Record[], -) { - await db.createCollection(name, { - validator: { $jsonSchema: closedSchema(fields) }, - validationLevel: 'strict', - validationAction: 'error', - }); - for (const field of unique) { - await db.collection(name).createIndex({ [field]: 1 }, { unique: true, name: `${field}_1` }); - } - await db.collection(name).insertMany(documents.map((document) => ({ ...document }))); -} - -async function planAndRun( - from: MongoContract, - to: MongoContract, - statements: readonly ResolvedMigrationStatement[] = [], -) { - await controlAdapter.initMarker(new MongoControlDriver(db, client), 'app', { - storageHash: from.storage.storageHash, - profileHash: 'p1', - }); - const planned = new MongoMigrationPlanner().plan({ - contract: to, - schema: await introspectSchema(db), - policy, - fromContract: from, - origin: { storageHash: from.storage.storageHash }, - statements, - frameworkComponents: [], - snapshotsImportPath: '../../snapshots', - }); - if (planned.kind !== 'success') throw new Error(JSON.stringify(planned.conflicts, null, 2)); - const operations = planned.plan.operations as readonly AnyMongoMigrationOperation[]; - const plan: MigrationPlan = { - targetId: 'mongo', - operations: JSON.parse(serializeMongoOps(operations)), - origin: { storageHash: from.storage.storageHash }, - destination: { storageHash: to.storage.storageHash }, - }; - const result = await new MongoMigrationRunner( - controlAdapter.createRunnerDependencies(new MongoControlDriver(db, client)), - ).execute({ - plan, - migrationEdges: [ - buildFabricatedMigrationEdge({ - currentMarkerStorageHash: from.storage.storageHash, - destinationStorageHash: to.storage.storageHash, - operationCount: operations.length, - }), - ], - destinationContract: to, - policy, - frameworkComponents: [], - }); - return { planned, result }; -} - -async function furtherPlanLabels(to: MongoContract): Promise { - const planned = new MongoMigrationPlanner().plan({ - contract: to, - schema: await introspectSchema(db), - policy, - fromContract: to, - origin: { storageHash: to.storage.storageHash }, - statements: [], - frameworkComponents: [], - snapshotsImportPath: '../../snapshots', - }); - if (planned.kind !== 'success') throw new Error(JSON.stringify(planned.conflicts)); - return (planned.plan.operations as readonly AnyMongoMigrationOperation[]).map( - (operation) => operation.label, - ); -} - -async function documents(name: string) { - return db - .collection(name) - .find({}, { projection: { _id: 0 }, sort: { _id: 1 } }) - .toArray(); -} - -function branchSchema(value: string, fields: readonly string[]) { - return { - properties: Object.fromEntries([ - ['_id', { bsonType: 'objectId' }], - ['type', { enum: [value] }], - ...fields.map((field) => [field, { bsonType: 'string' }]), - ]), - required: ['type'], - additionalProperties: false, - }; -} - -function tasksValidator(base: readonly string[], bug: readonly string[]) { - return { - bsonType: 'object', - required: ['_id', 'type'], - properties: Object.fromEntries([ - ['_id', { bsonType: 'objectId' }], - ['type', { bsonType: 'string' }], - ...base.map((field) => [field, { bsonType: 'string' }]), - ]), - oneOf: [branchSchema('bug', [...base, ...bug]), branchSchema('feature', [...base, 'priority'])], - }; -} - -function tasks(storageHash: string, base: readonly string[], bug: readonly string[]) { - return contract( - storageHash, - { - Task: { - collection: 'tasks', - fields: ['_id', 'type', ...base], - variants: { Bug: 'bug', Feature: 'feature' }, - }, - Bug: { collection: 'tasks', fields: bug, base: 'Task' }, - Feature: { collection: 'tasks', fields: ['priority'], base: 'Task' }, - }, - { - tasks: new MongoCollection({ - validator: { - jsonSchema: tasksValidator(base, bug), - validationLevel: 'strict', - validationAction: 'error', - }, - }), - }, - ); -} - -async function createTasksWithNote(): Promise { - await db.createCollection('tasks', { - validator: { $jsonSchema: tasksValidator(['note'], []) }, - validationLevel: 'strict', - validationAction: 'error', - }); - await db.collection('tasks').insertMany([ - { type: 'bug', note: 'b' }, - { type: 'feature', note: 'f' }, - ]); -} - describe('MongoMigrationPlanner and runner - statements', () => { it( 'renames a model and a uniquely indexed field behind a closed validator, keeping every document', async () => { - await createClosed('profiles', ['name'], ['name'], [{ name: 'Ada' }, { name: 'Bo' }]); + await live.createClosed('profiles', ['name'], ['name'], [{ name: 'Ada' }, { name: 'Bo' }]); const from = contract( 'from', { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, @@ -301,7 +33,7 @@ describe('MongoMigrationPlanner and runner - statements', () => { { users: closedCollection(['fullName'], ['fullName']) }, ); - const { planned, result } = await planAndRun(from, to, [ + const { planned, result } = await live.planAndRun(from, to, [ { kind: 'rename', entity: 'model', @@ -320,19 +52,25 @@ describe('MongoMigrationPlanner and runner - statements', () => { expect(planned.kind === 'success' ? planned.dataLoss : undefined).toEqual([]); expect({ collections: ( - await db.listCollections({ name: { $in: ['profiles', 'users'] } }).toArray() + await live + .db() + .listCollections({ name: { $in: ['profiles', 'users'] } }) + .toArray() ).map((c) => c.name), - users: await documents('users'), - further: await furtherPlanLabels(to), + users: await live.documents('users'), + further: await live.furtherPlanLabels(to), }).toEqual({ collections: ['users'], users: [{ fullName: 'Ada' }, { fullName: 'Bo' }], further: [], }); await expect( - db.collection('users').updateOne({ fullName: 'Ada' }, { $set: { fullName: 'Ada L.' } }), + live + .db() + .collection('users') + .updateOne({ fullName: 'Ada' }, { $set: { fullName: 'Ada L.' } }), ).resolves.toMatchObject({ modifiedCount: 1 }); - await expect(db.collection('users').insertOne({ fullName: 'Bo' })).rejects.toThrow( + await expect(live.db().collection('users').insertOne({ fullName: 'Bo' })).rejects.toThrow( /duplicate key/, ); }, @@ -342,7 +80,7 @@ describe('MongoMigrationPlanner and runner - statements', () => { it( 'removes a field the destination no longer has from every document', async () => { - await createClosed( + await live.createClosed( 'users', ['email', 'nickname'], [], @@ -362,7 +100,7 @@ describe('MongoMigrationPlanner and runner - statements', () => { { users: closedCollection(['email']) }, ); - const { planned, result } = await planAndRun(from, to); + const { planned, result } = await live.planAndRun(from, to); expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); expect(planned.kind === 'success' ? planned.dataLoss : undefined).toEqual([ @@ -371,114 +109,18 @@ describe('MongoMigrationPlanner and runner - statements', () => { subject: { kind: 'field', namespaceId: NS, model: 'User', field: 'nickname' }, }, ]); - expect({ users: await documents('users'), further: await furtherPlanLabels(to) }).toEqual({ + expect({ + users: await live.documents('users'), + further: await live.furtherPlanLabels(to), + }).toEqual({ users: [{ email: 'a@x' }, { email: 'b@x' }], further: [], }); await expect( - db.collection('users').updateOne({ email: 'a@x' }, { $set: { email: 'a@y' } }), - ).resolves.toMatchObject({ modifiedCount: 1 }); - }, - timeouts.spinUpMongoMemoryServer, - ); - - it( - "renames a variant's field only in its own documents, so another variant's field of that name keeps its values", - async () => { - await db.collection('tasks').insertMany([ - { type: 'bug', score: 3 }, - { type: 'feature', score: 5 }, - ]); - const tasks = (storageHash: string, bugField: string) => - contract( - storageHash, - { - Task: { - collection: 'tasks', - fields: ['_id', 'type'], - variants: { Bug: 'bug', Feature: 'feature' }, - }, - Bug: { collection: 'tasks', fields: [bugField], base: 'Task' }, - Feature: { collection: 'tasks', fields: ['score'], base: 'Task' }, - }, - { tasks: new MongoCollection({}) }, - ); - - const { result } = await planAndRun(tasks('from', 'score'), tasks('to', 'severity'), [ - { - kind: 'rename', - entity: 'field', - from: { namespaceId: NS, model: 'Bug', field: 'score' }, - to: { namespaceId: NS, model: 'Bug', field: 'severity' }, - }, - ]); - - expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); - expect(await documents('tasks')).toEqual([ - { type: 'bug', severity: 3 }, - { type: 'feature', score: 5 }, - ]); - }, - timeouts.spinUpMongoMemoryServer, - ); - - it( - 'removes a base field moved to one variant from the documents of every other variant', - async () => { - await createTasksWithNote(); - const to = tasks('to', [], ['note']); - - const { planned, result } = await planAndRun(tasks('from', ['note'], []), to); - - expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); - expect(planned.kind === 'success' ? planned.dataLoss : undefined).toEqual([ - { - operationIndex: 0, - subject: { kind: 'field', namespaceId: NS, model: 'Task', field: 'note' }, - }, - ]); - expect({ tasks: await documents('tasks'), further: await furtherPlanLabels(to) }).toEqual({ - tasks: [{ type: 'bug', note: 'b' }, { type: 'feature' }], - further: [], - }); - await expect( - db.collection('tasks').updateOne({ type: 'feature' }, { $set: { priority: 'p1' } }), - ).resolves.toMatchObject({ modifiedCount: 1 }); - await expect( - db.collection('tasks').updateOne({ type: 'bug' }, { $set: { note: 'b2' } }), - ).resolves.toMatchObject({ modifiedCount: 1 }); - }, - timeouts.spinUpMongoMemoryServer, - ); - - it( - 'renames a base field moved to one variant only in the documents of the other variants', - async () => { - await createTasksWithNote(); - const to = tasks('to', ['remark'], ['note']); - - const { result } = await planAndRun(tasks('from', ['note'], []), to, [ - { - kind: 'rename', - entity: 'field', - from: { namespaceId: NS, model: 'Task', field: 'note' }, - to: { namespaceId: NS, model: 'Task', field: 'remark' }, - }, - ]); - - expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); - expect({ tasks: await documents('tasks'), further: await furtherPlanLabels(to) }).toEqual({ - tasks: [ - { type: 'bug', note: 'b' }, - { type: 'feature', remark: 'f' }, - ], - further: [], - }); - await expect( - db.collection('tasks').updateOne({ type: 'feature' }, { $set: { priority: 'p1' } }), - ).resolves.toMatchObject({ modifiedCount: 1 }); - await expect( - db.collection('tasks').updateOne({ type: 'bug' }, { $set: { note: 'b2' } }), + live + .db() + .collection('users') + .updateOne({ email: 'a@x' }, { $set: { email: 'a@y' } }), ).resolves.toMatchObject({ modifiedCount: 1 }); }, timeouts.spinUpMongoMemoryServer, @@ -497,10 +139,10 @@ describe('MongoMigrationPlanner and runner - statements', () => { async () => { const to = readings('to', 'Reading', 'readings'); - const { result } = await planAndRun(contract('from', {}, {}), to); + const { result } = await live.planAndRun(contract('from', {}, {}), to); expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); - expect(await furtherPlanLabels(to)).toEqual([]); + expect(await live.furtherPlanLabels(to)).toEqual([]); }, timeouts.spinUpMongoMemoryServer, ); @@ -510,11 +152,14 @@ describe('MongoMigrationPlanner and runner - statements', () => { async () => { const from = readings('from', 'Reading', 'readings'); const to = readings('to', 'Measurement', 'measurements'); - await db.createCollection('readings', { timeseries: { timeField: 'at' } }); - await db.collection('readings').insertOne({ at: new Date('2026-01-01T00:00:00Z') }); + await live.db().createCollection('readings', { timeseries: { timeField: 'at' } }); + await live + .db() + .collection('readings') + .insertOne({ at: new Date('2026-01-01T00:00:00Z') }); const refused = new MongoMigrationPlanner().plan({ contract: to, - schema: await introspectSchema(db), + schema: await introspectSchema(live.db()), policy, fromContract: from, origin: { storageHash: 'from' }, @@ -533,25 +178,27 @@ describe('MongoMigrationPlanner and runner - statements', () => { 'MongoDB cannot rename a time-series collection. Leave out this statement. To keep the documents, first create collection "measurements" yourself with the same time-series options and copy them into it; the plan then drops "readings" once --delete names the model.', ); - await db.createCollection('measurements', { timeseries: { timeField: 'at' } }); - const copied = await db + await live.db().createCollection('measurements', { timeseries: { timeField: 'at' } }); + const copied = await live + .db() .collection('readings') .find({}, { projection: { _id: 0 } }) .toArray(); - await db.collection('measurements').insertMany(copied); - const { planned, result } = await planAndRun(from, to); + await live.db().collection('measurements').insertMany(copied); + const { planned, result } = await live.planAndRun(from, to); expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); expect( planned.kind === 'success' ? planned.dataLoss.map(({ subject }) => subject) : [], ).toEqual([{ kind: 'model', namespaceId: NS, model: 'Reading' }]); expect({ - measurements: await db + measurements: await live + .db() .collection('measurements') .find({}, { projection: { _id: 0 } }) .toArray(), - readings: await db.listCollections({ name: 'readings' }).toArray(), - further: await furtherPlanLabels(to), + readings: await live.db().listCollections({ name: 'readings' }).toArray(), + further: await live.furtherPlanLabels(to), }).toEqual({ measurements: [{ at: new Date('2026-01-01T00:00:00Z') }], readings: [], diff --git a/test/integration/test/mongo/target-runner/runner-statements.variants.test.ts b/test/integration/test/mongo/target-runner/runner-statements.variants.test.ts new file mode 100644 index 000000000000..c9634ddc83e1 --- /dev/null +++ b/test/integration/test/mongo/target-runner/runner-statements.variants.test.ts @@ -0,0 +1,195 @@ +import { MongoCollection } from '@internal/mongo-contract'; +import { timeouts } from '@repo/test-utils'; +import { describe, expect, it } from 'vitest'; +import { contract, NS, useStatementsDatabase } from './runner-statements-fixtures'; + +const live = useStatementsDatabase('runner_statements_variants_test'); + +function branchSchema(value: string, fields: readonly string[]) { + return { + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ['type', { enum: [value] }], + ...fields.map((field) => [field, { bsonType: 'string' }]), + ]), + required: ['type'], + additionalProperties: false, + }; +} + +function tasksValidator(base: readonly string[], bug: readonly string[]) { + return { + bsonType: 'object', + required: ['_id', 'type'], + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ['type', { bsonType: 'string' }], + ...base.map((field) => [field, { bsonType: 'string' }]), + ]), + oneOf: [branchSchema('bug', [...base, ...bug]), branchSchema('feature', [...base, 'priority'])], + }; +} + +function tasks(storageHash: string, base: readonly string[], bug: readonly string[]) { + return contract( + storageHash, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type', ...base], + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: bug, base: 'Task' }, + Feature: { collection: 'tasks', fields: ['priority'], base: 'Task' }, + }, + { + tasks: new MongoCollection({ + validator: { + jsonSchema: tasksValidator(base, bug), + validationLevel: 'strict', + validationAction: 'error', + }, + }), + }, + ); +} + +async function createTasksWithNote(): Promise { + await live.db().createCollection('tasks', { + validator: { $jsonSchema: tasksValidator(['note'], []) }, + validationLevel: 'strict', + validationAction: 'error', + }); + await live + .db() + .collection('tasks') + .insertMany([ + { type: 'bug', note: 'b' }, + { type: 'feature', note: 'f' }, + ]); +} + +describe('MongoMigrationPlanner and runner - statements on variants', () => { + it( + "renames a variant's field only in its own documents, so another variant's field of that name keeps its values", + async () => { + await live + .db() + .collection('tasks') + .insertMany([ + { type: 'bug', score: 3 }, + { type: 'feature', score: 5 }, + ]); + const tasks = (storageHash: string, bugField: string) => + contract( + storageHash, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type'], + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: [bugField], base: 'Task' }, + Feature: { collection: 'tasks', fields: ['score'], base: 'Task' }, + }, + { tasks: new MongoCollection({}) }, + ); + + const { result } = await live.planAndRun(tasks('from', 'score'), tasks('to', 'severity'), [ + { + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: 'Bug', field: 'score' }, + to: { namespaceId: NS, model: 'Bug', field: 'severity' }, + }, + ]); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect(await live.documents('tasks')).toEqual([ + { type: 'bug', severity: 3 }, + { type: 'feature', score: 5 }, + ]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'removes a base field moved to one variant from the documents of every other variant', + async () => { + await createTasksWithNote(); + const to = tasks('to', [], ['note']); + + const { planned, result } = await live.planAndRun(tasks('from', ['note'], []), to); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); + expect(planned.kind === 'success' ? planned.dataLoss : undefined).toEqual([ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Task', field: 'note' }, + }, + ]); + expect({ + tasks: await live.documents('tasks'), + further: await live.furtherPlanLabels(to), + }).toEqual({ + tasks: [{ type: 'bug', note: 'b' }, { type: 'feature' }], + further: [], + }); + await expect( + live + .db() + .collection('tasks') + .updateOne({ type: 'feature' }, { $set: { priority: 'p1' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + live + .db() + .collection('tasks') + .updateOne({ type: 'bug' }, { $set: { note: 'b2' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'renames a base field moved to one variant only in the documents of the other variants', + async () => { + await createTasksWithNote(); + const to = tasks('to', ['remark'], ['note']); + + const { result } = await live.planAndRun(tasks('from', ['note'], []), to, [ + { + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: 'Task', field: 'note' }, + to: { namespaceId: NS, model: 'Task', field: 'remark' }, + }, + ]); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); + expect({ + tasks: await live.documents('tasks'), + further: await live.furtherPlanLabels(to), + }).toEqual({ + tasks: [ + { type: 'bug', note: 'b' }, + { type: 'feature', remark: 'f' }, + ], + further: [], + }); + await expect( + live + .db() + .collection('tasks') + .updateOne({ type: 'feature' }, { $set: { priority: 'p1' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + live + .db() + .collection('tasks') + .updateOne({ type: 'bug' }, { $set: { note: 'b2' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + }, + timeouts.spinUpMongoMemoryServer, + ); +}); From 5824bc95e4bb1d7d256105dcd4d0909f1c29c725 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:44:25 +0200 Subject: [PATCH 24/57] TML-3478: a removed field is unset only in a managed collection planFieldRemovals and planValidatorFieldRemovals skip a collection whose control policy is external, observed or tolerated, so --delete never writes to a collection Prisma does not manage. Signed-off-by: willbot Signed-off-by: Will Madden --- .../src/core/migrations/mongo-planner.ts | 2 +- .../migrations/mongo-statement-planning.ts | 5 ++- .../mongo-planner.statements.fields.test.ts | 32 +++++++++++++++++++ 3 files changed, 37 insertions(+), 2 deletions(-) diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts index 320caec7d50b..715c0b922a8f 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts @@ -369,7 +369,7 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> const renamedTo = new Map(collectionRenames.map((call) => [call.from, call.to])); const removals: readonly FieldRemoval[] = fromContract === null - ? planValidatorFieldRemovals({ origin: schema, destination: destinationIR }) + ? planValidatorFieldRemovals({ origin: schema, destination: destinationIR, contract }) : planFieldRemovals({ statements: options.statements, fromContract, diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index 04bf8fdffebd..2956476cbcbb 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -439,7 +439,7 @@ function rootDocumentsLosing( /** * Each field an origin model stores that its destination model no longer stores, in a collection * the plan keeps: removing it is data loss, carried out by unsetting it from the documents that - * hold it. A variant's field is unset only from that variant's documents, unless its destination + * hold it, in a collection whose control policy is `managed`. A variant's field is unset only from that variant's documents, unless its destination * base now stores it. A base field is unset from the documents of every variant that does not now * store it itself. */ @@ -447,6 +447,7 @@ export function planFieldRemovals(input: FieldRemovalInput): readonly FieldRemov return modelCoordinates(input.fromContract).flatMap((origin) => { const collection = input.collectionAfterRenames(collectionOf(input.fromContract, origin)); if (!input.keptCollections.has(collection)) return []; + if (collectionControlPolicy(input.contract, collection) !== 'managed') return []; const destination = destinationModelOf(origin, input); if (destination === undefined) return []; const model = modelAt(input.fromContract, origin); @@ -502,8 +503,10 @@ function branchPropertyNames(schema: unknown): readonly string[] { export function planValidatorFieldRemovals(input: { readonly origin: MongoSchemaIR; readonly destination: MongoSchemaIR; + readonly contract: MongoContract; }): readonly FieldRemoval[] { return input.origin.collections.flatMap((originCollection) => { + if (collectionControlPolicy(input.contract, originCollection.name) !== 'managed') return []; const destination = input.destination.collection(originCollection.name); if (destination?.validator === undefined || originCollection.validator === undefined) return []; const kept = new Set([ diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts index 8164e477343a..83779e9d6c30 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts @@ -1,3 +1,4 @@ +import type { ControlPolicy } from '@internal/contract/types'; import { describe, expect, it } from 'vitest'; import { contract, @@ -280,3 +281,34 @@ describe('MongoMigrationPlanner, removed fields', () => { ).toEqual([]); }); }); + +describe('MongoMigrationPlanner, removed fields in a collection that is not managed', () => { + const schema = [schemaCollection('auth_users', { required: ['email', 'legacy'] })]; + const authUsers = (seed: string, fields: readonly string[], control: ControlPolicy) => + contract( + seed, + { AuthUser: { collection: 'auth_users', fields: ['_id', ...fields] } }, + { auth_users: { required: fields, control } }, + ); + + for (const control of ['external', 'observed', 'tolerated'] as const) { + it(`unsets nothing and asks nothing when the collection is ${control}`, () => { + const result = planned( + plan({ + from: authUsers('from', ['email', 'legacy'], control), + to: authUsers('to', ['email'], control), + schema, + }), + ); + const noOrigin = planned( + plan({ from: null, to: authUsers('to', ['email'], control), schema }), + ); + expect({ + unsets: [...result.labels, ...noOrigin.labels].filter((label) => + label.startsWith('Remove field'), + ), + dataLoss: [...result.dataLoss, ...noOrigin.dataLoss], + }).toEqual({ unsets: [], dataLoss: [] }); + }); + } +}); From 99480e2be03c26215b9fa962e6c602f41a33c18c Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:45:14 +0200 Subject: [PATCH 25/57] TML-3478: a variant rename that changes its discriminator value is refused Its documents would keep the old value and fail the new validator, and this release does not rewrite discriminator values. The refusal says to keep the value with @@base; a value change is answered with --convert in a later release. Signed-off-by: willbot Signed-off-by: Will Madden --- .../subsystems/7. Migration System.md | 2 +- docs/reference/error-reference.md | 2 +- .../migrations/mongo-statement-planning.ts | 27 ++++++++++ .../mongo-planner.statements.refusals.test.ts | 51 +++++++++++++++++++ 4 files changed, 80 insertions(+), 2 deletions(-) diff --git a/docs/architecture docs/subsystems/7. Migration System.md b/docs/architecture docs/subsystems/7. Migration System.md index 9d9a616171d4..8767e7b5ae93 100644 --- a/docs/architecture docs/subsystems/7. Migration System.md +++ b/docs/architecture docs/subsystems/7. Migration System.md @@ -225,7 +225,7 @@ The MongoDB planner does the same with collections and documents ([`mongo-statem - A model move to another namespace resolves, but the planner refuses it with `statementRefused`. - A model that keeps its name but changes its table cannot be renamed by a statement, because the model has the same name on both sides. Rename the table by hand with `this.renameTable(...)` in its own `migration.ts`. A field statement on such a model is refused with the same advice. - Only models and fields can be renamed. Value objects, their fields, enum values and namespaces cannot. -- On MongoDB, a statement cannot rename `_id`, a field whose stored name MongoDB reads as a path or an operator (it contains `.` or starts with `$`), a time-series collection, or a field inside a value object. A collection rename onto a name the database already has, or onto a name MongoDB rejects, is refused too. +- On MongoDB, a statement cannot rename a variant whose discriminator value changes, since its documents would keep the old value; it cannot rename `_id`, a field whose stored name MongoDB reads as a path or an operator (it contains `.` or starts with `$`), a time-series collection, or a field inside a value object. A collection rename onto a name the database already has, or onto a name MongoDB rejects, is refused too. - **[Placeholder: decision pending]** On a MongoDB contract without a validator, such as one written with the TypeScript contract builder, renaming or removing a field does not change the storage hash. What `migration plan` and `migrate` do with such a change is described here once it is decided. #### Delete and allow statements diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index be5a3a85fe77..cd17c21195b2 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -1576,7 +1576,7 @@ An apply command (`db migrate`/`db update`) cannot find a path through the on-di ### MIGRATION.PLANNING_FAILED -Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, replace the drop and create in its `migration.ts` with `renameCollection(...)`, run `node migration.ts` again, and apply it with `migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. +Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; renaming a variant whose discriminator value changes (the message says to keep the value with `@@base(, "")`); a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, replace the drop and create in its `migration.ts` with `renameCollection(...)`, run `node migration.ts` again, and apply it with `migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. `migration plan` raises it when the planner produces no operations. The conflict's kind names the case, and its `why` is the next action. With an earlier contract (`noDatabaseChange`), the contract changed in a way the database does not see, for example after a Prisma upgrade that changes how `contract.json` records values. If you deploy with migrations, run `prisma migration new --from `, with the earlier contract's hash from the message, to write a migration with no operations, then `prisma db migrate`. If you manage the database with `db init` or `db update`, run `prisma db sign` on each database instead. With no earlier contract (`nothingToCreate`), the contract describes nothing to create; if the database needs something the contract does not describe, run `prisma migration new` and write the operations in `migration.ts` by hand. `migration new` writes a migration only to the emitted contract, so with `--to` the message says to emit the target first, and for a database managed with `db update` it names `prisma db sign `. When the migrations directory is empty and the `db` ref points at a contract that describes nothing to create (`nothingToBaseline`), there is no baseline migration to plan, and no command writes one with no operations yet; keep using `db update`. diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index 2956476cbcbb..b53ea58ac4e7 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -222,12 +222,39 @@ function controlPolicyRefusal( }; } +/** + * The reason and advice when a statement renames a variant whose discriminator value changes: its + * documents keep the old value, which the new validator does not accept. + */ +function discriminatorValueChange( + statement: ResolvedModelRenameStatement, + input: { readonly fromContract: ContractWithDomain; readonly contract: MongoContract }, +): readonly [reason: string, why: string] | undefined { + const before = variantFilter(input.fromContract, statement.from); + const after = variantFilter(input.contract, statement.to); + const base = modelAt(input.contract, statement.to)?.base; + if (before === undefined || after === undefined || base === undefined) return undefined; + const [field, oldValue] = Object.entries(before)[0] ?? []; + const newValue = Object.values(after)[0]; + if (field === undefined || oldValue === newValue) return undefined; + const fromName = modelDisplayName(statement.from); + const toName = modelDisplayName(statement.to); + return [ + `its discriminator value changes from ${JSON.stringify(oldValue)} to ${JSON.stringify(newValue)}`, + `The documents of "${fromName}" hold ${field} ${JSON.stringify(oldValue)}, and this release does not rewrite a discriminator value, so they would no longer match "${toName}" and would fail validation. Keep the value: declare "${toName}" with @@base(${base.model}, ${JSON.stringify(oldValue)}). A later release changes a stored value with --convert.`, + ]; +} + function planModelStatement( statement: ResolvedModelRenameStatement, label: string, schema: MongoSchemaIR, input: { readonly fromContract: ContractWithDomain; readonly contract: MongoContract }, ): StatementOutcome { + const discriminatorChange = discriminatorValueChange(statement, input); + if (discriminatorChange !== undefined) { + return { kind: 'refused', conflict: refused(statement, label, ...discriminatorChange) }; + } const from = collectionOf(input.fromContract, statement.from); const to = collectionOf(input.contract, statement.to); if (from === to) return { kind: 'unchanged' }; diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts index c6a1409a69df..479cc7b2016c 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts @@ -203,3 +203,54 @@ describe('MongoMigrationPlanner, statements it refuses', () => { ); }); }); + +describe('MongoMigrationPlanner, a variant rename whose discriminator value changes', () => { + const tasks = (seed: string, variant: string, value: string) => + contract( + seed, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type'], + discriminator: 'type', + variants: { [variant]: value, Feature: 'feature' }, + }, + [variant]: { collection: 'tasks', fields: ['score'], base: 'Task' }, + Feature: { collection: 'tasks', fields: ['priority'], base: 'Task' }, + }, + { tasks: {} }, + ); + + it('is refused, and says to keep the value', () => { + expect( + refusal( + plan({ + from: tasks('from', 'Bug', 'bug'), + to: tasks('to', 'Defect', 'defect'), + schema: [schemaCollection('tasks')], + statements: [renameModel('Bug', 'Defect')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: + 'Cannot rename model "Bug" to "Defect": its discriminator value changes from "bug" to "defect"', + why: 'The documents of "Bug" hold type "bug", and this release does not rewrite a discriminator value, so they would no longer match "Defect" and would fail validation. Keep the value: declare "Defect" with @@base(Task, "bug"). A later release changes a stored value with --convert.', + }, + ]); + }); + + it('plans nothing when the value is kept', () => { + const keep = tasks('to', 'Defect', 'bug'); + const result = plan({ + from: tasks('from', 'Bug', 'bug'), + to: keep, + schema: [schemaCollection('tasks')], + statements: [renameModel('Bug', 'Defect')], + }); + expect( + result.kind === 'success' ? result.appliedStatements[0]?.operationIndexes : result, + ).toEqual([]); + }); +}); From eab9bbfc83281865e922dbb080023ee38c52ef8a Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:47:18 +0200 Subject: [PATCH 26/57] TML-3478: a rewrite the database user may not run past validation names the privilege to grant When MongoDB refuses a document rewrite with error 13 (Unauthorized), the runner fails with MIGRATION.EXECUTION_FAILED naming the bypassDocumentValidation action and the dbAdmin and restore roles that grant it. Other failures pass through unchanged. Signed-off-by: willbot Signed-off-by: Will Madden --- docs/reference/error-reference.md | 2 +- .../src/core/migrations/mongo-runner.ts | 42 ++++++++++++++- .../1-mongo-target/test/mongo-runner.test.ts | 52 ++++++++++++++++++- 3 files changed, 91 insertions(+), 5 deletions(-) diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index cd17c21195b2..6206a1e7ac73 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -1448,7 +1448,7 @@ The per-space migration planner received the same contract-space id more than on ### MIGRATION.EXECUTION_FAILED -A migration operation's SQL step failed while being executed against the database during apply (`db init`, `db update`, `db migrate`). The envelope carries the database error detail so you can see which statement failed and why. Payload: `operationId`, `stepDescription`, `sql`, `sqlState`, `constraint`, `table`, `column`, `detail`. +A migration operation's SQL step failed while being executed against the database during apply (`db init`, `db update`, `db migrate`). The envelope carries the database error detail so you can see which statement failed and why. Payload: `operationId`, `stepDescription`, `sql`, `sqlState`, `constraint`, `table`, `column`, `detail`. On MongoDB it is raised when a field rename or removal rewrites documents with `bypassDocumentValidation` and the database user lacks that action (MongoDB error 13, `Unauthorized`), as a `readWrite` user on a hosted cluster does; the message says to grant the `bypassDocumentValidation` action, for example with the `dbAdmin` or `restore` role, and to run the command again. Payload: `operationId`, `mongoCode`, `codeName`. ### MIGRATION.FIELD_NAME_UNSUPPORTED diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts index 8c2534759200..4532ba4d4312 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts @@ -28,6 +28,7 @@ import type { MongoMigrationCheck, } from '@internal/mongo-query-ast/control'; import { isMongoDmlOperation } from '@internal/mongo-query-ast/control'; +import { type MongoQueryPlan, RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; import { blindCast } from '@internal/utils/casts'; import { InternalError } from '@internal/utils/internal-error'; import { notOk, ok, type Result } from '@internal/utils/result'; @@ -80,6 +81,37 @@ function runnerFailure( }); } +const MONGO_UNAUTHORIZED = 13; + +/** + * The failure for a rewrite that skips validation when the database user lacks the + * `bypassDocumentValidation` action, as a `readWrite` user on a hosted cluster does. + */ +function bypassValidationRefusal( + op: MongoDmlOperation, + plan: MongoQueryPlan, + error: unknown, +): MongoMigrationRunnerResult | undefined { + if (!(plan.command instanceof RawUpdateManyCommand) || !plan.command.bypassDocumentValidation) { + return undefined; + } + if (typeof error !== 'object' || error === null) return undefined; + if (Reflect.get(error, 'code') !== MONGO_UNAUTHORIZED) return undefined; + const message: unknown = Reflect.get(error, 'message'); + return runnerFailure( + 'MIGRATION.EXECUTION_FAILED', + `Operation ${op.id} failed: the database user may not skip document validation`, + { + why: `Renaming or removing a field rewrites documents with bypassDocumentValidation, because the old validator requires the old field. MongoDB refused it: ${String(message)}. Grant the user the bypassDocumentValidation action on the database, for example with the dbAdmin or restore role, then run the command again; operations that already ran are skipped.`, + meta: { + operationId: op.id, + mongoCode: MONGO_UNAUTHORIZED, + codeName: Reflect.get(error, 'codeName'), + }, + }, + ); +} + export class MongoMigrationRunner { constructor(private readonly deps: MongoRunnerDependencies) {} @@ -335,8 +367,14 @@ export class MongoMigrationRunner { for (const plan of op.run) { const wireCommand = await adapter.lower(plan, {}); - for await (const _ of driver.execute(wireCommand)) { - /* consume */ + try { + for await (const _ of driver.execute(wireCommand)) { + /* consume */ + } + } catch (error) { + const refusal = bypassValidationRefusal(op, plan, error); + if (refusal === undefined) throw error; + return { executed: false, failure: refusal }; } } diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts index 0b08446305a4..8416039507f3 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts @@ -39,8 +39,13 @@ import type { AnyMongoDmlWireCommand, AnyMongoWireCommand, } from '@internal/mongo-wire'; +import { MongoServerError } from 'mongodb'; import { describe, expect, it } from 'vitest'; -import { createCollection, dataTransform } from '../src/core/migrations/migration-factories'; +import { + createCollection, + dataTransform, + unsetField, +} from '../src/core/migrations/migration-factories'; import { serializeMongoOps } from '../src/core/migrations/mongo-ops-serializer'; import { MongoMigrationRunner } from '../src/core/migrations/mongo-runner'; @@ -68,9 +73,14 @@ class EventLog { class StubMongoDriver implements MongoDriver { readonly executeCalls: WireCommand[] = []; private readonly responses: Row[][] = []; + private failure: unknown; constructor(private readonly log?: EventLog) {} + failUpdateManyWith(error: unknown): void { + this.failure = error; + } + queueResponse(rows: Row[]): void { this.responses.push(rows); } @@ -78,8 +88,10 @@ class StubMongoDriver implements MongoDriver { execute(wireCommand: WireCommand): AsyncIterable { this.executeCalls.push(wireCommand); this.log?.record(`dml:${wireCommand.kind}:${wireCommand.collection}`); - const rows = this.responses.shift() ?? []; + const failure = wireCommand.kind === 'updateMany' ? this.failure : undefined; + const rows = failure === undefined ? (this.responses.shift() ?? []) : []; return (async function* () { + if (failure !== undefined) throw failure; for (const row of rows) yield row as R; })(); } @@ -779,3 +791,39 @@ describe('MongoMigrationRunner - per-edge ledger', () => { expect((ledgerEntries[0]!.operations as unknown[]).length).toBe(1); }); }); + +describe('MongoMigrationRunner, a document rewrite without the privilege to skip validation', () => { + const unauthorized = () => + new MongoServerError({ + ok: 0, + code: 13, + codeName: 'Unauthorized', + errmsg: + 'not authorized on app to execute command { update: "users", updates: [ { q: { nickname: { $exists: true } }, u: { $unset: { nickname: "" } }, multi: true } ], ordered: true, bypassDocumentValidation: true, $db: "app" }', + }); + + it('names the bypassDocumentValidation action and the roles that grant it', async () => { + const harness = makeHarness(); + harness.driver.queueResponse([{ _id: 1 }]); + harness.driver.failUpdateManyWith(unauthorized()); + + const result = await execute(harness, [unsetField('users', 'nickname')]); + + expect(result.assertNotOk()).toEqual({ + code: 'MIGRATION.EXECUTION_FAILED', + summary: + 'Operation field.users.nickname.unset failed: the database user may not skip document validation', + why: 'Renaming or removing a field rewrites documents with bypassDocumentValidation, because the old validator requires the old field. MongoDB refused it: not authorized on app to execute command { update: "users", updates: [ { q: { nickname: { $exists: true } }, u: { $unset: { nickname: "" } }, multi: true } ], ordered: true, bypassDocumentValidation: true, $db: "app" }. Grant the user the bypassDocumentValidation action on the database, for example with the dbAdmin or restore role, then run the command again; operations that already ran are skipped.', + meta: { operationId: 'field.users.nickname.unset', mongoCode: 13, codeName: 'Unauthorized' }, + }); + }); + + it('lets any other failure of a data transform through unchanged', async () => { + const harness = makeHarness(); + harness.driver.failUpdateManyWith(unauthorized()); + + await expect( + execute(harness, [dataTransform('backfill-status', { run: () => makeRunPlan() })]), + ).rejects.toBeInstanceOf(MongoServerError); + }); +}); From 0af4ca2f727cc26e90438365450a1e821600f8a5 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:47:43 +0200 Subject: [PATCH 27/57] TML-3478: the app fragment covers the preview's getCollection form, time-series db update, and the new refusals Signed-off-by: willbot Signed-off-by: Will Madden --- .../app/instructions.md | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md b/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md index 7ea062ffca72..54c2b51d5286 100644 --- a/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md +++ b/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md @@ -6,6 +6,16 @@ changes: - id: mongo-removed-field-asks summary: | On MongoDB, removing a field from a model is now data loss. `migration plan` and `db update` ask about it, and fail with `CLI.CONSENT_REQUIRED` where nobody can answer, until `--delete .` or a `--rename` answers it. `--delete` removes the field from every document that holds it. + - id: mongo-preview-get-collection + summary: | + A MongoDB plan's preview writes each collection as `db.getCollection("users")`, for example `db.getCollection("users").createIndex(...)` where it wrote `db.users.createIndex(...)`. `ops.json` does not change. + detection: + glob: "**/*.{ts,mts,cts,js,mjs,cjs,snap}" + matches: + - 'db\.[A-Za-z_][\w]*\.(?:createIndex|dropIndex|drop)\(' + - id: mongo-time-series-update + summary: | + `db update` on a database with a MongoDB time-series collection no longer fails with "Cannot change immutable collection option 'timeseries'" when nothing changed. - id: storage-question-wording summary: | A data-loss question about data named by its storage name, because the origin contract is unknown, ends with `--delete loses that data` instead of `--delete loses its rows`. @@ -28,7 +38,7 @@ prisma db update --rename Profile:User --rename User.name:User.fullName - A model rename whose collection name changes renames the collection (`renameCollection`), keeping its documents, indexes and validator. - A field rename renames the field in every document that holds it (`renameField`), skipping validation. A variant's field is renamed only in that variant's documents. -- The planner refuses, with the reason and what to do instead: `_id`, a field whose stored name contains `.` or starts with `$`, a time-series collection, a target collection that already exists or that MongoDB rejects, and a collection whose control policy is not `managed`. +- The planner refuses, with the reason and what to do instead: a variant rename whose discriminator value changes, `_id`, a field whose stored name contains `.` or starts with `$`, a time-series collection, a target collection that already exists or that MongoDB rejects, and a collection whose control policy is not `managed`. A field removed from a model whose collection is not `managed` is not asked about and not removed. If you renamed a collection or a field by hand in `mongosh` because `--rename` was refused, nothing changes for migrations you already applied. From now on, give the `--rename` statement instead. @@ -40,6 +50,14 @@ Removing a field from a MongoDB model used to update the collection's validator In a script or CI job that runs `prisma db update --no-interactive` or `prisma migration plan` after a field was removed, add `--delete .` for each removed field, or `--rename` if the field was renamed. When `db update` has no snapshot of the contract the database is at, the field is named by its collection: `--delete profiles.nickname`. +## `mongo-preview-get-collection` + +A test or a script that compares the preview of a MongoDB plan (`migration plan`, `db update --dry-run`, `migration show`) expects `db.getCollection("")` in place of `db.`, so a collection name such as `my-coll` is valid `mongosh`. Plans and `ops.json` are unchanged. + +## `mongo-time-series-update` + +A repeated `db update` on a database whose contract declares a time-series collection used to fail with "Cannot change immutable collection option 'timeseries'", because MongoDB adds defaults such as `bucketMaxSpanSeconds` that the contract leaves out. The planner now compares only the options the contract declares, so the command succeeds and plans nothing. A workaround that skipped `db update` for such a database is no longer needed. + ## `storage-question-wording` A test that asserts the text of a data-loss question about data named by its storage name expects `--delete loses that data` where it expected `--delete loses its rows`. From 2bcb0cd8da88029957c0a913f9e99fbed54a495d Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:55:50 +0200 Subject: [PATCH 28/57] TML-3478: ADR 188 uses the MongoDmlOperation and MongoDmlCheck names Signed-off-by: willbot Signed-off-by: Will Madden --- .../ADR 188 - MongoDB migration operation model.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md index 6fe746dfeccd..4e72c5a2d6eb 100644 --- a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md +++ b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md @@ -126,21 +126,21 @@ This gives checks the same expressive power as MongoDB query filters, without in The envelope described above covers DDL operations — creating indexes, modifying collections. Data transform operations (backfills, renames, seed inserts) use the same three-phase envelope but carry different payloads. ```ts -interface MongoDataTransformOperation extends MigrationPlanOperation { +interface MongoDmlOperation extends MigrationPlanOperation { readonly operationClass: MigrationOperationClass; readonly name: string; - readonly precheck: readonly MongoDataTransformCheck[]; + readonly precheck: readonly MongoDmlCheck[]; readonly run: readonly MongoQueryPlan[]; - readonly postcheck: readonly MongoDataTransformCheck[]; + readonly postcheck: readonly MongoDmlCheck[]; } ``` -A transform built with `dataTransform`, whether a user writes it or the planner scaffolds it with a placeholder, has `operationClass: 'data'`. The `renameField` and `unsetField` rewrites carry the class of their effect, whoever writes them, as every other operation does: `renameField` is `widening` and `unsetField` is `destructive`. So `db update`, which does not allow `data`, can run a rename. The runner and the serializer tell a transform from a DDL operation by its shape (`run` against `execute`), not by its class. The `run` array contains `MongoQueryPlan` objects — aggregation pipelines or raw commands (`rawUpdateMany`, `rawInsertOne`, etc.) — rather than DDL commands. The `precheck` and `postcheck` arrays contain `MongoDataTransformCheck` objects instead of `MongoMigrationCheck`. +A transform built with `dataTransform`, whether a user writes it or the planner scaffolds it with a placeholder, has `operationClass: 'data'`. The `renameField` and `unsetField` rewrites carry the class of their effect, whoever writes them, as every other operation does: `renameField` is `widening` and `unsetField` is `destructive`. So `db update`, which does not allow `data`, can run a rename. The runner and the serializer tell a transform from a DDL operation by its shape (`run` against `execute`), not by its class. The `run` array contains `MongoQueryPlan` objects — aggregation pipelines or raw commands (`rawUpdateMany`, `rawInsertOne`, etc.) — rather than DDL commands. The `precheck` and `postcheck` arrays contain `MongoDmlCheck` objects instead of `MongoMigrationCheck`. -`MongoDataTransformCheck` has the same four-field shape as `MongoMigrationCheck`: +`MongoDmlCheck` has the same four-field shape as `MongoMigrationCheck`: ```ts -interface MongoDataTransformCheck { +interface MongoDmlCheck { readonly description: string; readonly source: MongoQueryPlan; readonly filter: MongoFilterExpr; @@ -175,7 +175,7 @@ Data transform `run` commands execute via `MongoAdapter.lower()` → `MongoDrive #### Relationship to the generic envelope -Both DDL and data transform operations are specializations of the generic three-phase migration operation envelope described in [ADR 191](ADR%20191%20-%20Generic%20three-phase%20migration%20operation%20envelope.md). The DDL variant fills the phases with inspection commands, DDL commands, and `MongoMigrationCheck`; the data transform variant fills them with query plans and `MongoDataTransformCheck`. The runner processes both through the same three-phase loop. +Both DDL and data transform operations are specializations of the generic three-phase migration operation envelope described in [ADR 191](ADR%20191%20-%20Generic%20three-phase%20migration%20operation%20envelope.md). The DDL variant fills the phases with inspection commands, DDL commands, and `MongoMigrationCheck`; the data transform variant fills them with query plans and `MongoDmlCheck`. The runner processes both through the same three-phase loop. ## Serialization From f69bccddeddda104799d98f70184acd1dfc173d9 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:58:00 +0200 Subject: [PATCH 29/57] TML-3478: ADR 188 calls the run-shaped operation a DML operation Signed-off-by: willbot Signed-off-by: Will Madden --- .../ADR 188 - MongoDB migration operation model.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md index 4e72c5a2d6eb..da46d13ffd0a 100644 --- a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md +++ b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md @@ -1,6 +1,6 @@ # ADR 188 — MongoDB migration operation model: data-driven commands and checks -> **Update — 2026-10:** document rewrites the planner writes for `--rename` and `--delete` statements are classed by their effect, not as `data`. See § Data transform operations. +> **Update — 2026-10:** document rewrites the planner writes for `--rename` and `--delete` statements are classed by their effect, not as `data`. See § DML operations and unified check shape. ## Grounding example @@ -121,9 +121,9 @@ interface MongoMigrationCheck { This gives checks the same expressive power as MongoDB query filters, without inventing a separate vocabulary. -### Data transform operations and unified check shape +### DML operations and unified check shape -The envelope described above covers DDL operations — creating indexes, modifying collections. Data transform operations (backfills, renames, seed inserts) use the same three-phase envelope but carry different payloads. +The envelope described above covers DDL operations — creating indexes, modifying collections. DML operations, which read and write documents (backfills, field renames and removals, seed inserts), use the same three-phase envelope but carry different payloads. A data transform is a DML operation built with `dataTransform`. ```ts interface MongoDmlOperation extends MigrationPlanOperation { @@ -150,7 +150,7 @@ interface MongoDmlCheck { The only difference is the `source` type: `MongoQueryPlan` (an aggregation or raw command) instead of `AnyMongoInspectionCommand`. Both use `MongoFilterExpr` for `filter` and `'exists' | 'notExists'` for `expect`. Both are evaluated by the same `FilterEvaluator` path in the runner — run the source query, apply the filter client-side, check against the expectation. -The two types are kept concrete rather than collapsed into a generic `MigrationCheck` because the source types need different executors — DDL checks dispatch through `MongoInspectionCommandVisitor`, data transform checks dispatch through `MongoAdapter.lower()` → `MongoDriver.execute()`. Two concrete types with a shared shape let the runner dispatch internally based on which operation type it's processing. +The two types are kept concrete rather than collapsed into a generic `MigrationCheck` because the source types need different executors — DDL checks dispatch through `MongoInspectionCommandVisitor`, DML checks dispatch through `MongoAdapter.lower()` → `MongoDriver.execute()`. Two concrete types with a shared shape let the runner dispatch internally based on which operation type it's processing. #### One factory derives both precheck and postcheck @@ -171,11 +171,11 @@ The factory resolves this into a precheck with `expect: 'exists'` (violations fo #### DML execution -Data transform `run` commands execute via `MongoAdapter.lower()` → `MongoDriver.execute()` — the same adapter+driver transport used for runtime queries. No bespoke DML executor is needed; the adapter lowers each `MongoQueryPlan` to a wire command, and the driver executes it against the database. +DML `run` commands execute via `MongoAdapter.lower()` → `MongoDriver.execute()` — the same adapter+driver transport used for runtime queries. No bespoke DML executor is needed; the adapter lowers each `MongoQueryPlan` to a wire command, and the driver executes it against the database. #### Relationship to the generic envelope -Both DDL and data transform operations are specializations of the generic three-phase migration operation envelope described in [ADR 191](ADR%20191%20-%20Generic%20three-phase%20migration%20operation%20envelope.md). The DDL variant fills the phases with inspection commands, DDL commands, and `MongoMigrationCheck`; the data transform variant fills them with query plans and `MongoDmlCheck`. The runner processes both through the same three-phase loop. +Both DDL and DML operations are specializations of the generic three-phase migration operation envelope described in [ADR 191](ADR%20191%20-%20Generic%20three-phase%20migration%20operation%20envelope.md). The DDL variant fills the phases with inspection commands, DDL commands, and `MongoMigrationCheck`; the DML variant fills them with query plans and `MongoDmlCheck`. The runner processes both through the same three-phase loop. ## Serialization From 6336910fe611a358c31d9b1aec569a44b62ab000 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 12:58:06 +0200 Subject: [PATCH 30/57] TML-3478: ADR 188 says DML operation where it tells shapes apart Signed-off-by: willbot Signed-off-by: Will Madden --- .../adrs/ADR 188 - MongoDB migration operation model.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md index da46d13ffd0a..831fbf40bf97 100644 --- a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md +++ b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md @@ -135,7 +135,7 @@ interface MongoDmlOperation extends MigrationPlanOperation { } ``` -A transform built with `dataTransform`, whether a user writes it or the planner scaffolds it with a placeholder, has `operationClass: 'data'`. The `renameField` and `unsetField` rewrites carry the class of their effect, whoever writes them, as every other operation does: `renameField` is `widening` and `unsetField` is `destructive`. So `db update`, which does not allow `data`, can run a rename. The runner and the serializer tell a transform from a DDL operation by its shape (`run` against `execute`), not by its class. The `run` array contains `MongoQueryPlan` objects — aggregation pipelines or raw commands (`rawUpdateMany`, `rawInsertOne`, etc.) — rather than DDL commands. The `precheck` and `postcheck` arrays contain `MongoDmlCheck` objects instead of `MongoMigrationCheck`. +A transform built with `dataTransform`, whether a user writes it or the planner scaffolds it with a placeholder, has `operationClass: 'data'`. The `renameField` and `unsetField` rewrites carry the class of their effect, whoever writes them, as every other operation does: `renameField` is `widening` and `unsetField` is `destructive`. So `db update`, which does not allow `data`, can run a rename. The runner and the serializer tell a DML operation from a DDL operation by its shape (`run` against `execute`), not by its class. The `run` array contains `MongoQueryPlan` objects — aggregation pipelines or raw commands (`rawUpdateMany`, `rawInsertOne`, etc.) — rather than DDL commands. The `precheck` and `postcheck` arrays contain `MongoDmlCheck` objects instead of `MongoMigrationCheck`. `MongoDmlCheck` has the same four-field shape as `MongoMigrationCheck`: From b7d84d072fadb7c14cfff24dac6b84d0e7e0f336 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 13:00:27 +0200 Subject: [PATCH 31/57] TML-3478: the privilege failure names MongoDB's error code, not its command echo The message says MongoDB refused the rewrite with error 13 (Unauthorized) instead of quoting the server's message, which echoes the whole command with session and cluster-time data. mongoCode and codeName stay in meta. Signed-off-by: willbot Signed-off-by: Will Madden --- .../1-mongo-target/src/core/migrations/mongo-runner.ts | 6 +++--- .../1-mongo-target/test/mongo-runner.test.ts | 9 ++++++--- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts index 4532ba4d4312..dde6ec3ead48 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts @@ -97,16 +97,16 @@ function bypassValidationRefusal( } if (typeof error !== 'object' || error === null) return undefined; if (Reflect.get(error, 'code') !== MONGO_UNAUTHORIZED) return undefined; - const message: unknown = Reflect.get(error, 'message'); + const codeName: unknown = Reflect.get(error, 'codeName'); return runnerFailure( 'MIGRATION.EXECUTION_FAILED', `Operation ${op.id} failed: the database user may not skip document validation`, { - why: `Renaming or removing a field rewrites documents with bypassDocumentValidation, because the old validator requires the old field. MongoDB refused it: ${String(message)}. Grant the user the bypassDocumentValidation action on the database, for example with the dbAdmin or restore role, then run the command again; operations that already ran are skipped.`, + why: `Renaming or removing a field rewrites documents with bypassDocumentValidation, because the old validator requires the old field. MongoDB refused it with error ${MONGO_UNAUTHORIZED} (${String(codeName)}). Grant the user the bypassDocumentValidation action on the database, for example with the dbAdmin or restore role, then run the command again; operations that already ran are skipped.`, meta: { operationId: op.id, mongoCode: MONGO_UNAUTHORIZED, - codeName: Reflect.get(error, 'codeName'), + codeName, }, }, ); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts index 8416039507f3..23b4bc1e1797 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts @@ -792,14 +792,16 @@ describe('MongoMigrationRunner - per-edge ledger', () => { }); }); +const COMMAND_ECHO = + 'not authorized on app to execute command { update: "users", ordered: true, bypassDocumentValidation: true, lsid: { id: UUID("0b2c8a4e-6f0d-4a7e-9d6f-2f1d5c3e8a91") }, txnNumber: 1, $clusterTime: { clusterTime: Timestamp(1791552000, 3), signature: { hash: BinData(0, 7F3C1D9E2A4B6C8D0E1F2A3B4C5D6E7F8A9B0C1D), keyId: 7421987654321098753 } }, $db: "app" }'; + describe('MongoMigrationRunner, a document rewrite without the privilege to skip validation', () => { const unauthorized = () => new MongoServerError({ ok: 0, code: 13, codeName: 'Unauthorized', - errmsg: - 'not authorized on app to execute command { update: "users", updates: [ { q: { nickname: { $exists: true } }, u: { $unset: { nickname: "" } }, multi: true } ], ordered: true, bypassDocumentValidation: true, $db: "app" }', + errmsg: COMMAND_ECHO, }); it('names the bypassDocumentValidation action and the roles that grant it', async () => { @@ -809,11 +811,12 @@ describe('MongoMigrationRunner, a document rewrite without the privilege to skip const result = await execute(harness, [unsetField('users', 'nickname')]); + expect(JSON.stringify(result.assertNotOk())).not.toContain('clusterTime'); expect(result.assertNotOk()).toEqual({ code: 'MIGRATION.EXECUTION_FAILED', summary: 'Operation field.users.nickname.unset failed: the database user may not skip document validation', - why: 'Renaming or removing a field rewrites documents with bypassDocumentValidation, because the old validator requires the old field. MongoDB refused it: not authorized on app to execute command { update: "users", updates: [ { q: { nickname: { $exists: true } }, u: { $unset: { nickname: "" } }, multi: true } ], ordered: true, bypassDocumentValidation: true, $db: "app" }. Grant the user the bypassDocumentValidation action on the database, for example with the dbAdmin or restore role, then run the command again; operations that already ran are skipped.', + why: 'Renaming or removing a field rewrites documents with bypassDocumentValidation, because the old validator requires the old field. MongoDB refused it with error 13 (Unauthorized). Grant the user the bypassDocumentValidation action on the database, for example with the dbAdmin or restore role, then run the command again; operations that already ran are skipped.', meta: { operationId: 'field.users.nickname.unset', mongoCode: 13, codeName: 'Unauthorized' }, }); }); From f55b1bb9e3486967e67239afed2d6ddd7a9535a9 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 13:01:38 +0200 Subject: [PATCH 32/57] TML-3478: manual QA script for MongoDB renames and deletes Signed-off-by: willbot Signed-off-by: Will Madden --- .../slices/mongo-renames/manual-qa.md | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 projects/migration-statements/slices/mongo-renames/manual-qa.md diff --git a/projects/migration-statements/slices/mongo-renames/manual-qa.md b/projects/migration-statements/slices/mongo-renames/manual-qa.md new file mode 100644 index 000000000000..f357bb1e9094 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/manual-qa.md @@ -0,0 +1,47 @@ +# Manual QA — slice 4a, renames and deletes on MongoDB (TML-3478) + +> **Be the user.** Run the built CLI against a real MongoDB (mongodb-memory-server replica set) in scratch projects. Read every message as someone who has never seen this feature. Judge what the journeys cannot: whether a refusal tells you what to type next, whether the questions make sense, and whether anything loses or breaks documents without you saying so. +> +> **Out of scope.** Do not re-run `pnpm test`, the journeys or CI lints. Decide from what the commands print and what the database holds, not from the source. +> +> **Spec:** `projects/migration-statements/slices/mongo-renames/spec.md` + +## What this script is testing + +**The problem.** MongoDB refused every `--rename` and printed steps to run by hand in `mongosh`. Removing a field asked nothing, and every document that still held the field then failed each update against Prisma's closed validator. + +**What changed.** +- `--rename Profile:User` renames the collection; `--rename User.name:User.fullName` renames the field in every document (only the variant's documents for a variant's field). +- Removing a field is data loss: it is asked about, and `--delete User.nickname` removes it from every document. +- Both `migration plan` and `db update` do this. +- Some renames are refused with a reason and steps: `_id`, dotted names, time-series collections, a discriminator value change, a collection name MongoDB rejects. + +**Audiences.** End users in a terminal and in scripts; agents using the `prisma-8` skill; extension authors reading the upgrade fragment. + +## Scenarios + +| # | Scenario | What it proves | +| - | -------- | -------------- | +| 1 | PSL project with documents. Rename model `Profile` to `User` (collection `profiles` to `users`), rename `name` to `fullName` (unique index), remove `nickname`. `migration plan` with no flags, non-interactive | The refusal names `nickname` and the flags; nothing is written | +| 2 | Same, in a real terminal, answering by typing `rename …` and `delete` | The questions read clearly; a wrong answer is explained and re-asked; the written `migration.ts` reads like hand-written code (`renameCollection`, `renameField`, `unsetField`) | +| 3 | Apply scenario 2's plan with `db migrate`; then update a document through the ORM | Documents under the new names, `nickname` gone, unique index on `fullName`, updates succeed, `db verify --schema-only` clean, a further plan empty | +| 4 | The same change through `db update`, then run it again | No prompt with the flags; the second run fails and says why | +| 5 | Polymorphic collection: base `Task`, variants `Bug` and `Feature` both store `score`. Rename `Bug.score` to `Bug.severity` | `Feature` documents keep `score`; every document accepts an update | +| 6 | Move a base field `Task.note` to `Bug`. Answer `delete`; in a second project answer `rename Task.note:Task.remark` | `Feature` documents lose `note` (or get `remark`); `Bug` documents keep `note` | +| 7 | `db update --db ` with no snapshot, removing a field | The question names `profiles.nickname` and says why; the printed `--delete` works | +| 8 | Each refusal: rename `_id`; a field mapped to `a.b`; a time-series collection; `Bug` renamed with its discriminator value changed; a collection mapped to `system.x` | Each names the reason; each piece of advice, followed as printed, ends in a correct state with no lost documents | +| 9 | A field removed from an `external` collection (TypeScript contract with `defaultControlPolicy: 'external'`) | Nothing asked, nothing removed | +| 10 | Interrupt a field rename half way (kill the process after some documents moved, or move one by hand), then run `db migrate` again | It finishes; no document holds both names | +| 11 | Read the app and extension upgrade fragments as their audiences **(judgement)** | A user knows what changed; an extension author knows every rename and removal | +| 12 | Ask the skill how to rename a MongoDB field **(judgement)** | `skills/prisma-8/references/migrations.md` leads to `--rename` in the right form | +| 13 | Exploratory: 20 minutes | Anything the scenarios missed | + +## Pre-flight + +1. In the worktree, `git status --short` shows no source changes. +2. `mise exec -- pnpm build`. +3. Scratch projects under `wip/qa-4a/`, one per scenario, with a mongodb-memory-server replica set (`examples/mongo-demo` shows the config). Stop every server by PID at the end. + +## Report + +Write `projects/migration-statements/manual-qa-reports/-qa-slice-4a.md`: a summary table with a verdict per scenario (✅ / ⚠️ / 🛑 Blocker), then each finding numbered `F` with the exact command and output. A blocker is anything that loses or breaks documents without the user saying so, or advice that does so when followed. From 596575119b07ba4a0a265e73be0d56ae2be3fb90 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 13:17:08 +0200 Subject: [PATCH 33/57] TML-3478: manual QA report for slice 4a Signed-off-by: willbot Signed-off-by: Will Madden --- .../2026-10-09-qa-slice-4a.md | 136 ++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 projects/migration-statements/manual-qa-reports/2026-10-09-qa-slice-4a.md diff --git a/projects/migration-statements/manual-qa-reports/2026-10-09-qa-slice-4a.md b/projects/migration-statements/manual-qa-reports/2026-10-09-qa-slice-4a.md new file mode 100644 index 000000000000..ee574b58f786 --- /dev/null +++ b/projects/migration-statements/manual-qa-reports/2026-10-09-qa-slice-4a.md @@ -0,0 +1,136 @@ +# Manual QA — slice 4a, renames and deletes on MongoDB (TML-3478) + +Branch `tml-3478-mongo-renames` at `f55b1bb9e3`. Built CLI (`packages/1-framework/3-tooling/cli/dist/bin.mjs`), one mongodb-memory-server replica set (MongoDB 8.2), one database per scratch project under `wip/qa-4a/`. Terminal scenarios ran in a real pseudo-terminal (140 columns). Each result was checked in the database. + +**No blocker.** No document was lost or broken without a `--delete`. Every printed piece of advice that I followed left the documents intact. + +## Summary + +| # | Scenario | Verdict | Findings | +| - | -------- | ------- | -------- | +| 1 | `migration plan`, no flags, not interactive | ✅ | The refusal names `Profile` first. After `--rename Profile:User` it names `User.name` and `User.nickname` with both flags. Nothing is written. | +| 2 | Same, in a terminal | ✅ | Wrong answers are explained and asked again. `migration.ts` uses `renameCollection`, `dropIndex`, `renameField`, `unsetField`, `createIndex`, `collMod`. | +| 3 | `db migrate`, then ORM update | ✅ | Documents in `users` with `fullName`, `nickname` gone, unique index `fullName_1`, 3 of 3 ORM updates succeed, `db verify --schema-only` clean, next plan empty. | +| 4 | `db update` twice | ⚠️ | F3 | +| 5 | Rename `Bug.score` in a shared collection | ✅ | `Feature` documents keep `score`; 4 of 4 updates succeed. | +| 6 | Move `Task.note` to `Bug` (delete; rename) | ⚠️ | F6. Data result correct both ways. | +| 7 | `db update --db` with no snapshot | ✅ | `--delete profiles.nickname` works. The longer route in the "why" (store a snapshot, then `--rename`) also works, followed as printed. See F8. | +| 8 | Refusals | ⚠️ | F1, F4, F5. `_id`, discriminator change and `system.x` refusals are clear, and their advice works. | +| 9 | Field removed from an `external` collection | ✅ | Nothing asked, nothing removed. A `--rename` there is refused with a clear reason. | +| 10 | Interrupted field rename | ✅ | `db migrate` finishes; no document holds both names. See F2 for a related case. | +| 11 | Upgrade fragments | ⚠️ | Clear for both audiences. The app fragment says a time-series collection is refused; for fields it is not (F1). | +| 12 | Skill: rename a MongoDB field | ✅ | `references/migrations.md` item 8 gives `--rename` in the right form and the `User._id` rule. Same time-series claim as F1. | +| 13 | Exploratory | ⚠️ | F2, F7, F9 | + +## Findings + +### F1 ⚠️ A field rename or removal on a time-series collection is not refused; `db update` fails with an unexpected error + +TypeScript contract, model `Reading` on time-series collection `readings` (`timeField: 'ts'`, `metaField: 'meta'`), field `value` renamed to `reading`. + +```text +$ prisma db update --no-interactive --rename Reading.value:Reading.reading +✘ [CLI.UNEXPECTED] Unexpected error + why: Unexpected error during db update: Cannot perform an update or delete on a time-series collection when querying on a field that is not the metaField 'meta' +→ Check the error message and try again + +$ prisma db update --no-interactive --delete Reading.value +✘ [CLI.UNEXPECTED] Unexpected error + why: Unexpected error during db update: Cannot perform an update or delete on a time-series collection when querying on a field that is not the metaField 'meta' +``` + +Without a flag the command asks about `Reading.value`. So no answer works, and the user cannot remove a field from a time-series model at all. Before this slice that removal went through silently. No documents changed. + +Expected: a planning refusal with a reason, as for the collection rename (`collection "readings" is a time-series collection`). The app fragment and `skills/prisma-8/references/migrations.md` both say statements refuse a time-series collection. This may overlap the pending decision about TypeScript contracts without a validator, but that decision is about `migrate`; this failure is in `db update`. + +### F2 ⚠️ A rename precheck failure does not say what is wrong, and the earlier index drop stays applied + +A document holds both `name` and `fullName` (written outside Prisma). Unique index on `name`. + +```text +$ prisma db update --no-interactive --rename Profile.name:Profile.fullName +✘ [MIGRATION.RUNNER_FAILED] Operation field.profiles.name.rename(fullName) failed during precheck + why: Migration runner failed +→ Inspect the reported conflict, reconcile schema drift if needed, then re-run `prisma db update` +``` + +The data is safe: no value was overwritten. But the message does not say that a document already holds `fullName`, or which one. The JSON adds only `runnerErrorCode: MIGRATION.PRECHECK_FAILED`. The unique index `name_1` was dropped before the precheck failed and is still gone. + +Expected: the reason ("1 document in `profiles` already holds `fullName`") and what to do. + +### F3 ⚠️ A repeated `db update` reports stale statements one at a time + +After a successful `db update --rename Profile:User --rename User.name:User.fullName --delete User.nickname`, the same command again: + +```text +✘ [MIGRATION.STATEMENT_UNRESOLVED] Cannot resolve statement "--rename Profile:User" + why: The origin contract already has "User" and has no "Profile", so this rename has already happened. ... +→ Leave out --rename Profile:User. +``` + +Without it, the next run names `--rename User.name:User.fullName`; without that, `CLI.CONSENT_UNUSED` names `--delete User.nickname`. Three runs to learn the three flags to drop. Each message is clear. Expected: all stale statements in one error. + +### F4 ⚠️ Dotted field name: the rename refusal's advice is incomplete + +PSL accepts `nick String? @map("a.b")`. The ORM cannot write it (`Document failed validation`), so data only gets there from outside Prisma. With a document holding `"a.b": "bobby"` and the map changed to `"c"`: + +```text +$ prisma db update --no-interactive --rename Profile.a.b:Profile.c +✘ [MIGRATION.PLANNING_FAILED] Migration planning failed + why: Cannot rename field "Profile.a.b" to "Profile.c": the stored field name "a.b" cannot be rewritten +→ MongoDB reads a name containing "." as a path into embedded documents, so a statement cannot move its values. Rename it in the database yourself, or leave out this statement. +``` + +If you leave the statement out, the plan updates the validator (`removed: a.b`) without asking. The values stay in the documents, so those documents then fail every update. "Rename it in the database yourself" gives no steps, and `$rename` cannot target a literal dotted name. Low impact because Prisma cannot write such a field. + +### F5 ⚠️ Following the time-series rename advice drops MongoDB's automatic index + +```text +→ MongoDB cannot rename a time-series collection. Leave out this statement. To keep the documents, first create collection "samples" yourself with the same time-series options and copy them into it; the plan then drops "readings" once --delete names the model. +``` + +Followed as printed: created `samples` with the same options, copied 2 documents, ran `db update --delete Reading`. Documents kept. The plan also ran `Drop index on samples (meta:1, ts:1)`. MongoDB creates that index automatically for a time-series collection with a `metaField`. Not data loss, but slower queries after following the advice. + +### F6 ⚠️ The question for a base field moved to a variant shows a raw MongoDB filter + +```text +◆ Remove field note from collection tasks where {"kind":{"$nin":["bug"]}} would lose the values of field "Task.note". (rename Task.note:Task. or delete) +``` + +The same text is in the plan list and in scenario 5 (`where {"kind":"bug"}`). The result is correct: `Feature` documents lose `note` (or get `remark`), `Bug` documents keep it. Expected plain words, for example "in documents that are not `Bug`". + +### F7 ⚠️ Changing only a model's `@@map` offers only `--delete` + +`model Profile { … @@map("profiles") }` changed to `@@map("people")`: + +```text +✘ [CLI.CONSENT_REQUIRED] "Profile" needs a statement, and the session is not interactive. + why: Drop collection profiles would lose the data of model "Profile". +→ Run the command again with --delete Profile +``` + +Followed as printed, this drops the documents (with the user's consent). The way to keep them is in the skill (item 8: hand-edit `renameCollection(...)` into `migration.ts`). I did that: plan with `--delete Profile`, replace the operations with `renameCollection('profiles', 'people')`, self-emit, `db migrate`. Documents kept. Expected: the CLI says how to keep the documents, as the skill does. + +### F8 ⚠️ The no-snapshot "why" packs three numbered steps into one line + +```text + why: Remove field nickname from collection profiles would lose the data in "profiles.nickname", named by its storage name because the origin contract is unknown; --delete loses that data. If it was renamed, keep its data instead: store the snapshot of the contract the database is at, then run this command again and answer with --rename. 1. Put the contract source back ... 2. Run `prisma db update --advance-ref --dry-run` ... 3. Put the new contract source back ... +``` + +The steps are correct (I followed them and `--rename Profile.nickname:Profile.alias` moved the values). They are hard to read as one paragraph of about 900 characters. + +### F9 ⚠️ A plain `system.x` collection gets an unexpected error + +The rename onto `system.x` is refused clearly. But `db update --delete Profile` for the same change fails with: + +```text +✘ [CLI.UNEXPECTED] Unexpected error + why: Unexpected error during db update: Invalid system namespace: s8h.system.x +``` + +No documents lost: the create fails before the drop. Probably outside this slice. + +## Notes outside the slice + +- `db migrate` did not advance the `db` ref (`advancedRef: null`), so the next `migration plan` failed with `MIGRATION.PLAN_ORIGIN_UNKNOWN` until `migration ref set db `. `db update` advances it. +- Without a `package.json` that lists `@prisma/orm-mongo`, `migration.ts` imports `@internal/target-mongo/migration` and cannot self-emit. With one it imports `@prisma/orm-mongo/target/migration`. This came from my scratch setup, not from the slice. From 5eab8fb2560f9ffab51eb9f2e66503a0a77cb76b Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 13:19:47 +0200 Subject: [PATCH 34/57] TML-3478: the no-snapshot recovery steps in a data-loss question are one per line The question ends with the recovery on its own lines, as MIGRATION.STATEMENT_ORIGIN_UNKNOWN prints the same steps, on SQL and MongoDB alike. Signed-off-by: willbot Signed-off-by: Will Madden --- .../cli/src/control-api/statements/plan-questions.ts | 4 ++-- .../3-tooling/cli/test/control-api/db-update.test.ts | 2 +- .../control-api/statements/ask-plan-questions.test.ts | 8 +++++++- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts b/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts index 94db5cbbee63..6ec9c1955a67 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts @@ -158,7 +158,7 @@ function lossText(subject: MigrationSubject, text: string, originKnown: boolean) export const ORIGIN_SNAPSHOT_RECOVERY = [ 'If it was renamed, keep its data instead: store the snapshot of the contract the database is at, then run this command again and answer with --rename.', ...STORE_ORIGIN_SNAPSHOT_STEPS.map((step) => step.replaceAll('{bin}', 'prisma')), -].join(' '); +].join('\n'); function sameCoordinate(statement: ResolvedMigrationStatement, subject: MigrationSubject): boolean { if (subject.kind === 'storage') return false; @@ -370,7 +370,7 @@ function withRecoveryOnFirstStorageQuestion( : -1; return asked.map(({ question }, index) => index === first - ? { ...question, question: `${question.question} ${ORIGIN_SNAPSHOT_RECOVERY}` } + ? { ...question, question: `${question.question}\n${ORIGIN_SNAPSHOT_RECOVERY}` } : question, ); } diff --git a/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts index 9dd6a7de515e..c48b7bf7585a 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts @@ -579,7 +579,7 @@ describe('executeDbUpdate', () => { expect(asked).toEqual([ { - question: `Drop column nickname from user would lose the data in "user.nickname", named by its storage name because the origin contract is unknown; --delete loses that data. ${ORIGIN_SNAPSHOT_RECOVERY}`, + question: `Drop column nickname from user would lose the data in "user.nickname", named by its storage name because the origin contract is unknown; --delete loses that data.\n${ORIGIN_SNAPSHOT_RECOVERY}`, subject: 'user.nickname', verbs: ['delete'], }, diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts index 87ec3a153bff..636cba080fdb 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts @@ -109,9 +109,15 @@ describe('questions about storage when the origin contract is unknown', () => { }); expect(asked).toEqual([ - `Drop table Legacy would lose the data in "Legacy", named by its storage name because the origin contract is unknown; --delete loses that data. ${ORIGIN_SNAPSHOT_RECOVERY}`, + `Drop table Legacy would lose the data in "Legacy", named by its storage name because the origin contract is unknown; --delete loses that data.\n${ORIGIN_SNAPSHOT_RECOVERY}`, 'Drop table Profile would lose the data in "Profile", named by its storage name because the origin contract is unknown; --delete loses that data.', ]); + expect(ORIGIN_SNAPSHOT_RECOVERY.split('\n').map((line) => line.slice(0, 2))).toEqual([ + 'If', + '1.', + '2.', + '3.', + ]); expect(ORIGIN_SNAPSHOT_RECOVERY).toContain('--dry-run'); expect(ORIGIN_SNAPSHOT_RECOVERY).toContain('`prisma db update --advance-ref --dry-run`'); expect(ORIGIN_SNAPSHOT_RECOVERY).toContain( From b3aab16e767505530d11e65f1092d103b3280f72 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 13:20:39 +0200 Subject: [PATCH 35/57] TML-3478: a field rewrite's label says which documents it touches in words A variant filter reads as "in documents whose type is \"bug\"" or "whose type is not \"bug\"" in the label, the plan list and the data-loss question. Operation ids keep the raw filter. Signed-off-by: willbot Signed-off-by: Will Madden --- .../core/migrations/migration-factories.ts | 28 ++++++++++++++++++- .../test/field-rewrites.test.ts | 14 +++++++++- .../mongo-planner.statements.fields.test.ts | 12 +++++--- 3 files changed, 48 insertions(+), 6 deletions(-) diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts index 0c9d0165b7ef..487d9c19fb8b 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts @@ -200,8 +200,34 @@ function idSuffix(filter: Document | undefined): string { return filter === undefined ? '' : `.where(${JSON.stringify(filter)})`; } +function valueList(values: readonly unknown[]): string { + return values.map((value) => JSON.stringify(value)).join(', '); +} + +/** A field condition in words: `type is "bug"`, `type is not "bug"`, `type is none of "bug", "task"`. */ +function conditionInWords(field: string, condition: unknown): string | undefined { + const excluded: unknown = + typeof condition === 'object' && condition !== null && !Array.isArray(condition) + ? Reflect.get(condition, '$nin') + : undefined; + if (Array.isArray(excluded)) { + return excluded.length === 1 + ? `${field} is not ${valueList(excluded)}` + : `${field} is none of ${valueList(excluded)}`; + } + return typeof condition === 'object' && condition !== null + ? undefined + : `${field} is ${JSON.stringify(condition)}`; +} + +/** The documents a filter selects, in words where the planner wrote it, for an operation's label. */ function labelSuffix(filter: Document | undefined): string { - return filter === undefined ? '' : ` where ${JSON.stringify(filter)}`; + if (filter === undefined) return ''; + const entries = Object.entries(filter); + const words = entries.map(([field, condition]) => conditionInWords(field, condition)); + return words.every((word) => word !== undefined) && words.length > 0 + ? ` in documents whose ${words.join(' and ')}` + : ` in documents matching ${JSON.stringify(filter)}`; } export function renameField( diff --git a/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts b/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts index 80555333c583..1041d1244e88 100644 --- a/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts @@ -68,7 +68,7 @@ describe('renameField', () => { postcheck: op.postcheck[0]?.source.command, }).toEqual({ id: 'field.posts.title.rename(headline).where({"type":"bug"})', - label: 'Rename field title to headline in collection posts where {"type":"bug"}', + label: 'Rename field title to headline in collection posts in documents whose type is "bug"', precheck: new RawAggregateCommand('posts', [ { $match: { @@ -136,6 +136,18 @@ describe('unsetField', () => { }); }); +describe('field rewrite labels', () => { + it('describe the documents a filter selects in words', () => { + expect([ + unsetField('tasks', 'note', { filter: { type: { $nin: ['bug', 'task'] } } }).label, + unsetField('tasks', 'note', { filter: { type: { $in: ['bug'] } } }).label, + ]).toEqual([ + 'Remove field note from collection tasks in documents whose type is none of "bug", "task"', + 'Remove field note from collection tasks in documents matching {"type":{"$in":["bug"]}}', + ]); + }); +}); + describe('field names a rewrite refuses', () => { const refusals = [ { field: 'a.b', reason: 'contains "."' }, diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts index 83779e9d6c30..33f92d3c73fc 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts @@ -98,7 +98,9 @@ describe('MongoMigrationPlanner, field statements', () => { statements: [renameField('Bug', 'score', 'Bug', 'severity')], }), ).labels, - ).toEqual(['Rename field score to severity in collection tasks where {"type":"bug"}']); + ).toEqual([ + 'Rename field score to severity in collection tasks in documents whose type is "bug"', + ]); }); it('plans nothing for a relation renamed to another relation, which documents do not store', () => { @@ -193,7 +195,7 @@ describe('MongoMigrationPlanner, removed fields', () => { }), ), ).toMatchObject({ - labels: ['Remove field score from collection tasks where {"type":"bug"}'], + labels: ['Remove field score from collection tasks in documents whose type is "bug"'], dataLoss: [ { operationIndex: 0, @@ -213,7 +215,7 @@ describe('MongoMigrationPlanner, removed fields', () => { }), ), ).toMatchObject({ - labels: ['Remove field note from collection tasks where {"type":{"$nin":["bug"]}}'], + labels: ['Remove field note from collection tasks in documents whose type is not "bug"'], dataLoss: [ { operationIndex: 0, @@ -234,7 +236,9 @@ describe('MongoMigrationPlanner, removed fields', () => { }), ), ).toMatchObject({ - labels: ['Rename field note to remark in collection tasks where {"type":{"$nin":["bug"]}}'], + labels: [ + 'Rename field note to remark in collection tasks in documents whose type is not "bug"', + ], applied: [[0]], dataLoss: [], }); From d53972b4216edf652a64664428d91d8f9b257fe6 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 13:22:47 +0200 Subject: [PATCH 36/57] TML-3478: a field rename stopped by documents holding the new name says how to find and fix them The failure names the collection and both fields, prints the mongosh find that lists the documents, says nothing was renamed, and that the operations before it already ran and a re-run after fixing the documents finishes the plan. A target-runner test follows the message to a finished plan. Signed-off-by: willbot Signed-off-by: Will Madden --- docs/reference/error-reference.md | 2 +- .../9-family/src/exports/control.ts | 1 + .../src/core/migrations/mongo-runner.ts | 53 +++++++++++++++--- .../1-mongo-target/test/mongo-runner.test.ts | 24 +++++++++ .../runner-field-rewrites.test.ts | 54 +++++++++++++++++++ 5 files changed, 125 insertions(+), 9 deletions(-) diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index 6206a1e7ac73..8dfbad706331 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -1602,7 +1602,7 @@ A `PostgresMigration` operation (e.g. `createTable`, `dataTransform`) was invoke ### MIGRATION.PRECHECK_FAILED -Before executing a migration operation, one of its precheck steps (a query expected to return true) did not hold: the database is not in the state the operation requires, so the apply stops and rolls back. Payload: `operationId`, `phase`, `stepDescription`. +Before executing a migration operation, one of its precheck steps (a query expected to return true) did not hold: the database is not in the state the operation requires, so the apply stops and rolls back. Payload: `operationId`, `phase`, `stepDescription`. On MongoDB, where operations are not wrapped in a transaction, the operations before it stay applied, and a re-run skips them. A field rename (`renameField`) stops here when documents already hold the new name as well as the old one: the message names the collection, gives the `mongosh` query that finds those documents, and says to fix them and run the command again. Payload on MongoDB: `operationId`, `name`. ### MIGRATION.PROVIDED_INVARIANTS_MISMATCH diff --git a/packages/2-mongo-family/9-family/src/exports/control.ts b/packages/2-mongo-family/9-family/src/exports/control.ts index 8386ea3617a7..9943492436af 100644 --- a/packages/2-mongo-family/9-family/src/exports/control.ts +++ b/packages/2-mongo-family/9-family/src/exports/control.ts @@ -6,6 +6,7 @@ export { } from '../core/control-instance'; export type { MongoControlTargetDescriptor } from '../core/control-target-descriptor'; export type { MongoControlExtensionDescriptor } from '../core/control-types'; +export { type FieldRewrite, fieldRewriteOf } from '../core/field-rewrite'; export { formatMongoOperations, mongoOperationsToPreview, diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts index dde6ec3ead48..17e9837834c0 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts @@ -1,5 +1,6 @@ import type { ContractMarkerRecord } from '@internal/contract/types'; import { errorRunnerFailed } from '@internal/errors/execution'; +import { fieldRewriteOf } from '@internal/family-mongo/control'; import type { MarkerOperations, MongoRunnerDependencies, @@ -28,7 +29,11 @@ import type { MongoMigrationCheck, } from '@internal/mongo-query-ast/control'; import { isMongoDmlOperation } from '@internal/mongo-query-ast/control'; -import { type MongoQueryPlan, RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; +import { + type MongoQueryPlan, + RawAggregateCommand, + RawUpdateManyCommand, +} from '@internal/mongo-query-ast/execution'; import { blindCast } from '@internal/utils/casts'; import { InternalError } from '@internal/utils/internal-error'; import { notOk, ok, type Result } from '@internal/utils/result'; @@ -81,6 +86,36 @@ function runnerFailure( }); } +/** + * The failure when a field rename's before-check finds documents that already hold the new name, + * which `$rename` would overwrite: what to look for and fix, and that the plan resumes. + */ +function renameOverwriteRefusal(op: MongoDmlOperation): MongoMigrationRunnerResult | undefined { + const rewrite = fieldRewriteOf(op); + if (rewrite === undefined) return undefined; + const renames: unknown = Reflect.get(rewrite.command.update, '$rename'); + const to: unknown = + typeof renames === 'object' && renames !== null + ? Reflect.get(renames, rewrite.field) + : undefined; + if (typeof to !== 'string') return undefined; + const [check] = op.precheck; + const source = check?.source.command; + const match: unknown = + source instanceof RawAggregateCommand + ? Reflect.get(source.pipeline[0] ?? {}, '$match') + : undefined; + const find = `db.getCollection(${JSON.stringify(rewrite.collection)}).find(${JSON.stringify(match ?? {})})`; + return runnerFailure( + 'MIGRATION.PRECHECK_FAILED', + `Operation ${op.id} stopped: some documents in collection ${rewrite.collection} already hold ${to} as well as ${rewrite.field}`, + { + why: `Renaming ${rewrite.field} to ${to} would overwrite their ${to}. Find them with ${find} in mongosh, and move or remove one of the two values in each. Nothing was renamed. The operations before this one in the plan already ran, such as dropping an index on ${rewrite.field}; once the documents are fixed, run the command again and the plan finishes from here.`, + meta: { operationId: op.id, name: op.name }, + }, + ); +} + const MONGO_UNAUTHORIZED = 13; /** @@ -354,13 +389,15 @@ export class MongoMigrationRunner { if (!passed) { return { executed: false, - failure: runnerFailure( - 'MIGRATION.PRECHECK_FAILED', - `Operation ${op.id} failed during precheck`, - { - meta: { operationId: op.id, name: op.name }, - }, - ), + failure: + renameOverwriteRefusal(op) ?? + runnerFailure( + 'MIGRATION.PRECHECK_FAILED', + `Operation ${op.id} failed during precheck`, + { + meta: { operationId: op.id, name: op.name }, + }, + ), }; } } diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts index 23b4bc1e1797..64861051f866 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts @@ -44,6 +44,7 @@ import { describe, expect, it } from 'vitest'; import { createCollection, dataTransform, + renameField, unsetField, } from '../src/core/migrations/migration-factories'; import { serializeMongoOps } from '../src/core/migrations/mongo-ops-serializer'; @@ -830,3 +831,26 @@ describe('MongoMigrationRunner, a document rewrite without the privilege to skip ).rejects.toBeInstanceOf(MongoServerError); }); }); + +describe('MongoMigrationRunner, a field rename stopped because documents already hold the new name', () => { + it('says which collection, how to find the documents, and that the operations before it already ran', async () => { + const harness = makeHarness(); + harness.driver.queueResponse([{ _id: 1 }]); + harness.driver.queueResponse([{ _id: 2 }]); + + const result = await execute(harness, [ + renameField('posts', 'title', 'headline', { filter: { type: 'bug' } }), + ]); + + expect(result.assertNotOk()).toEqual({ + code: 'MIGRATION.PRECHECK_FAILED', + summary: + 'Operation field.posts.title.rename(headline).where({"type":"bug"}) stopped: some documents in collection posts already hold headline as well as title', + why: 'Renaming title to headline would overwrite their headline. Find them with db.getCollection("posts").find({"$and":[{"type":"bug"},{"title":{"$exists":true}},{"headline":{"$exists":true}}]}) in mongosh, and move or remove one of the two values in each. Nothing was renamed. The operations before this one in the plan already ran, such as dropping an index on title; once the documents are fixed, run the command again and the plan finishes from here.', + meta: { + operationId: 'field.posts.title.rename(headline).where({"type":"bug"})', + name: 'field.posts.title.rename(headline).where({"type":"bug"})', + }, + }); + }); +}); diff --git a/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts b/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts index a27591d0a243..8679842e0bed 100644 --- a/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts +++ b/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts @@ -226,6 +226,60 @@ describe('MongoMigrationRunner - field rewrites', () => { timeouts.spinUpMongoMemoryServer, ); + it( + 'finishes the plan once the documents its before-check names are fixed as its message says', + async () => { + await createClosedUsers(); + await db + .collection('users') + .insertOne({ name: 'Dee', fullName: 'Dee D.' }, { bypassDocumentValidation: true }); + + const stopped = await run(renameNamePlan, usersWithFullName); + if (stopped.ok) throw new Error('Expected the rename to stop at its before-check'); + const printedFind = /db\.getCollection\("users"\)\.find\((.*)\) in mongosh/.exec( + stopped.failure.why ?? '', + ); + const found = await db + .collection('users') + .find(JSON.parse(printedFind?.[1] ?? 'null'), { projection: { _id: 0 } }) + .toArray(); + expect({ + summary: stopped.failure.summary, + found, + indexes: (await db.collection('users').indexes()).map(({ name }) => name), + }).toEqual({ + summary: + 'Operation field.users.name.rename(fullName) stopped: some documents in collection users already hold fullName as well as name', + found: [{ name: 'Dee', fullName: 'Dee D.' }], + indexes: ['_id_'], + }); + + await db.collection('users').updateMany( + JSON.parse(printedFind?.[1] ?? 'null'), + { $unset: { fullName: '' } }, + { + bypassDocumentValidation: true, + }, + ); + const finished = await run(renameNamePlan, usersWithFullName); + + expect(finished.ok ? finished.value.operationsExecuted : finished.failure).toBe(3); + expect({ + documents: await documents('users'), + indexes: (await db.collection('users').indexes()).map(({ name }) => name), + }).toEqual({ + documents: [ + { fullName: 'Ada' }, + { fullName: 'Bo' }, + { fullName: 'Cy' }, + { fullName: 'Dee' }, + ], + indexes: ['_id_', 'fullName_1'], + }); + }, + timeouts.spinUpMongoMemoryServer, + ); + it( 'finishes a rename that stopped part way, where some documents already hold only the new name', async () => { From 39a18df2e748bcd069a4a42d7f385433ad04eb09 Mon Sep 17 00:00:00 2001 From: willbot Date: Fri, 9 Oct 2026 13:24:31 +0200 Subject: [PATCH 37/57] TML-3478: renaming or removing a field of a time-series collection is refused at plan time MongoDB updates time-series documents only through the metaField, so renameField and unsetField would fail with a raw server error. A field statement is refused with advice to keep the stored name, and a removed field fails planning with a fieldRemovalRefused conflict that says to keep the field in the contract. Following either advice plans nothing. Signed-off-by: willbot Signed-off-by: Will Madden --- .../subsystems/7. Migration System.md | 2 +- docs/reference/error-reference.md | 2 +- .../src/core/migrations/mongo-planner.ts | 3 + .../migrations/mongo-statement-planning.ts | 31 +++++++++- .../mongo-planner.statements.refusals.test.ts | 49 +++++++++++++++ .../target-runner/runner-statements.test.ts | 59 +++++++++++++++++++ 6 files changed, 143 insertions(+), 3 deletions(-) diff --git a/docs/architecture docs/subsystems/7. Migration System.md b/docs/architecture docs/subsystems/7. Migration System.md index 8767e7b5ae93..efac11bd9d37 100644 --- a/docs/architecture docs/subsystems/7. Migration System.md +++ b/docs/architecture docs/subsystems/7. Migration System.md @@ -225,7 +225,7 @@ The MongoDB planner does the same with collections and documents ([`mongo-statem - A model move to another namespace resolves, but the planner refuses it with `statementRefused`. - A model that keeps its name but changes its table cannot be renamed by a statement, because the model has the same name on both sides. Rename the table by hand with `this.renameTable(...)` in its own `migration.ts`. A field statement on such a model is refused with the same advice. - Only models and fields can be renamed. Value objects, their fields, enum values and namespaces cannot. -- On MongoDB, a statement cannot rename a variant whose discriminator value changes, since its documents would keep the old value; it cannot rename `_id`, a field whose stored name MongoDB reads as a path or an operator (it contains `.` or starts with `$`), a time-series collection, or a field inside a value object. A collection rename onto a name the database already has, or onto a name MongoDB rejects, is refused too. +- On MongoDB, a statement cannot rename a variant whose discriminator value changes, since its documents would keep the old value; it cannot rename `_id`, a field whose stored name MongoDB reads as a path or an operator (it contains `.` or starts with `$`), a time-series collection or a field of one, or a field inside a value object. A field removed from a model on a time-series collection fails planning with a `fieldRemovalRefused` conflict, because MongoDB cannot remove it from those documents; keeping the field in the contract plans nothing. A collection rename onto a name the database already has, or onto a name MongoDB rejects, is refused too. - **[Placeholder: decision pending]** On a MongoDB contract without a validator, such as one written with the TypeScript contract builder, renaming or removing a field does not change the storage hash. What `migration plan` and `migrate` do with such a change is described here once it is decided. #### Delete and allow statements diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index 8dfbad706331..4472d1f0823f 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -1576,7 +1576,7 @@ An apply command (`db migrate`/`db update`) cannot find a path through the on-di ### MIGRATION.PLANNING_FAILED -Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; renaming a variant whose discriminator value changes (the message says to keep the value with `@@base(, "")`); a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, replace the drop and create in its `migration.ts` with `renameCollection(...)`, run `node migration.ts` again, and apply it with `migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. +Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; renaming a variant whose discriminator value changes (the message says to keep the value with `@@base(, "")`); a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); renaming a field of a time-series collection, since MongoDB updates its documents only through the metaField (the message says to keep the field's stored name); removing a field of a time-series collection, a conflict of kind `fieldRemovalRefused` (the message says to keep the field in the contract, optional if new documents leave it out); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, replace the drop and create in its `migration.ts` with `renameCollection(...)`, run `node migration.ts` again, and apply it with `migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. `migration plan` raises it when the planner produces no operations. The conflict's kind names the case, and its `why` is the next action. With an earlier contract (`noDatabaseChange`), the contract changed in a way the database does not see, for example after a Prisma upgrade that changes how `contract.json` records values. If you deploy with migrations, run `prisma migration new --from `, with the earlier contract's hash from the message, to write a migration with no operations, then `prisma db migrate`. If you manage the database with `db init` or `db update`, run `prisma db sign` on each database instead. With no earlier contract (`nothingToCreate`), the contract describes nothing to create; if the database needs something the contract does not describe, run `prisma migration new` and write the operations in `migration.ts` by hand. `migration new` writes a migration only to the emitted contract, so with `--to` the message says to emit the target first, and for a database managed with `db update` it names `prisma db sign `. When the migrations directory is empty and the `db` ref points at a contract that describes nothing to create (`nothingToBaseline`), there is no baseline migration to plan, and no command writes one with no operations yet; keep using `db update`. diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts index 715c0b922a8f..c33a45b8ab7e 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts @@ -31,6 +31,7 @@ import { planStatements, planValidatorFieldRemovals, type StatementCalls, + timeSeriesRemovalRefusals, } from './mongo-statement-planning'; import type { OpFactoryCall } from './op-factory-call'; import { @@ -379,6 +380,8 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> schema.collectionNames.filter((name) => destinationIR.collection(name) !== undefined), ), }); + const removalRefusals = timeSeriesRemovalRefusals(removals, schema); + if (removalRefusals.length > 0) return { kind: 'failure', conflicts: removalRefusals }; const result = this.planCalls({ ...options, statementCalls: { diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index b53ea58ac4e7..cbe462d20ff7 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -139,6 +139,28 @@ function refused( const LEAVE_OUT = 'Leave out this statement.'; +const TIME_SERIES_UPDATES = + 'MongoDB updates the documents of a time-series collection only through its metaField'; + +/** + * A removal the plan cannot carry out because its collection is a time-series collection, whose + * documents MongoDB does not let an update change except through the metaField. + */ +export function timeSeriesRemovalRefusals( + removals: readonly FieldRemoval[], + schema: MongoSchemaIR, +): readonly MigrationPlannerConflict[] { + return removals + .filter( + (removal) => schema.collection(removal.call.collection)?.options?.timeseries !== undefined, + ) + .map((removal) => ({ + kind: 'fieldRemovalRefused', + summary: `Cannot remove field ${JSON.stringify(removal.call.field)} from collection ${JSON.stringify(removal.call.collection)}: it is a time-series collection`, + why: `${TIME_SERIES_UPDATES}, so the field cannot be removed from the documents that hold it. Keep the field in the contract, as an optional field if new documents leave it out, then run this command again.`, + })); +} + /** * Plans the rename each statement states, in order: a model statement whose collection changes renames the * collection in the schema the diff then starts from, and a field statement whose stored name @@ -162,7 +184,7 @@ export function planStatements(input: { const outcome = statement.entity === 'model' ? planModelStatement(statement, label, schema, input) - : planFieldStatement(statement, label, renamedCollections, input); + : planFieldStatement(statement, label, renamedCollections, schema, input); if (outcome.kind === 'refused') { conflicts.push(outcome.conflict); callOfStatement.push(undefined); @@ -297,6 +319,7 @@ function planFieldStatement( statement: ResolvedFieldRenameStatement, label: string, renamedCollections: ReadonlyMap, + schema: MongoSchemaIR, input: { readonly fromContract: ContractWithDomain; readonly contract: MongoContract }, ): StatementOutcome { const refuse = (reason: string, why: string): StatementOutcome => ({ @@ -345,6 +368,12 @@ function planFieldStatement( ); } if (statement.from.field === statement.to.field) return { kind: 'unchanged' }; + if (schema.collection(collection)?.options?.timeseries !== undefined) { + return refuse( + `collection "${collection}" is a time-series collection`, + `${TIME_SERIES_UPDATES}, so a statement cannot move the values of a field in them. Keep the stored name: name the field ${JSON.stringify(statement.from.field)} in the contract (with @map(${JSON.stringify(statement.from.field)}) in PSL), then run this command again without this statement.`, + ); + } const policyRefusal = controlPolicyRefusal(statement, label, input.contract, collection); if (policyRefusal !== undefined) return policyRefusal; const filter = diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts index 479cc7b2016c..322ae3898eaf 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts @@ -254,3 +254,52 @@ describe('MongoMigrationPlanner, a variant rename whose discriminator value chan ).toEqual([]); }); }); + +describe('MongoMigrationPlanner, a field of a time-series collection', () => { + const readings = (seed: string, fields: readonly string[]) => + contract( + seed, + { Reading: { collection: 'readings', fields: ['_id', 'at', ...fields] } }, + { readings: { timeField: 'at' } }, + ); + const schema = [schemaCollection('readings', { timeseries: true })]; + + it('cannot be renamed, and the refusal says to keep its stored name', () => { + expect( + refusal( + plan({ + from: readings('from', ['value']), + to: readings('to', ['reading']), + schema, + statements: [renameField('Reading', 'value', 'Reading', 'reading')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: + 'Cannot rename field "Reading.value" to "Reading.reading": collection "readings" is a time-series collection', + why: 'MongoDB updates the documents of a time-series collection only through its metaField, so a statement cannot move the values of a field in them. Keep the stored name: name the field "value" in the contract (with @map("value") in PSL), then run this command again without this statement.', + }, + ]); + }); + + it('plans nothing once the stored name is kept, as the refusal says', () => { + expect( + plan({ from: readings('from', ['value']), to: readings('to', ['value']), schema }).kind, + ).toBe('success'); + }); + + it('cannot be removed, and the refusal says to keep it', () => { + expect( + refusal(plan({ from: readings('from', ['value']), to: readings('to', []), schema })), + ).toEqual([ + { + kind: 'fieldRemovalRefused', + summary: + 'Cannot remove field "value" from collection "readings": it is a time-series collection', + why: 'MongoDB updates the documents of a time-series collection only through its metaField, so the field cannot be removed from the documents that hold it. Keep the field in the contract, as an optional field if new documents leave it out, then run this command again.', + }, + ]); + }); +}); diff --git a/test/integration/test/mongo/target-runner/runner-statements.test.ts b/test/integration/test/mongo/target-runner/runner-statements.test.ts index 82fc33f8213e..c8298b2f5964 100644 --- a/test/integration/test/mongo/target-runner/runner-statements.test.ts +++ b/test/integration/test/mongo/target-runner/runner-statements.test.ts @@ -207,4 +207,63 @@ describe('MongoMigrationPlanner and runner - statements', () => { }, timeouts.spinUpMongoMemoryServer, ); + + it( + 'refuses at plan time to rename or remove a field of a live time-series collection, and plans nothing once it is kept', + async () => { + const withFields = (storageHash: string, fields: readonly string[]) => + contract( + storageHash, + { Reading: { collection: 'readings', fields: ['_id', 'at', ...fields] } }, + { + readings: new MongoCollection({ options: { timeseries: { timeField: 'at' } } }), + }, + ); + await live.db().createCollection('readings', { timeseries: { timeField: 'at' } }); + await live + .db() + .collection('readings') + .insertOne({ at: new Date('2026-01-01'), value: 1 }); + const planFor = async ( + to: ReturnType, + statements: Parameters[0]['statements'] = [], + ) => + new MongoMigrationPlanner().plan({ + contract: to, + schema: await introspectSchema(live.db()), + policy, + fromContract: withFields('from', ['value']), + origin: { storageHash: 'from' }, + statements, + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + + const renamed = await planFor(withFields('to', ['reading']), [ + { + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: 'Reading', field: 'value' }, + to: { namespaceId: NS, model: 'Reading', field: 'reading' }, + }, + ]); + const removed = await planFor(withFields('to', [])); + const kept = await planFor(withFields('to', ['value'])); + + expect({ + renamed: renamed.kind === 'failure' ? renamed.conflicts.map(({ summary }) => summary) : [], + removed: removed.kind === 'failure' ? removed.conflicts.map(({ summary }) => summary) : [], + kept: kept.kind === 'success' ? kept.plan.operations.length : kept.conflicts, + }).toEqual({ + renamed: [ + 'Cannot rename field "Reading.value" to "Reading.reading": collection "readings" is a time-series collection', + ], + removed: [ + 'Cannot remove field "value" from collection "readings": it is a time-series collection', + ], + kept: 0, + }); + }, + timeouts.spinUpMongoMemoryServer, + ); }); From 0e2821060a94e632ff13fb18a966c79408aba7a8 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 12:27:38 +0200 Subject: [PATCH 38/57] TML-3478: the question about a dropped collection whose model keeps its name says how to keep the documents No statement can rename the collection of a model that keeps its name, so the question offered only --delete. The MongoDB planner now attaches advice to that drop: rename the collection in mongosh before db update, or replace the drop and create in migration.ts with renameCollection. The framework carries the advice as an optional keepDataAdvice on each data loss and ends the question with it, so no MongoDB words enter the framework. Signed-off-by: willbot Signed-off-by: Will Madden --- .../src/control/migration-statements.ts | 8 ++- .../src/exports/control.ts | 1 + .../control-api/operations/migration-plan.ts | 7 +-- .../control-api/statements/plan-questions.ts | 13 +++-- .../statements/data-loss-question.test.ts | 22 ++++++++ .../src/core/migrations/mongo-planner.ts | 52 +++++++++++++++---- .../migrations/mongo-statement-planning.ts | 18 ++++++- .../test/mongo-planner.data-loss.test.ts | 34 +++++++++++- ...ete-statements-migration.mongo.e2e.test.ts | 45 +++++++++++++++- 9 files changed, 180 insertions(+), 20 deletions(-) diff --git a/packages/1-framework/1-core/framework-components/src/control/migration-statements.ts b/packages/1-framework/1-core/framework-components/src/control/migration-statements.ts index d646cfd14a32..0802c673b292 100644 --- a/packages/1-framework/1-core/framework-components/src/control/migration-statements.ts +++ b/packages/1-framework/1-core/framework-components/src/control/migration-statements.ts @@ -68,6 +68,12 @@ export interface MigrationOperationSubject { readonly subject: MigrationSubject; } +/** An operation that loses the data of its subject. */ +export interface MigrationDataLoss extends MigrationOperationSubject { + /** How to keep the data instead, in the target's words, when no statement can say so. */ + readonly keepDataAdvice?: string; +} + /** An operation that changes who can read or write the rows of its subject. */ export interface MigrationAccessChange extends MigrationOperationSubject { /** @@ -79,7 +85,7 @@ export interface MigrationAccessChange extends MigrationOperationSubject { /** What the operations of a plan lose and whose access they change, each by its position in the plan. */ export interface MigrationPlanSubjects< - TEntry extends MigrationOperationSubject = MigrationOperationSubject, + TEntry extends MigrationOperationSubject = MigrationDataLoss, TAccess extends MigrationAccessChange = MigrationAccessChange, > { /** Each operation that loses data, in plan order, with what it loses. */ diff --git a/packages/1-framework/1-core/framework-components/src/exports/control.ts b/packages/1-framework/1-core/framework-components/src/exports/control.ts index d76f0c3756da..bdb2edc8dbfc 100644 --- a/packages/1-framework/1-core/framework-components/src/exports/control.ts +++ b/packages/1-framework/1-core/framework-components/src/exports/control.ts @@ -127,6 +127,7 @@ export type { FieldCoordinate, FieldCoordinateJson, MigrationAccessChange, + MigrationDataLoss, MigrationOperationSubject, MigrationPlanSubjects, MigrationStatementJson, diff --git a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts index 5f5afa4b6176..ff281ab0262d 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts @@ -52,7 +52,7 @@ import { snapshotVerifierFor } from '../../utils/snapshot-content-verification'; import { type AnswerPlanQuestions, askPlanQuestions, - type PlannedSubject, + type PlannedLoss, subjectText, } from '../statements/plan-questions'; import { @@ -112,7 +112,7 @@ type PlannerSuccess = { readonly hasPlaceholders: boolean; readonly appliedStatements: readonly AppliedStatementReport[]; /** The operations that would lose data, positioned in the plan's operations. */ - readonly dataLoss: readonly PlannedSubject[]; + readonly dataLoss: readonly PlannedLoss[]; }; /** The origin of a plan from an empty database: no models, so no statement resolves. */ @@ -216,9 +216,10 @@ async function runPlannerLeg( plannedOps, migrationTsContent: plannerResult.plan.renderTypeScript(resolveImportSpecifier), hasPlaceholders, - dataLoss: plannerResult.dataLoss.map(({ operationIndex, subject }) => ({ + dataLoss: plannerResult.dataLoss.map(({ operationIndex, subject, keepDataAdvice }) => ({ operationIndex: position(operationIndex), subject, + ...ifDefined('keepDataAdvice', keepDataAdvice), label: resolved[operationIndex]?.label ?? subjectText(subject, { origin: fromContract ?? EMPTY_ORIGIN, renames: [] }), diff --git a/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts b/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts index 6ec9c1955a67..18c2a43b7457 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts @@ -59,6 +59,12 @@ export interface PlannedSubject { readonly subject: MigrationSubject; } +/** An operation of a plan that loses its subject's data. */ +export interface PlannedLoss extends PlannedSubject { + /** How to keep the data instead, in the target's words; the question ends with it. */ + readonly keepDataAdvice?: string; +} + /** An operation of a plan that changes who can read or write its subject's rows. */ export interface PlannedAccessChange extends PlannedSubject { readonly widens: boolean; @@ -66,7 +72,7 @@ export interface PlannedAccessChange extends PlannedSubject { /** What a plan would lose, and whose access it would change. */ export interface PlannedQuestions { - readonly dataLoss: readonly PlannedSubject[]; + readonly dataLoss: readonly PlannedLoss[]; readonly accessWidening: readonly PlannedAccessChange[]; } @@ -239,7 +245,7 @@ function missingFieldModel( * after a statement renames the model, so the question says to pass one. */ export function dataLossQuestion( - loss: PlannedSubject, + loss: PlannedLoss, contracts: StatementContracts & { readonly destination: ContractWithDomain; readonly originKnown: boolean; @@ -257,6 +263,7 @@ export function dataLossQuestion( question: [ `${loss.label} ${lossText(loss.subject, subject, contracts.originKnown)}.`, ...(missingModel === undefined ? [] : [modelRenameAdvice(missingModel, contracts)]), + ...(loss.keepDataAdvice === undefined ? [] : [loss.keepDataAdvice]), ].join(' '), subject, verbs: renamable ? ['rename', 'delete'] : ['delete'], @@ -362,7 +369,7 @@ function checkAnswers(questions: readonly PlanQuestion[], answers: readonly Plan } function withRecoveryOnFirstStorageQuestion( - asked: readonly { readonly loss: PlannedSubject; readonly question: PlanQuestion }[], + asked: readonly { readonly loss: PlannedLoss; readonly question: PlanQuestion }[], recoveryApplies: boolean, ): readonly PlanQuestion[] { const first = recoveryApplies diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts index bdf3b335ad5d..56acf775cb40 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts @@ -140,3 +140,25 @@ describe('dataLossQuestion for a field whose model the destination names differe }); }); }); + +describe('dataLossQuestion with advice from the target', () => { + it('ends the question with how to keep the data, and still offers only delete', () => { + const app = asNamespaceId('app'); + const users = contractOf({ app: { models: { User: { fields: ['email'] } } } }); + const question = dataLossQuestion( + { + operationIndex: 0, + label: 'Drop storage users', + subject: { kind: 'model', namespaceId: app, model: 'User' }, + keepDataAdvice: 'To keep its data, move it by hand.', + }, + { origin: users, destination: users, renames: [], originKnown: true }, + ); + + expect({ question: question.question, verbs: question.verbs }).toEqual({ + question: + 'Drop storage users would lose the data of model "User". To keep its data, move it by hand.', + verbs: ['delete'], + }); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts index c33a45b8ab7e..6341b1e4ef7a 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts @@ -2,6 +2,7 @@ import { asNamespaceId, type Contract, type ContractWithDomain } from '@internal import { contractToMongoSchemaIR } from '@internal/family-mongo/control'; import type { TargetBoundComponentDescriptor } from '@internal/framework-components/components'; import type { + MigrationDataLoss, MigrationOperationPolicy, MigrationOperationSubject, MigrationPlanner, @@ -27,6 +28,7 @@ import { blindCast } from '@internal/utils/casts'; import { collectionOf, type FieldRemoval, + keepCollectionByHandAdvice, planFieldRemovals, planStatements, planValidatorFieldRemovals, @@ -168,22 +170,54 @@ function modelStoredIn( return (storing.find(({ isRoot }) => isRoot) ?? storing[0])?.coordinate; } -/** Each collection drop of the plan, with the model whose documents it loses. */ +/** + * Each collection drop of the plan, with the model whose documents it loses. When that model keeps + * its name and the plan creates its new collection, the drop says how to rename the collection by + * hand instead. + */ function collectionDrops( calls: readonly OpFactoryCall[], fromContract: ContractWithDomain | null, -): readonly MigrationOperationSubject[] { - return calls.flatMap((call, operationIndex) => { + contract: ContractWithDomain, +): readonly MigrationDataLoss[] { + const created = new Set( + calls.flatMap((call) => (call instanceof CreateCollectionCall ? [call.collection] : [])), + ); + return calls.flatMap((call, operationIndex): readonly MigrationDataLoss[] => { if (!(call instanceof DropCollectionCall)) return []; const model = fromContract === null ? undefined : modelStoredIn(fromContract, call.collection); - const subject: MigrationSubject = - model === undefined - ? { kind: 'storage', name: call.collection } - : { kind: 'model', ...model }; - return [{ operationIndex, subject }]; + if (model === undefined) { + return [{ operationIndex, subject: { kind: 'storage', name: call.collection } }]; + } + const subject: MigrationSubject = { kind: 'model', ...model }; + const destinationCollection = modelStoredInDestination(contract, model); + return [ + destinationCollection !== undefined && created.has(destinationCollection) + ? { + operationIndex, + subject, + keepDataAdvice: keepCollectionByHandAdvice( + model, + call.collection, + destinationCollection, + ), + } + : { operationIndex, subject }, + ]; }); } +/** The collection the destination stores `model` in, when the destination still has that model. */ +function modelStoredInDestination( + contract: ContractWithDomain, + model: ModelCoordinate, +): string | undefined { + const namespace = contract.domain.namespaces[model.namespaceId]; + return namespace !== undefined && Object.hasOwn(namespace.models, model.model) + ? collectionOf(contract, model) + : undefined; +} + export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> { planCalls(options: { readonly contract: unknown; @@ -410,7 +444,7 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> const call = callOfStatement[index]; return { statement, operationIndexes: call === undefined ? [] : [calls.indexOf(call)] }; }), - dataLoss: [...collectionDrops(calls, fromContract), ...fieldLoss].sort( + dataLoss: [...collectionDrops(calls, fromContract, contract), ...fieldLoss].sort( (a, b) => a.operationIndex - b.operationIndex, ), accessWidening: [], diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index cbe462d20ff7..e30041351b65 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -393,7 +393,23 @@ function collectionChangeAdvice( if (!sameModel(statement.from, statement.to)) { return `Add --rename ${modelDisplayName(statement.from)}:${modelDisplayName(statement.to)} before this statement, so the collection is renamed with its documents.`; } - return `A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(statement.from)} to answer its question; in its migration.ts replace the drop of "${from}" and the create of "${to}" with renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}), imported with the other operations; run node migration.ts in the migration's directory, so it writes ops.json again; then apply it with migrate. Then change the field and run this statement again.`; + return `A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. ${renameCollectionByHandSteps(statement.from, from, to)} Then change the field and run this statement again.`; +} + +function renameCollectionByHandSteps(model: ModelCoordinate, from: string, to: string): string { + return `With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(model)} to answer its question; in its migration.ts replace the drop of "${from}" and the create of "${to}" with renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}), imported with the other operations; run node migration.ts in the migration's directory, so it writes ops.json again; then apply it with migrate.`; +} + +/** + * How to keep the documents of a model that keeps its name while its collection changes from + * `from` to `to`: no statement can rename that collection, so the user renames it by hand. + */ +export function keepCollectionByHandAdvice( + model: ModelCoordinate, + from: string, + to: string, +): string { + return `To keep its documents, rename collection "${from}" to "${to}" instead; a statement cannot rename the collection of a model that keeps its name. ${renameCollectionByHandSteps(model, from, to)}`; } export interface FieldRemoval { diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts index 6100bb55cc34..b44387d37aac 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts @@ -61,9 +61,12 @@ function contractWith( } as unknown as MongoContract; } -function plan(fromContract: MongoContract | null) { +function plan( + fromContract: MongoContract | null, + contract: MongoContract = contractWith({ User: 'users' }, 'to'), +) { const result = new MongoMigrationPlanner().plan({ - contract: contractWith({ User: 'users' }, 'to'), + contract, schema: new MongoSchemaIR([ new MongoSchemaCollection({ name: 'users' }), new MongoSchemaCollection({ @@ -142,4 +145,31 @@ describe('MongoDB planner, data loss', () => { }, ]); }); + + it('advises renaming the collection by hand when only the collection of a model that keeps its name changes', () => { + const planned = plan( + contractWith({ User: 'users', Event: 'events' }, 'from'), + contractWith({ User: 'users', Event: 'occasions' }, 'to'), + ); + + expect( + planned.dataLoss.map(({ operationIndex, subject, keepDataAdvice }) => ({ + operation: planned.operations[operationIndex], + subject, + keepDataAdvice, + })), + ).toEqual([ + { + operation: 'Drop collection events', + subject: { kind: 'model', namespaceId: asNamespaceId('__unbound__'), model: 'Event' }, + keepDataAdvice: + 'To keep its documents, rename collection "events" to "occasions" instead; a statement cannot rename the collection of a model that keeps its name. With db update, rename the collection in the database yourself, for example db.getCollection("events").renameCollection("occasions") in mongosh, then run db update. With migration plan, plan it, passing --delete Event to answer its question; in its migration.ts replace the drop of "events" and the create of "occasions" with renameCollection("events", "occasions"), imported with the other operations; run node migration.ts in the migration\'s directory, so it writes ops.json again; then apply it with migrate.', + }, + ]); + }); + + it('gives no advice when the model of the dropped collection is gone', () => { + const planned = plan(contractWith({ User: 'users', Event: 'events' }, 'from')); + expect(planned.dataLoss.map(({ keepDataAdvice }) => keepDataAdvice)).toEqual([undefined]); + }); }); diff --git a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts index 8038185e6dd1..18fef319cea4 100644 --- a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts +++ b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts @@ -3,7 +3,7 @@ * * Dropping a collection loses its documents, so `migration plan` and `db update` ask what the drop means; `--delete ` answers. The MongoDB planner refuses rename statements in this release. * - * Journey S1 removes `Event` through `migration plan`: refused without `--delete Event`, planned with it, and applied by `migrate`. Journey S2 does the same through `db update`, where `--confirm ` no longer consents. Journey S3 adds a required field with no refusal. Journey S4 gives a rename, which renames the collection and keeps its documents. Journey S5 follows, as printed, the refusal of a field rename whose model keeps its name while its collection changes. + * Journey S1 removes `Event` through `migration plan`: refused without `--delete Event`, planned with it, and applied by `migrate`. Journey S2 does the same through `db update`, where `--confirm ` no longer consents. Journey S3 adds a required field with no refusal. Journey S4 gives a rename, which renames the collection and keeps its documents. Journey S5 follows, as printed, the refusal of a field rename whose model keeps its name while its collection changes. Journey S6 follows, through `db update`, the question about a collection whose model keeps its name, and keeps its documents. */ import { copyFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; @@ -336,6 +336,12 @@ describe('Journeys (MongoDB): statements that consent to data loss', { useFixture(ctx, 'contract-events-remapped.ts'); await emit(ctx, 'S5.06'); + const asked = await runMigrationPlan(ctx, ['--from', origin, '--json']); + expect(asked.exitCode, 'S5.06: the drop of events needs a statement').toBe(2); + expect( + JSON.stringify(engineError(asked)), + 'S5.06: the question says how to keep the documents', + ).toContain('replace the drop of \\"events\\" and the create of \\"occasions\\"'); const remap = await runMigrationPlan(ctx, [ '--name', 'remap', @@ -395,4 +401,41 @@ describe('Journeys (MongoDB): statements that consent to data loss', { { title: 'launch' }, ]); }); + + it('S6: db update keeps the documents when the question about a collection whose model keeps its name is followed', async () => { + const dbName = 'mongo_remap_update'; + const ctx = setupProject(withDatabase(replSet.getUri(), dbName)); + created.add(ctx.testDir); + await emit(ctx, 'S6.01'); + const create = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(create.exitCode, `S6.02: db update creates the collections: ${create.stderr}`).toBe(0); + await client.db(dbName).collection('events').insertOne({ name: 'launch' }); + useFixture(ctx, 'contract-events-remapped.ts'); + await emit(ctx, 'S6.03'); + + const refused = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(refused.exitCode, 'S6.04: the drop of events needs a statement').toBe(2); + expect(engineError(refused)?.code, 'S6.04').toBe('CLI.CONSENT_REQUIRED'); + expect( + JSON.stringify(engineError(refused)), + 'S6.04: the question says how to keep the documents', + ).toContain( + 'db.getCollection(\\"events\\").renameCollection(\\"occasions\\") in mongosh, then run db update', + ); + expect(await collections(dbName), 'S6.04: nothing dropped').toContain('events'); + + await client.db(dbName).collection('events').rename('occasions'); + const updated = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(updated.exitCode, `S6.05: db update after the rename: ${updated.stderr}`).toBe(0); + expect(await collections(dbName), 'S6.05: the collection is renamed').toEqual([ + 'occasions', + 'users', + ]); + const documents = await client + .db(dbName) + .collection('occasions') + .find({}, { projection: { _id: 0 } }) + .toArray(); + expect(documents, 'S6.05: its document is kept').toEqual([{ name: 'launch' }]); + }); }); From 66a3168529f53b0058f1fa1dc499c9c28e9d9376 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 12:27:53 +0200 Subject: [PATCH 39/57] TML-3478: a journey checks that the no-snapshot recovery steps print one per line The steps were already joined by newlines, and the CLI engine prints each line of a why on its own line. Journey R3 now asserts the question, the lead-in and steps 1 to 3 arrive as separate lines. Signed-off-by: willbot Signed-off-by: Will Madden --- .../cli-journeys/rename-delete-statements.mongo.e2e.test.ts | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts b/test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts index 768791ef0e44..10b741d23e08 100644 --- a/test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts +++ b/test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts @@ -343,6 +343,12 @@ describe('Journeys (MongoDB): --rename and --delete', { expect(refused.exitCode, 'R3.04: the removed field needs a statement').toBe(2); const printed = printedFlags(refused, 'delete'); expect(printed, 'R3.04: named by its collection').toEqual(['--delete', 'profiles.nickname']); + expect( + String(engineError(refused)?.why) + .split('\n') + .map((line) => line.slice(0, 3)), + 'R3.04: the question, then each way to keep the data on a line of its own', + ).toEqual(['Rem', 'If ', '1. ', '2. ', '3. ']); const updated = await runDbUpdate(ctx, ['--no-interactive', ...printed]); expect(updated.exitCode, `R3.05: db update with the printed flag: ${updated.stderr}`).toBe(0); From 92414364ce5d0c2775b99082d2d37f47c992db76 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 12:55:02 +0200 Subject: [PATCH 40/57] TML-3478: advice to self-emit a migration names the command that works from the project directory The MongoDB advice for renaming a collection by hand, and the advice when migration plan has nothing to create, said to run node migration.ts in the migration directory. There the config is not found, ops.json stays as planned, and db migrate applies the original drop. Both now say to run node migrations/app//migration.ts from the project directory. Journey S5 runs that command as the question prints it. Signed-off-by: willbot Signed-off-by: Will Madden --- .../3-tooling/cli/src/utils/cli-errors.ts | 4 +-- .../orm/migration-plan.no-operations.test.ts | 4 +-- .../migrations/mongo-statement-planning.ts | 2 +- .../test/mongo-planner.data-loss.test.ts | 2 +- .../mongo-planner.statements.refusals.test.ts | 2 +- ...ete-statements-migration.mongo.e2e.test.ts | 8 +++--- .../test/utils/journey-test-helpers.ts | 25 +++++++++++++++++++ 7 files changed, 36 insertions(+), 11 deletions(-) diff --git a/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts b/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts index 76a0603b3b50..3f84d5773da9 100644 --- a/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts +++ b/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts @@ -779,8 +779,8 @@ function noOperationsConflict( summary: 'This contract describes nothing migration plan can create, so there is no first migration to plan.', why: destination.isEmitted - ? 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then run `node migration.ts` in that directory to write `ops.json`.' - : '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then run `node migration.ts` in that directory to write `ops.json`.', + ? 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then, from the project directory, run `node migrations/app//migration.ts`, naming the directory it wrote, to write `ops.json`.' + : '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then, from the project directory, run `node migrations/app//migration.ts`, naming the directory it wrote, to write `ops.json`.', }; case 'baseline': return { diff --git a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts index 404374aaeab0..5d548651266f 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts @@ -191,7 +191,7 @@ describe('migration plan when the planner produces no operations', () => { planningFailed({ kind: 'nothingToCreate', summary: NOTHING_TO_CREATE, - why: 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then run `node migration.ts` in that directory to write `ops.json`.', + why: 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then, from the project directory, run `node migrations/app//migration.ts`, naming the directory it wrote, to write `ops.json`.', }), ); expect({ exitCode: scaffold.exitCode, migrations: await migrationsOf(project) }).toEqual({ @@ -208,7 +208,7 @@ describe('migration plan when the planner produces no operations', () => { planningFailed({ kind: 'nothingToCreate', summary: NOTHING_TO_CREATE, - why: '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then run `node migration.ts` in that directory to write `ops.json`.', + why: '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then, from the project directory, run `node migrations/app//migration.ts`, naming the directory it wrote, to write `ops.json`.', }), ); }); diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index e30041351b65..e16b5a834be4 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -397,7 +397,7 @@ function collectionChangeAdvice( } function renameCollectionByHandSteps(model: ModelCoordinate, from: string, to: string): string { - return `With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(model)} to answer its question; in its migration.ts replace the drop of "${from}" and the create of "${to}" with renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}), imported with the other operations; run node migration.ts in the migration's directory, so it writes ops.json again; then apply it with migrate.`; + return `With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(model)} to answer its question; in its migration.ts replace the drop of "${from}" and the create of "${to}" with renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}), imported with the other operations; from the project directory, run node migrations/app//migration.ts, naming the directory migration plan wrote, so it writes ops.json again; then apply it with prisma db migrate.`; } /** diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts index b44387d37aac..3958de1fe5c1 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts @@ -163,7 +163,7 @@ describe('MongoDB planner, data loss', () => { operation: 'Drop collection events', subject: { kind: 'model', namespaceId: asNamespaceId('__unbound__'), model: 'Event' }, keepDataAdvice: - 'To keep its documents, rename collection "events" to "occasions" instead; a statement cannot rename the collection of a model that keeps its name. With db update, rename the collection in the database yourself, for example db.getCollection("events").renameCollection("occasions") in mongosh, then run db update. With migration plan, plan it, passing --delete Event to answer its question; in its migration.ts replace the drop of "events" and the create of "occasions" with renameCollection("events", "occasions"), imported with the other operations; run node migration.ts in the migration\'s directory, so it writes ops.json again; then apply it with migrate.', + 'To keep its documents, rename collection "events" to "occasions" instead; a statement cannot rename the collection of a model that keeps its name. With db update, rename the collection in the database yourself, for example db.getCollection("events").renameCollection("occasions") in mongosh, then run db update. With migration plan, plan it, passing --delete Event to answer its question; in its migration.ts replace the drop of "events" and the create of "occasions" with renameCollection("events", "occasions"), imported with the other operations; from the project directory, run node migrations/app//migration.ts, naming the directory migration plan wrote, so it writes ops.json again; then apply it with prisma db migrate.', }, ]); }); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts index 322ae3898eaf..7bcd139151c7 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts @@ -199,7 +199,7 @@ describe('MongoMigrationPlanner, statements it refuses', () => { }), )[0]?.why, ).toBe( - 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question; in its migration.ts replace the drop of "profiles" and the create of "users" with renameCollection("profiles", "users"), imported with the other operations; run node migration.ts in the migration\'s directory, so it writes ops.json again; then apply it with migrate. Then change the field and run this statement again.', + 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question; in its migration.ts replace the drop of "profiles" and the create of "users" with renameCollection("profiles", "users"), imported with the other operations; from the project directory, run node migrations/app//migration.ts, naming the directory migration plan wrote, so it writes ops.json again; then apply it with prisma db migrate. Then change the field and run this statement again.', ); }); }); diff --git a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts index 18fef319cea4..dabb30a3bdc3 100644 --- a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts +++ b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts @@ -23,7 +23,7 @@ import { runDbUpdate, runMigrate, runMigrationPlan, - selfEmitMigration, + runPrintedSelfEmit, } from '../utils/journey-test-helpers'; const FIXTURES_DIR = join(fixtureAppDir, 'fixtures/mongo-cli-journeys'); @@ -331,7 +331,7 @@ describe('Journeys (MongoDB): statements that consent to data loss', { ]); expect(refused.exitCode, 'S5.05: the field rename is refused').not.toBe(0); expect(JSON.stringify(engineError(refused)?.meta), 'S5.05: the advice').toContain( - 'run node migration.ts in the migration', + 'from the project directory, run node migrations/app//migration.ts', ); useFixture(ctx, 'contract-events-remapped.ts'); @@ -369,8 +369,8 @@ describe('Journeys (MongoDB): statements that consent to data loss', { ); expect(edited, `S5.08: the drop is gone:\n${edited}`).not.toMatch(/dropCollection\(['"]events/); writeFileSync(migrationTs, edited, 'utf-8'); - const emitted = await selfEmitMigration(ctx, ['--dir', `migrations/app/${remapDir}`]); - expect(emitted.exitCode, `S5.09: node migration.ts: ${emitted.stderr}`).toBe(0); + const emitted = runPrintedSelfEmit(ctx, String(engineError(asked)?.why), remapDir); + expect(emitted.exitCode, `S5.09: the printed node command: ${emitted.stderr}`).toBe(0); const applyRemap = await runMigrate(ctx); expect(applyRemap.exitCode, `S5.10: migrate: ${applyRemap.stderr}`).toBe(0); diff --git a/test/integration/test/utils/journey-test-helpers.ts b/test/integration/test/utils/journey-test-helpers.ts index 82abd945eb55..4f4aa9d702c3 100644 --- a/test/integration/test/utils/journey-test-helpers.ts +++ b/test/integration/test/utils/journey-test-helpers.ts @@ -6,6 +6,7 @@ * so journey tests stay concise and readable. */ +import { spawnSync } from 'node:child_process'; import { copyFileSync, existsSync, @@ -550,6 +551,30 @@ export async function selfEmitMigration( return runMigrationFile(migrationTs, args, ctx.testDir); } +/** + * Finds the `node migrations/app//migration.ts` command `message` prints, names the + * migration directory in it, and runs it as a user would type it in the project directory. + */ +export function runPrintedSelfEmit( + ctx: JourneyContext, + message: string, + migrationDir: string, +): CommandResult { + const printed = /\bnode (migrations\/app\/\/migration\.ts)/.exec(message)?.[1]; + if (printed === undefined) { + return { + exitCode: -1, + stdout: '', + stderr: `the message prints no node migrations/app//migration.ts command: ${message}`, + }; + } + const run = spawnSync(process.execPath, [printed.replace('', migrationDir)], { + cwd: ctx.testDir, + encoding: 'utf-8', + }); + return { exitCode: run.status ?? -1, stdout: run.stdout, stderr: run.stderr }; +} + /** * Runs `migration plan` (which scaffolds a draft migration.ts), then runs * that migration.ts in-process so it writes its own ops.json and From 7fec13d86f3aefc76a4c9b246d0b78c6cd848fe5 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 12:55:07 +0200 Subject: [PATCH 41/57] TML-3478: on Postgres and SQLite, the question about a dropped table whose model keeps its name says how to keep its rows When only @@map changes, no statement can rename the table, and the question offered only --delete. The SQL family now fills keepDataAdvice for that drop, with the target writing the renameTable call and the SQL: rename the table by hand before db update, or write the rename as its own migration with migration new and run it from the project directory. Journeys M1 and M2 follow both routes as printed on SQLite and Postgres and keep the rows. Signed-off-by: willbot Signed-off-by: Will Madden --- .../src/core/migrations/operation-subjects.ts | 37 +++- .../src/core/migrations/statement-planning.ts | 15 +- .../2-sql/9-family/src/exports/control.ts | 1 + .../9-family/test/operation-subjects.test.ts | 75 +++++++- .../postgres/src/core/migrations/planner.ts | 5 +- .../src/core/migrations/table-rename-calls.ts | 9 + .../migrations/operation-subjects.test.ts | 8 +- .../sqlite/src/core/migrations/planner.ts | 5 +- .../src/core/migrations/table-rename-calls.ts | 13 +- .../migrations/operation-subjects.test.ts | 3 +- .../cli-journeys/table-map-change.e2e.test.ts | 169 +++++++++++++++++ .../table-map-change.sqlite.e2e.test.ts | 172 ++++++++++++++++++ 12 files changed, 494 insertions(+), 18 deletions(-) create mode 100644 test/integration/test/cli-journeys/table-map-change.e2e.test.ts create mode 100644 test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts diff --git a/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts b/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts index e6c77bcc9696..cfec57ed17c0 100644 --- a/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts +++ b/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts @@ -1,7 +1,7 @@ import { asNamespaceId, type ContractWithDomain } from '@internal/contract/types'; import type { MigrationAccessChange, - MigrationOperationSubject, + MigrationDataLoss, MigrationPlanSubjects, MigrationSubject, ModelCoordinate, @@ -13,6 +13,7 @@ import { fieldRenameStorageEffect, isSqlModelStorage, modelRenameStorageEffect, + type TableRenameByHand, } from './statement-planning'; /** @@ -78,6 +79,7 @@ interface SubjectContext { readonly contract: ContractWithDomain; /** The rename statements the plan applied, whose renames a target names the new way. */ readonly statements: readonly ResolvedMigrationStatement[]; + readonly byHand: TableRenameByHand; } function originTable( @@ -165,6 +167,29 @@ function subjectOf(target: SubjectStorage, context: SubjectContext): MigrationSu return storage; } +/** + * How to keep the rows of a dropped table whose model the destination keeps under the same name in + * another table of the same namespace: no statement can rename that table, so the user renames it + * by hand. None when the origin contract already stores a model in that table. + */ +function keepTableAdvice( + target: SubjectStorage, + subject: MigrationSubject, + context: SubjectContext, +): string | undefined { + if (subject.kind !== 'model' || target.table === undefined || context.fromContract === null) { + return undefined; + } + const storage = + context.contract.domain.namespaces[subject.namespaceId]?.models[subject.model]?.storage; + if (storage === undefined || !isSqlModelStorage(storage)) return undefined; + const { namespaceId, table: from } = target.table; + if (storage.namespaceId !== namespaceId || storage.table === from) return undefined; + if (modelsStoring(context.fromContract, namespaceId, storage.table).length > 0) return undefined; + const rename = { namespaceId, from, to: storage.table }; + return `To keep its rows, rename table "${from}" to "${storage.table}" instead; a statement cannot rename the table of a model that keeps its name. With db update, rename the table in the database yourself with ${context.byHand.tableRenameByHand(rename).join('; ')}, then run prisma db update. With migration plan, write the rename as a migration of its own instead: run prisma migration new --name --from , add ${context.byHand.renderTableRename(rename)} to the operations of its migration.ts, from the project directory run node migrations/app//migration.ts, naming the directory migration new wrote, so it writes ops.json, then run prisma db migrate.`; +} + /** * The subject of each operation that loses data and each that widens access, at its position in * the plan. A subject is a model or a field of the origin contract when one stores it, found under @@ -174,12 +199,18 @@ export function subjectsOfCalls( calls: readonly CallSubjects[], context: SubjectContext, ): MigrationPlanSubjects { - const dataLoss: MigrationOperationSubject[] = []; + const dataLoss: MigrationDataLoss[] = []; const accessWidening: MigrationAccessChange[] = []; let operationIndex = 0; for (const call of calls) { for (const target of call.dataLoss) { - dataLoss.push({ operationIndex, subject: subjectOf(target, context) }); + const subject = subjectOf(target, context); + const keepDataAdvice = keepTableAdvice(target, subject, context); + dataLoss.push( + keepDataAdvice === undefined + ? { operationIndex, subject } + : { operationIndex, subject, keepDataAdvice }, + ); } for (const target of call.accessWidening) { accessWidening.push({ diff --git a/packages/2-sql/9-family/src/core/migrations/statement-planning.ts b/packages/2-sql/9-family/src/core/migrations/statement-planning.ts index 5ca1f6a0b836..4148b868af89 100644 --- a/packages/2-sql/9-family/src/core/migrations/statement-planning.ts +++ b/packages/2-sql/9-family/src/core/migrations/statement-planning.ts @@ -156,8 +156,17 @@ export interface CallWithCompanions extends SingleOperationCall { readonly companions: readonly SingleOperationCall[]; } +/** How a user renames a table by hand, written for the target. */ +export interface TableRenameByHand { + /** The `renameTable` call a user writes in `migration.ts` to make `rename` by hand. */ + renderTableRename(rename: TableRename): string; + /** The SQL statements a user runs against the database to make `rename` by hand. */ + tableRenameByHand(rename: TableRename): readonly string[]; +} + /** What a target supplies for planning statements against its working schema. */ -export interface StatementPlanningTarget { +export interface StatementPlanningTarget + extends TableRenameByHand { /** The tables of the working schema as earlier statements have left it. */ tables(): SchemaTables; /** The call that renames a table, with its companions, computed against the working schema. */ @@ -166,10 +175,6 @@ export interface StatementPlanningTarget { renameColumnCall(rename: ColumnRename): TCall; /** Applies a call to the working schema. */ apply(call: TCall): void; - /** The `renameTable` call a user writes in `migration.ts` to make `rename` by hand. */ - renderTableRename(rename: TableRename): string; - /** The SQL statements a user runs against the database to make `rename` by hand. */ - tableRenameByHand(rename: TableRename): readonly string[]; } function operationsOf(call: CallWithCompanions): readonly SingleOperationCall[] { diff --git a/packages/2-sql/9-family/src/exports/control.ts b/packages/2-sql/9-family/src/exports/control.ts index 72217d9595c7..173a9f084df4 100644 --- a/packages/2-sql/9-family/src/exports/control.ts +++ b/packages/2-sql/9-family/src/exports/control.ts @@ -97,6 +97,7 @@ export type { PlannedStatements, SingleOperationCall, StatementPlanningTarget, + TableRenameByHand, } from '../core/migrations/statement-planning'; export { fieldRenameStorageEffect, diff --git a/packages/2-sql/9-family/test/operation-subjects.test.ts b/packages/2-sql/9-family/test/operation-subjects.test.ts index b8383d007b47..ff40e9697cad 100644 --- a/packages/2-sql/9-family/test/operation-subjects.test.ts +++ b/packages/2-sql/9-family/test/operation-subjects.test.ts @@ -8,10 +8,16 @@ import { subjectsOfCalls, unknownCallNames, } from '../src/core/migrations/operation-subjects'; +import type { TableRenameByHand } from '../src/core/migrations/statement-planning'; import { contractOf, renameField, renameModel } from './statement-fixtures'; const app = asNamespaceId('app'); +const byHand: TableRenameByHand = { + renderTableRename: ({ from, to }) => `this.renameTable("${from}", "${to}")`, + tableRenameByHand: ({ from, to }) => [`ALTER TABLE "${from}" RENAME TO "${to}"`], +}; + function table(name: string, storageName = name): SubjectStorage { return { storageName, table: { namespaceId: 'app', table: name, column: undefined } }; } @@ -40,6 +46,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([unchanged, losing(table('legacy')), losing(column('user', 'full_name'))], { fromContract: origin, contract: contractOf({ User: { table: 'user', fields: { id: 'id' } } }), + byHand, statements: [], }), ).toEqual({ @@ -58,7 +65,7 @@ describe('subjectsOfCalls', () => { expect( subjectsOfCalls( [{ operationCount: 3, dataLoss: [], accessWidening: [] }, losing(table('legacy'))], - { fromContract: origin, contract: origin, statements: [] }, + { fromContract: origin, contract: origin, byHand, statements: [] }, ).dataLoss, ).toEqual([ { operationIndex: 3, subject: { kind: 'model', namespaceId: app, model: 'Legacy' } }, @@ -76,6 +83,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([losing(column('account', 'display_name'))], { fromContract: from, contract: destination, + byHand, statements: [ renameModel('User', 'Account'), renameField('User', 'name', 'displayName', 'Account'), @@ -96,6 +104,7 @@ describe('subjectsOfCalls', () => { { fromContract: origin, contract: origin, + byHand, statements: [], }, ).dataLoss, @@ -110,6 +119,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([losing(table('legacy'), column('user', 'full_name'))], { fromContract: null, contract: origin, + byHand, statements: [], }).dataLoss, ).toEqual([ @@ -123,6 +133,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([losing({ storageName: 'public.mood', table: undefined })], { fromContract: origin, contract: origin, + byHand, statements: [], }).dataLoss, ).toEqual([{ operationIndex: 0, subject: { kind: 'storage', name: 'public.mood' } }]); @@ -139,7 +150,7 @@ describe('subjectsOfCalls', () => { accessWidening: [{ ...table('user'), widens: false }], }, ], - { fromContract: origin, contract: origin, statements: [] }, + { fromContract: origin, contract: origin, byHand, statements: [] }, ), ).toEqual({ dataLoss: [], @@ -165,6 +176,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([losing(table('users')), losing(column('users', 'email'))], { fromContract: sharing, contract: sharing, + byHand, statements: [], }).dataLoss.map(({ subject }) => subject), ).toEqual([ @@ -178,6 +190,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([losing(column('users', 'level'))], { fromContract: sharing, contract: sharing, + byHand, statements: [], }).dataLoss.map(({ subject }) => subject), ).toEqual([{ kind: 'field', namespaceId: app, model: 'Admin', field: 'level' }]); @@ -209,7 +222,7 @@ describe('fieldEventStorage', () => { fieldEventStorage({ namespaceId: 'app', tableName: 'user', columnName: 'nickname' }), ), ], - { fromContract: origin, contract: origin, statements: [] }, + { fromContract: origin, contract: origin, byHand, statements: [] }, ); expect(subjects.dataLoss.map(({ subject }) => subject)).toEqual([ { kind: 'field', namespaceId: app, model: 'User', field: 'name' }, @@ -221,3 +234,59 @@ describe('fieldEventStorage', () => { ).toBe('user.bio'); }); }); + +describe('subjectsOfCalls for a table whose model keeps its name', () => { + const from = contractOf({ Profile: { table: 'profiles', fields: { id: 'id' } } }); + + it('says how to rename the table by hand to keep its rows', () => { + const to = contractOf({ Profile: { table: 'people', fields: { id: 'id' } } }); + + expect( + subjectsOfCalls([losing(table('profiles'))], { + fromContract: from, + contract: to, + statements: [], + byHand, + }).dataLoss, + ).toEqual([ + { + operationIndex: 0, + subject: { kind: 'model', namespaceId: app, model: 'Profile' }, + keepDataAdvice: + 'To keep its rows, rename table "profiles" to "people" instead; a statement cannot rename the table of a model that keeps its name. With db update, rename the table in the database yourself with ALTER TABLE "profiles" RENAME TO "people", then run prisma db update. With migration plan, write the rename as a migration of its own instead: run prisma migration new --name --from , add this.renameTable("profiles", "people") to the operations of its migration.ts, from the project directory run node migrations/app//migration.ts, naming the directory migration new wrote, so it writes ops.json, then run prisma db migrate.', + }, + ]); + }); + + it('gives no advice when the destination has no model of that name', () => { + const to = contractOf({ Person: { table: 'people', fields: { id: 'id' } } }); + + expect( + subjectsOfCalls([losing(table('profiles'))], { + fromContract: from, + contract: to, + statements: [], + byHand, + }).dataLoss, + ).toEqual([ + { operationIndex: 0, subject: { kind: 'model', namespaceId: app, model: 'Profile' } }, + ]); + }); + + it('gives no advice when the new table already holds another model of the origin', () => { + const withPeople = contractOf({ + Profile: { table: 'profiles', fields: { id: 'id' } }, + Person: { table: 'people', fields: { id: 'id' } }, + }); + const to = contractOf({ Profile: { table: 'people', fields: { id: 'id' } } }); + + expect( + subjectsOfCalls([losing(table('profiles'))], { + fromContract: withPeople, + contract: to, + statements: [], + byHand, + }).dataLoss.map(({ keepDataAdvice }) => keepDataAdvice), + ).toEqual([undefined]); + }); +}); diff --git a/packages/3-targets/3-targets/postgres/src/core/migrations/planner.ts b/packages/3-targets/3-targets/postgres/src/core/migrations/planner.ts index e68e81317db8..77fcb7e4d882 100644 --- a/packages/3-targets/3-targets/postgres/src/core/migrations/planner.ts +++ b/packages/3-targets/3-targets/postgres/src/core/migrations/planner.ts @@ -86,6 +86,7 @@ import { resolveDdlSchemaForNamespaceStorage } from './resolve-ddl-schema'; import { postgresSchemaTables } from './schema-tables'; import { postgresColumnRenameCall, + postgresTableRenameByHand, postgresTableRenameCall, renameTableByHandStatements, renderRenameTableCall, @@ -474,6 +475,7 @@ export class PostgresMigrationPlanner implements MigrationPlanner<'sql', 'postgr fromContract: options.fromContract, contract: options.contract, statements: options.statements, + byHand: postgresTableRenameByHand(options.contract), }, ), ...(warnings.length > 0 ? { warnings: Object.freeze(warnings) } : {}), @@ -742,8 +744,7 @@ export class PostgresMigrationPlanner implements MigrationPlanner<'sql', 'postgr frameworkComponents: options.frameworkComponents, }), apply: (call) => working.apply(call), - renderTableRename: renderRenameTableCall, - tableRenameByHand: (rename) => renameTableByHandStatements(options.contract, rename), + ...postgresTableRenameByHand(options.contract), }, }); if (!planned.ok) return planned; diff --git a/packages/3-targets/3-targets/postgres/src/core/migrations/table-rename-calls.ts b/packages/3-targets/3-targets/postgres/src/core/migrations/table-rename-calls.ts index 638def676022..2cd3d9c2457f 100644 --- a/packages/3-targets/3-targets/postgres/src/core/migrations/table-rename-calls.ts +++ b/packages/3-targets/3-targets/postgres/src/core/migrations/table-rename-calls.ts @@ -3,6 +3,7 @@ import type { ColumnRename, MigrationOperationPolicy, TableRename, + TableRenameByHand, TableRenameRequest, } from '@internal/family-sql/control'; import type { TargetBoundComponentDescriptor } from '@internal/framework-components/components'; @@ -58,6 +59,14 @@ export function renameTableByHandStatements( ]; } +/** How a user renames a table of `contract` by hand on Postgres. */ +export function postgresTableRenameByHand(contract: Contract): TableRenameByHand { + return { + renderTableRename: renderRenameTableCall, + tableRenameByHand: (rename) => renameTableByHandStatements(contract, rename), + }; +} + function tableNode( schema: PostgresDatabaseSchemaNode, ddlSchema: string, diff --git a/packages/3-targets/3-targets/postgres/test/migrations/operation-subjects.test.ts b/packages/3-targets/3-targets/postgres/test/migrations/operation-subjects.test.ts index edaa5399aceb..dc6c8c955655 100644 --- a/packages/3-targets/3-targets/postgres/test/migrations/operation-subjects.test.ts +++ b/packages/3-targets/3-targets/postgres/test/migrations/operation-subjects.test.ts @@ -5,6 +5,7 @@ import { SqlStorage, StorageTable } from '@internal/sql-contract/types'; import { applicationDomainOf } from '@repo/test-utils'; import { describe, expect, it } from 'vitest'; import { postgresCallSubjects } from '../../src/core/migrations/operation-subjects'; +import { postgresTableRenameByHand } from '../../src/core/migrations/table-rename-calls'; import { PostgresSchema } from '../../src/core/postgres-schema'; const text = { dataType: 'pg/text', codecId: 'pg/text@1', nullable: false }; @@ -70,7 +71,12 @@ function subjectsOf(calls: readonly OpFactoryCall[], fieldEvents: readonly Field contract, fieldEvents: new Map(fieldEvents.map((fieldEvent) => [fieldEvent.call, fieldEvent])), }), - { fromContract: contract, contract, statements: [] }, + { + fromContract: contract, + contract, + byHand: postgresTableRenameByHand(contract), + statements: [], + }, ).dataLoss; } diff --git a/packages/3-targets/3-targets/sqlite/src/core/migrations/planner.ts b/packages/3-targets/3-targets/sqlite/src/core/migrations/planner.ts index 3c8b12fe76c6..5d9ba18a93e3 100644 --- a/packages/3-targets/3-targets/sqlite/src/core/migrations/planner.ts +++ b/packages/3-targets/3-targets/sqlite/src/core/migrations/planner.ts @@ -57,6 +57,7 @@ import { renameTableByHandStatements, renderRenameTableCall, sqliteColumnRenameCall, + sqliteTableRenameByHand, sqliteTableRenameCall, } from './table-rename-calls'; import { createWorkingSchema } from './working-schema'; @@ -264,6 +265,7 @@ export class SqliteMigrationPlanner fromContract: options.fromContract, contract: options.contract, statements: options.statements, + byHand: sqliteTableRenameByHand, }, ), }; @@ -303,8 +305,7 @@ export class SqliteMigrationPlanner frameworkComponents: options.frameworkComponents, }), apply: (call) => working.apply(call), - renderTableRename: renderRenameTableCall, - tableRenameByHand: renameTableByHandStatements, + ...sqliteTableRenameByHand, }, }); if (!planned.ok) return planned; diff --git a/packages/3-targets/3-targets/sqlite/src/core/migrations/table-rename-calls.ts b/packages/3-targets/3-targets/sqlite/src/core/migrations/table-rename-calls.ts index 6d8e07b3cee5..db4901e14515 100644 --- a/packages/3-targets/3-targets/sqlite/src/core/migrations/table-rename-calls.ts +++ b/packages/3-targets/3-targets/sqlite/src/core/migrations/table-rename-calls.ts @@ -1,5 +1,10 @@ import type { Contract } from '@internal/contract/types'; -import type { ColumnRename, TableRename, TableRenameRequest } from '@internal/family-sql/control'; +import type { + ColumnRename, + TableRename, + TableRenameByHand, + TableRenameRequest, +} from '@internal/family-sql/control'; import type { TargetBoundComponentDescriptor } from '@internal/framework-components/components'; import type { SqlStorage } from '@internal/sql-contract/types'; import type { SqlSchemaIR } from '@internal/sql-schema-ir/types'; @@ -20,6 +25,12 @@ export function renderRenameTableCall(rename: TableRenameRequest): string { return new RenameTableCall(rename.from, rename.to, []).renderTypeScript(); } +/** How a user renames a table by hand on SQLite. */ +export const sqliteTableRenameByHand: TableRenameByHand = { + renderTableRename: renderRenameTableCall, + tableRenameByHand: renameTableByHandStatements, +}; + /** * The call that renames a table, carrying as companions a drop and a create under the new name of * each index whose wire name derives from the old table name, since SQLite cannot rename an index. diff --git a/packages/3-targets/3-targets/sqlite/test/migrations/operation-subjects.test.ts b/packages/3-targets/3-targets/sqlite/test/migrations/operation-subjects.test.ts index 3e326cf0cce2..a5a0c59dd49e 100644 --- a/packages/3-targets/3-targets/sqlite/test/migrations/operation-subjects.test.ts +++ b/packages/3-targets/3-targets/sqlite/test/migrations/operation-subjects.test.ts @@ -6,6 +6,7 @@ import { SqlStorage, StorageTable } from '@internal/sql-contract/types'; import { applicationDomainOf } from '@repo/test-utils'; import { describe, expect, it } from 'vitest'; import { sqliteCallSubjects } from '../../src/core/migrations/operation-subjects'; +import { sqliteTableRenameByHand } from '../../src/core/migrations/table-rename-calls'; import { sqliteCreateNamespace } from '../../src/core/sqlite-unbound-database'; const text = { dataType: 'sqlite/text', codecId: 'sqlite/text@1', nullable: false }; @@ -71,7 +72,7 @@ function subjectsOf(calls: readonly OpFactoryCall[], fieldEvents: readonly Field calls, new Map(fieldEvents.map((fieldEvent) => [fieldEvent.call, fieldEvent])), ), - { fromContract: contract, contract, statements: [] }, + { fromContract: contract, contract, byHand: sqliteTableRenameByHand, statements: [] }, ).dataLoss; } diff --git a/test/integration/test/cli-journeys/table-map-change.e2e.test.ts b/test/integration/test/cli-journeys/table-map-change.e2e.test.ts new file mode 100644 index 000000000000..cd1765f712a7 --- /dev/null +++ b/test/integration/test/cli-journeys/table-map-change.e2e.test.ts @@ -0,0 +1,169 @@ +/** + * A model that keeps its name while `@@map` gives it a new table (Postgres). + * + * The Postgres twin of `table-map-change.sqlite.e2e.test.ts`. + * + * No statement can rename that table, so the question about the dropped table says how to keep its rows. Journey M1 follows the `migration plan` route as printed: a migration of its own written with `migration new`, the printed `renameTable` call and the printed `node` command, then `db migrate`. Journey M2 follows the `db update` route: the printed SQL, then `db update`. + */ + +import { readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'pathe'; +import { describe, expect, it } from 'vitest'; +import { withTempDir } from '../utils/cli-test-helpers'; +import { + type EngineCommandResult, + engineError, + type JourneyContext, + latestMigrationDirName, + latestMigrationToHash, + parseJsonOutput, + planMigrationAndSelfEmit, + runContractEmit, + runDbUpdate, + runDbVerify, + runMigrate, + runMigrationNew, + runMigrationPlan, + runPrintedSelfEmit, + setupJourney, + sql, + timeouts, + useDevDatabase, +} from '../utils/journey-test-helpers'; + +const FROM_PSL = `// use prisma-8 + +model Profile { + id Int @id + name String @unique + + @@map("profiles") +} +`; + +const TO_PSL = FROM_PSL.replace('@@map("profiles")', '@@map("people")'); + +async function emitContract(ctx: JourneyContext, psl: string, label: string): Promise { + writeFileSync(join(ctx.testDir, 'contract.prisma'), psl, 'utf-8'); + const emit = await runContractEmit(ctx); + expect(emit.exitCode, `${label}: emit: ${emit.stderr}`).toBe(0); +} + +function questionOf(result: EngineCommandResult): string { + return String(engineError(result)?.why); +} + +async function expectRowsKept(connectionString: string, label: string): Promise { + const rows = await sql(connectionString, 'SELECT id, name FROM "public"."people" ORDER BY id'); + const tables = await sql( + connectionString, + "SELECT tablename FROM pg_tables WHERE schemaname = 'public' AND tablename NOT LIKE '\\_prisma%' ORDER BY tablename", + ); + expect( + { rows: rows.rows, tables: tables.rows.map((row) => row['tablename']) }, + `${label}: the rows are in the renamed table`, + ).toEqual({ + rows: [ + { id: 1, name: 'alice' }, + { id: 2, name: 'bob' }, + ], + tables: ['people'], + }); +} + +async function seedRows(connectionString: string): Promise { + await sql( + connectionString, + `INSERT INTO "public"."profiles" (id, name) VALUES (1, 'alice'), (2, 'bob')`, + ); +} + +withTempDir(({ createTempDir }) => { + describe('Journey M1 (Postgres): the migration plan route keeps the rows', () => { + const db = useDevDatabase(); + + it( + 'writes the rename as a migration of its own, as the question prints it', + async () => { + const ctx = setupJourney({ + connectionString: db.connectionString, + createTempDir, + contractMode: 'psl', + }); + await emitContract(ctx, FROM_PSL, 'M1.01'); + const initial = await planMigrationAndSelfEmit(ctx, ['--name', 'initial']); + expect(initial.exitCode, `M1.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(ctx)).exitCode, 'M1.03: migrate initial').toBe(0); + await seedRows(db.connectionString); + const origin = latestMigrationDirName(ctx); + const originHash = latestMigrationToHash(ctx); + await emitContract(ctx, TO_PSL, 'M1.04'); + + const refused = await runMigrationPlan(ctx, ['--from', origin, '--json']); + expect(refused.exitCode, 'M1.05: the drop of profiles needs a statement').toBe(2); + const question = questionOf(refused); + expect(question, 'M1.05: the question says how to keep the rows').toContain( + 'To keep its rows, rename table "profiles" to "people" instead', + ); + + const scaffold = await runMigrationNew(ctx, ['--name', 'people', '--from', originHash]); + expect(scaffold.exitCode, `M1.06: migration new: ${scaffold.stderr}`).toBe(0); + const dir = latestMigrationDirName(ctx); + const call = /add (\.\.\.this\.renameTable\(.*?\)) to the operations/.exec(question)?.[1]; + expect(call, `M1.07: the printed renameTable call in: ${question}`).toBeDefined(); + const migrationTs = join(ctx.testDir, 'migrations/app', dir, 'migration.ts'); + const scaffolded = readFileSync(migrationTs, 'utf-8'); + const edited = scaffolded.replace(/return \[[\s\S]*?\];/, () => `return [${call}];`); + expect(edited, 'M1.07: the call is in the operations').not.toBe(scaffolded); + writeFileSync(migrationTs, edited, 'utf-8'); + const emitted = runPrintedSelfEmit(ctx, question, dir); + expect(emitted.exitCode, `M1.08: the printed node command: ${emitted.stderr}`).toBe(0); + + const applied = await runMigrate(ctx); + expect(applied.exitCode, `M1.09: db migrate: ${applied.stderr}`).toBe(0); + await expectRowsKept(db.connectionString, 'M1.10'); + const verify = await runDbVerify(ctx, ['--schema-only']); + expect(verify.exitCode, `M1.10: db verify --schema-only: ${verify.stderr}`).toBe(0); + const further = await runMigrationPlan(ctx, ['--from', dir, '--json']); + expect(further.exitCode, `M1.11: a further plan: ${further.stderr}`).toBe(0); + expect(parseJsonOutput<{ noOp: boolean }>(further).noOp, 'M1.11: it is empty').toBe(true); + }, + timeouts.spinUpPpgDev, + ); + }); + + describe('Journey M2 (Postgres): the db update route keeps the rows', () => { + const db = useDevDatabase(); + + it( + 'renames the table with the printed SQL, then db update finds nothing that loses data', + async () => { + const ctx = setupJourney({ + connectionString: db.connectionString, + createTempDir, + contractMode: 'psl', + }); + await emitContract(ctx, FROM_PSL, 'M2.01'); + const create = await runDbUpdate(ctx, ['--json']); + expect(create.exitCode, `M2.02: db update creates the table: ${create.stderr}`).toBe(0); + await seedRows(db.connectionString); + await emitContract(ctx, TO_PSL, 'M2.03'); + + const refused = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(refused.exitCode, 'M2.04: the drop of profiles needs a statement').toBe(2); + const printed = /yourself with (.*?), then run prisma db update/.exec( + questionOf(refused), + )?.[1]; + expect(printed, 'M2.04: the printed SQL').toBeDefined(); + await sql(db.connectionString, String(printed)); + + const updated = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(updated.exitCode, `M2.05: db update: ${updated.stderr}`).toBe(0); + await expectRowsKept(db.connectionString, 'M2.06'); + const verify = await runDbVerify(ctx, ['--schema-only']); + expect(verify.exitCode, `M2.06: db verify --schema-only: ${verify.stderr}`).toBe(0); + }, + timeouts.spinUpPpgDev, + ); + }); +}); diff --git a/test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts b/test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts new file mode 100644 index 000000000000..e400d32607cf --- /dev/null +++ b/test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts @@ -0,0 +1,172 @@ +/** + * A model that keeps its name while `@@map` gives it a new table (SQLite). + * + * No statement can rename that table, so the question about the dropped table says how to keep its rows. Journey M1 follows the `migration plan` route as printed: a migration of its own written with `migration new`, the printed `renameTable` call and the printed `node` command, then `db migrate`. Journey M2 follows the `db update` route: the printed SQL, then `db update`. + */ + +import { readFileSync, writeFileSync } from 'node:fs'; +import { DatabaseSync } from 'node:sqlite'; +import { join } from 'pathe'; +import { describe, expect, it } from 'vitest'; +import { withTempDir, writeProjectManifest } from '../utils/cli-test-helpers'; +import { + type EngineCommandResult, + engineError, + type JourneyContext, + latestMigrationDirName, + latestMigrationToHash, + parseJsonOutput, + planMigrationAndSelfEmit, + runContractEmit, + runDbUpdate, + runDbVerify, + runMigrate, + runMigrationNew, + runMigrationPlan, + runPrintedSelfEmit, + sqlitePslConfigFixture, + timeouts, +} from '../utils/journey-test-helpers'; + +const FROM_PSL = `// use prisma-8 + +model Profile { + id Int @id + name String @unique + + @@map("profiles") +} +`; + +const TO_PSL = FROM_PSL.replace('@@map("profiles")', '@@map("people")'); + +type SqliteJourney = JourneyContext & { readonly dbPath: string }; + +function setupSqliteJourney(createTempDir: () => string): SqliteJourney { + const testDir = createTempDir(); + const dbPath = join(testDir, 'journey.db'); + const configPath = join(testDir, 'prisma.config.ts'); + const config = readFileSync(sqlitePslConfigFixture, 'utf-8').replace('{{DB_PATH}}', () => dbPath); + writeFileSync(configPath, config, 'utf-8'); + writeProjectManifest(testDir); + return { testDir, configPath, outputDir: testDir, dbPath }; +} + +function withDatabase(dbPath: string, run: (db: DatabaseSync) => T): T { + const db = new DatabaseSync(dbPath); + try { + return run(db); + } finally { + db.close(); + } +} + +async function emitContract(ctx: JourneyContext, psl: string, label: string): Promise { + writeFileSync(join(ctx.testDir, 'contract.prisma'), psl, 'utf-8'); + const emit = await runContractEmit(ctx); + expect(emit.exitCode, `${label}: emit: ${emit.stderr}`).toBe(0); +} + +function questionOf(result: EngineCommandResult): string { + return String(engineError(result)?.why); +} + +function expectRowsKept(ctx: SqliteJourney, label: string): void { + const state = withDatabase(ctx.dbPath, (db) => ({ + rows: db.prepare(`SELECT id, name FROM "people" ORDER BY id`).all(), + tables: db + .prepare(`SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name`) + .all() + .map((row) => row['name']) + .filter((name) => !/^(_prisma|sqlite_)/.test(String(name))), + })); + expect(state, `${label}: the rows are in the renamed table`).toEqual({ + rows: [ + { id: 1, name: 'alice' }, + { id: 2, name: 'bob' }, + ], + tables: ['people'], + }); +} + +function seedRows(ctx: SqliteJourney): void { + withDatabase(ctx.dbPath, (db) => + db.exec(`INSERT INTO "profiles" (id, name) VALUES (1, 'alice'), (2, 'bob')`), + ); +} + +withTempDir(({ createTempDir }) => { + describe('Journey M1 (SQLite): the migration plan route keeps the rows', () => { + it( + 'writes the rename as a migration of its own, as the question prints it', + async () => { + const ctx = setupSqliteJourney(createTempDir); + await emitContract(ctx, FROM_PSL, 'M1.01'); + const initial = await planMigrationAndSelfEmit(ctx, ['--name', 'initial']); + expect(initial.exitCode, `M1.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(ctx)).exitCode, 'M1.03: migrate initial').toBe(0); + seedRows(ctx); + const origin = latestMigrationDirName(ctx); + const originHash = latestMigrationToHash(ctx); + await emitContract(ctx, TO_PSL, 'M1.04'); + + const refused = await runMigrationPlan(ctx, ['--from', origin, '--json']); + expect(refused.exitCode, 'M1.05: the drop of profiles needs a statement').toBe(2); + const question = questionOf(refused); + expect(question, 'M1.05: the question says how to keep the rows').toContain( + 'To keep its rows, rename table "profiles" to "people" instead', + ); + + const scaffold = await runMigrationNew(ctx, ['--name', 'people', '--from', originHash]); + expect(scaffold.exitCode, `M1.06: migration new: ${scaffold.stderr}`).toBe(0); + const dir = latestMigrationDirName(ctx); + const call = /add (\.\.\.this\.renameTable\(.*?\)) to the operations/.exec(question)?.[1]; + expect(call, `M1.07: the printed renameTable call in: ${question}`).toBeDefined(); + const migrationTs = join(ctx.testDir, 'migrations/app', dir, 'migration.ts'); + const scaffolded = readFileSync(migrationTs, 'utf-8'); + const edited = scaffolded.replace(/return \[[\s\S]*?\];/, () => `return [${call}];`); + expect(edited, 'M1.07: the call is in the operations').not.toBe(scaffolded); + writeFileSync(migrationTs, edited, 'utf-8'); + const emitted = runPrintedSelfEmit(ctx, question, dir); + expect(emitted.exitCode, `M1.08: the printed node command: ${emitted.stderr}`).toBe(0); + + const applied = await runMigrate(ctx); + expect(applied.exitCode, `M1.09: db migrate: ${applied.stderr}`).toBe(0); + expectRowsKept(ctx, 'M1.10'); + const verify = await runDbVerify(ctx, ['--schema-only']); + expect(verify.exitCode, `M1.10: db verify --schema-only: ${verify.stderr}`).toBe(0); + const further = await runMigrationPlan(ctx, ['--from', dir, '--json']); + expect(further.exitCode, `M1.11: a further plan: ${further.stderr}`).toBe(0); + expect(parseJsonOutput<{ noOp: boolean }>(further).noOp, 'M1.11: it is empty').toBe(true); + }, + timeouts.spinUpPpgDev, + ); + }); + + describe('Journey M2 (SQLite): the db update route keeps the rows', () => { + it( + 'renames the table with the printed SQL, then db update finds nothing that loses data', + async () => { + const ctx = setupSqliteJourney(createTempDir); + await emitContract(ctx, FROM_PSL, 'M2.01'); + const create = await runDbUpdate(ctx, ['--json']); + expect(create.exitCode, `M2.02: db update creates the table: ${create.stderr}`).toBe(0); + seedRows(ctx); + await emitContract(ctx, TO_PSL, 'M2.03'); + + const refused = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(refused.exitCode, 'M2.04: the drop of profiles needs a statement').toBe(2); + const sql = /yourself with (.*?), then run prisma db update/.exec(questionOf(refused))?.[1]; + expect(sql, 'M2.04: the printed SQL').toBeDefined(); + withDatabase(ctx.dbPath, (db) => db.exec(String(sql))); + + const updated = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(updated.exitCode, `M2.05: db update: ${updated.stderr}`).toBe(0); + expectRowsKept(ctx, 'M2.06'); + const verify = await runDbVerify(ctx, ['--schema-only']); + expect(verify.exitCode, `M2.06: db verify --schema-only: ${verify.stderr}`).toBe(0); + }, + timeouts.spinUpPpgDev, + ); + }); +}); From 613bf7c574e348d52bb65f78e2fead749d9a445e Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 13:06:22 +0200 Subject: [PATCH 42/57] TML-3478: migration plan refuses a MongoDB field change that leaves the storage hash unchanged A contract written in TypeScript gives its collections no validator, so field names are not in the storage hash, and a migration that only renames or removes a field would start and end at the same hash, which db migrate refuses. The MongoDB planner now refuses such a change when it plans a migration package, naming the field and saying to use db update or a hand-written dataTransform migration. db update plans without an origin and is not refused. The refusal lives in same-hash-field-changes.ts so it can be removed once TypeScript contracts get a validator. The Migration System doc states the limit. Signed-off-by: willbot Signed-off-by: Will Madden --- .../subsystems/7. Migration System.md | 2 +- .../src/core/migrations/mongo-planner.ts | 13 +- .../migrations/same-hash-field-changes.ts | 55 ++++++ ...go-planner.same-hash-field-changes.test.ts | 95 ++++++++++ .../test/mongo-statement-fixtures.ts | 5 +- .../same-hash-field-changes.mongo.e2e.test.ts | 163 ++++++++++++++++++ .../contract-events-name-removed.ts | 22 +++ .../contract-events-name-renamed.ts | 23 +++ 8 files changed, 375 insertions(+), 3 deletions(-) create mode 100644 packages/3-mongo-target/1-mongo-target/src/core/migrations/same-hash-field-changes.ts create mode 100644 packages/3-mongo-target/1-mongo-target/test/mongo-planner.same-hash-field-changes.test.ts create mode 100644 test/integration/test/cli-journeys/same-hash-field-changes.mongo.e2e.test.ts create mode 100644 test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-removed.ts create mode 100644 test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-renamed.ts diff --git a/docs/architecture docs/subsystems/7. Migration System.md b/docs/architecture docs/subsystems/7. Migration System.md index efac11bd9d37..202c4dc30fbe 100644 --- a/docs/architecture docs/subsystems/7. Migration System.md +++ b/docs/architecture docs/subsystems/7. Migration System.md @@ -226,7 +226,7 @@ The MongoDB planner does the same with collections and documents ([`mongo-statem - A model that keeps its name but changes its table cannot be renamed by a statement, because the model has the same name on both sides. Rename the table by hand with `this.renameTable(...)` in its own `migration.ts`. A field statement on such a model is refused with the same advice. - Only models and fields can be renamed. Value objects, their fields, enum values and namespaces cannot. - On MongoDB, a statement cannot rename a variant whose discriminator value changes, since its documents would keep the old value; it cannot rename `_id`, a field whose stored name MongoDB reads as a path or an operator (it contains `.` or starts with `$`), a time-series collection or a field of one, or a field inside a value object. A field removed from a model on a time-series collection fails planning with a `fieldRemovalRefused` conflict, because MongoDB cannot remove it from those documents; keeping the field in the contract plans nothing. A collection rename onto a name the database already has, or onto a name MongoDB rejects, is refused too. -- **[Placeholder: decision pending]** On a MongoDB contract without a validator, such as one written with the TypeScript contract builder, renaming or removing a field does not change the storage hash. What `migration plan` and `migrate` do with such a change is described here once it is decided. +- On a MongoDB contract that gives a collection no validator, which today is every contract written with the TypeScript contract builder, field names are not part of the storage hash. A field rename or removal alone then leaves the hash unchanged, and `db migrate` refuses a migration that starts and ends at the same hash unless it runs a `dataTransform`. So `migration plan` refuses a field rename statement there before it writes anything (`sameHashFieldChangeRefusals` in [`same-hash-field-changes.ts`](../../../packages/3-mongo-target/1-mongo-target/src/core/migrations/same-hash-field-changes.ts)), and a field removal, which no statement names, leaves the hash unchanged, so the plan finds no change. `db update` writes no migration and carries out both. A model rename that changes the collection name still works, since the collection name is in the hash. A later release gives contracts written in TypeScript the same validator as PSL contracts, and this limit goes away. #### Delete and allow statements diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts index 6341b1e4ef7a..1ce01da465c6 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts @@ -46,6 +46,7 @@ import { schemaIndexToCreateIndexOptions, } from './op-factory-call'; import { PlannerProducedMongoMigration } from './planner-produced-migration'; +import { sameHashFieldChangeRefusals } from './same-hash-field-changes'; function buildIndexLookupKey(index: MongoSchemaIndex): string { const keys = index.keys.map((k) => `${k.field}:${k.direction}`).join(','); @@ -414,7 +415,17 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> schema.collectionNames.filter((name) => destinationIR.collection(name) !== undefined), ), }); - const removalRefusals = timeSeriesRemovalRefusals(removals, schema); + const removalRefusals = [ + ...timeSeriesRemovalRefusals(removals, schema), + ...sameHashFieldChangeRefusals({ + origin: options.origin, + fromContract: fromContract ?? contract, + contract, + statements: options.statements, + callOfStatement, + removals, + }), + ]; if (removalRefusals.length > 0) return { kind: 'failure', conflicts: removalRefusals }; const result = this.planCalls({ ...options, diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/same-hash-field-changes.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/same-hash-field-changes.ts new file mode 100644 index 000000000000..cad2d5f25a42 --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/same-hash-field-changes.ts @@ -0,0 +1,55 @@ +import type { ContractWithDomain } from '@internal/contract/types'; +import { + describeMigrationStatement, + type MigrationPlannerConflict, + type PlanOrigin, + type ResolvedMigrationStatement, +} from '@internal/framework-components/control'; +import type { MongoContract } from '@internal/mongo-contract'; +import type { FieldRemoval } from './mongo-statement-planning'; +import { type OpFactoryCall, RenameFieldCall } from './op-factory-call'; + +const SAME_HASH = 'the migration would start and end at the same contract hash'; + +function why(collection: string): string { + return `The contract gives collection ${JSON.stringify(collection)} no validator, so the names of its fields are not part of the contract's storage hash; a contract written in TypeScript has none yet. A migration that only renames or removes a field would start and end at the same hash, and db migrate refuses such a migration. Use prisma db update for this change instead: it carries the change out with the same --rename or --delete and writes no migration. Or write the migration by hand: run prisma migration new --from , and carry the change out in a dataTransform operation of its migration.ts, since db migrate accepts a migration that starts and ends at the same hash when it runs a dataTransform.`; +} + +/** + * Refuses, for a migration package, each field rename and field removal of a plan whose contract + * storage hash does not change: the collection has no validator in the contract, so its field + * names are not in the hash, and `db migrate` refuses a migration that starts and ends at the same + * hash. `db update` writes no migration and plans without an origin, so it is not refused. + */ +export function sameHashFieldChangeRefusals(input: { + readonly origin: PlanOrigin | null; + readonly fromContract: ContractWithDomain; + readonly contract: MongoContract; + readonly statements: readonly ResolvedMigrationStatement[]; + readonly callOfStatement: readonly (OpFactoryCall | undefined)[]; + readonly removals: readonly FieldRemoval[]; +}): readonly MigrationPlannerConflict[] { + if (input.origin === null || input.origin.storageHash !== input.contract.storage.storageHash) { + return []; + } + const renames = input.statements.flatMap((statement, index): MigrationPlannerConflict[] => { + const call = input.callOfStatement[index]; + if (!(call instanceof RenameFieldCall)) return []; + return [ + { + kind: 'statementRefused', + summary: `Cannot ${describeMigrationStatement(statement, input.fromContract, input.contract)}: ${SAME_HASH}`, + why: why(call.collection), + refusedStatement: statement, + }, + ]; + }); + const removals = input.removals.map( + ({ call }): MigrationPlannerConflict => ({ + kind: 'fieldRemovalRefused', + summary: `Cannot remove field ${JSON.stringify(call.field)} from collection ${JSON.stringify(call.collection)}: ${SAME_HASH}`, + why: why(call.collection), + }), + ); + return [...renames, ...removals]; +} diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.same-hash-field-changes.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.same-hash-field-changes.test.ts new file mode 100644 index 000000000000..3c5645006321 --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.same-hash-field-changes.test.ts @@ -0,0 +1,95 @@ +import { describe, expect, it } from 'vitest'; +import { + contract, + plan, + planned, + refusal, + renameField, + renameModel, + schemaCollection, +} from './mongo-statement-fixtures'; + +const WHY = + 'The contract gives collection "profiles" no validator, so the names of its fields are not part of the contract\'s storage hash; a contract written in TypeScript has none yet. A migration that only renames or removes a field would start and end at the same hash, and db migrate refuses such a migration. Use prisma db update for this change instead: it carries the change out with the same --rename or --delete and writes no migration. Or write the migration by hand: run prisma migration new --from , and carry the change out in a dataTransform operation of its migration.ts, since db migrate accepts a migration that starts and ends at the same hash when it runs a dataTransform.'; + +const withName = contract( + 'same', + { Profile: { collection: 'profiles', fields: ['_id', 'name', 'nickname'] } }, + { profiles: {} }, +); +const schema = [schemaCollection('profiles')]; +const migrationOrigin = { storageHash: withName.storage.storageHash }; + +describe('MongoMigrationPlanner, field changes that leave the storage hash unchanged', () => { + it('refuses a field rename for a migration package', () => { + const to = contract( + 'same', + { Profile: { collection: 'profiles', fields: ['_id', 'fullName', 'nickname'] } }, + { profiles: {} }, + ); + expect( + refusal( + plan({ + from: withName, + to, + schema, + origin: migrationOrigin, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: + 'Cannot rename field "Profile.name" to "Profile.fullName": the migration would start and end at the same contract hash', + why: WHY, + }, + ]); + }); + + it('refuses a field removal for a migration package', () => { + const to = contract( + 'same', + { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: {} }, + ); + expect(refusal(plan({ from: withName, to, schema, origin: migrationOrigin }))).toEqual([ + { + kind: 'fieldRemovalRefused', + summary: + 'Cannot remove field "nickname" from collection "profiles": the migration would start and end at the same contract hash', + why: WHY, + }, + ]); + }); + + it('plans the same change for db update, which plans without an origin', () => { + const to = contract( + 'same', + { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: {} }, + ); + expect(planned(plan({ from: withName, to, schema })).labels).toEqual([ + 'Remove field nickname from collection profiles', + ]); + }); + + it('plans a model rename that changes the collection, whose name is in the hash', () => { + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'name', 'nickname'] } }, + { users: {} }, + ); + expect( + planned( + plan({ + from: withName, + to, + schema, + origin: migrationOrigin, + statements: [renameModel('Profile', 'User')], + }), + ).labels, + ).toEqual(['Rename collection profiles to users']); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts index b78398411db1..405b7899ac17 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts @@ -1,6 +1,7 @@ import { asNamespaceId, type ControlPolicy } from '@internal/contract/types'; import type { MigrationOperationPolicy, + PlanOrigin, ResolvedMigrationStatement, } from '@internal/framework-components/control'; import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; @@ -168,13 +169,15 @@ export function plan(input: { readonly schema: readonly MongoSchemaCollection[]; readonly statements?: readonly ResolvedMigrationStatement[]; readonly policy?: MigrationOperationPolicy; + /** The origin a migration package asserts; `db update` plans without one. */ + readonly origin?: PlanOrigin | null; }) { return new MongoMigrationPlanner().plan({ contract: input.to, schema: new MongoSchemaIR(input.schema), policy: input.policy ?? ALL_CLASSES_POLICY, fromContract: input.from, - origin: null, + origin: input.origin ?? null, statements: input.statements ?? [], frameworkComponents: [], snapshotsImportPath: '../../snapshots', diff --git a/test/integration/test/cli-journeys/same-hash-field-changes.mongo.e2e.test.ts b/test/integration/test/cli-journeys/same-hash-field-changes.mongo.e2e.test.ts new file mode 100644 index 000000000000..b6347e50b9cc --- /dev/null +++ b/test/integration/test/cli-journeys/same-hash-field-changes.mongo.e2e.test.ts @@ -0,0 +1,163 @@ +/** + * Field changes on a MongoDB contract written in TypeScript, which gives its collections no validator. + * + * Field names are then not in the contract's storage hash, so a migration that only renames or removes a field would start and end at the same hash, which `db migrate` refuses. `migration plan` refuses the change before it writes anything; `db update` carries it out. Journey H1 renames `Event.name` and follows the refusal's advice through `db update`. Journey H2 removes the field: no statement names a removal, so `migration plan` finds no change and writes nothing, and `db update` carries it out. + */ + +import { copyFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { rm } from 'node:fs/promises'; +import { join } from 'node:path'; +import { timeouts } from '@repo/test-utils'; +import { MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, afterEach, beforeAll, describe, expect, it } from 'vitest'; +import { fixtureAppDir } from '../utils/cli-test-helpers'; +import { + engineError, + getMigrationDirs, + type JourneyContext, + latestMigrationDirName, + parseJsonOutput, + runContractEmit, + runDbUpdate, + runMigrate, + runMigrationPlan, +} from '../utils/journey-test-helpers'; + +const FIXTURES_DIR = join(fixtureAppDir, 'fixtures/mongo-cli-journeys'); + +function withDatabase(baseUri: string, dbName: string): string { + const url = new URL(baseUri); + url.pathname = `/${dbName}`; + return url.toString(); +} + +function setupProject(connectionString: string): JourneyContext { + const testDir = join( + fixtureAppDir, + `test-mongo-same-hash-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + const outputDir = join(testDir, 'output'); + mkdirSync(outputDir, { recursive: true }); + mkdirSync(join(testDir, 'migrations'), { recursive: true }); + writeFileSync( + join(testDir, 'package.json'), + JSON.stringify({ + name: 'mongo-same-hash-journey', + private: true, + type: 'module', + dependencies: { '@prisma/orm-mongo': 'workspace:0.16.0' }, + }), + 'utf-8', + ); + copyFileSync(join(FIXTURES_DIR, 'contract-with-events.ts'), join(testDir, 'contract.ts')); + const config = readFileSync(join(FIXTURES_DIR, 'prisma.config.with-db.ts'), 'utf-8').replace( + /\{\{DB_URL\}\}/g, + () => connectionString, + ); + const configPath = join(testDir, 'prisma.config.ts'); + writeFileSync(configPath, config, 'utf-8'); + return { testDir, configPath, outputDir }; +} + +async function useFixture(ctx: JourneyContext, name: string, label: string): Promise { + copyFileSync(join(FIXTURES_DIR, name), join(ctx.testDir, 'contract.ts')); + const emitted = await runContractEmit(ctx); + expect(emitted.exitCode, `${label}: emit: ${emitted.stderr}`).toBe(0); +} + +describe('Journeys (MongoDB): field changes that leave the storage hash unchanged', { + timeout: timeouts.spinUpMongoMemoryServer, +}, () => { + let replSet: MongoMemoryReplSet; + let client: MongoClient; + const created = new Set(); + + beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + instanceOpts: [ + { launchTimeout: timeouts.spinUpMongoMemoryServer, storageEngine: 'wiredTiger' }, + ], + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + }, timeouts.spinUpMongoMemoryServer); + + afterEach(async () => { + for (const dir of created) { + await rm(dir, { recursive: true, force: true }).catch(() => {}); + } + created.clear(); + }); + + afterAll(async () => { + await client?.close().catch(() => {}); + await replSet?.stop().catch(() => {}); + }, timeouts.spinUpMongoMemoryServer); + + async function migratedProject(dbName: string, label: string) { + const ctx = setupProject(withDatabase(replSet.getUri(), dbName)); + created.add(ctx.testDir); + await useFixture(ctx, 'contract-with-events.ts', `${label}.01`); + const initial = await runMigrationPlan(ctx, ['--name', 'initial']); + expect(initial.exitCode, `${label}.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(ctx)).exitCode, `${label}.03: migrate initial`).toBe(0); + await client.db(dbName).collection('events').insertOne({ name: 'launch' }); + return { ctx, origin: latestMigrationDirName(ctx) }; + } + + async function events(dbName: string) { + return client + .db(dbName) + .collection('events') + .find({}, { projection: { _id: 0 } }) + .toArray(); + } + + it('H1: migration plan refuses a field rename, and db update carries it out as advised', async () => { + const dbName = 'mongo_same_hash_rename'; + const { ctx, origin } = await migratedProject(dbName, 'H1'); + await useFixture(ctx, 'contract-events-name-renamed.ts', 'H1.04'); + const rename = ['--rename', 'Event.name:Event.title']; + + const refused = await runMigrationPlan(ctx, ['--from', origin, ...rename, '--json']); + expect(refused.exitCode, 'H1.05: the rename is refused').not.toBe(0); + const error = JSON.stringify(engineError(refused)); + expect(error, 'H1.05: names the field and the reason').toContain( + 'Cannot rename field \\"Event.name\\" to \\"Event.title\\": the migration would start and end at the same contract hash', + ); + expect(error, 'H1.05: says what works').toContain('Use prisma db update for this change'); + expect(getMigrationDirs(ctx), 'H1.05: nothing written').toHaveLength(1); + + const updated = await runDbUpdate(ctx, ['--no-interactive', ...rename, '--json']); + expect(updated.exitCode, `H1.06: db update: ${updated.stderr}`).toBe(0); + expect(await events(dbName), 'H1.06: the value is under the new name').toEqual([ + { title: 'launch' }, + ]); + }); + + it('H2: migration plan finds no change in a field removal, and db update carries it out', async () => { + const dbName = 'mongo_same_hash_remove'; + const { ctx, origin } = await migratedProject(dbName, 'H2'); + await useFixture(ctx, 'contract-events-name-removed.ts', 'H2.04'); + + const unstated = await runMigrationPlan(ctx, ['--from', origin, '--json']); + expect(unstated.exitCode, `H2.05: without a statement: ${unstated.stderr}`).toBe(0); + expect( + parseJsonOutput<{ noOp: boolean }>(unstated).noOp, + 'H2.05: the hash is unchanged, so the plan sees no change', + ).toBe(true); + + expect(getMigrationDirs(ctx), 'H2.05: nothing written').toHaveLength(1); + + const updated = await runDbUpdate(ctx, [ + '--no-interactive', + '--delete', + 'Event.name', + '--json', + ]); + expect(updated.exitCode, `H2.06: db update: ${updated.stderr}`).toBe(0); + expect(await events(dbName), 'H2.06: the field is gone').toEqual([{}]); + }); +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-removed.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-removed.ts new file mode 100644 index 000000000000..2cd00ec063df --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-removed.ts @@ -0,0 +1,22 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'events', + fields: { + _id: field.objectId(), + }, +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-renamed.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-renamed.ts new file mode 100644 index 000000000000..b710a78b811d --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-renamed.ts @@ -0,0 +1,23 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'events', + fields: { + _id: field.objectId(), + title: field.string(), + }, +}); + +export const contract = defineContract({ + models: { User, Event }, +}); From cf3879e0b76aea27655d4af262e9cb0409335f1a Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 13:11:26 +0200 Subject: [PATCH 43/57] TML-3478: every instruction to run a migration.ts gives the command from the project directory node migration.ts finds prisma.config.ts only from the project directory. The migration plan placeholder summaries now name each written migration, as node migrations/app//migration.ts run from this directory; the migration new help text, the MongoDB family README, the Migration System and MongoDB Family docs, and two upgrade instructions give the same form. Signed-off-by: willbot Signed-off-by: Will Madden --- .../subsystems/10. MongoDB Family.md | 2 +- .../subsystems/7. Migration System.md | 4 ++-- .../control-api/operations/migration-plan.ts | 23 +++++++++++-------- .../3-tooling/cli/src/orm/migration/new.ts | 5 ++-- .../cli/test/orm/migration-plan.test.ts | 17 +++++++++++++- packages/2-mongo-family/9-family/README.md | 2 +- .../instructions.md | 2 +- .../instructions.md | 2 +- 8 files changed, 39 insertions(+), 18 deletions(-) diff --git a/docs/architecture docs/subsystems/10. MongoDB Family.md b/docs/architecture docs/subsystems/10. MongoDB Family.md index e5a2e95157f2..8753b8af3241 100644 --- a/docs/architecture docs/subsystems/10. MongoDB Family.md +++ b/docs/architecture docs/subsystems/10. MongoDB Family.md @@ -482,7 +482,7 @@ For polymorphic models (`@@discriminator` on the base, `@@base` on each variant) 1. Run `migration plan` (or `migration new`). The planner fills in everything it can derive from the contract diff. 2. Open `migration.ts`. Replace any `placeholder(slot)` calls with real query plans (typically inside a `dataTransform({ check, run })` operation, where the planner can scaffold the structure but not the body). Hand-author additional operations as needed. -3. Re-emit by running the file directly: `./migration.ts` or `node migration.ts`. `Migration.run(...)` detects it is the main module and rewrites `ops.json` and the `migrationId` in `migration.json`. Unfilled `placeholder(...)` calls fail here with the structured error `PN-MIG-2001` rather than producing silent nonsense ([ADR 200](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md)). +3. Re-emit by running the file from the project directory, where `prisma.config.ts` is: `node migrations/app//migration.ts`. `Migration.run(...)` detects it is the main module and rewrites `ops.json` and the `migrationId` in `migration.json`. Unfilled `placeholder(...)` calls fail here with the structured error `PN-MIG-2001` rather than producing silent nonsense ([ADR 200](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md)). 4. Commit both files. CI / `migrate` reads `ops.json` only — TypeScript is never executed at apply time. Apply-time hash verification rehashes `(manifest, ops)` and aborts on mismatch, so post-emit hand edits to `ops.json` (or any FS corruption) are caught before the runner touches the database. See [Subsystem 7 — Hash verification](7.%20Migration%20System.md#hash-verification). What the runner guarantees about the seam: diff --git a/docs/architecture docs/subsystems/7. Migration System.md b/docs/architecture docs/subsystems/7. Migration System.md index 202c4dc30fbe..090d05a348e8 100644 --- a/docs/architecture docs/subsystems/7. Migration System.md +++ b/docs/architecture docs/subsystems/7. Migration System.md @@ -84,7 +84,7 @@ flowchart LR 3. **Scaffold.** The CLI calls `plan.renderTypeScript(resolveImportSpecifier)` and writes the result to `migration.ts`. The resolver comes from the consuming project's own `package.json` (`projectImportRoot` in `@internal/cli`), so the package names in the scaffold are ones that project can resolve — the same resolver `contract.d.ts` emission gets, for the same reason: both are files the application keeps. See [ADR 242 — Public npm surface](../adrs/ADR%20242%20-%20Public%20npm%20surface%20-%20single%20@prisma%20scope%20with%20consolidated%20publish%20packages.md). The destination contract is snapshotted into the migration directory so queries are typed against the schema at planning time. See [ADR 197 — Migration packages snapshot their own contract](../adrs/ADR%20197%20-%20Migration%20packages%20snapshot%20their%20own%20contract.md). Unfilled slots (data-transform bodies the planner can't derive) use `placeholder(slot)` — a `never`-returning function that throws a structured `PN-MIG-2001` error at emit time. See [ADR 200 — Placeholder utility for scaffolded migration slots](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md). -4. **Edit.** The developer opens `migration.ts`, replaces placeholders with real queries, and iterates by running the file directly (`./migration.ts` or `node migration.ts`) thanks to the shebang and `Migration.run(...)`. +4. **Edit.** The developer opens `migration.ts`, replaces placeholders with real queries, and iterates by running the file from the project directory (`node migrations/app//migration.ts`) thanks to the shebang and `Migration.run(...)`. 5. **Emit.** Every migration package is **fully attested on disk**: `migration plan` writes `migration.ts`, `migration.json` (with a content-addressed `migrationHash`), and `ops.json` (the planned ops, or `[]` if the planner could not lower any calls because of unfilled placeholders). After the developer fills any `placeholder(...)` slots, **self-emission** rewrites `ops.json` and the `migrationHash`: running the file via `./migration.ts` (shebang) or `node migration.ts` causes `Migration.run(...)` to detect it is the main module and re-emit artifacts in-process. See [ADR 196 — In-process emit for class-flow targets](../adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md). @@ -600,7 +600,7 @@ The intermediate state ends when the developer runs the package's `migration.ts` A user who is unsure whether a migration is in the intermediate state has several signals: -- `prisma migration plan` returns `pendingPlaceholders: true` and prints a `⚠` line whose body says *"edit migration.ts then run `node migration.ts` to self-emit"*, followed by the operations that resolved and the statements applied. The same envelope is in the JSON output for tooling, with those operations in `operations`. +- `prisma migration plan` returns `pendingPlaceholders: true` and prints a `⚠` line whose body says to edit `migration.ts`, then run `node migrations/app//migration.ts` (with the directory it wrote) from the project directory to self-emit, followed by the operations that resolved and the statements applied. The same envelope is in the JSON output for tooling, with those operations in `operations`. - `ops.json` literally contains `[]`, and `migration.ts` mentions `placeholder(...)` (planner-led) or has no operation calls in its `plan()` body (custom flow). - Running `migration.ts` while a placeholder is still unfilled fails with `PN-MIG-2001` — the structured error names the slot. - `prisma db migrate` against a placeholder-blocked package will execute the lock + verify path but apply zero ops; the destination-hash post-check then surfaces `MIGRATION.POSTCHECK_FAILED` (state mismatch). This is the runner-side fallback diagnostic when an intermediate-state migration leaks past the CLI signals above. diff --git a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts index ff281ab0262d..25961f399a1b 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts @@ -115,6 +115,11 @@ type PlannerSuccess = { readonly dataLoss: readonly PlannedLoss[]; }; +function placeholderSummary(planned: string, dirs: readonly string[]): string { + const commands = dirs.map((dir) => `\`node ${join(dir, 'migration.ts')}\``).join(' and '); + return `Planned ${planned} with placeholder(s) — edit migration.ts, then run ${commands} from this directory to self-emit`; +} + /** The origin of a plan from an empty database: no models, so no statement resolves. */ const EMPTY_ORIGIN: ContractWithDomain = { domain: { namespaces: {} } }; @@ -864,8 +869,7 @@ async function executeMigrationPlanCommandInner( ...(warnings.length > 0 ? { warnings } : {}), pendingPlaceholders: true, appliedStatements: [...statementsWithoutOperations, ...deletes], - summary: - 'Planned baseline with placeholder(s) — edit migration.ts then run `node migration.ts` to self-emit', + summary: placeholderSummary('baseline', [baselineDir]), timings: { total: Date.now() - startTime }, }; return ok(result); @@ -907,21 +911,22 @@ async function executeMigrationPlanCommandInner( const baselineOps = baselineLeg.value.hasPlaceholders ? [] : baselineLeg.value.plannedOps; const deltaOps = deltaLeg.value.hasPlaceholders ? [] : deltaLeg.value.plannedOps; if (baselineLeg.value.hasPlaceholders || deltaLeg.value.hasPlaceholders) { + const baselineDir = relative(cwd, baselinePackageDir); + const deltaDir = relative(cwd, deltaPackageDir); const result: MigrationPlanResult = { ok: true, noOp: false, from: fromHash, to: toStorageHash, - dir: relative(cwd, deltaPackageDir), - baselineDir: relative(cwd, baselinePackageDir), + dir: deltaDir, + baselineDir, operations: operationSummaries(deltaLeg.value.plannedOps), baselineOperations: operationSummaries(baselineLeg.value.plannedOps), emittedExtensionDirs, ...(warnings.length > 0 ? { warnings } : {}), pendingPlaceholders: true, appliedStatements: [...deltaLeg.value.appliedStatements, ...deletes], - summary: - 'Planned baseline + migration with placeholder(s) — edit migration.ts then run `node migration.ts` to self-emit', + summary: placeholderSummary('baseline + migration', [baselineDir, deltaDir]), timings: { total: Date.now() - startTime }, }; return ok(result); @@ -1020,20 +1025,20 @@ async function executeMigrationPlanCommandInner( await writeDestinationSnapshot(toStorageHash); if (deltaLeg.value.hasPlaceholders) { + const dir = relative(cwd, packageDir); const result: MigrationPlanResult = { ok: true, noOp: false, from: fromHash, to: toStorageHash, - dir: relative(cwd, packageDir), + dir, operations: operationSummaries(deltaLeg.value.plannedOps), emittedExtensionDirs, ...(warnings.length > 0 ? { warnings } : {}), pendingPlaceholders: true, ...(fromDefaulted ? { fromDefaulted } : {}), appliedStatements: [...deltaLeg.value.appliedStatements, ...deletes], - summary: - 'Planned migration with placeholder(s) — edit migration.ts then run `node migration.ts` to self-emit', + summary: placeholderSummary('migration', [dir]), timings: { total: Date.now() - startTime }, }; return ok(result); diff --git a/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts b/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts index ff44ce7a5ad4..3eda14cba908 100644 --- a/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts +++ b/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts @@ -64,8 +64,9 @@ export function createMigrationNewCommand(createClient: CreateControlClient) { summary: 'Scaffold a new migration for manual authoring', description: 'Creates a migration package with a migration.ts file for manual authoring.\n' + - 'Write the migration body in migration.ts, then run the file with Node\n' + - '(`node migration.ts`) to self-emit ops.json and attest the package.\n' + + 'Write the migration body in migration.ts, then run the file with Node from the\n' + + 'project directory (`node migrations/app//migration.ts`) to self-emit\n' + + 'ops.json and attest the package.\n' + 'Offline — does not consult the database.', examples: [ 'migration new --name split-name', diff --git a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts index 231205172693..5b5e10d7af79 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts @@ -802,10 +802,20 @@ describe('migration plan', () => { expect(run.exitCode).toBe(0); const data = run.presented?.data as { readonly pendingPlaceholders: boolean; + readonly dir: string; + readonly baselineDir: string; + readonly summary: string; readonly operations: readonly { readonly id: string }[]; readonly appliedStatements: readonly { readonly operationIndexes: readonly number[] }[]; }; expect(data.pendingPlaceholders).toBe(true); + expect([data.baselineDir, data.dir]).toEqual([ + expect.stringMatching(/^migrations\/app\/[^/]+$/), + expect.stringMatching(/^migrations\/app\/[^/]+$/), + ]); + expect(data.summary).toBe( + `Planned baseline + migration with placeholder(s) — edit migration.ts, then run \`node ${data.baselineDir}/migration.ts\` and \`node ${data.dir}/migration.ts\` from this directory to self-emit`, + ); expect( data.appliedStatements.map(({ operationIndexes }) => operationIndexes.map((index) => data.operations[index]?.id), @@ -832,7 +842,12 @@ describe('migration plan', () => { { cwd: project.dir }, ); expect(answered.exitCode).toBe(0); - expect(answered.presented?.data).toMatchObject({ pendingPlaceholders: true }); + const data = answered.presented?.data as { readonly dir: string }; + expect(data.dir).toMatch(/^migrations\/app\/[^/]+$/); + expect(data).toMatchObject({ + pendingPlaceholders: true, + summary: `Planned migration with placeholder(s) — edit migration.ts, then run \`node ${data.dir}/migration.ts\` from this directory to self-emit`, + }); }); it('asks a question a re-plan brings up in a second batch', async () => { diff --git a/packages/2-mongo-family/9-family/README.md b/packages/2-mongo-family/9-family/README.md index cae5f029e46b..2d972838cc69 100644 --- a/packages/2-mongo-family/9-family/README.md +++ b/packages/2-mongo-family/9-family/README.md @@ -130,7 +130,7 @@ export default AddUsersCollection; MigrationCLI.run(import.meta.url, AddUsersCollection); ``` -Run `node migration.ts` to produce `ops.json` and `migration.json`. Use `--dry-run` to preview without writing. +Run `node migrations/app//migration.ts` from the project directory to produce `ops.json` and `migration.json`. Use `--dry-run` to preview without writing. ## Package Structure diff --git a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md index b5cc00c3c2ad..d76a696160b5 100644 --- a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md +++ b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md @@ -817,7 +817,7 @@ The `migration.ts` that `migration plan` writes for a new SQLite table now gives A migration operation that changes an existing default on PostgreSQL checked afterwards only that the column has a default. The old default passes that check, and the runner skips an operation whose check already passes, so the default stayed as it was and verification then failed with `MIGRATION.SCHEMA_VERIFY_FAILED`. Such an operation now has no check afterwards and always runs; setting a default twice changes nothing. -A migration package an earlier version planned keeps the old check in `ops.json`. If one changes a default and you have not applied it, write its `ops.json` again before you apply it with `prisma db migrate`. Either delete the package and run `prisma migration plan` again, or rewrite its `setDefault` call in `migration.ts` to the form [`migration-ts-column-defaults`](#migration-ts-column-defaults) shows and then run the file (`node migration.ts`). Run unchanged, the file stops with `MIGRATION.OPERATION_OPTION_REMOVED`, because its `setDefault` still passes `defaultSql`. `db update` plans again each time, so it needs nothing. +A migration package an earlier version planned keeps the old check in `ops.json`. If one changes a default and you have not applied it, write its `ops.json` again before you apply it with `prisma db migrate`. Either delete the package and run `prisma migration plan` again, or rewrite its `setDefault` call in `migration.ts` to the form [`migration-ts-column-defaults`](#migration-ts-column-defaults) shows and then run the file from the project directory (`node migrations/app//migration.ts`). Run unchanged, the file stops with `MIGRATION.OPERATION_OPTION_REMOVED`, because its `setDefault` still passes `defaultSql`. `db update` plans again each time, so it needs nothing. ## `cli-error-from-caught` diff --git a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md index dce208bdbc40..01ea70bd1ae2 100644 --- a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md +++ b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md @@ -585,7 +585,7 @@ The planner and the migration factories now class an operation as `destructive` A migration that is already applied needs nothing: `prisma db migrate` applies its `ops.json` as written, and the database ledger records the migration by its hash. -For each migration package that no database has applied yet and whose `migration.ts` calls `setNotNull` (Postgres), or `dropIndex`, `setValidation` or `collMod` without an `operationClass` (MongoDB), run its `migration.ts` again (`node migration.ts`) so that `ops.json` records the new class. This rewrites `ops.json` and the `migrationHash` in `migration.json`. Do not run it again for a package a database has applied: its new hash would no longer match the hash the database's ledger recorded for it. +For each migration package that no database has applied yet and whose `migration.ts` calls `setNotNull` (Postgres), or `dropIndex`, `setValidation` or `collMod` without an `operationClass` (MongoDB), run its `migration.ts` again from the project directory (`node migrations/app//migration.ts`) so that `ops.json` records the new class. This rewrites `ops.json` and the `migrationHash` in `migration.json`. Do not run it again for a package a database has applied: its new hash would no longer match the hash the database's ledger recorded for it. ## `migration-plan-refuses-data-loss` From 83effa73deda2f3854b28d5f4da771775e1064c1 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 14:14:33 +0200 Subject: [PATCH 44/57] TML-3478: the MongoDB family doc and the error reference give the self-emit command from the project directory Signed-off-by: willbot Signed-off-by: Will Madden --- docs/architecture docs/subsystems/10. MongoDB Family.md | 6 +++--- docs/reference/error-reference.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/architecture docs/subsystems/10. MongoDB Family.md b/docs/architecture docs/subsystems/10. MongoDB Family.md index 8753b8af3241..c952c3c0b0aa 100644 --- a/docs/architecture docs/subsystems/10. MongoDB Family.md +++ b/docs/architecture docs/subsystems/10. MongoDB Family.md @@ -8,7 +8,7 @@ This document explains what makes MongoDB different from SQL, how each differenc **Responsibilities:** MongoDB-specific contract types (`MongoContract`), execution pipeline (`MongoQueryPlan`, `MongoDriver`, `MongoRuntime`), codec registry (BSON codecs), ORM client, and contract deserialization and validation (`MongoContractSerializer.deserializeContract()`). -**Migrations are supported.** MongoDB does have server-side schema state — indexes, JSON Schema validators, collection options (capped, time-series, collation) — and Mongo participates in the migration system on the same footing as SQL targets. `prisma migration new --target mongo` and `prisma migration plan --target mongo` scaffold a runnable `migration.ts` (extending `Migration` from `@internal/family-mongo/migration`). Two paths produce identical `ops.json` artifacts — the authoritative migration contract (see [ADR 192](../adrs/ADR%20192%20-%20ops.json%20is%20the%20migration%20contract.md)): inline emit at the end of `migration plan` and self-emit via `./migration.ts` (see [ADR 193](../adrs/ADR%20193%20-%20Class-flow%20as%20the%20canonical%20migration%20authoring%20strategy.md) for the authoring strategy). Data migrations are authored as `dataTransform({ check, run })` operations whose pipelines invariant-guard the transition (see [ADR 176](../adrs/ADR%20176%20-%20Data%20migrations%20as%20invariant-guarded%20transitions.md)). See the [Migration System](7.%20Migration%20System.md) subsystem doc for the full canonical pipeline. +**Migrations are supported.** MongoDB does have server-side schema state — indexes, JSON Schema validators, collection options (capped, time-series, collation) — and Mongo participates in the migration system on the same footing as SQL targets. `prisma migration new --target mongo` and `prisma migration plan --target mongo` scaffold a runnable `migration.ts` (extending `Migration` from `@internal/family-mongo/migration`). Two paths produce identical `ops.json` artifacts — the authoritative migration contract (see [ADR 192](../adrs/ADR%20192%20-%20ops.json%20is%20the%20migration%20contract.md)): inline emit at the end of `migration plan` and self-emit by running `node migrations/app//migration.ts` from the project directory (see [ADR 193](../adrs/ADR%20193%20-%20Class-flow%20as%20the%20canonical%20migration%20authoring%20strategy.md) for the authoring strategy). Data migrations are authored as `dataTransform({ check, run })` operations whose pipelines invariant-guard the transition (see [ADR 176](../adrs/ADR%20176%20-%20Data%20migrations%20as%20invariant-guarded%20transitions.md)). See the [Migration System](7.%20Migration%20System.md) subsystem doc for the full canonical pipeline. ## At a glance @@ -435,7 +435,7 @@ Both commands write into `migrations/_/`. The destination contr The Mongo *target* (not the framework) owns the scaffolding pipeline end-to-end. The framework defines the shape of a migration package and the runner contract; everything family-specific — what factories exist, how the planner IR renders into a runnable file, what the diff produces — lives under `packages/3-mongo-target/` and `packages/2-mongo-family/`. -Concretely, the Mongo planner produces an `OpFactoryCall[]` IR (a discriminated union of frozen AST classes — `CreateIndexCall`, `DropIndexCall`, `CreateCollectionCall`, `DropCollectionCall`, `RenameCollectionCall`, `RenameFieldCall`, `UnsetFieldCall`, `CollModCall`) and feeds it to two renderers: an operation renderer that calls the matching factory functions to materialize `MongoMigrationPlanOperation[]` for `ops.json`, and a TypeScript renderer that emits a `migration.ts` calling the same factories. Both outputs derive from the same IR, so re-running `./migration.ts` after editing it produces an `ops.json` that matches the planner's output for the unedited regions. See [ADR 195 — Planner IR with two renderers](../adrs/ADR%20195%20-%20Planner%20IR%20with%20two%20renderers.md) and [ADR 194 — Plans carry their own authoring surface](../adrs/ADR%20194%20-%20Plans%20carry%20their%20own%20authoring%20surface.md). +Concretely, the Mongo planner produces an `OpFactoryCall[]` IR (a discriminated union of frozen AST classes — `CreateIndexCall`, `DropIndexCall`, `CreateCollectionCall`, `DropCollectionCall`, `RenameCollectionCall`, `RenameFieldCall`, `UnsetFieldCall`, `CollModCall`) and feeds it to two renderers: an operation renderer that calls the matching factory functions to materialize `MongoMigrationPlanOperation[]` for `ops.json`, and a TypeScript renderer that emits a `migration.ts` calling the same factories. Both outputs derive from the same IR, so running it again after editing it (`node migrations/app//migration.ts` from the project directory) produces an `ops.json` that matches the planner's output for the unedited regions. See [ADR 195 — Planner IR with two renderers](../adrs/ADR%20195%20-%20Planner%20IR%20with%20two%20renderers.md) and [ADR 194 — Plans carry their own authoring surface](../adrs/ADR%20194%20-%20Plans%20carry%20their%20own%20authoring%20surface.md). The factories the scaffold imports live in two layered modules: @@ -487,7 +487,7 @@ For polymorphic models (`@@discriminator` on the base, `@@base` on each variant) What the runner guarantees about the seam: -- **Round-trippable.** Re-running `./migration.ts` on an unedited scaffold produces byte-identical `ops.json` (the operation renderer and TypeScript renderer share the same IR). +- **Round-trippable.** Running `node migrations/app//migration.ts` again on an unedited scaffold produces byte-identical `ops.json` (the operation renderer and TypeScript renderer share the same IR). - **No drift between source and contract.** `migrationId` is content-addressed over `(manifest, ops)`, so any edit to either side that changes the canonical operations updates `migrationId` and is caught at apply time. - **Hand-authored operations are first-class.** Operations added by the developer (e.g. extra `dataTransform` calls, custom `collMod` adjustments) flow through the same factories the planner uses; there is no "planner-only" vs "user-only" distinction in `ops.json`. diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index 1c8faf40322a..0d420496ea0d 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -1590,7 +1590,7 @@ An apply command (`db migrate`/`db update`) cannot find a path through the on-di ### MIGRATION.PLANNING_FAILED -Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; renaming a variant whose discriminator value changes (the message says to keep the value with `@@base(, "")`); a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); renaming a field of a time-series collection, since MongoDB updates its documents only through the metaField (the message says to keep the field's stored name); removing a field of a time-series collection, a conflict of kind `fieldRemovalRefused` (the message says to keep the field in the contract, optional if new documents leave it out); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, replace the drop and create in its `migration.ts` with `renameCollection(...)`, run `node migration.ts` again, and apply it with `migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. +Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run `node migrations/app//migration.ts` from the project directory, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; renaming a variant whose discriminator value changes (the message says to keep the value with `@@base(, "")`); a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); renaming a field of a time-series collection, since MongoDB updates its documents only through the metaField (the message says to keep the field's stored name); removing a field of a time-series collection, a conflict of kind `fieldRemovalRefused` (the message says to keep the field in the contract, optional if new documents leave it out); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, replace the drop and create in its `migration.ts` with `renameCollection(...)`, run `node migrations/app//migration.ts` from the project directory, and apply it with `db migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. `migration plan` raises it when the planner produces no operations. The conflict's kind names the case, and its `why` is the next action. With an earlier contract (`noDatabaseChange`), the contract changed in a way the database does not see, for example after a Prisma upgrade that changes how `contract.json` records values. If you deploy with migrations, run `prisma migration new --from `, with the earlier contract's hash from the message, to write a migration with no operations, then `prisma db migrate`. If you manage the database with `db init` or `db update`, run `prisma db sign` on each database instead. With no earlier contract (`nothingToCreate`), the contract describes nothing to create; if the database needs something the contract does not describe, run `prisma migration new` and write the operations in `migration.ts` by hand. `migration new` writes a migration only to the emitted contract, so with `--to` the message says to emit the target first, and for a database managed with `db update` it names `prisma db sign `. When the migrations directory is empty and the `db` ref points at a contract that describes nothing to create (`nothingToBaseline`), there is no baseline migration to plan, and no command writes one with no operations yet; keep using `db update`. From bc2ca122b509eebf9a2c73c5d8a33f5e7611d155 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 14:15:27 +0200 Subject: [PATCH 45/57] TML-3478: a three-part statement side that reads both as namespace.Model.field and as a dotted field is refused as ambiguous Signed-off-by: willbot Signed-off-by: Will Madden --- .../statements/resolve-statements.ts | 9 +++++++-- .../statements/resolve-field-statements.test.ts | 17 +++++++++++++++++ 2 files changed, 24 insertions(+), 2 deletions(-) diff --git a/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts b/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts index 339e539685cc..111cf4225b3e 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts @@ -359,9 +359,14 @@ function readSide( if (segments.length === 3) { const [first, second, third] = segments; const qualified = readField(state, side, first, second, third); - if (qualified.ok) return qualified; const dotted = readField(state, side, undefined, first, `${second}.${third}`); - return dotted.ok ? dotted : qualified; + if (qualified.ok && dotted.ok) { + return notOk({ + reason: `"${first}.${second}.${third}" resolves both as namespace "${first}" model "${second}" field "${third}" and as model "${first}" field "${second}.${third}"`, + fix: `Give the field a stored name without "." with @map, or rename namespace "${first}" or model "${first}", so that only one reading resolves.`, + }); + } + return qualified.ok || !dotted.ok ? qualified : dotted; } const [first, second] = segments; const asModel = readModel(state, side, first, second); diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts index a084eff32c3e..3aeb3bcd26a2 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts @@ -291,4 +291,21 @@ describe('a field whose stored name holds a dot', () => { }, ]); }); + + it('refuses Model.a.b when it also resolves as namespace.Model.field', () => { + const origin = contractOf({ + a: { models: { b: { fields: ['c'] } } }, + app: { models: { a: { fields: ['b.c'] } } }, + }); + const destination = contractOf({ + a: { models: { b: { fields: ['d'] } } }, + app: { models: { a: { fields: ['b.d'] } } }, + }); + expectFailure( + resolve(['a.b.c:a.b.d'], origin, destination), + UNRESOLVED, + 'namespace "a" model "b" field "c"', + 'model "a" field "b.c"', + ); + }); }); From 2bfe86606a02e8f7f6a1d1336e1da3201dad39d5 Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 14:22:05 +0200 Subject: [PATCH 46/57] TML-3478: removing a declared time-series or clustered-index option is reported as an immutable change The planner ignored every origin key the destination does not name, so dropping metaField or a clustered index name planned nothing. It now ignores only the keys MongoDB fills in when they are absent: granularity "seconds", bucketMaxSpanSeconds and bucketRoundingSeconds for a time-series collection, and v, key, unique and the name "_id_" for a clustered index, as listCollections reports them. Signed-off-by: willbot Signed-off-by: Will Madden --- .../src/core/migrations/mongo-planner.ts | 47 ++++++++++++---- .../test/mongo-planner.statements.test.ts | 46 ++++++++++++++++ .../1-mongo-target/test/mongo-planner.test.ts | 53 +++++++++++++++++++ 3 files changed, 137 insertions(+), 9 deletions(-) diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts index 1ce01da465c6..6441b53f4e90 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts @@ -115,13 +115,41 @@ function validatorUpdateLabel( } /** - * The origin's value of an option the server fills in, keeping only the keys the destination - * names: MongoDB adds defaults such as a time-series collection's `bucketMaxSpanSeconds`, which a - * contract that leaves them out does not change. + * The keys of an option MongoDB fills in when a collection is created without them, each with a + * test for the value it fills in. `listCollections` reports a time-series collection's granularity + * as "seconds" and its bucketMaxSpanSeconds when they were not given, and bucketRoundingSeconds only + * for custom bucketing, which a contract cannot state; and a clustered index's v, key and unique, and + * its name as "_id_". */ -function asDeclared(origin: unknown, dest: unknown): unknown { - if (!isPlainObject(origin) || !isPlainObject(dest)) return origin; - return Object.fromEntries(Object.entries(origin).filter(([key]) => Object.hasOwn(dest, key))); +const SERVER_FILLED: Readonly< + Record<'timeseries' | 'clusteredIndex', Readonly boolean>>> +> = { + timeseries: { + granularity: (value) => value === 'seconds', + bucketMaxSpanSeconds: () => true, + bucketRoundingSeconds: () => true, + }, + clusteredIndex: { + v: () => true, + key: () => true, + unique: () => true, + name: (value) => value === '_id_', + }, +}; + +/** The origin's value of an option without the keys the server filled in and the destination leaves out. */ +function withoutServerFilled( + option: keyof typeof SERVER_FILLED, + origin: unknown, + dest: unknown, +): unknown { + if (!isPlainObject(origin)) return origin; + const filled = SERVER_FILLED[option]; + return Object.fromEntries( + Object.entries(origin).filter( + ([key, value]) => (isPlainObject(dest) && Object.hasOwn(dest, key)) || !filled[key]?.(value), + ), + ); } function hasImmutableOptionChange( @@ -130,14 +158,15 @@ function hasImmutableOptionChange( ): string | undefined { if (canonicalize(origin?.capped) !== canonicalize(dest?.capped)) return 'capped'; if ( - canonicalize(asDeclared(origin?.timeseries, dest?.timeseries)) !== + canonicalize(withoutServerFilled('timeseries', origin?.timeseries, dest?.timeseries)) !== canonicalize(dest?.timeseries) ) return 'timeseries'; if (canonicalize(origin?.collation) !== canonicalize(dest?.collation)) return 'collation'; if ( - canonicalize(asDeclared(origin?.clusteredIndex, dest?.clusteredIndex)) !== - canonicalize(dest?.clusteredIndex) + canonicalize( + withoutServerFilled('clusteredIndex', origin?.clusteredIndex, dest?.clusteredIndex), + ) !== canonicalize(dest?.clusteredIndex) ) return 'clusteredIndex'; return undefined; diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts index 741013bf8e24..f90594f4ff6a 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts @@ -75,4 +75,50 @@ describe('MongoMigrationPlanner, time-series options the server fills in', () => }); expect(planned(plan({ from: readings, to: readings, schema: [live] })).labels).toEqual([]); }); + + const readings = contract( + 'to', + { Reading: { collection: 'readings', fields: ['_id', 'at'] } }, + { readings: { timeField: 'at' } }, + ); + + function liveReadings(extra: Record) { + const timeseries = { timeField: 'at', bucketMaxSpanSeconds: 3600, ...extra }; + return new MongoSchemaCollection({ + name: 'readings', + options: new MongoSchemaCollectionOptions({ timeseries }), + }); + } + + it('reports a conflict when the contract removes a declared metaField', () => { + const result = plan({ + from: readings, + to: readings, + schema: [liveReadings({ metaField: 'meta', granularity: 'seconds' })], + }); + expect(result.kind === 'failure' ? result.conflicts.map(({ summary }) => summary) : []).toEqual( + [expect.stringContaining('timeseries')], + ); + }); + + it('reports a conflict when the contract removes a granularity other than the server default', () => { + const result = plan({ + from: readings, + to: readings, + schema: [liveReadings({ granularity: 'hours' })], + }); + expect(result.kind).toBe('failure'); + }); + + it('plans nothing for custom bucketing set outside the contract', () => { + expect( + planned( + plan({ + from: readings, + to: readings, + schema: [liveReadings({ bucketRoundingSeconds: 3600 })], + }), + ).labels, + ).toEqual([]); + }); }); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts index 23589dc23340..397ec529ab10 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts @@ -1347,6 +1347,59 @@ describe('MongoMigrationPlanner', () => { expect(result.conflicts.some((c) => c.summary.includes('clusteredIndex'))).toBe(true); }); + function liveClusteredIndex(name: string) { + const clusteredIndex = { v: 2, key: { _id: 1 }, unique: true, name }; + return clusteredIndex; + } + + it('plans nothing for the clustered index fields the server fills in', () => { + const contract = makeContract({ events: { options: { clusteredIndex: {} } } }); + const origin = new MongoSchemaIR([ + new MongoSchemaCollection({ + name: 'events', + options: new MongoSchemaCollectionOptions({ + clusteredIndex: liveClusteredIndex('_id_'), + }), + }), + ]); + const result = planner.plan({ + contract, + schema: origin, + policy: ALL_CLASSES_POLICY, + fromContract: null, + origin: null, + statements: [], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + expect(result.kind).toBe('success'); + }); + + it('reports conflict when the contract removes a declared clustered index name', () => { + const contract = makeContract({ events: { options: { clusteredIndex: {} } } }); + const origin = new MongoSchemaIR([ + new MongoSchemaCollection({ + name: 'events', + options: new MongoSchemaCollectionOptions({ + clusteredIndex: liveClusteredIndex('byId'), + }), + }), + ]); + const result = planner.plan({ + contract, + schema: origin, + policy: ALL_CLASSES_POLICY, + fromContract: null, + origin: null, + statements: [], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + expect( + result.kind === 'failure' ? result.conflicts.map(({ summary }) => summary) : [], + ).toEqual([expect.stringContaining('clusteredIndex')]); + }); + it('reports conflict when timeseries changes', () => { const contract = makeContract({ events: { From 1d984fcddad9f6449934553ab80db77aa8fda70d Mon Sep 17 00:00:00 2001 From: willbot Date: Mon, 12 Oct 2026 14:24:58 +0200 Subject: [PATCH 47/57] TML-3478: the advice to keep a dropped collection also covers a new collection the plan creates only with indexes The advice was given only when a createCollection call created the destination collection. A collection with indexes but no options or validator is created by its first createIndex, so its drop offered only --delete. The check now asks whether the destination collection is absent from the planned origin schema and present in the destination. Signed-off-by: willbot Signed-off-by: Will Madden --- .../src/core/migrations/mongo-planner.ts | 23 ++++++++----- .../test/mongo-planner.data-loss.test.ts | 33 +++++++++++++++++++ 2 files changed, 47 insertions(+), 9 deletions(-) diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts index 6441b53f4e90..35eb603ee7ab 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts @@ -202,17 +202,18 @@ function modelStoredIn( /** * Each collection drop of the plan, with the model whose documents it loses. When that model keeps - * its name and the plan creates its new collection, the drop says how to rename the collection by - * hand instead. + * its name and the plan brings its new collection into existence, the drop says how to rename the + * collection by hand instead. */ function collectionDrops( calls: readonly OpFactoryCall[], fromContract: ContractWithDomain | null, contract: ContractWithDomain, + collections: { readonly origin: MongoSchemaIR; readonly destination: MongoSchemaIR }, ): readonly MigrationDataLoss[] { - const created = new Set( - calls.flatMap((call) => (call instanceof CreateCollectionCall ? [call.collection] : [])), - ); + const created = (collection: string) => + collections.origin.collection(collection) === undefined && + collections.destination.collection(collection) !== undefined; return calls.flatMap((call, operationIndex): readonly MigrationDataLoss[] => { if (!(call instanceof DropCollectionCall)) return []; const model = fromContract === null ? undefined : modelStoredIn(fromContract, call.collection); @@ -222,7 +223,7 @@ function collectionDrops( const subject: MigrationSubject = { kind: 'model', ...model }; const destinationCollection = modelStoredInDestination(contract, model); return [ - destinationCollection !== undefined && created.has(destinationCollection) + destinationCollection !== undefined && created(destinationCollection) ? { operationIndex, subject, @@ -484,9 +485,13 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> const call = callOfStatement[index]; return { statement, operationIndexes: call === undefined ? [] : [calls.indexOf(call)] }; }), - dataLoss: [...collectionDrops(calls, fromContract, contract), ...fieldLoss].sort( - (a, b) => a.operationIndex - b.operationIndex, - ), + dataLoss: [ + ...collectionDrops(calls, fromContract, contract, { + origin: schema, + destination: destinationIR, + }), + ...fieldLoss, + ].sort((a, b) => a.operationIndex - b.operationIndex), accessWidening: [], }; } diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts index 3958de1fe5c1..28993bff58bb 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts @@ -6,6 +6,12 @@ import { expectDataLossMatchesDestructive } from '@repo/test-utils/data-loss-exp import { describe, expect, it } from 'vitest'; import { MongoMigrationPlanner } from '../src/core/migrations/mongo-planner'; import type { PlannerProducedMongoMigration } from '../src/core/migrations/planner-produced-migration'; +import { + contract, + planned, + plan as planStatementsFixture, + schemaCollection, +} from './mongo-statement-fixtures'; const ALL_CLASSES_POLICY: MigrationOperationPolicy = { allowedOperationClasses: ['additive', 'widening', 'destructive', 'data'], @@ -172,4 +178,31 @@ describe('MongoDB planner, data loss', () => { const planned = plan(contractWith({ User: 'users', Event: 'events' }, 'from')); expect(planned.dataLoss.map(({ keepDataAdvice }) => keepDataAdvice)).toEqual([undefined]); }); + + it('advises renaming the collection by hand when the plan creates the new collection only with its indexes', () => { + const result = planStatementsFixture({ + from: contract( + 'from', + { Event: { collection: 'events', fields: ['_id', 'name'] } }, + { events: { unique: ['name'] } }, + ), + to: contract( + 'to', + { Event: { collection: 'occasions', fields: ['_id', 'name'] } }, + { occasions: { unique: ['name'] } }, + ), + schema: [schemaCollection('events', { unique: ['name'] })], + }); + const { labels, dataLoss } = planned(result); + + expect({ + creates: labels.filter((label) => label.startsWith('Create')), + advice: dataLoss.map(({ keepDataAdvice }) => keepDataAdvice), + }).toEqual({ + creates: ['Create index on occasions (name:1)'], + advice: [ + expect.stringMatching(/^To keep its documents, rename collection "events" to "occasions"/), + ], + }); + }); }); From f6720eb4c41482c1c2e6da6c864692a8d1c2dca3 Mon Sep 17 00:00:00 2001 From: willbot Date: Tue, 13 Oct 2026 08:13:54 +0200 Subject: [PATCH 48/57] TML-3478: node migration.ts finds the config above its own directory, and advice no longer names a migrations path runMigration looked for prisma.config.ts in the working directory, though its doc said it walked up from the migration file. It now loads the nearest config above the migration directory unless --config names one. Advice in the CLI, the SQL family and the MongoDB target says to run node on the migration.ts, without the migrations/app path or a working directory. Signed-off-by: willbot Signed-off-by: Will Madden --- .../subsystems/10. MongoDB Family.md | 8 +- .../subsystems/7. Migration System.md | 4 +- docs/reference/error-reference.md | 2 +- .../3-tooling/cli/src/migration-cli.ts | 10 ++- .../3-tooling/cli/src/orm/migration/new.ts | 5 +- .../3-tooling/cli/src/utils/cli-errors.ts | 4 +- .../3-tooling/cli/test/migration-cli.test.ts | 19 ++++- .../orm/migration-plan.no-operations.test.ts | 4 +- packages/2-mongo-family/9-family/README.md | 2 +- .../src/core/migrations/operation-subjects.ts | 2 +- .../9-family/test/operation-subjects.test.ts | 2 +- .../migrations/mongo-statement-planning.ts | 2 +- .../test/mongo-planner.data-loss.test.ts | 2 +- .../mongo-planner.statements.refusals.test.ts | 2 +- .../instructions.md | 2 +- .../instructions.md | 2 +- ...ete-statements-migration.mongo.e2e.test.ts | 8 +- ...tion-file-config-lookup.sqlite.e2e.test.ts | 75 +++++++++++++++++++ .../cli-journeys/table-map-change.e2e.test.ts | 8 +- .../table-map-change.sqlite.e2e.test.ts | 28 ++++--- .../test/utils/journey-test-helpers.ts | 27 +++---- .../app/instructions.md | 7 ++ 22 files changed, 165 insertions(+), 60 deletions(-) create mode 100644 test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts diff --git a/docs/architecture docs/subsystems/10. MongoDB Family.md b/docs/architecture docs/subsystems/10. MongoDB Family.md index c952c3c0b0aa..39ea455040d6 100644 --- a/docs/architecture docs/subsystems/10. MongoDB Family.md +++ b/docs/architecture docs/subsystems/10. MongoDB Family.md @@ -8,7 +8,7 @@ This document explains what makes MongoDB different from SQL, how each differenc **Responsibilities:** MongoDB-specific contract types (`MongoContract`), execution pipeline (`MongoQueryPlan`, `MongoDriver`, `MongoRuntime`), codec registry (BSON codecs), ORM client, and contract deserialization and validation (`MongoContractSerializer.deserializeContract()`). -**Migrations are supported.** MongoDB does have server-side schema state — indexes, JSON Schema validators, collection options (capped, time-series, collation) — and Mongo participates in the migration system on the same footing as SQL targets. `prisma migration new --target mongo` and `prisma migration plan --target mongo` scaffold a runnable `migration.ts` (extending `Migration` from `@internal/family-mongo/migration`). Two paths produce identical `ops.json` artifacts — the authoritative migration contract (see [ADR 192](../adrs/ADR%20192%20-%20ops.json%20is%20the%20migration%20contract.md)): inline emit at the end of `migration plan` and self-emit by running `node migrations/app//migration.ts` from the project directory (see [ADR 193](../adrs/ADR%20193%20-%20Class-flow%20as%20the%20canonical%20migration%20authoring%20strategy.md) for the authoring strategy). Data migrations are authored as `dataTransform({ check, run })` operations whose pipelines invariant-guard the transition (see [ADR 176](../adrs/ADR%20176%20-%20Data%20migrations%20as%20invariant-guarded%20transitions.md)). See the [Migration System](7.%20Migration%20System.md) subsystem doc for the full canonical pipeline. +**Migrations are supported.** MongoDB does have server-side schema state — indexes, JSON Schema validators, collection options (capped, time-series, collation) — and Mongo participates in the migration system on the same footing as SQL targets. `prisma migration new --target mongo` and `prisma migration plan --target mongo` scaffold a runnable `migration.ts` (extending `Migration` from `@internal/family-mongo/migration`). Two paths produce identical `ops.json` artifacts — the authoritative migration contract (see [ADR 192](../adrs/ADR%20192%20-%20ops.json%20is%20the%20migration%20contract.md)): inline emit at the end of `migration plan` and self-emit by running `node /migration.ts` from any directory (see [ADR 193](../adrs/ADR%20193%20-%20Class-flow%20as%20the%20canonical%20migration%20authoring%20strategy.md) for the authoring strategy). Data migrations are authored as `dataTransform({ check, run })` operations whose pipelines invariant-guard the transition (see [ADR 176](../adrs/ADR%20176%20-%20Data%20migrations%20as%20invariant-guarded%20transitions.md)). See the [Migration System](7.%20Migration%20System.md) subsystem doc for the full canonical pipeline. ## At a glance @@ -435,7 +435,7 @@ Both commands write into `migrations/_/`. The destination contr The Mongo *target* (not the framework) owns the scaffolding pipeline end-to-end. The framework defines the shape of a migration package and the runner contract; everything family-specific — what factories exist, how the planner IR renders into a runnable file, what the diff produces — lives under `packages/3-mongo-target/` and `packages/2-mongo-family/`. -Concretely, the Mongo planner produces an `OpFactoryCall[]` IR (a discriminated union of frozen AST classes — `CreateIndexCall`, `DropIndexCall`, `CreateCollectionCall`, `DropCollectionCall`, `RenameCollectionCall`, `RenameFieldCall`, `UnsetFieldCall`, `CollModCall`) and feeds it to two renderers: an operation renderer that calls the matching factory functions to materialize `MongoMigrationPlanOperation[]` for `ops.json`, and a TypeScript renderer that emits a `migration.ts` calling the same factories. Both outputs derive from the same IR, so running it again after editing it (`node migrations/app//migration.ts` from the project directory) produces an `ops.json` that matches the planner's output for the unedited regions. See [ADR 195 — Planner IR with two renderers](../adrs/ADR%20195%20-%20Planner%20IR%20with%20two%20renderers.md) and [ADR 194 — Plans carry their own authoring surface](../adrs/ADR%20194%20-%20Plans%20carry%20their%20own%20authoring%20surface.md). +Concretely, the Mongo planner produces an `OpFactoryCall[]` IR (a discriminated union of frozen AST classes — `CreateIndexCall`, `DropIndexCall`, `CreateCollectionCall`, `DropCollectionCall`, `RenameCollectionCall`, `RenameFieldCall`, `UnsetFieldCall`, `CollModCall`) and feeds it to two renderers: an operation renderer that calls the matching factory functions to materialize `MongoMigrationPlanOperation[]` for `ops.json`, and a TypeScript renderer that emits a `migration.ts` calling the same factories. Both outputs derive from the same IR, so running it again after editing it (`node /migration.ts`) produces an `ops.json` that matches the planner's output for the unedited regions. See [ADR 195 — Planner IR with two renderers](../adrs/ADR%20195%20-%20Planner%20IR%20with%20two%20renderers.md) and [ADR 194 — Plans carry their own authoring surface](../adrs/ADR%20194%20-%20Plans%20carry%20their%20own%20authoring%20surface.md). The factories the scaffold imports live in two layered modules: @@ -482,12 +482,12 @@ For polymorphic models (`@@discriminator` on the base, `@@base` on each variant) 1. Run `migration plan` (or `migration new`). The planner fills in everything it can derive from the contract diff. 2. Open `migration.ts`. Replace any `placeholder(slot)` calls with real query plans (typically inside a `dataTransform({ check, run })` operation, where the planner can scaffold the structure but not the body). Hand-author additional operations as needed. -3. Re-emit by running the file from the project directory, where `prisma.config.ts` is: `node migrations/app//migration.ts`. `Migration.run(...)` detects it is the main module and rewrites `ops.json` and the `migrationId` in `migration.json`. Unfilled `placeholder(...)` calls fail here with the structured error `PN-MIG-2001` rather than producing silent nonsense ([ADR 200](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md)). +3. Re-emit by running the file with node from any directory: `node /migration.ts`. It loads the nearest `prisma.config.ts` above the migration's directory, or the one `--config` names. `Migration.run(...)` detects it is the main module and rewrites `ops.json` and the `migrationId` in `migration.json`. Unfilled `placeholder(...)` calls fail here with the structured error `PN-MIG-2001` rather than producing silent nonsense ([ADR 200](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md)). 4. Commit both files. CI / `migrate` reads `ops.json` only — TypeScript is never executed at apply time. Apply-time hash verification rehashes `(manifest, ops)` and aborts on mismatch, so post-emit hand edits to `ops.json` (or any FS corruption) are caught before the runner touches the database. See [Subsystem 7 — Hash verification](7.%20Migration%20System.md#hash-verification). What the runner guarantees about the seam: -- **Round-trippable.** Running `node migrations/app//migration.ts` again on an unedited scaffold produces byte-identical `ops.json` (the operation renderer and TypeScript renderer share the same IR). +- **Round-trippable.** Running `node /migration.ts` again on an unedited scaffold produces byte-identical `ops.json` (the operation renderer and TypeScript renderer share the same IR). - **No drift between source and contract.** `migrationId` is content-addressed over `(manifest, ops)`, so any edit to either side that changes the canonical operations updates `migrationId` and is caught at apply time. - **Hand-authored operations are first-class.** Operations added by the developer (e.g. extra `dataTransform` calls, custom `collMod` adjustments) flow through the same factories the planner uses; there is no "planner-only" vs "user-only" distinction in `ops.json`. diff --git a/docs/architecture docs/subsystems/7. Migration System.md b/docs/architecture docs/subsystems/7. Migration System.md index 4278e565018b..286c9a00e02a 100644 --- a/docs/architecture docs/subsystems/7. Migration System.md +++ b/docs/architecture docs/subsystems/7. Migration System.md @@ -84,7 +84,7 @@ flowchart LR 3. **Scaffold.** The CLI calls `plan.renderTypeScript(resolveImportSpecifier)` and writes the result to `migration.ts`. The resolver comes from the consuming project's own `package.json` (`projectImportRoot` in `@internal/cli`), so the package names in the scaffold are ones that project can resolve — the same resolver `contract.d.ts` emission gets, for the same reason: both are files the application keeps. See [ADR 242 — Public npm surface](../adrs/ADR%20242%20-%20Public%20npm%20surface%20-%20single%20@prisma%20scope%20with%20consolidated%20publish%20packages.md). The destination contract is snapshotted into the migration directory so queries are typed against the schema at planning time. See [ADR 197 — Migration packages snapshot their own contract](../adrs/ADR%20197%20-%20Migration%20packages%20snapshot%20their%20own%20contract.md). Unfilled slots (data-transform bodies the planner can't derive) use `placeholder(slot)` — a `never`-returning function that throws a structured `PN-MIG-2001` error at emit time. See [ADR 200 — Placeholder utility for scaffolded migration slots](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md). -4. **Edit.** The developer opens `migration.ts`, replaces placeholders with real queries, and iterates by running the file from the project directory (`node migrations/app//migration.ts`) thanks to the shebang and `Migration.run(...)`. +4. **Edit.** The developer opens `migration.ts`, replaces placeholders with real queries, and iterates by running the file with node from any directory (`node /migration.ts`; it loads the nearest `prisma.config.ts` above the migration's directory) thanks to the shebang and `Migration.run(...)`. 5. **Emit.** Every migration package is **fully attested on disk**: `migration plan` writes `migration.ts`, `migration.json` (with a content-addressed `migrationHash`), and `ops.json` (the planned ops, or `[]` if the planner could not lower any calls because of unfilled placeholders). After the developer fills any `placeholder(...)` slots, **self-emission** rewrites `ops.json` and the `migrationHash`: running the file via `./migration.ts` (shebang) or `node migration.ts` causes `Migration.run(...)` to detect it is the main module and re-emit artifacts in-process. See [ADR 196 — In-process emit for class-flow targets](../adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md). @@ -600,7 +600,7 @@ The intermediate state ends when the developer runs the package's `migration.ts` A user who is unsure whether a migration is in the intermediate state has several signals: -- `prisma migration plan` returns `pendingPlaceholders: true` and prints a `⚠` line whose body says to edit `migration.ts`, then run `node migrations/app//migration.ts` (with the directory it wrote) from the project directory to self-emit, followed by the operations that resolved and the statements applied. The same envelope is in the JSON output for tooling, with those operations in `operations`. +- `prisma migration plan` returns `pendingPlaceholders: true` and prints a `⚠` line whose body says to edit `migration.ts`, then run `node /migration.ts`, with the directory it wrote, to self-emit, followed by the operations that resolved and the statements applied. The same envelope is in the JSON output for tooling, with those operations in `operations`. - `ops.json` literally contains `[]`, and `migration.ts` mentions `placeholder(...)` (planner-led) or has no operation calls in its `plan()` body (custom flow). - Running `migration.ts` while a placeholder is still unfilled fails with `PN-MIG-2001` — the structured error names the slot. - `prisma db migrate` against a placeholder-blocked package will execute the lock + verify path but apply zero ops; the destination-hash post-check then surfaces `MIGRATION.POSTCHECK_FAILED` (state mismatch). This is the runner-side fallback diagnostic when an intermediate-state migration leaks past the CLI signals above. diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index 0d420496ea0d..b9edbfd32f0d 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -1590,7 +1590,7 @@ An apply command (`db migrate`/`db update`) cannot find a path through the on-di ### MIGRATION.PLANNING_FAILED -Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run `node migrations/app//migration.ts` from the project directory, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; renaming a variant whose discriminator value changes (the message says to keep the value with `@@base(, "")`); a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); renaming a field of a time-series collection, since MongoDB updates its documents only through the metaField (the message says to keep the field's stored name); removing a field of a time-series collection, a conflict of kind `fieldRemovalRefused` (the message says to keep the field in the contract, optional if new documents leave it out); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, replace the drop and create in its `migration.ts` with `renameCollection(...)`, run `node migrations/app//migration.ts` from the project directory, and apply it with `db migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. +Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts` with `node`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; renaming a variant whose discriminator value changes (the message says to keep the value with `@@base(, "")`); a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); renaming a field of a time-series collection, since MongoDB updates its documents only through the metaField (the message says to keep the field's stored name); removing a field of a time-series collection, a conflict of kind `fieldRemovalRefused` (the message says to keep the field in the contract, optional if new documents leave it out); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, replace the drop and create in its `migration.ts` with `renameCollection(...)`, run its `migration.ts` with `node`, and apply it with `db migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. `migration plan` raises it when the planner produces no operations. The conflict's kind names the case, and its `why` is the next action. With an earlier contract (`noDatabaseChange`), the contract changed in a way the database does not see, for example after a Prisma upgrade that changes how `contract.json` records values. If you deploy with migrations, run `prisma migration new --from `, with the earlier contract's hash from the message, to write a migration with no operations, then `prisma db migrate`. If you manage the database with `db init` or `db update`, run `prisma db sign` on each database instead. With no earlier contract (`nothingToCreate`), the contract describes nothing to create; if the database needs something the contract does not describe, run `prisma migration new` and write the operations in `migration.ts` by hand. `migration new` writes a migration only to the emitted contract, so with `--to` the message says to emit the target first, and for a database managed with `db update` it names `prisma db sign `. When the migrations directory is empty and the `db` ref points at a contract that describes nothing to create (`nothingToBaseline`), there is no baseline migration to plan, and no command writes one with no operations yet; keep using `db update`. diff --git a/packages/1-framework/3-tooling/cli/src/migration-cli.ts b/packages/1-framework/3-tooling/cli/src/migration-cli.ts index b3c43afb9c85..5ed6baa933c2 100644 --- a/packages/1-framework/3-tooling/cli/src/migration-cli.ts +++ b/packages/1-framework/3-tooling/cli/src/migration-cli.ts @@ -18,8 +18,9 @@ * 2. Parses CLI args (`--help`, `--dry-run`, `--config `) via * [clipanion](https://github.com/arcanis/clipanion). * 3. Loads the project's `prisma.config.ts` via the same - * `loadConfigForSections` the CLI commands use, walking up from the - * migration file's directory. + * `loadConfigForSections` the CLI commands use. Without `--config`, it + * loads the nearest config found walking up from the migration file's + * directory, so the file runs from any working directory. * 4. Probe-instantiates the migration class without a stack so it can read * `targetId` and verify it matches `config.target.targetId` * (`MIGRATION.TARGET_MISMATCH` on mismatch) before any stack-driven adapter @@ -46,7 +47,7 @@ import { readFileSync, realpathSync, writeFileSync } from 'node:fs'; import type { Writable } from 'node:stream'; import { fileURLToPath } from 'node:url'; -import { loadConfigForSections } from '@internal/config-loader'; +import { findNearestConfigPathForFile, loadConfigForSections } from '@internal/config-loader'; import { type CliStructuredError, errorMigrationCliInvalidConfigArg, @@ -549,7 +550,8 @@ async function runMigration( const migrationFile = fileURLToPath(importMetaUrl); const migrationDir = dirname(migrationFile); - const configResult = await loadConfigForSections(parsed.config, [ + const configPath = parsed.config ?? (await findNearestConfigPathForFile(migrationFile)); + const configResult = await loadConfigForSections(configPath, [ 'family', 'target', 'adapter', diff --git a/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts b/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts index 3eda14cba908..f65ed7fa7352 100644 --- a/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts +++ b/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts @@ -64,9 +64,8 @@ export function createMigrationNewCommand(createClient: CreateControlClient) { summary: 'Scaffold a new migration for manual authoring', description: 'Creates a migration package with a migration.ts file for manual authoring.\n' + - 'Write the migration body in migration.ts, then run the file with Node from the\n' + - 'project directory (`node migrations/app//migration.ts`) to self-emit\n' + - 'ops.json and attest the package.\n' + + 'Write the migration body in migration.ts, then run the file with Node, as this\n' + + 'command prints, to self-emit ops.json and attest the package.\n' + 'Offline — does not consult the database.', examples: [ 'migration new --name split-name', diff --git a/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts b/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts index 3f84d5773da9..93a3b9dc780b 100644 --- a/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts +++ b/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts @@ -779,8 +779,8 @@ function noOperationsConflict( summary: 'This contract describes nothing migration plan can create, so there is no first migration to plan.', why: destination.isEmitted - ? 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then, from the project directory, run `node migrations/app//migration.ts`, naming the directory it wrote, to write `ops.json`.' - : '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then, from the project directory, run `node migrations/app//migration.ts`, naming the directory it wrote, to write `ops.json`.', + ? 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then run that `migration.ts` with `node` to write `ops.json`.' + : '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then run that `migration.ts` with `node` to write `ops.json`.', }; case 'baseline': return { diff --git a/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts b/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts index 5f0753d6955f..179f168d5b50 100644 --- a/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts +++ b/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts @@ -19,13 +19,15 @@ import { structuredError } from '@internal/utils/structured-error'; import { join } from 'pathe'; import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -const { loadConfigMock, createControlStackMock } = vi.hoisted(() => ({ +const { loadConfigMock, findConfigMock, createControlStackMock } = vi.hoisted(() => ({ loadConfigMock: vi.fn(), + findConfigMock: vi.fn(async (): Promise => undefined), createControlStackMock: vi.fn(), })); vi.mock('@internal/config-loader', () => ({ loadConfigForSections: loadConfigMock, + findNearestConfigPathForFile: findConfigMock, })); vi.mock('@internal/framework-components/control', async () => { @@ -352,6 +354,21 @@ describe('MigrationCLI.run', () => { expect(loadConfigMock).toHaveBeenCalledWith('/explicit/config.ts', expect.any(Array)); }); + it('loads the config nearest the migration file when --config is not given', async () => { + findConfigMock.mockResolvedValueOnce('/project/prisma.config.ts'); + loadConfigMock.mockResolvedValue(ok(okConfig)); + createControlStackMock.mockReturnValue({ adapter: { create: () => ({}) } }); + + await MigrationCLI.run(pathToFileURL(migrationFile).href, FakeMigration, { + argv: entrypointArgv(), + stdout: new BufferStream(), + stderr: new BufferStream(), + }); + + expect(findConfigMock).toHaveBeenCalledWith(migrationFile); + expect(loadConfigMock).toHaveBeenCalledWith('/project/prisma.config.ts', expect.any(Array)); + }); + it('forwards --config= (equals form) to loadConfig', async () => { loadConfigMock.mockResolvedValue(ok(okConfig)); createControlStackMock.mockReturnValue({ adapter: { create: () => ({}) } }); diff --git a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts index 5d548651266f..05eee6b097cf 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts @@ -191,7 +191,7 @@ describe('migration plan when the planner produces no operations', () => { planningFailed({ kind: 'nothingToCreate', summary: NOTHING_TO_CREATE, - why: 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then, from the project directory, run `node migrations/app//migration.ts`, naming the directory it wrote, to write `ops.json`.', + why: 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then run that `migration.ts` with `node` to write `ops.json`.', }), ); expect({ exitCode: scaffold.exitCode, migrations: await migrationsOf(project) }).toEqual({ @@ -208,7 +208,7 @@ describe('migration plan when the planner produces no operations', () => { planningFailed({ kind: 'nothingToCreate', summary: NOTHING_TO_CREATE, - why: '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then, from the project directory, run `node migrations/app//migration.ts`, naming the directory it wrote, to write `ops.json`.', + why: '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then run that `migration.ts` with `node` to write `ops.json`.', }), ); }); diff --git a/packages/2-mongo-family/9-family/README.md b/packages/2-mongo-family/9-family/README.md index 2d972838cc69..70d34744ee34 100644 --- a/packages/2-mongo-family/9-family/README.md +++ b/packages/2-mongo-family/9-family/README.md @@ -130,7 +130,7 @@ export default AddUsersCollection; MigrationCLI.run(import.meta.url, AddUsersCollection); ``` -Run `node migrations/app//migration.ts` from the project directory to produce `ops.json` and `migration.json`. Use `--dry-run` to preview without writing. +Run `node /migration.ts` from any directory to produce `ops.json` and `migration.json`. Use `--dry-run` to preview without writing. ## Package Structure diff --git a/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts b/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts index cfec57ed17c0..86d3025604d7 100644 --- a/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts +++ b/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts @@ -187,7 +187,7 @@ function keepTableAdvice( if (storage.namespaceId !== namespaceId || storage.table === from) return undefined; if (modelsStoring(context.fromContract, namespaceId, storage.table).length > 0) return undefined; const rename = { namespaceId, from, to: storage.table }; - return `To keep its rows, rename table "${from}" to "${storage.table}" instead; a statement cannot rename the table of a model that keeps its name. With db update, rename the table in the database yourself with ${context.byHand.tableRenameByHand(rename).join('; ')}, then run prisma db update. With migration plan, write the rename as a migration of its own instead: run prisma migration new --name --from , add ${context.byHand.renderTableRename(rename)} to the operations of its migration.ts, from the project directory run node migrations/app//migration.ts, naming the directory migration new wrote, so it writes ops.json, then run prisma db migrate.`; + return `To keep its rows, rename table "${from}" to "${storage.table}" instead; a statement cannot rename the table of a model that keeps its name. With db update, rename the table in the database yourself with ${context.byHand.tableRenameByHand(rename).join('; ')}, then run prisma db update. With migration plan, write the rename as a migration of its own instead: run prisma migration new --name --from , add ${context.byHand.renderTableRename(rename)} to the operations of its migration.ts, run node on that migration.ts so it writes ops.json, then run prisma db migrate.`; } /** diff --git a/packages/2-sql/9-family/test/operation-subjects.test.ts b/packages/2-sql/9-family/test/operation-subjects.test.ts index ff40e9697cad..b9853cbb219a 100644 --- a/packages/2-sql/9-family/test/operation-subjects.test.ts +++ b/packages/2-sql/9-family/test/operation-subjects.test.ts @@ -253,7 +253,7 @@ describe('subjectsOfCalls for a table whose model keeps its name', () => { operationIndex: 0, subject: { kind: 'model', namespaceId: app, model: 'Profile' }, keepDataAdvice: - 'To keep its rows, rename table "profiles" to "people" instead; a statement cannot rename the table of a model that keeps its name. With db update, rename the table in the database yourself with ALTER TABLE "profiles" RENAME TO "people", then run prisma db update. With migration plan, write the rename as a migration of its own instead: run prisma migration new --name --from , add this.renameTable("profiles", "people") to the operations of its migration.ts, from the project directory run node migrations/app//migration.ts, naming the directory migration new wrote, so it writes ops.json, then run prisma db migrate.', + 'To keep its rows, rename table "profiles" to "people" instead; a statement cannot rename the table of a model that keeps its name. With db update, rename the table in the database yourself with ALTER TABLE "profiles" RENAME TO "people", then run prisma db update. With migration plan, write the rename as a migration of its own instead: run prisma migration new --name --from , add this.renameTable("profiles", "people") to the operations of its migration.ts, run node on that migration.ts so it writes ops.json, then run prisma db migrate.', }, ]); }); diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index e16b5a834be4..c4999ce7508b 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -397,7 +397,7 @@ function collectionChangeAdvice( } function renameCollectionByHandSteps(model: ModelCoordinate, from: string, to: string): string { - return `With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(model)} to answer its question; in its migration.ts replace the drop of "${from}" and the create of "${to}" with renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}), imported with the other operations; from the project directory, run node migrations/app//migration.ts, naming the directory migration plan wrote, so it writes ops.json again; then apply it with prisma db migrate.`; + return `With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(model)} to answer its question; in its migration.ts replace the drop of "${from}" and the create of "${to}" with renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}), imported with the other operations; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate.`; } /** diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts index 28993bff58bb..22e8a536c748 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts @@ -169,7 +169,7 @@ describe('MongoDB planner, data loss', () => { operation: 'Drop collection events', subject: { kind: 'model', namespaceId: asNamespaceId('__unbound__'), model: 'Event' }, keepDataAdvice: - 'To keep its documents, rename collection "events" to "occasions" instead; a statement cannot rename the collection of a model that keeps its name. With db update, rename the collection in the database yourself, for example db.getCollection("events").renameCollection("occasions") in mongosh, then run db update. With migration plan, plan it, passing --delete Event to answer its question; in its migration.ts replace the drop of "events" and the create of "occasions" with renameCollection("events", "occasions"), imported with the other operations; from the project directory, run node migrations/app//migration.ts, naming the directory migration plan wrote, so it writes ops.json again; then apply it with prisma db migrate.', + 'To keep its documents, rename collection "events" to "occasions" instead; a statement cannot rename the collection of a model that keeps its name. With db update, rename the collection in the database yourself, for example db.getCollection("events").renameCollection("occasions") in mongosh, then run db update. With migration plan, plan it, passing --delete Event to answer its question; in its migration.ts replace the drop of "events" and the create of "occasions" with renameCollection("events", "occasions"), imported with the other operations; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate.', }, ]); }); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts index 7bcd139151c7..d56355c9f3c5 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts @@ -199,7 +199,7 @@ describe('MongoMigrationPlanner, statements it refuses', () => { }), )[0]?.why, ).toBe( - 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question; in its migration.ts replace the drop of "profiles" and the create of "users" with renameCollection("profiles", "users"), imported with the other operations; from the project directory, run node migrations/app//migration.ts, naming the directory migration plan wrote, so it writes ops.json again; then apply it with prisma db migrate. Then change the field and run this statement again.', + 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question; in its migration.ts replace the drop of "profiles" and the create of "users" with renameCollection("profiles", "users"), imported with the other operations; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate. Then change the field and run this statement again.', ); }); }); diff --git a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md index b17bde1936a8..cf39ebd8b754 100644 --- a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md +++ b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md @@ -820,7 +820,7 @@ The `migration.ts` that `migration plan` writes for a new SQLite table now gives A migration operation that changes an existing default on PostgreSQL checked afterwards only that the column has a default. The old default passes that check, and the runner skips an operation whose check already passes, so the default stayed as it was and verification then failed with `MIGRATION.SCHEMA_VERIFY_FAILED`. Such an operation now has no check afterwards and always runs; setting a default twice changes nothing. -A migration package an earlier version planned keeps the old check in `ops.json`. If one changes a default and you have not applied it, write its `ops.json` again before you apply it with `prisma db migrate`. Either delete the package and run `prisma migration plan` again, or rewrite its `setDefault` call in `migration.ts` to the form [`migration-ts-column-defaults`](#migration-ts-column-defaults) shows and then run the file from the project directory (`node migrations/app//migration.ts`). Run unchanged, the file stops with `MIGRATION.OPERATION_OPTION_REMOVED`, because its `setDefault` still passes `defaultSql`. `db update` plans again each time, so it needs nothing. +A migration package an earlier version planned keeps the old check in `ops.json`. If one changes a default and you have not applied it, write its `ops.json` again before you apply it with `prisma db migrate`. Either delete the package and run `prisma migration plan` again, or rewrite its `setDefault` call in `migration.ts` to the form [`migration-ts-column-defaults`](#migration-ts-column-defaults) shows and then run the file with node from the directory that holds `prisma.config.ts` (`node /migration.ts`). Run unchanged, the file stops with `MIGRATION.OPERATION_OPTION_REMOVED`, because its `setDefault` still passes `defaultSql`. `db update` plans again each time, so it needs nothing. ## `cli-error-from-caught` diff --git a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md index 01ea70bd1ae2..7dd66236c1aa 100644 --- a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md +++ b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md @@ -585,7 +585,7 @@ The planner and the migration factories now class an operation as `destructive` A migration that is already applied needs nothing: `prisma db migrate` applies its `ops.json` as written, and the database ledger records the migration by its hash. -For each migration package that no database has applied yet and whose `migration.ts` calls `setNotNull` (Postgres), or `dropIndex`, `setValidation` or `collMod` without an `operationClass` (MongoDB), run its `migration.ts` again from the project directory (`node migrations/app//migration.ts`) so that `ops.json` records the new class. This rewrites `ops.json` and the `migrationHash` in `migration.json`. Do not run it again for a package a database has applied: its new hash would no longer match the hash the database's ledger recorded for it. +For each migration package that no database has applied yet and whose `migration.ts` calls `setNotNull` (Postgres), or `dropIndex`, `setValidation` or `collMod` without an `operationClass` (MongoDB), run its `migration.ts` again with node from the directory that holds `prisma.config.ts` (`node /migration.ts`) so that `ops.json` records the new class. This rewrites `ops.json` and the `migrationHash` in `migration.json`. Do not run it again for a package a database has applied: its new hash would no longer match the hash the database's ledger recorded for it. ## `migration-plan-refuses-data-loss` diff --git a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts index dabb30a3bdc3..e6e9f25ad2ca 100644 --- a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts +++ b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts @@ -8,7 +8,7 @@ import { copyFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; import { rm } from 'node:fs/promises'; -import { join } from 'node:path'; +import { dirname, join } from 'node:path'; import { timeouts } from '@repo/test-utils'; import { MongoClient } from 'mongodb'; import { MongoMemoryReplSet } from 'mongodb-memory-server'; @@ -331,7 +331,7 @@ describe('Journeys (MongoDB): statements that consent to data loss', { ]); expect(refused.exitCode, 'S5.05: the field rename is refused').not.toBe(0); expect(JSON.stringify(engineError(refused)?.meta), 'S5.05: the advice').toContain( - 'from the project directory, run node migrations/app//migration.ts', + 'run node on that migration.ts so it writes ops.json again', ); useFixture(ctx, 'contract-events-remapped.ts'); @@ -369,8 +369,8 @@ describe('Journeys (MongoDB): statements that consent to data loss', { ); expect(edited, `S5.08: the drop is gone:\n${edited}`).not.toMatch(/dropCollection\(['"]events/); writeFileSync(migrationTs, edited, 'utf-8'); - const emitted = runPrintedSelfEmit(ctx, String(engineError(asked)?.why), remapDir); - expect(emitted.exitCode, `S5.09: the printed node command: ${emitted.stderr}`).toBe(0); + const emitted = runPrintedSelfEmit(String(engineError(asked)?.why), dirname(migrationTs)); + expect(emitted.exitCode, `S5.09: node on migration.ts, as printed: ${emitted.stderr}`).toBe(0); const applyRemap = await runMigrate(ctx); expect(applyRemap.exitCode, `S5.10: migrate: ${applyRemap.stderr}`).toBe(0); diff --git a/test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts b/test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts new file mode 100644 index 000000000000..a7a8c002cbac --- /dev/null +++ b/test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts @@ -0,0 +1,75 @@ +/** + * `node migration.ts` finds the project's config by walking up from the migration's own directory, so it runs from any working directory, including when `migrations.dir` is not the default. + */ + +import { spawnSync } from 'node:child_process'; +import { readdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { join, relative } from 'pathe'; +import { describe, expect, it } from 'vitest'; +import { withTempDir, writeProjectManifest } from '../utils/cli-test-helpers'; +import { + type JourneyContext, + runContractEmit, + runMigrationPlan, + sqlitePslConfigFixture, + timeouts, +} from '../utils/journey-test-helpers'; + +const PSL = `// use prisma-8 + +model Profile { + id Int @id + name String +} +`; + +function setupProject(createTempDir: () => string): JourneyContext { + const testDir = createTempDir(); + const configPath = join(testDir, 'prisma.config.ts'); + const config = readFileSync(sqlitePslConfigFixture, 'utf-8') + .replace('{{DB_PATH}}', () => join(testDir, 'journey.db')) + .replace("migrations: { dir: 'migrations' }", "migrations: { dir: './db/migrations' }"); + writeFileSync(configPath, config, 'utf-8'); + writeFileSync(join(testDir, 'contract.prisma'), PSL, 'utf-8'); + writeProjectManifest(testDir); + return { testDir, configPath, outputDir: testDir }; +} + +function runNode(file: string, cwd: string) { + const run = spawnSync(process.execPath, [file], { cwd, encoding: 'utf-8' }); + return { exitCode: run.status, stdout: run.stdout, stderr: run.stderr }; +} + +withTempDir(({ createTempDir }) => { + describe('node migration.ts', () => { + it( + 'finds the config from the migration directory and from the project directory', + async () => { + const ctx = setupProject(createTempDir); + const emitted = await runContractEmit(ctx); + expect(emitted.exitCode, `emit: ${emitted.stderr}`).toBe(0); + const planned = await runMigrationPlan(ctx, ['--name', 'initial']); + expect(planned.exitCode, `plan: ${planned.stderr}`).toBe(0); + const appDir = join(ctx.testDir, 'db/migrations/app'); + const [dir] = readdirSync(appDir).filter((name) => name !== 'refs'); + const migrationDir = join(appDir, String(dir)); + + const fromMigrationDir = runNode('migration.ts', migrationDir); + expect(fromMigrationDir, 'from the migration directory').toMatchObject({ + exitCode: 0, + stdout: expect.stringContaining('Wrote ops.json + migration.json'), + }); + + const fromProjectDir = runNode( + relative(ctx.testDir, join(migrationDir, 'migration.ts')), + ctx.testDir, + ); + expect(fromProjectDir, 'from the project directory').toMatchObject({ + exitCode: 0, + stdout: expect.stringContaining('Wrote ops.json + migration.json'), + }); + }, + timeouts.spinUpPpgDev, + ); + }); +}); diff --git a/test/integration/test/cli-journeys/table-map-change.e2e.test.ts b/test/integration/test/cli-journeys/table-map-change.e2e.test.ts index cd1765f712a7..b1116400736a 100644 --- a/test/integration/test/cli-journeys/table-map-change.e2e.test.ts +++ b/test/integration/test/cli-journeys/table-map-change.e2e.test.ts @@ -7,7 +7,7 @@ */ import { readFileSync, writeFileSync } from 'node:fs'; -import { join } from 'pathe'; +import { dirname, join } from 'pathe'; import { describe, expect, it } from 'vitest'; import { withTempDir } from '../utils/cli-test-helpers'; import { @@ -116,8 +116,10 @@ withTempDir(({ createTempDir }) => { const edited = scaffolded.replace(/return \[[\s\S]*?\];/, () => `return [${call}];`); expect(edited, 'M1.07: the call is in the operations').not.toBe(scaffolded); writeFileSync(migrationTs, edited, 'utf-8'); - const emitted = runPrintedSelfEmit(ctx, question, dir); - expect(emitted.exitCode, `M1.08: the printed node command: ${emitted.stderr}`).toBe(0); + const emitted = runPrintedSelfEmit(question, dirname(migrationTs)); + expect(emitted.exitCode, `M1.08: node on migration.ts, as printed: ${emitted.stderr}`).toBe( + 0, + ); const applied = await runMigrate(ctx); expect(applied.exitCode, `M1.09: db migrate: ${applied.stderr}`).toBe(0); diff --git a/test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts b/test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts index e400d32607cf..d236edd704a1 100644 --- a/test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts +++ b/test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts @@ -1,12 +1,12 @@ /** * A model that keeps its name while `@@map` gives it a new table (SQLite). * - * No statement can rename that table, so the question about the dropped table says how to keep its rows. Journey M1 follows the `migration plan` route as printed: a migration of its own written with `migration new`, the printed `renameTable` call and the printed `node` command, then `db migrate`. Journey M2 follows the `db update` route: the printed SQL, then `db update`. + * No statement can rename that table, so the question about the dropped table says how to keep its rows. Journey M1 follows the `migration plan` route as printed, in a project whose `migrations.dir` is `./db/migrations`: a migration of its own written with `migration new`, the printed `renameTable` call, node run on its `migration.ts`, then `db migrate`. Journey M2 follows the `db update` route: the printed SQL, then `db update`. */ import { readFileSync, writeFileSync } from 'node:fs'; import { DatabaseSync } from 'node:sqlite'; -import { join } from 'pathe'; +import { dirname, join } from 'pathe'; import { describe, expect, it } from 'vitest'; import { withTempDir, writeProjectManifest } from '../utils/cli-test-helpers'; import { @@ -42,14 +42,22 @@ const TO_PSL = FROM_PSL.replace('@@map("profiles")', '@@map("people")'); type SqliteJourney = JourneyContext & { readonly dbPath: string }; -function setupSqliteJourney(createTempDir: () => string): SqliteJourney { +function setupSqliteJourney( + createTempDir: () => string, + migrationsDir = 'migrations', +): SqliteJourney { const testDir = createTempDir(); const dbPath = join(testDir, 'journey.db'); const configPath = join(testDir, 'prisma.config.ts'); - const config = readFileSync(sqlitePslConfigFixture, 'utf-8').replace('{{DB_PATH}}', () => dbPath); + const config = readFileSync(sqlitePslConfigFixture, 'utf-8') + .replace('{{DB_PATH}}', () => dbPath) + .replace( + "migrations: { dir: 'migrations' }", + () => `migrations: { dir: './${migrationsDir}' }`, + ); writeFileSync(configPath, config, 'utf-8'); writeProjectManifest(testDir); - return { testDir, configPath, outputDir: testDir, dbPath }; + return { testDir, configPath, outputDir: testDir, dbPath, migrationsDir }; } function withDatabase(dbPath: string, run: (db: DatabaseSync) => T): T { @@ -100,7 +108,7 @@ withTempDir(({ createTempDir }) => { it( 'writes the rename as a migration of its own, as the question prints it', async () => { - const ctx = setupSqliteJourney(createTempDir); + const ctx = setupSqliteJourney(createTempDir, 'db/migrations'); await emitContract(ctx, FROM_PSL, 'M1.01'); const initial = await planMigrationAndSelfEmit(ctx, ['--name', 'initial']); expect(initial.exitCode, `M1.02: plan initial: ${initial.stderr}`).toBe(0); @@ -122,13 +130,15 @@ withTempDir(({ createTempDir }) => { const dir = latestMigrationDirName(ctx); const call = /add (\.\.\.this\.renameTable\(.*?\)) to the operations/.exec(question)?.[1]; expect(call, `M1.07: the printed renameTable call in: ${question}`).toBeDefined(); - const migrationTs = join(ctx.testDir, 'migrations/app', dir, 'migration.ts'); + const migrationTs = join(ctx.testDir, 'db/migrations/app', dir, 'migration.ts'); const scaffolded = readFileSync(migrationTs, 'utf-8'); const edited = scaffolded.replace(/return \[[\s\S]*?\];/, () => `return [${call}];`); expect(edited, 'M1.07: the call is in the operations').not.toBe(scaffolded); writeFileSync(migrationTs, edited, 'utf-8'); - const emitted = runPrintedSelfEmit(ctx, question, dir); - expect(emitted.exitCode, `M1.08: the printed node command: ${emitted.stderr}`).toBe(0); + const emitted = runPrintedSelfEmit(question, dirname(migrationTs)); + expect(emitted.exitCode, `M1.08: node on migration.ts, as printed: ${emitted.stderr}`).toBe( + 0, + ); const applied = await runMigrate(ctx); expect(applied.exitCode, `M1.09: db migrate: ${applied.stderr}`).toBe(0); diff --git a/test/integration/test/utils/journey-test-helpers.ts b/test/integration/test/utils/journey-test-helpers.ts index 4f4aa9d702c3..2adbb72a89c4 100644 --- a/test/integration/test/utils/journey-test-helpers.ts +++ b/test/integration/test/utils/journey-test-helpers.ts @@ -55,6 +55,8 @@ export interface JourneyContext { testDir: string; configPath: string; outputDir: string; + /** The config's `migrations.dir`, relative to `testDir`, when it is not `migrations`. */ + migrationsDir?: string; } // Re-export timeouts so journey tests can import from a single module. @@ -404,7 +406,7 @@ export async function authorMigration( throw new Error('authorMigration: the scaffold has no operations array to fill'); } writeFileSync(migrationTsPath, authored, 'utf-8'); - const emit = await selfEmitMigration(ctx, ['--dir', `migrations/app/${dirName}`]); + const emit = await selfEmitMigration(ctx, ['--dir', join(appMigrationsDir(ctx), dirName)]); return { dirName, emit }; } @@ -552,26 +554,17 @@ export async function selfEmitMigration( } /** - * Finds the `node migrations/app//migration.ts` command `message` prints, names the - * migration directory in it, and runs it as a user would type it in the project directory. + * Follows a message that says to run a migration's `migration.ts` with node: runs `node migration.ts` inside `packageDir`, the migration package directory, as a user would. */ -export function runPrintedSelfEmit( - ctx: JourneyContext, - message: string, - migrationDir: string, -): CommandResult { - const printed = /\bnode (migrations\/app\/\/migration\.ts)/.exec(message)?.[1]; - if (printed === undefined) { +export function runPrintedSelfEmit(message: string, packageDir: string): CommandResult { + if (!/\bnode on that migration\.ts\b/.test(message)) { return { exitCode: -1, stdout: '', - stderr: `the message prints no node migrations/app//migration.ts command: ${message}`, + stderr: `the message does not say to run node on that migration.ts: ${message}`, }; } - const run = spawnSync(process.execPath, [printed.replace('', migrationDir)], { - cwd: ctx.testDir, - encoding: 'utf-8', - }); + const run = spawnSync(process.execPath, ['migration.ts'], { cwd: packageDir, encoding: 'utf-8' }); return { exitCode: run.status ?? -1, stdout: run.stdout, stderr: run.stderr }; } @@ -592,7 +585,7 @@ export async function planMigrationAndSelfEmit( if (planResult.exitCode !== 0) return planResult; const latest = getLatestMigrationDir(ctx); if (!latest) return planResult; - const emitResult = await selfEmitMigration(ctx, ['--dir', `migrations/app/${latest}`]); + const emitResult = await selfEmitMigration(ctx, ['--dir', join(appMigrationsDir(ctx), latest)]); if (emitResult.exitCode !== 0) { return { ...planResult, @@ -770,7 +763,7 @@ export function migrationStatusAppSpace( * test root (`migrations/app/`). */ function appMigrationsDir(ctx: JourneyContext): string { - return join(ctx.testDir, 'migrations', 'app'); + return join(ctx.testDir, ctx.migrationsDir ?? 'migrations', 'app'); } /** diff --git a/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md b/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md index 54c2b51d5286..866e529a26eb 100644 --- a/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md +++ b/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md @@ -23,6 +23,9 @@ changes: glob: "**/*.{ts,mts,cts,js,mjs,cjs}" matches: - '--delete loses its rows' + - id: migration-ts-finds-its-config + summary: | + Running a `migration.ts` with node loads the nearest `prisma.config.ts` above the migration's own directory, so `node migration.ts` works from any directory. It used to look only in the working directory, and failed with `CONFIG.FILE_NOT_FOUND` anywhere else unless `--config` named the file. --- # MongoDB carries out `--rename` and `--delete` @@ -61,3 +64,7 @@ A repeated `db update` on a database whose contract declares a time-series colle ## `storage-question-wording` A test that asserts the text of a data-loss question about data named by its storage name expects `--delete loses that data` where it expected `--delete loses its rows`. + +## `migration-ts-finds-its-config` + +`node migration.ts`, run inside the migration's directory, now finds the project's `prisma.config.ts`, as does `node /migration.ts` run from any other directory. A command that runs the file from the project directory keeps working unchanged. A script that passed `--config` only because it ran the file from another directory can drop it; `--config` still overrides the lookup. Messages that tell you to run a `migration.ts` now say to run it with node, without a path or a working directory. From 92eb09350795c5d5d5ef9acf9d75abaace93e27f Mon Sep 17 00:00:00 2001 From: willbot Date: Tue, 13 Oct 2026 08:16:01 +0200 Subject: [PATCH 49/57] TML-3478: the advice to keep a collection's documents puts renameCollection first and works when no create exists When the new collection is created only by its indexes, the plan has no createCollection to replace, and renameCollection placed where the drop was fails its precheck. The advice now says to make renameCollection the first operation, delete the drop, and delete the create if there is one. A journey follows it for a collection with a unique index, through migration plan and db migrate. Signed-off-by: willbot Signed-off-by: Will Madden --- docs/reference/error-reference.md | 2 +- .../migrations/mongo-statement-planning.ts | 2 +- .../test/mongo-planner.data-loss.test.ts | 4 +- .../mongo-planner.statements.refusals.test.ts | 2 +- ...ete-statements-migration.mongo.e2e.test.ts | 21 ++- .../keep-documents-advice.mongo.e2e.test.ts | 157 ++++++++++++++++++ .../contract-indexed-events-remapped.ts | 24 +++ .../contract-with-indexed-events.ts | 24 +++ .../test/utils/journey-test-helpers.ts | 46 +++++ 9 files changed, 266 insertions(+), 16 deletions(-) create mode 100644 test/integration/test/cli-journeys/keep-documents-advice.mongo.e2e.test.ts create mode 100644 test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-indexed-events-remapped.ts create mode 100644 test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-with-indexed-events.ts diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index b9edbfd32f0d..a7cda78fd3fc 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -1590,7 +1590,7 @@ An apply command (`db migrate`/`db update`) cannot find a path through the on-di ### MIGRATION.PLANNING_FAILED -Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts` with `node`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; renaming a variant whose discriminator value changes (the message says to keep the value with `@@base(, "")`); a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); renaming a field of a time-series collection, since MongoDB updates its documents only through the metaField (the message says to keep the field's stored name); removing a field of a time-series collection, a conflict of kind `fieldRemovalRefused` (the message says to keep the field in the contract, optional if new documents leave it out); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, replace the drop and create in its `migration.ts` with `renameCollection(...)`, run its `migration.ts` with `node`, and apply it with `db migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. +Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts` with `node`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; renaming a variant whose discriminator value changes (the message says to keep the value with `@@base(, "")`); a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); renaming a field of a time-series collection, since MongoDB updates its documents only through the metaField (the message says to keep the field's stored name); removing a field of a time-series collection, a conflict of kind `fieldRemovalRefused` (the message says to keep the field in the contract, optional if new documents leave it out); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, make `renameCollection(...)` the first operation in its `migration.ts` and delete the drop, and the create if there is one, run its `migration.ts` with `node`, and apply it with `db migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. `migration plan` raises it when the planner produces no operations. The conflict's kind names the case, and its `why` is the next action. With an earlier contract (`noDatabaseChange`), the contract changed in a way the database does not see, for example after a Prisma upgrade that changes how `contract.json` records values. If you deploy with migrations, run `prisma migration new --from `, with the earlier contract's hash from the message, to write a migration with no operations, then `prisma db migrate`. If you manage the database with `db init` or `db update`, run `prisma db sign` on each database instead. With no earlier contract (`nothingToCreate`), the contract describes nothing to create; if the database needs something the contract does not describe, run `prisma migration new` and write the operations in `migration.ts` by hand. `migration new` writes a migration only to the emitted contract, so with `--to` the message says to emit the target first, and for a database managed with `db update` it names `prisma db sign `. When the migrations directory is empty and the `db` ref points at a contract that describes nothing to create (`nothingToBaseline`), there is no baseline migration to plan, and no command writes one with no operations yet; keep using `db update`. diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts index c4999ce7508b..5667143d6dea 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -397,7 +397,7 @@ function collectionChangeAdvice( } function renameCollectionByHandSteps(model: ModelCoordinate, from: string, to: string): string { - return `With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(model)} to answer its question; in its migration.ts replace the drop of "${from}" and the create of "${to}" with renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}), imported with the other operations; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate.`; + return `With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(model)} to answer its question; in its migration.ts make renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}) the first operation, imported with the other operations, delete the drop of "${from}", and delete the create of "${to}" if there is one; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate.`; } /** diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts index 22e8a536c748..82f019d6cb15 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts @@ -169,7 +169,7 @@ describe('MongoDB planner, data loss', () => { operation: 'Drop collection events', subject: { kind: 'model', namespaceId: asNamespaceId('__unbound__'), model: 'Event' }, keepDataAdvice: - 'To keep its documents, rename collection "events" to "occasions" instead; a statement cannot rename the collection of a model that keeps its name. With db update, rename the collection in the database yourself, for example db.getCollection("events").renameCollection("occasions") in mongosh, then run db update. With migration plan, plan it, passing --delete Event to answer its question; in its migration.ts replace the drop of "events" and the create of "occasions" with renameCollection("events", "occasions"), imported with the other operations; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate.', + 'To keep its documents, rename collection "events" to "occasions" instead; a statement cannot rename the collection of a model that keeps its name. With db update, rename the collection in the database yourself, for example db.getCollection("events").renameCollection("occasions") in mongosh, then run db update. With migration plan, plan it, passing --delete Event to answer its question; in its migration.ts make renameCollection("events", "occasions") the first operation, imported with the other operations, delete the drop of "events", and delete the create of "occasions" if there is one; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate.', }, ]); }); @@ -201,7 +201,7 @@ describe('MongoDB planner, data loss', () => { }).toEqual({ creates: ['Create index on occasions (name:1)'], advice: [ - expect.stringMatching(/^To keep its documents, rename collection "events" to "occasions"/), + 'To keep its documents, rename collection "events" to "occasions" instead; a statement cannot rename the collection of a model that keeps its name. With db update, rename the collection in the database yourself, for example db.getCollection("events").renameCollection("occasions") in mongosh, then run db update. With migration plan, plan it, passing --delete Event to answer its question; in its migration.ts make renameCollection("events", "occasions") the first operation, imported with the other operations, delete the drop of "events", and delete the create of "occasions" if there is one; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate.', ], }); }); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts index d56355c9f3c5..6c0ee354068e 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts @@ -199,7 +199,7 @@ describe('MongoMigrationPlanner, statements it refuses', () => { }), )[0]?.why, ).toBe( - 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question; in its migration.ts replace the drop of "profiles" and the create of "users" with renameCollection("profiles", "users"), imported with the other operations; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate. Then change the field and run this statement again.', + 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question; in its migration.ts make renameCollection("profiles", "users") the first operation, imported with the other operations, delete the drop of "profiles", and delete the create of "users" if there is one; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate. Then change the field and run this statement again.', ); }); }); diff --git a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts index e6e9f25ad2ca..09fce4a4c07e 100644 --- a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts +++ b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts @@ -16,6 +16,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it } from 'vitest'; import { fixtureAppDir } from '../utils/cli-test-helpers'; import { engineError, + followRenameCollectionAdvice, type JourneyContext, latestMigrationDirName, parseJsonOutput, @@ -341,7 +342,9 @@ describe('Journeys (MongoDB): statements that consent to data loss', { expect( JSON.stringify(engineError(asked)), 'S5.06: the question says how to keep the documents', - ).toContain('replace the drop of \\"events\\" and the create of \\"occasions\\"'); + ).toContain( + 'make renameCollection(\\"events\\", \\"occasions\\") the first operation, imported with the other operations, delete the drop of \\"events\\", and delete the create of \\"occasions\\" if there is one', + ); const remap = await runMigrationPlan(ctx, [ '--name', 'remap', @@ -357,17 +360,13 @@ describe('Journeys (MongoDB): statements that consent to data loss', { const remapDir = latestMigrationDirName(ctx); const migrationTs = join(ctx.testDir, 'migrations/app', remapDir, 'migration.ts'); const planned = readFileSync(migrationTs, 'utf-8'); - const edited = planned - .replace( - /createCollection\(['"]occasions['"][^)]*\)/, - 'renameCollection("events", "occasions")', - ) - .replace(/,?\s*dropCollection\(['"]events['"]\)/, '') - .replace(/\bdropCollection,/, 'dropCollection,\n renameCollection,'); - expect(edited, `S5.08: migration.ts is edited as the advice says:\n${planned}`).toContain( - 'renameCollection("events", "occasions")', + const edited = followRenameCollectionAdvice(planned, 'events', 'occasions'); + expect(edited, `S5.08: migration.ts is edited as the advice says:\n${edited}`).toMatch( + /return \[renameCollection\("events", "occasions"\)/, + ); + expect(edited, `S5.08: the drop and the create are gone:\n${edited}`).not.toMatch( + /dropCollection\(['"]events|createCollection\(['"]occasions/, ); - expect(edited, `S5.08: the drop is gone:\n${edited}`).not.toMatch(/dropCollection\(['"]events/); writeFileSync(migrationTs, edited, 'utf-8'); const emitted = runPrintedSelfEmit(String(engineError(asked)?.why), dirname(migrationTs)); expect(emitted.exitCode, `S5.09: node on migration.ts, as printed: ${emitted.stderr}`).toBe(0); diff --git a/test/integration/test/cli-journeys/keep-documents-advice.mongo.e2e.test.ts b/test/integration/test/cli-journeys/keep-documents-advice.mongo.e2e.test.ts new file mode 100644 index 000000000000..68de094734c1 --- /dev/null +++ b/test/integration/test/cli-journeys/keep-documents-advice.mongo.e2e.test.ts @@ -0,0 +1,157 @@ +/** + * The advice to keep the documents of a collection whose model keeps its name, followed as printed when the plan creates the new collection only with its index (MongoDB). + * + * `Event` moves from `events` to `occasions` and has a unique index on `name`, so the plan has no `createCollection("occasions")`: `createIndex` creates it. The journey plans with `--delete Event`, edits `migration.ts` as the question says, runs node on it, and applies it with `db migrate`. + */ + +import { copyFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { rm } from 'node:fs/promises'; +import { dirname, join } from 'node:path'; +import { timeouts } from '@repo/test-utils'; +import { MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { fixtureAppDir } from '../utils/cli-test-helpers'; +import { + engineError, + followRenameCollectionAdvice, + type JourneyContext, + latestMigrationDirName, + parseJsonOutput, + runContractEmit, + runDbVerify, + runMigrate, + runMigrationPlan, + runPrintedSelfEmit, +} from '../utils/journey-test-helpers'; + +const FIXTURES_DIR = join(fixtureAppDir, 'fixtures/mongo-cli-journeys'); +const DB_NAME = 'mongo_keep_indexed_documents'; + +function setupProject(connectionString: string): JourneyContext { + const testDir = join( + fixtureAppDir, + `test-mongo-keep-documents-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + const outputDir = join(testDir, 'output'); + mkdirSync(outputDir, { recursive: true }); + mkdirSync(join(testDir, 'migrations'), { recursive: true }); + writeFileSync( + join(testDir, 'package.json'), + JSON.stringify({ + name: 'mongo-keep-documents-journey', + private: true, + type: 'module', + dependencies: { '@prisma/orm-mongo': 'workspace:0.16.0' }, + }), + 'utf-8', + ); + const config = readFileSync(join(FIXTURES_DIR, 'prisma.config.with-db.ts'), 'utf-8').replace( + /\{\{DB_URL\}\}/g, + () => connectionString, + ); + const configPath = join(testDir, 'prisma.config.ts'); + writeFileSync(configPath, config, 'utf-8'); + return { testDir, configPath, outputDir }; +} + +async function emitFixture(ctx: JourneyContext, name: string, label: string): Promise { + copyFileSync(join(FIXTURES_DIR, name), join(ctx.testDir, 'contract.ts')); + const emitted = await runContractEmit(ctx); + expect(emitted.exitCode, `${label}: emit: ${emitted.stderr}`).toBe(0); +} + +describe('Journey (MongoDB): keeping the documents of a collection created only by its index', { + timeout: timeouts.spinUpMongoMemoryServer, +}, () => { + let replSet: MongoMemoryReplSet; + let client: MongoClient; + let ctx: JourneyContext | undefined; + + beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + instanceOpts: [ + { launchTimeout: timeouts.spinUpMongoMemoryServer, storageEngine: 'wiredTiger' }, + ], + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + }, timeouts.spinUpMongoMemoryServer); + + afterAll(async () => { + if (ctx !== undefined) await rm(ctx.testDir, { recursive: true, force: true }).catch(() => {}); + await client?.close().catch(() => {}); + await replSet?.stop().catch(() => {}); + }, timeouts.spinUpMongoMemoryServer); + + it('renames the collection with its documents and index when migration.ts is edited as printed', async () => { + const url = new URL(replSet.getUri()); + url.pathname = `/${DB_NAME}`; + const project = setupProject(url.toString()); + ctx = project; + await emitFixture(project, 'contract-with-indexed-events.ts', 'K.01'); + const initial = await runMigrationPlan(project, ['--name', 'initial']); + expect(initial.exitCode, `K.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(project)).exitCode, 'K.03: migrate initial').toBe(0); + const db = client.db(DB_NAME); + await db.collection('events').insertMany([{ name: 'launch' }, { name: 'party' }]); + const origin = latestMigrationDirName(project); + + await emitFixture(project, 'contract-indexed-events-remapped.ts', 'K.04'); + const asked = await runMigrationPlan(project, ['--from', origin, '--json']); + expect(asked.exitCode, 'K.05: the drop of events needs a statement').toBe(2); + const question = String(engineError(asked)?.why); + expect(question, 'K.05: the question says where the rename goes').toContain( + 'make renameCollection("events", "occasions") the first operation, imported with the other operations, delete the drop of "events", and delete the create of "occasions" if there is one', + ); + + const remap = await runMigrationPlan(project, [ + '--name', + 'remap', + '--from', + origin, + '--delete', + 'Event', + ]); + expect(remap.exitCode, `K.06: plan with --delete Event: ${remap.stderr}`).toBe(0); + const remapDir = latestMigrationDirName(project); + const migrationTs = join(project.testDir, 'migrations/app', remapDir, 'migration.ts'); + const planned = readFileSync(migrationTs, 'utf-8'); + expect(planned, 'K.07: the index creates occasions, no createCollection does').not.toMatch( + /createCollection\(/, + ); + writeFileSync(migrationTs, followRenameCollectionAdvice(planned, 'events', 'occasions')); + const emitted = runPrintedSelfEmit(question, dirname(migrationTs)); + expect(emitted.exitCode, `K.08: node on migration.ts, as printed: ${emitted.stderr}`).toBe(0); + + const applied = await runMigrate(project); + expect(applied.exitCode, `K.09: db migrate: ${applied.stderr}`).toBe(0); + const collections = (await db.listCollections().toArray()) + .map(({ name }) => name) + .filter((name) => !name.startsWith('_prisma')) + .sort(); + const documents = await db + .collection('occasions') + .find({}, { projection: { _id: 0 } }) + .sort({ name: 1 }) + .toArray(); + const indexes = (await db.collection('occasions').indexes()).map(({ key, unique }) => ({ + key, + unique: unique === true, + })); + expect({ collections, documents, indexes }, 'K.10: the documents and the index moved').toEqual({ + collections: ['occasions', 'users'], + documents: [{ name: 'launch' }, { name: 'party' }], + indexes: [ + { key: { _id: 1 }, unique: false }, + { key: { name: 1 }, unique: true }, + ], + }); + const verify = await runDbVerify(project, ['--schema-only']); + expect(verify.exitCode, `K.11: db verify --schema-only: ${verify.stderr}`).toBe(0); + const further = await runMigrationPlan(project, ['--from', remapDir, '--json']); + expect(further.exitCode, `K.12: a further plan: ${further.stderr}`).toBe(0); + expect(parseJsonOutput<{ noOp: boolean }>(further).noOp, 'K.12: it is empty').toBe(true); + }); +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-indexed-events-remapped.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-indexed-events-remapped.ts new file mode 100644 index 000000000000..a9efe8e66c4c --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-indexed-events-remapped.ts @@ -0,0 +1,24 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'occasions', + fields: { + _id: field.objectId(), + name: field.string(), + }, + indexes: [index({ name: 1 }, { unique: true })], +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-with-indexed-events.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-with-indexed-events.ts new file mode 100644 index 000000000000..6ccf183f7c74 --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-with-indexed-events.ts @@ -0,0 +1,24 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'events', + fields: { + _id: field.objectId(), + name: field.string(), + }, + indexes: [index({ name: 1 }, { unique: true })], +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/utils/journey-test-helpers.ts b/test/integration/test/utils/journey-test-helpers.ts index 2adbb72a89c4..40e357a7d796 100644 --- a/test/integration/test/utils/journey-test-helpers.ts +++ b/test/integration/test/utils/journey-test-helpers.ts @@ -568,6 +568,52 @@ export function runPrintedSelfEmit(message: string, packageDir: string): Command return { exitCode: run.status ?? -1, stdout: run.stdout, stderr: run.stderr }; } +/** + * Splits the body of a `return [ ... ]` operations list into its top-level entries, keeping each entry's source text. + */ +function operationEntries(body: string): string[] { + const entries: string[] = []; + let depth = 0; + let quote: string | undefined; + let start = 0; + for (let i = 0; i < body.length; i++) { + const char = body[i]!; + if (quote !== undefined) { + if (char === '\\') i++; + else if (char === quote) quote = undefined; + } else if (char === "'" || char === '"' || char === '`') quote = char; + else if ('([{'.includes(char)) depth++; + else if (')]}'.includes(char)) depth--; + else if (char === ',' && depth === 0) { + entries.push(body.slice(start, i)); + start = i + 1; + } + } + entries.push(body.slice(start)); + return entries.map((entry) => entry.trim()).filter((entry) => entry !== ''); +} + +/** + * Edits a MongoDB `migration.ts` as the advice to keep a dropped collection's documents says: `renameCollection(from, to)` becomes the first operation, imported with the others, and the drop of `from` and any create of `to` are deleted. + */ +export function followRenameCollectionAdvice(source: string, from: string, to: string): string { + const operations = /(override get operations\(\) \{\s*return \[)([\s\S]*?)(\];\s*\})/; + const match = operations.exec(source); + if (match === null) throw new Error(`no operations list in:\n${source}`); + const isCall = (entry: string, factory: string, collection: string) => + new RegExp(`^${factory}\\(\\s*['"]${collection}['"]`).test(entry); + const kept = operationEntries(match[2]!).filter( + (entry) => !isCall(entry, 'dropCollection', from) && !isCall(entry, 'createCollection', to), + ); + const entries = [`renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)})`, ...kept]; + return source + .replace( + operations, + (_all, open: string, _body, close: string) => `${open}${entries.join(', ')}${close}`, + ) + .replace(/import \{/, 'import { renameCollection,'); +} + /** * Runs `migration plan` (which scaffolds a draft migration.ts), then runs * that migration.ts in-process so it writes its own ops.json and From b81602829e174f15d0b1ee5e0c7f40021c1c348c Mon Sep 17 00:00:00 2001 From: willbot Date: Tue, 13 Oct 2026 08:17:48 +0200 Subject: [PATCH 50/57] TML-3478: a collection rename stopped because the new collection exists says so and names both collections The runner reported only "failed during precheck". It now says the new collection already exists, that nothing was renamed, and what to do: drop it, or put the rename first when an earlier operation of the same migration created it. Signed-off-by: willbot Signed-off-by: Will Madden --- docs/reference/error-reference.md | 2 +- .../src/core/migrations/mongo-runner.ts | 56 ++++++++++++++++--- .../1-mongo-target/test/mongo-runner.test.ts | 22 +++++++- .../runner-rename-collection.test.ts | 8 ++- 4 files changed, 76 insertions(+), 12 deletions(-) diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index a7cda78fd3fc..72f3cb1560e4 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -1616,7 +1616,7 @@ A `PostgresMigration` operation (e.g. `createTable`, `dataTransform`) was invoke ### MIGRATION.PRECHECK_FAILED -Before executing a migration operation, one of its precheck steps (a query expected to return true) did not hold: the database is not in the state the operation requires, so the apply stops and rolls back. Payload: `operationId`, `phase`, `stepDescription`. On MongoDB, where operations are not wrapped in a transaction, the operations before it stay applied, and a re-run skips them. A field rename (`renameField`) stops here when documents already hold the new name as well as the old one: the message names the collection, gives the `mongosh` query that finds those documents, and says to fix them and run the command again. Payload on MongoDB: `operationId`, `name`. +Before executing a migration operation, one of its precheck steps (a query expected to return true) did not hold: the database is not in the state the operation requires, so the apply stops and rolls back. Payload: `operationId`, `phase`, `stepDescription`. On MongoDB, where operations are not wrapped in a transaction, the operations before it stay applied, and a re-run skips them. A field rename (`renameField`) stops here when documents already hold the new name as well as the old one: the message names the collection, gives the `mongosh` query that finds those documents, and says to fix them and run the command again. A collection rename (`renameCollection`) stops here when the new collection already exists: the message names both collections, says nothing was renamed, and says to drop the new collection, or, when an earlier operation of the same migration created it, to make the rename the first operation of its `migration.ts`. Payload on MongoDB: `operationId`, `name` (a field rename only). ### MIGRATION.PROVIDED_INVARIANTS_MISMATCH diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts index 17e9837834c0..0f08afcb938e 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts @@ -27,8 +27,9 @@ import type { MongoDmlOperation, MongoInspectionCommandVisitor, MongoMigrationCheck, + MongoMigrationPlanOperation, } from '@internal/mongo-query-ast/control'; -import { isMongoDmlOperation } from '@internal/mongo-query-ast/control'; +import { isMongoDmlOperation, RenameCollectionCommand } from '@internal/mongo-query-ast/control'; import { type MongoQueryPlan, RawAggregateCommand, @@ -116,6 +117,30 @@ function renameOverwriteRefusal(op: MongoDmlOperation): MongoMigrationRunnerResu ); } +/** + * The failure when a collection rename's before-check finds that the new collection already exists. + */ +function renameTargetExistsRefusal( + op: MongoMigrationPlanOperation, + failed: MongoMigrationCheck, +): MongoMigrationRunnerResult | undefined { + const rename = op.execute.find( + (step) => step.command instanceof RenameCollectionCommand, + )?.command; + if (!(rename instanceof RenameCollectionCommand) || failed.expect !== 'notExists') { + return undefined; + } + const { collection: from, to } = rename; + return runnerFailure( + 'MIGRATION.PRECHECK_FAILED', + `Operation ${op.id} stopped: collection ${to} already exists, so collection ${from} cannot be renamed to it`, + { + why: `Nothing was renamed. If ${to} holds nothing you need, drop it, then run the command again; operations that already ran are skipped. If an earlier operation of the same migration created ${to}, as a createIndex on ${to} does, make renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}) the first operation of its migration.ts, run node on that migration.ts, and then drop ${to} and run the command again.`, + meta: { operationId: op.id }, + }, + ); +} + const MONGO_UNAUTHORIZED = 13; /** @@ -210,16 +235,19 @@ export class MongoMigrationRunner { } if (runPrechecks) { - const precheckResult = await this.evaluateChecks( + const failed = await this.firstFailingCheck( operation.precheck, inspectionExecutor, filterEvaluator, ); - if (!precheckResult) { - return runnerFailure( - 'MIGRATION.PRECHECK_FAILED', - `Operation ${operation.id} failed during precheck`, - { meta: { operationId: operation.id } }, + if (failed !== undefined) { + return ( + renameTargetExistsRefusal(operation, failed) ?? + runnerFailure( + 'MIGRATION.PRECHECK_FAILED', + `Operation ${operation.id} failed during precheck`, + { meta: { operationId: operation.id } }, + ) ); } } @@ -475,13 +503,23 @@ export class MongoMigrationRunner { inspectionExecutor: MongoInspectionCommandVisitor[]>>, filterEvaluator: FilterEvaluator, ): Promise { + return ( + (await this.firstFailingCheck(checks, inspectionExecutor, filterEvaluator)) === undefined + ); + } + + private async firstFailingCheck( + checks: readonly MongoMigrationCheck[], + inspectionExecutor: MongoInspectionCommandVisitor[]>>, + filterEvaluator: FilterEvaluator, + ): Promise { for (const check of checks) { const documents = await check.source.accept(inspectionExecutor); const matchFound = documents.some((doc) => filterEvaluator.evaluate(check.filter, doc)); const passed = check.expect === 'exists' ? matchFound : !matchFound; - if (!passed) return false; + if (!passed) return check; } - return true; + return undefined; } private async allChecksSatisfied( diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts index 64861051f866..6cda4c426540 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts @@ -44,6 +44,7 @@ import { describe, expect, it } from 'vitest'; import { createCollection, dataTransform, + renameCollection, renameField, unsetField, } from '../src/core/migrations/migration-factories'; @@ -144,6 +145,7 @@ class StubMongoAdapter implements MongoAdapter { class StubInspectionExecutor implements MongoInspectionCommandVisitor> { readonly calls: AnyMongoInspectionCommand[] = []; + collections: Row[] = []; async listIndexes(command: AnyMongoInspectionCommand): Promise { this.calls.push(command); @@ -151,7 +153,7 @@ class StubInspectionExecutor implements MongoInspectionCommandVisitor { this.calls.push(command); - return []; + return this.collections; } } @@ -854,3 +856,21 @@ describe('MongoMigrationRunner, a field rename stopped because documents already }); }); }); + +describe('MongoMigrationRunner, a collection rename stopped because the new collection exists', () => { + it('says that the new collection exists, names both collections, and that nothing was renamed', async () => { + const harness = makeHarness(); + harness.inspectionExecutor.collections = [{ name: 'events' }, { name: 'occasions' }]; + + const result = await execute(harness, [renameCollection('events', 'occasions')]); + + expect(harness.ddlCalls).toEqual([]); + expect(result.assertNotOk()).toEqual({ + code: 'MIGRATION.PRECHECK_FAILED', + summary: + 'Operation collection.events.rename(occasions) stopped: collection occasions already exists, so collection events cannot be renamed to it', + why: 'Nothing was renamed. If occasions holds nothing you need, drop it, then run the command again; operations that already ran are skipped. If an earlier operation of the same migration created occasions, as a createIndex on occasions does, make renameCollection("events", "occasions") the first operation of its migration.ts, run node on that migration.ts, and then drop occasions and run the command again.', + meta: { operationId: 'collection.events.rename(occasions)' }, + }); + }); +}); diff --git a/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts b/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts index c3fe19453ace..a07d1e821f2a 100644 --- a/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts +++ b/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts @@ -196,7 +196,13 @@ describe('MongoMigrationRunner - renameCollection', () => { const result = await runRename(); - expect(result.ok ? result.value : result.failure.code).toBe('MIGRATION.PRECHECK_FAILED'); + expect( + result.ok ? result.value : { code: result.failure.code, summary: result.failure.summary }, + ).toEqual({ + code: 'MIGRATION.PRECHECK_FAILED', + summary: + 'Operation collection.profiles.rename(users) stopped: collection users already exists, so collection profiles cannot be renamed to it', + }); expect({ names: await collectionNames(), profiles: await emails('profiles'), From 3f0de5c8b258b8f26347fed5e0f77de04697e39a Mon Sep 17 00:00:00 2001 From: willbot Date: Tue, 13 Oct 2026 08:18:10 +0200 Subject: [PATCH 51/57] TML-3478: deferred.md ties keepDataAdvice to the missing statement that renames a model's storage Signed-off-by: willbot Signed-off-by: Will Madden --- projects/migration-statements/deferred.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index e79746a3f208..cf9f00379269 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -12,6 +12,8 @@ A user keeps `model User` and changes `@@map("users")` to `@@map("app_users")`. **Decided 2026-10-07 by Will:** no statement for this case in this project. Slice 2's refusal keeps slice 1's manual steps for both commands. A statement that can name storage objects rather than models and fields is wanted later; Will floated `--rename table/users:app_users` as an illustration only, and the syntax needs a critical design discussion and a survey of established prior art before anything is chosen. +**What depends on it:** `MigrationDataLoss.keepDataAdvice` (framework components, `control/migration-statements.ts`) exists only because this statement is missing. The SQL family (`keepTableAdvice` in `operation-subjects.ts`) and the MongoDB target (`keepCollectionByHandAdvice`, reached from `collectionDrops` in `mongo-planner.ts`) use it to print manual steps for this one case. When the statement lands, the planner answers this case with it, and both advice texts go, with the field if nothing else uses it. Slice 3a (PR #30695) adds `label` to the same type, `MigrationDataLoss`; whichever of the two PRs merges second combines the two changes. + ## A model move across namespaces (`auth.User:public.User`) **Found:** slice 1, dispatch 3 (2026-10-06). Resolves but is refused with `statementRejected`; needs a `set schema` operation with its working-schema step and companion names. Scheduled for slice 3 with the namespace renames (recorded in `plan.md`). From 94bbe66e3ff6c49ff9537ced1bd0005e30fc89ad Mon Sep 17 00:00:00 2001 From: willbot Date: Tue, 13 Oct 2026 08:47:21 +0200 Subject: [PATCH 52/57] TML-3478: without --config, migration.ts uses the config whose migrations directory holds it The nearest prisma.config.ts above the migration could belong to another project. The file now uses the nearest config only if the migration is inside its migrations directory, then the working directory's config on the same condition, and otherwise fails with CLI.MIGRATION_CONFIG_NOT_FOUND, naming each config it checked. migration new and migration plan repeat the --config they were given in the node command they print, and the re-emit hint of migration package errors says to add it. Signed-off-by: willbot Signed-off-by: Will Madden --- .../subsystems/10. MongoDB Family.md | 2 +- .../subsystems/7. Migration System.md | 2 +- docs/reference/error-reference.md | 4 + .../1-framework/1-core/errors/src/control.ts | 37 +++++ .../1-core/errors/src/exports/control.ts | 3 +- .../1-core/errors/test/control.test.ts | 34 ++++ .../control-api/operations/migration-plan.ts | 23 ++- .../3-tooling/cli/src/migration-cli.ts | 22 ++- .../3-tooling/cli/src/orm/migration/new.ts | 5 +- .../3-tooling/cli/src/orm/migration/plan.ts | 9 +- .../cli/src/utils/migration-file-config.ts | 69 ++++++++ .../cli/src/utils/self-emit-command.ts | 6 + .../3-tooling/cli/test/migration-cli.test.ts | 28 ++-- .../cli/test/orm/migration-new.test.ts | 17 ++ .../cli/test/orm/migration-plan.test.ts | 26 +++ .../test/utils/migration-file-config.test.ts | 153 ++++++++++++++++++ .../config-loader/src/exports/index.ts | 1 + .../3-tooling/config-loader/src/load.ts | 2 +- .../3-tooling/migration/src/errors.ts | 2 +- .../3-tooling/migration/test/errors.test.ts | 8 + ...tion-file-config-lookup.sqlite.e2e.test.ts | 118 ++++++++++---- .../test/utils/cli-test-helpers.ts | 6 +- .../app/instructions.md | 6 +- 23 files changed, 507 insertions(+), 76 deletions(-) create mode 100644 packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts create mode 100644 packages/1-framework/3-tooling/cli/src/utils/self-emit-command.ts create mode 100644 packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts diff --git a/docs/architecture docs/subsystems/10. MongoDB Family.md b/docs/architecture docs/subsystems/10. MongoDB Family.md index 39ea455040d6..ecec7ff2b0de 100644 --- a/docs/architecture docs/subsystems/10. MongoDB Family.md +++ b/docs/architecture docs/subsystems/10. MongoDB Family.md @@ -482,7 +482,7 @@ For polymorphic models (`@@discriminator` on the base, `@@base` on each variant) 1. Run `migration plan` (or `migration new`). The planner fills in everything it can derive from the contract diff. 2. Open `migration.ts`. Replace any `placeholder(slot)` calls with real query plans (typically inside a `dataTransform({ check, run })` operation, where the planner can scaffold the structure but not the body). Hand-author additional operations as needed. -3. Re-emit by running the file with node from any directory: `node /migration.ts`. It loads the nearest `prisma.config.ts` above the migration's directory, or the one `--config` names. `Migration.run(...)` detects it is the main module and rewrites `ops.json` and the `migrationId` in `migration.json`. Unfilled `placeholder(...)` calls fail here with the structured error `PN-MIG-2001` rather than producing silent nonsense ([ADR 200](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md)). +3. Re-emit by running the file with node from any directory: `node /migration.ts`. It loads the config `--config` names. Without `--config`, it uses the config that owns the migration, whose migrations directory holds the file: the nearest `prisma.config.ts` above the migration's directory if it owns it, else the `prisma.config.ts` in the working directory if that one does; otherwise it stops with `CLI.MIGRATION_CONFIG_NOT_FOUND`. `Migration.run(...)` detects it is the main module and rewrites `ops.json` and the `migrationId` in `migration.json`. Unfilled `placeholder(...)` calls fail here with the structured error `PN-MIG-2001` rather than producing silent nonsense ([ADR 200](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md)). 4. Commit both files. CI / `migrate` reads `ops.json` only — TypeScript is never executed at apply time. Apply-time hash verification rehashes `(manifest, ops)` and aborts on mismatch, so post-emit hand edits to `ops.json` (or any FS corruption) are caught before the runner touches the database. See [Subsystem 7 — Hash verification](7.%20Migration%20System.md#hash-verification). What the runner guarantees about the seam: diff --git a/docs/architecture docs/subsystems/7. Migration System.md b/docs/architecture docs/subsystems/7. Migration System.md index 286c9a00e02a..25b064fe6883 100644 --- a/docs/architecture docs/subsystems/7. Migration System.md +++ b/docs/architecture docs/subsystems/7. Migration System.md @@ -84,7 +84,7 @@ flowchart LR 3. **Scaffold.** The CLI calls `plan.renderTypeScript(resolveImportSpecifier)` and writes the result to `migration.ts`. The resolver comes from the consuming project's own `package.json` (`projectImportRoot` in `@internal/cli`), so the package names in the scaffold are ones that project can resolve — the same resolver `contract.d.ts` emission gets, for the same reason: both are files the application keeps. See [ADR 242 — Public npm surface](../adrs/ADR%20242%20-%20Public%20npm%20surface%20-%20single%20@prisma%20scope%20with%20consolidated%20publish%20packages.md). The destination contract is snapshotted into the migration directory so queries are typed against the schema at planning time. See [ADR 197 — Migration packages snapshot their own contract](../adrs/ADR%20197%20-%20Migration%20packages%20snapshot%20their%20own%20contract.md). Unfilled slots (data-transform bodies the planner can't derive) use `placeholder(slot)` — a `never`-returning function that throws a structured `PN-MIG-2001` error at emit time. See [ADR 200 — Placeholder utility for scaffolded migration slots](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md). -4. **Edit.** The developer opens `migration.ts`, replaces placeholders with real queries, and iterates by running the file with node from any directory (`node /migration.ts`; it loads the nearest `prisma.config.ts` above the migration's directory) thanks to the shebang and `Migration.run(...)`. +4. **Edit.** The developer opens `migration.ts`, replaces placeholders with real queries, and iterates by running the file with node (`node /migration.ts`) thanks to the shebang and `Migration.run(...)`. The file loads the config `--config` names. Without `--config`, it uses the config that owns the migration, whose migrations directory holds the file: the nearest `prisma.config.ts` above the migration's directory if it owns it, else the `prisma.config.ts` in the working directory if that one does; otherwise it stops with `CLI.MIGRATION_CONFIG_NOT_FOUND`. 5. **Emit.** Every migration package is **fully attested on disk**: `migration plan` writes `migration.ts`, `migration.json` (with a content-addressed `migrationHash`), and `ops.json` (the planned ops, or `[]` if the planner could not lower any calls because of unfilled placeholders). After the developer fills any `placeholder(...)` slots, **self-emission** rewrites `ops.json` and the `migrationHash`: running the file via `./migration.ts` (shebang) or `node migration.ts` causes `Migration.run(...)` to detect it is the main module and re-emit artifacts in-process. See [ADR 196 — In-process emit for class-flow targets](../adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md). diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index 72f3cb1560e4..2a3cb4a5b67e 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -211,6 +211,10 @@ The main CLI received a `--format` value other than `pretty` or `json`. Raised d Reserved: a command was asked for a `--json` sub-format it does not support; the error lists the formats the command does accept. Declared in the shared error factories but not raised by any command today. Payload: `command`, `format`, `supportedFormats`. +### CLI.MIGRATION_CONFIG_NOT_FOUND + +Running a `migration.ts` with node and no `--config` found no config that owns the migration. Without `--config`, the file uses the config whose migrations directory holds it: the nearest `prisma.config.ts` above the migration's directory if it owns it, else the `prisma.config.ts` in the working directory if that one does. The message names each config it checked, with the migrations directory it keeps or the reason it failed to load, and says to run the file again with `--config `. Payload: `migrationDir`, `checked`. + ### CLI.OUTPUT_FORMAT_CONFLICT The main CLI received mutually exclusive output flags: `--format pretty` together with `--json`. Use `--format json` or `--json` alone for JSON output. Payload: none. diff --git a/packages/1-framework/1-core/errors/src/control.ts b/packages/1-framework/1-core/errors/src/control.ts index c51c516bba87..06704d637b30 100644 --- a/packages/1-framework/1-core/errors/src/control.ts +++ b/packages/1-framework/1-core/errors/src/control.ts @@ -443,6 +443,43 @@ export function errorTargetMigrationNotSupported(options?: { ); } +/** A config the migration-file CLI checked while looking for the one that owns the migration. */ +export interface CheckedMigrationConfig { + readonly configPath: string; + readonly migrationsDir?: string; + readonly loadFailure?: string; +} + +/** + * Without `--config`, the migration-file CLI found no config whose migrations directory holds the migration. + */ +export function errorMigrationCliConfigNotFound(options: { + readonly migrationDir: string; + readonly checked: readonly CheckedMigrationConfig[]; +}): CliStructuredError { + const rule = `Without --config, a migration.ts uses the prisma.config.ts whose migrations directory holds it: the nearest one above ${options.migrationDir}, else the one in the working directory.`; + const found = options.checked.map(({ configPath, migrationsDir, loadFailure }) => + loadFailure === undefined + ? `${configPath} keeps migrations in ${migrationsDir}` + : `${configPath} failed to load (${loadFailure})`, + ); + return new CliStructuredError( + 'CLI.MIGRATION_CONFIG_NOT_FOUND', + 'No config found for this migration', + { + why: + found.length === 0 + ? `${rule} There is neither.` + : `${rule} None of them does: ${found.join('; ')}.`, + fix: 'Run it again with `--config `.', + meta: { + migrationDir: options.migrationDir, + checked: options.checked.map(({ configPath }) => configPath), + }, + }, + ); +} + /** * The migration-file CLI received `--config` without a path argument (either * a bare trailing `--config`, or `--config` followed by another flag like diff --git a/packages/1-framework/1-core/errors/src/exports/control.ts b/packages/1-framework/1-core/errors/src/exports/control.ts index 4eb66fbbac67..67bb6d7d42f4 100644 --- a/packages/1-framework/1-core/errors/src/exports/control.ts +++ b/packages/1-framework/1-core/errors/src/exports/control.ts @@ -1,4 +1,4 @@ -export type { CliErrorConflict, CliErrorEnvelope } from '../control'; +export type { CheckedMigrationConfig, CliErrorConflict, CliErrorEnvelope } from '../control'; export { CliStructuredError, errorConfigEvaluationFailed, @@ -15,6 +15,7 @@ export { errorFileNotFound, errorInvalidOutputFormat, errorJsonFormatNotSupported, + errorMigrationCliConfigNotFound, errorMigrationCliInvalidConfigArg, errorMigrationCliUnknownFlag, errorMigrationPlanningFailed, diff --git a/packages/1-framework/1-core/errors/test/control.test.ts b/packages/1-framework/1-core/errors/test/control.test.ts index ddb9111dba89..c8ccd8d01d02 100644 --- a/packages/1-framework/1-core/errors/test/control.test.ts +++ b/packages/1-framework/1-core/errors/test/control.test.ts @@ -19,6 +19,7 @@ import { errorFileNotFound, errorInvalidOutputFormat, errorJsonFormatNotSupported, + errorMigrationCliConfigNotFound, errorMigrationCliInvalidConfigArg, errorMigrationCliUnknownFlag, errorMigrationPlanningFailed, @@ -523,6 +524,39 @@ describe('Config Errors', () => { expect(error.where).toBeUndefined(); }); + it('errorMigrationCliConfigNotFound names each config it checked and why it does not own the migration', () => { + const error = errorMigrationCliConfigNotFound({ + migrationDir: '/repo/inner/migrations/app/20260101T0000_initial', + checked: [ + { configPath: '/repo/prisma.config.ts', migrationsDir: '/repo/migrations' }, + { configPath: '/work/prisma.config.ts', loadFailure: 'Failed to evaluate config' }, + ], + }); + expect({ + code: error.code, + message: error.message, + why: error.why, + fix: error.fix, + meta: error.meta, + }).toEqual({ + code: 'CLI.MIGRATION_CONFIG_NOT_FOUND', + message: 'No config found for this migration', + why: 'Without --config, a migration.ts uses the prisma.config.ts whose migrations directory holds it: the nearest one above /repo/inner/migrations/app/20260101T0000_initial, else the one in the working directory. None of them does: /repo/prisma.config.ts keeps migrations in /repo/migrations; /work/prisma.config.ts failed to load (Failed to evaluate config).', + fix: 'Run it again with `--config `.', + meta: { + migrationDir: '/repo/inner/migrations/app/20260101T0000_initial', + checked: ['/repo/prisma.config.ts', '/work/prisma.config.ts'], + }, + }); + }); + + it('errorMigrationCliConfigNotFound says when it found no config at all', () => { + const error = errorMigrationCliConfigNotFound({ migrationDir: '/m', checked: [] }); + expect(error.why).toBe( + 'Without --config, a migration.ts uses the prisma.config.ts whose migrations directory holds it: the nearest one above /m, else the one in the working directory. There is neither.', + ); + }); + it('errorMigrationCliInvalidConfigArg creates correct error for missing path', () => { const error = errorMigrationCliInvalidConfigArg(); expect(error.code).toBe('CLI.CONFIG_ARG_MISSING_PATH'); diff --git a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts index 25961f399a1b..3a66f4e6a07d 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts @@ -85,6 +85,8 @@ export interface MigrationPlanOptions { readonly cwd: string; /** The project's directory, normally the validated config's `baseDir`; locates the project manifest. */ readonly projectDir?: string; + /** The `--config` the command was given, repeated in the command that runs a written `migration.ts`. */ + readonly configPath?: string; readonly name?: string; readonly from?: string; readonly to?: string; @@ -115,8 +117,15 @@ type PlannerSuccess = { readonly dataLoss: readonly PlannedLoss[]; }; -function placeholderSummary(planned: string, dirs: readonly string[]): string { - const commands = dirs.map((dir) => `\`node ${join(dir, 'migration.ts')}\``).join(' and '); +function placeholderSummary( + planned: string, + dirs: readonly string[], + configPath: string | undefined, +): string { + const config = configPath === undefined ? '' : ` --config "${configPath}"`; + const commands = dirs + .map((dir) => `\`node ${join(dir, 'migration.ts')}${config}\``) + .join(' and '); return `Planned ${planned} with placeholder(s) — edit migration.ts, then run ${commands} from this directory to self-emit`; } @@ -869,7 +878,7 @@ async function executeMigrationPlanCommandInner( ...(warnings.length > 0 ? { warnings } : {}), pendingPlaceholders: true, appliedStatements: [...statementsWithoutOperations, ...deletes], - summary: placeholderSummary('baseline', [baselineDir]), + summary: placeholderSummary('baseline', [baselineDir], options.configPath), timings: { total: Date.now() - startTime }, }; return ok(result); @@ -926,7 +935,11 @@ async function executeMigrationPlanCommandInner( ...(warnings.length > 0 ? { warnings } : {}), pendingPlaceholders: true, appliedStatements: [...deltaLeg.value.appliedStatements, ...deletes], - summary: placeholderSummary('baseline + migration', [baselineDir, deltaDir]), + summary: placeholderSummary( + 'baseline + migration', + [baselineDir, deltaDir], + options.configPath, + ), timings: { total: Date.now() - startTime }, }; return ok(result); @@ -1038,7 +1051,7 @@ async function executeMigrationPlanCommandInner( pendingPlaceholders: true, ...(fromDefaulted ? { fromDefaulted } : {}), appliedStatements: [...deltaLeg.value.appliedStatements, ...deletes], - summary: placeholderSummary('migration', [dir]), + summary: placeholderSummary('migration', [dir], options.configPath), timings: { total: Date.now() - startTime }, }; return ok(result); diff --git a/packages/1-framework/3-tooling/cli/src/migration-cli.ts b/packages/1-framework/3-tooling/cli/src/migration-cli.ts index 5ed6baa933c2..4d34bd47fbb8 100644 --- a/packages/1-framework/3-tooling/cli/src/migration-cli.ts +++ b/packages/1-framework/3-tooling/cli/src/migration-cli.ts @@ -17,10 +17,8 @@ * 1. Detects whether the file is the direct entrypoint (no-op when imported). * 2. Parses CLI args (`--help`, `--dry-run`, `--config `) via * [clipanion](https://github.com/arcanis/clipanion). - * 3. Loads the project's `prisma.config.ts` via the same - * `loadConfigForSections` the CLI commands use. Without `--config`, it - * loads the nearest config found walking up from the migration file's - * directory, so the file runs from any working directory. + * 3. Loads the config `--config` names. Without it, loads the config that + * owns the migration (see `loadMigrationFileConfig`). * 4. Probe-instantiates the migration class without a stack so it can read * `targetId` and verify it matches `config.target.targetId` * (`MIGRATION.TARGET_MISMATCH` on mismatch) before any stack-driven adapter @@ -47,7 +45,6 @@ import { readFileSync, realpathSync, writeFileSync } from 'node:fs'; import type { Writable } from 'node:stream'; import { fileURLToPath } from 'node:url'; -import { findNearestConfigPathForFile, loadConfigForSections } from '@internal/config-loader'; import { type CliStructuredError, errorMigrationCliInvalidConfigArg, @@ -62,6 +59,7 @@ import { isStructuredError } from '@internal/utils/structured-error'; import { Cli, Command, Option, UsageError } from 'clipanion'; import { dirname, join } from 'pathe'; import { errorFromCaught } from './control-api/operations/caught-errors'; +import { loadMigrationFileConfig } from './utils/migration-file-config'; /** * Constructor shape accepted by `MigrationCLI.run`. `Migration` subclasses @@ -550,14 +548,12 @@ async function runMigration( const migrationFile = fileURLToPath(importMetaUrl); const migrationDir = dirname(migrationFile); - const configPath = parsed.config ?? (await findNearestConfigPathForFile(migrationFile)); - const configResult = await loadConfigForSections(configPath, [ - 'family', - 'target', - 'adapter', - 'driver', - 'extensions', - ]); + const configResult = await loadMigrationFileConfig({ + configPath: parsed.config, + migrationFile, + cwd: process.cwd(), + sections: ['family', 'target', 'adapter', 'driver', 'extensions'], + }); if (!configResult.ok) { throw configResult.failure; } diff --git a/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts b/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts index f65ed7fa7352..aa5ea17ede34 100644 --- a/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts +++ b/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts @@ -10,6 +10,7 @@ import type { MigrationNewResult } from '../../control-api/operations/migration- import { executeMigrationNewCommand } from '../../control-api/operations/migration-new'; import type { CreateControlClient } from '../../control-api/types'; import { runCommandAction } from '../../utils/next-actions'; +import { selfEmitCommand } from '../../utils/self-emit-command'; import { defineOrmCommand } from '../define-command'; import { normalizeError } from '../normalize-error'; import { appMigrationsDirFor, baseDirFor, contractPathFor, displayPath } from './paths'; @@ -18,6 +19,7 @@ function newPresentations(inputs: { readonly document: MigrationNewResult; readonly contractPath: string; readonly appMigrationsRelative: string; + readonly configPath: string | undefined; }): Presentations { const { document } = inputs; const migrationTs = join(document.dir, 'migration.ts'); @@ -52,7 +54,7 @@ function newPresentations(inputs: { { kind: 'edit-file', label: `Write the migration body in ${migrationTs}` }, runCommandAction( 'Run it to self-emit ops.json and attest the package', - `node "${migrationTs}"`, + selfEmitCommand(migrationTs, inputs.configPath), ), ], }; @@ -114,6 +116,7 @@ export function createMigrationNewCommand(createClient: CreateControlClient) { contractPath: contractPath === undefined ? '(unset)' : displayPath(contractPath, ctx.cwd), appMigrationsRelative: displayPath(appMigrationsDirFor(ctx.config), ctx.cwd), + configPath: ctx.configPath, }), ), ); diff --git a/packages/1-framework/3-tooling/cli/src/orm/migration/plan.ts b/packages/1-framework/3-tooling/cli/src/orm/migration/plan.ts index abd2bc4afd54..5116aa75569d 100644 --- a/packages/1-framework/3-tooling/cli/src/orm/migration/plan.ts +++ b/packages/1-framework/3-tooling/cli/src/orm/migration/plan.ts @@ -16,6 +16,7 @@ import { } from '../../utils/contract-ref-forms'; import { previewBlockHeader } from '../../utils/formatters/migrations'; import { runCommandAction } from '../../utils/next-actions'; +import { selfEmitCommand } from '../../utils/self-emit-command'; import { defineOrmCommand } from '../define-command'; import { normalizeError } from '../normalize-error'; import { promptPlanQuestions } from '../plan-question-prompt'; @@ -190,6 +191,7 @@ function planBlocks(result: MigrationPlanResult, migrationsRelative: string): re function planNextActions( result: MigrationPlanResult, migrationsRelative: string, + configPath: string | undefined, ): readonly NextAction[] { if (result.pendingPlaceholders === true) { const stubFiles = [ @@ -206,7 +208,7 @@ function planNextActions( }, runCommandAction( 'Run it to self-emit ops.json and attest the package', - `node "${migrationTs}"`, + selfEmitCommand(migrationTs, configPath), ), ]; } @@ -234,6 +236,7 @@ function planPresentations(inputs: { readonly from: string | undefined; readonly to: string | undefined; readonly name: string | undefined; + readonly configPath: string | undefined; }): Presentations { return { stdout: () => [], @@ -252,7 +255,7 @@ function planPresentations(inputs: { ...planBlocks(inputs.document, inputs.migrationsRelative), ], json: () => inputs.document, - next: () => planNextActions(inputs.document, inputs.migrationsRelative), + next: () => planNextActions(inputs.document, inputs.migrationsRelative, inputs.configPath), }; } @@ -312,6 +315,7 @@ export function createMigrationPlanCommand(createClient: CreateControlClient) { config: ctx.config, cwd: ctx.cwd, projectDir: baseDirFor(ctx.config), + ...ifDefined('configPath', ctx.configPath), ...ifDefined('name', args.flags.name), ...ifDefined('from', args.flags.from), ...ifDefined('to', args.flags.to), @@ -353,6 +357,7 @@ export function createMigrationPlanCommand(createClient: CreateControlClient) { from: args.flags.from, to: args.flags.to, name: args.flags.name, + configPath: ctx.configPath, }), ), ); diff --git a/packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts b/packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts new file mode 100644 index 000000000000..e679fa794f83 --- /dev/null +++ b/packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts @@ -0,0 +1,69 @@ +import { existsSync, realpathSync } from 'node:fs'; +import type { PrismaNextConfig } from '@internal/config/config-types'; +import { + CONFIG_FILENAME, + type ConfigSection, + findNearestConfigPathForFile, + loadConfig, + loadConfigForSections, + requireConfigSections, +} from '@internal/config-loader'; +import { + type CheckedMigrationConfig, + type CliStructuredError, + errorMigrationCliConfigNotFound, +} from '@internal/errors/control'; +import { notOk, ok, type Result } from '@internal/utils/result'; +import { dirname, isAbsolute, join, relative, resolve } from 'pathe'; + +/** + * The config a `migration.ts` runs with. `--config` names it outright. Without it, the config is the one that owns the migration, whose migrations directory holds the file: the nearest `prisma.config.ts` above the migration, else the one in the working directory. + */ +export async function loadMigrationFileConfig(options: { + readonly configPath: string | undefined; + readonly migrationFile: string; + readonly cwd: string; + readonly sections: readonly ConfigSection[]; +}): Promise> { + const { configPath, migrationFile, cwd, sections } = options; + if (configPath !== undefined) { + return loadConfigForSections(configPath, sections); + } + const workingDirectoryConfig = join(cwd, CONFIG_FILENAME); + const candidates = [ + ...new Set([ + await findNearestConfigPathForFile(migrationFile), + existsSync(workingDirectoryConfig) ? workingDirectoryConfig : undefined, + ]), + ].filter((candidate) => candidate !== undefined); + const checked: CheckedMigrationConfig[] = []; + for (const candidate of candidates) { + const loaded = await loadConfig(candidate); + const config = loaded.ok + ? requireConfigSections(loaded.value, [...sections, 'migrations']) + : loaded; + if (!config.ok) { + checked.push({ configPath: candidate, loadFailure: config.failure.message }); + continue; + } + const migrationsDir = migrationsDirOf(config.value, candidate); + if (contains(migrationsDir, migrationFile)) { + return ok(config.value); + } + checked.push({ configPath: candidate, migrationsDir }); + } + return notOk(errorMigrationCliConfigNotFound({ migrationDir: dirname(migrationFile), checked })); +} + +function migrationsDirOf(config: PrismaNextConfig, configPath: string): string { + return resolve(config.baseDir ?? dirname(configPath), config.migrations?.dir ?? 'migrations'); +} + +function contains(dir: string, file: string): boolean { + const path = relative(realPath(dir), realPath(file)); + return path !== '' && !path.startsWith('..') && !isAbsolute(path); +} + +function realPath(path: string): string { + return existsSync(path) ? realpathSync(path) : path; +} diff --git a/packages/1-framework/3-tooling/cli/src/utils/self-emit-command.ts b/packages/1-framework/3-tooling/cli/src/utils/self-emit-command.ts new file mode 100644 index 000000000000..7cd7f26921ab --- /dev/null +++ b/packages/1-framework/3-tooling/cli/src/utils/self-emit-command.ts @@ -0,0 +1,6 @@ +/** The command that runs a `migration.ts`, repeating the `--config` the CLI was given so it loads the same config. */ +export function selfEmitCommand(migrationTs: string, configPath: string | undefined): string { + return configPath === undefined + ? `node "${migrationTs}"` + : `node "${migrationTs}" --config "${configPath}"`; +} diff --git a/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts b/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts index 179f168d5b50..6a51e2add92f 100644 --- a/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts +++ b/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts @@ -19,15 +19,13 @@ import { structuredError } from '@internal/utils/structured-error'; import { join } from 'pathe'; import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; -const { loadConfigMock, findConfigMock, createControlStackMock } = vi.hoisted(() => ({ +const { loadConfigMock, createControlStackMock } = vi.hoisted(() => ({ loadConfigMock: vi.fn(), - findConfigMock: vi.fn(async (): Promise => undefined), createControlStackMock: vi.fn(), })); -vi.mock('@internal/config-loader', () => ({ - loadConfigForSections: loadConfigMock, - findNearestConfigPathForFile: findConfigMock, +vi.mock('../src/utils/migration-file-config', () => ({ + loadMigrationFileConfig: loadConfigMock, })); vi.mock('@internal/framework-components/control', async () => { @@ -41,6 +39,7 @@ vi.resetModules(); const { MigrationCLI } = await import('../src/migration-cli'); afterAll(() => { + vi.doUnmock('../src/utils/migration-file-config'); vi.resetModules(); }); @@ -351,11 +350,12 @@ describe('MigrationCLI.run', () => { }); expect(exitCode).toBe(0); - expect(loadConfigMock).toHaveBeenCalledWith('/explicit/config.ts', expect.any(Array)); + expect(loadConfigMock).toHaveBeenCalledWith( + expect.objectContaining({ configPath: '/explicit/config.ts' }), + ); }); - it('loads the config nearest the migration file when --config is not given', async () => { - findConfigMock.mockResolvedValueOnce('/project/prisma.config.ts'); + it('loads the config that owns the migration file when --config is not given', async () => { loadConfigMock.mockResolvedValue(ok(okConfig)); createControlStackMock.mockReturnValue({ adapter: { create: () => ({}) } }); @@ -365,8 +365,12 @@ describe('MigrationCLI.run', () => { stderr: new BufferStream(), }); - expect(findConfigMock).toHaveBeenCalledWith(migrationFile); - expect(loadConfigMock).toHaveBeenCalledWith('/project/prisma.config.ts', expect.any(Array)); + expect(loadConfigMock).toHaveBeenCalledWith({ + configPath: undefined, + migrationFile, + cwd: process.cwd(), + sections: expect.any(Array), + }); }); it('forwards --config= (equals form) to loadConfig', async () => { @@ -382,7 +386,9 @@ describe('MigrationCLI.run', () => { }); expect(exitCode).toBe(0); - expect(loadConfigMock).toHaveBeenCalledWith('/equals/config.ts', expect.any(Array)); + expect(loadConfigMock).toHaveBeenCalledWith( + expect.objectContaining({ configPath: '/equals/config.ts' }), + ); }); it('preserves createdAt from a previously-scaffolded migration.json', async () => { diff --git a/packages/1-framework/3-tooling/cli/test/orm/migration-new.test.ts b/packages/1-framework/3-tooling/cli/test/orm/migration-new.test.ts index 0dc4c233d8ea..c02fcfdb1a1b 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/migration-new.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/migration-new.test.ts @@ -142,6 +142,23 @@ describe('migration new', () => { ]); }); + it('repeats the --config it was given in the command that runs the stub', async () => { + const project = await createOfflineProject({ storageHash: HASH_TO }); + + const run = await harness(project).run(['migration', 'new', '--config', 'prisma.config.ts'], { + cwd: project.dir, + isTty: { stdout: true }, + }); + const [dirName] = await scaffoldedDirs(project); + const migrationTs = join('migrations', 'app', dirName ?? '', 'migration.ts'); + + expect(run.presented?.presentation.next?.at(-1)).toEqual({ + kind: 'run-command', + label: 'Run it to self-emit ops.json and attest the package', + command: `node "${migrationTs}" --config "prisma.config.ts"`, + }); + }); + it('writes nothing to stdout in human mode', async () => { const project = await createOfflineProject({ storageHash: HASH_TO }); diff --git a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts index 5b5e10d7af79..149f5efcd0a7 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts @@ -850,6 +850,32 @@ describe('migration plan', () => { }); }); + it('repeats the --config it was given in the commands that run the stub', async () => { + const project = await losingProject(); + const script: FakePlannerScript = { + operations: [ADDITIVE_OP, DESTRUCTIVE_OP], + operationsAccessorThrows: errorUnfilledPlaceholder('backfill'), + dataLoss: [LEGACY_LOSS], + }; + + const answered = await harness(project, { script }).run( + ['migration', 'plan', '--delete', 'Legacy', '--config', 'prisma.config.ts', '--json'], + { cwd: project.dir }, + ); + const data = answered.presented?.data as { readonly dir: string; readonly summary: string }; + expect({ + summary: data.summary, + next: answered.presented?.presentation.next?.at(-1), + }).toEqual({ + summary: `Planned migration with placeholder(s) — edit migration.ts, then run \`node ${data.dir}/migration.ts --config "prisma.config.ts"\` from this directory to self-emit`, + next: { + kind: 'run-command', + label: 'Run it to self-emit ops.json and attest the package', + command: `node "${data.dir}/migration.ts" --config "prisma.config.ts"`, + }, + }); + }); + it('asks a question a re-plan brings up in a second batch', async () => { const project = await losingProject(); diff --git a/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts b/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts new file mode 100644 index 000000000000..c800f11e2443 --- /dev/null +++ b/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts @@ -0,0 +1,153 @@ +import { mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { errorConfigFileNotFound } from '@internal/errors/control'; +import { notOk, ok } from '@internal/utils/result'; +import { join } from 'pathe'; +import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +const { loadConfigMock, loadConfigForSectionsMock } = vi.hoisted(() => ({ + loadConfigMock: vi.fn(), + loadConfigForSectionsMock: vi.fn(), +})); + +vi.mock('@internal/config-loader', async () => { + const actual = + await vi.importActual('@internal/config-loader'); + return { + ...actual, + loadConfig: loadConfigMock, + loadConfigForSections: loadConfigForSectionsMock, + }; +}); + +const { loadMigrationFileConfig } = await import('../../src/utils/migration-file-config'); + +afterAll(() => { + vi.doUnmock('@internal/config-loader'); + vi.resetModules(); +}); + +const SECTIONS = ['family', 'target'] as const; + +let root: string; + +beforeEach(() => { + root = realpathSync(mkdtempSync(join(tmpdir(), 'migration-file-config-'))); + loadConfigMock.mockReset(); + loadConfigForSectionsMock.mockReset(); +}); + +afterEach(() => { + rmSync(root, { recursive: true, force: true }); +}); + +function writeConfig(dir: string): string { + mkdirSync(dir, { recursive: true }); + const path = join(dir, 'prisma.config.ts'); + writeFileSync(path, ''); + return path; +} + +function writeMigration(dir: string): string { + mkdirSync(dir, { recursive: true }); + const path = join(dir, 'migration.ts'); + writeFileSync(path, ''); + return path; +} + +/** Each config path loads a config that keeps its migrations in the given directory. */ +function configsKeepingMigrationsIn(dirs: Record): void { + loadConfigMock.mockImplementation(async (configPath: string) => { + const dir = dirs[configPath]; + return dir === undefined + ? notOk(errorConfigFileNotFound(configPath)) + : ok({ config: { configPath, migrations: { dir } }, diagnostics: [] }); + }); +} + +describe('loadMigrationFileConfig', () => { + it('loads the config --config names, without checking that it owns the migration', async () => { + loadConfigForSectionsMock.mockResolvedValue(ok({ named: true })); + + const loaded = await loadMigrationFileConfig({ + configPath: 'custom.config.ts', + migrationFile: join(root, 'elsewhere/migration.ts'), + cwd: root, + sections: SECTIONS, + }); + + expect(loaded.assertOk()).toEqual({ named: true }); + expect(loadConfigForSectionsMock).toHaveBeenCalledWith('custom.config.ts', SECTIONS); + expect(loadConfigMock).not.toHaveBeenCalled(); + }); + + it('uses the nearest config above the migration when its migrations directory holds it', async () => { + const config = writeConfig(root); + const migrationFile = writeMigration(join(root, 'db/migrations/app/0001_initial')); + configsKeepingMigrationsIn({ [config]: join(root, 'db/migrations') }); + + const loaded = await loadMigrationFileConfig({ + configPath: undefined, + migrationFile, + cwd: join(root, 'db'), + sections: SECTIONS, + }); + + expect(loaded.assertOk()).toMatchObject({ configPath: config }); + }); + + it('falls back to the working directory config when the nearest does not own the migration', async () => { + const outer = writeConfig(root); + const project = writeConfig(join(root, 'app')); + const migrationFile = writeMigration(join(root, 'shared-db/migrations/app/0001_initial')); + configsKeepingMigrationsIn({ + [outer]: join(root, 'migrations'), + [project]: join(root, 'shared-db/migrations'), + }); + + const loaded = await loadMigrationFileConfig({ + configPath: undefined, + migrationFile, + cwd: join(root, 'app'), + sections: SECTIONS, + }); + + expect(loaded.assertOk()).toMatchObject({ configPath: project }); + }); + + it('falls back to the working directory config when the nearest fails to load', async () => { + const outer = writeConfig(root); + const project = writeConfig(join(root, 'app')); + const migrationFile = writeMigration(join(root, 'shared-db/migrations/app/0001_initial')); + configsKeepingMigrationsIn({ [project]: join(root, 'shared-db/migrations') }); + + const loaded = await loadMigrationFileConfig({ + configPath: undefined, + migrationFile, + cwd: join(root, 'app'), + sections: SECTIONS, + }); + + expect(loaded.assertOk()).toMatchObject({ configPath: project }); + expect(loadConfigMock).toHaveBeenCalledWith(outer); + }); + + it('refuses, naming each config it checked, when none owns the migration', async () => { + const outer = writeConfig(root); + const migrationDir = join(root, 'inner/migrations/app/0001_initial'); + const migrationFile = writeMigration(migrationDir); + configsKeepingMigrationsIn({ [outer]: join(root, 'migrations') }); + + const loaded = await loadMigrationFileConfig({ + configPath: undefined, + migrationFile, + cwd: migrationDir, + sections: SECTIONS, + }); + + expect(loaded.ok ? loaded.value : loaded.failure.toEnvelope()).toMatchObject({ + code: 'CLI.MIGRATION_CONFIG_NOT_FOUND', + meta: { migrationDir, checked: [outer] }, + }); + }); +}); diff --git a/packages/1-framework/3-tooling/config-loader/src/exports/index.ts b/packages/1-framework/3-tooling/config-loader/src/exports/index.ts index a45ee4656385..973a41994af3 100644 --- a/packages/1-framework/3-tooling/config-loader/src/exports/index.ts +++ b/packages/1-framework/3-tooling/config-loader/src/exports/index.ts @@ -3,6 +3,7 @@ export { expandContractInputs, globContractInputMatching } from '../expand-contr export type { ConfigFile, ConfigFiles, LoadedConfig } from '../load'; export { buildLoadedConfig, + CONFIG_FILENAME, evaluateConfigModule, findNearestConfigPathForFile, loadConfig, diff --git a/packages/1-framework/3-tooling/config-loader/src/load.ts b/packages/1-framework/3-tooling/config-loader/src/load.ts index 458dd99e8930..0b2ec8abc281 100644 --- a/packages/1-framework/3-tooling/config-loader/src/load.ts +++ b/packages/1-framework/3-tooling/config-loader/src/load.ts @@ -25,7 +25,7 @@ import { validateOrmSectionExcept, } from './orm-section'; -const CONFIG_FILENAME = 'prisma.config.ts'; +export const CONFIG_FILENAME = 'prisma.config.ts'; export type { ConfigSection }; diff --git a/packages/1-framework/3-tooling/migration/src/errors.ts b/packages/1-framework/3-tooling/migration/src/errors.ts index b2201406e4ae..2ceac23dcd84 100644 --- a/packages/1-framework/3-tooling/migration/src/errors.ts +++ b/packages/1-framework/3-tooling/migration/src/errors.ts @@ -16,7 +16,7 @@ import type { MigrationGraph } from './graph'; */ function reemitHint(dir: string, fallback?: string): string { const relativeDir = relative(process.cwd(), dir); - const reemit = `Re-emit the package by running \`node "${relativeDir}/migration.ts"\``; + const reemit = `Re-emit the package by running \`node "${relativeDir}/migration.ts"\`, adding the \`--config \` you gave this command, if any`; return fallback ? `${reemit}, ${fallback}` : `${reemit}.`; } diff --git a/packages/1-framework/3-tooling/migration/test/errors.test.ts b/packages/1-framework/3-tooling/migration/test/errors.test.ts index 9bfee2759e06..ac1e87705f8d 100644 --- a/packages/1-framework/3-tooling/migration/test/errors.test.ts +++ b/packages/1-framework/3-tooling/migration/test/errors.test.ts @@ -1,4 +1,5 @@ import { CliStructuredError } from '@internal/errors/control'; +import { join } from 'pathe'; import { describe, expect, it } from 'vitest'; import { errorBundleNotFoundForGraphNode, @@ -252,6 +253,13 @@ describe('MigrationToolsError base type', () => { expect(MigrationToolsError.is(err)).toBe(true); }); + it('tells the user to repeat --config when re-emitting a package', () => { + const err = errorMissingFile('ops.json', join(process.cwd(), 'db/migrations/app/0001_init')); + expect(err.fix).toBe( + 'Re-emit the package by running `node "db/migrations/app/0001_init/migration.ts"`, adding the `--config ` you gave this command, if any, or delete the directory if the migration is unwanted and the source TypeScript is gone.', + ); + }); + it('exposes toEnvelope() with code, summary, why, fix, nextActions, and meta', () => { const err = errorDirectoryExists('/tmp/m/20260101_init'); expect(err.toEnvelope()).toEqual({ diff --git a/test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts b/test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts index a7a8c002cbac..39d4b3ad2e90 100644 --- a/test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts +++ b/test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts @@ -1,10 +1,10 @@ /** - * `node migration.ts` finds the project's config by walking up from the migration's own directory, so it runs from any working directory, including when `migrations.dir` is not the default. + * Without `--config`, `node migration.ts` uses the config that owns the migration: the one whose migrations directory holds the file. It tries the nearest `prisma.config.ts` above the migration, then the one in the working directory, and refuses when neither owns the file. */ import { spawnSync } from 'node:child_process'; -import { readdirSync, readFileSync, writeFileSync } from 'node:fs'; -import { join, relative } from 'pathe'; +import { mkdirSync, readdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join, relative } from 'pathe'; import { describe, expect, it } from 'vitest'; import { withTempDir, writeProjectManifest } from '../utils/cli-test-helpers'; import { @@ -23,51 +23,101 @@ model Profile { } `; -function setupProject(createTempDir: () => string): JourneyContext { - const testDir = createTempDir(); - const configPath = join(testDir, 'prisma.config.ts'); +const THROWING_CONFIG = "throw new Error('the outer prisma.config.ts was loaded');\n"; + +/** Writes a SQLite project into `dir` whose config, named `configName`, keeps migrations in `migrationsDir`. */ +function writeProject(dir: string, migrationsDir: string, configName = 'prisma.config.ts') { + mkdirSync(dir, { recursive: true }); + const configPath = join(dir, configName); const config = readFileSync(sqlitePslConfigFixture, 'utf-8') - .replace('{{DB_PATH}}', () => join(testDir, 'journey.db')) - .replace("migrations: { dir: 'migrations' }", "migrations: { dir: './db/migrations' }"); + .replace('{{DB_PATH}}', () => join(dir, 'journey.db')) + .replace("migrations: { dir: 'migrations' }", () => `migrations: { dir: '${migrationsDir}' }`); writeFileSync(configPath, config, 'utf-8'); - writeFileSync(join(testDir, 'contract.prisma'), PSL, 'utf-8'); - writeProjectManifest(testDir); - return { testDir, configPath, outputDir: testDir }; + writeFileSync(join(dir, 'contract.prisma'), PSL, 'utf-8'); + writeProjectManifest(dir); + return { testDir: dir, configPath, outputDir: dir } satisfies JourneyContext; +} + +/** Emits the contract, plans the first migration, and returns the migration's directory. */ +async function planFirstMigration(ctx: JourneyContext, appDir: string): Promise { + const emitted = await runContractEmit(ctx); + expect(emitted.exitCode, `emit: ${emitted.stderr}`).toBe(0); + const planned = await runMigrationPlan(ctx, ['--name', 'initial']); + expect(planned.exitCode, `plan: ${planned.stderr}`).toBe(0); + const [dir] = readdirSync(appDir).filter((name) => name !== 'refs'); + return join(appDir, String(dir)); } -function runNode(file: string, cwd: string) { - const run = spawnSync(process.execPath, [file], { cwd, encoding: 'utf-8' }); +function runNode(args: readonly string[], cwd: string) { + const run = spawnSync(process.execPath, args, { cwd, encoding: 'utf-8' }); return { exitCode: run.status, stdout: run.stdout, stderr: run.stderr }; } +const WROTE = { exitCode: 0, stdout: expect.stringContaining('Wrote ops.json + migration.json') }; + withTempDir(({ createTempDir }) => { describe('node migration.ts', () => { it( - 'finds the config from the migration directory and from the project directory', + 'finds the config from the migration directory, the project directory and its parent', async () => { - const ctx = setupProject(createTempDir); - const emitted = await runContractEmit(ctx); - expect(emitted.exitCode, `emit: ${emitted.stderr}`).toBe(0); - const planned = await runMigrationPlan(ctx, ['--name', 'initial']); - expect(planned.exitCode, `plan: ${planned.stderr}`).toBe(0); - const appDir = join(ctx.testDir, 'db/migrations/app'); - const [dir] = readdirSync(appDir).filter((name) => name !== 'refs'); - const migrationDir = join(appDir, String(dir)); + const ctx = writeProject(createTempDir(), './db/migrations'); + const migrationDir = await planFirstMigration(ctx, join(ctx.testDir, 'db/migrations/app')); + const file = join(migrationDir, 'migration.ts'); - const fromMigrationDir = runNode('migration.ts', migrationDir); - expect(fromMigrationDir, 'from the migration directory').toMatchObject({ - exitCode: 0, - stdout: expect.stringContaining('Wrote ops.json + migration.json'), - }); + expect( + runNode(['migration.ts'], migrationDir), + 'from the migration directory', + ).toMatchObject(WROTE); + expect( + runNode([relative(ctx.testDir, file)], ctx.testDir), + 'from the project directory', + ).toMatchObject(WROTE); + const parent = dirname(ctx.testDir); + expect(runNode([relative(parent, file)], parent), 'from the parent').toMatchObject(WROTE); + }, + timeouts.spinUpPpgDev, + ); - const fromProjectDir = runNode( - relative(ctx.testDir, join(migrationDir, 'migration.ts')), - ctx.testDir, - ); - expect(fromProjectDir, 'from the project directory').toMatchObject({ - exitCode: 0, - stdout: expect.stringContaining('Wrote ops.json + migration.json'), + it( + 'uses the working directory config when the migrations directory is outside the project', + async () => { + const root = createTempDir(); + writeFileSync(join(root, 'prisma.config.ts'), THROWING_CONFIG, 'utf-8'); + const ctx = writeProject(join(root, 'app'), '../shared-db/migrations'); + const migrationDir = await planFirstMigration(ctx, join(root, 'shared-db/migrations/app')); + + expect( + runNode([relative(ctx.testDir, join(migrationDir, 'migration.ts'))], ctx.testDir), + 'from the project directory', + ).toMatchObject(WROTE); + }, + timeouts.spinUpPpgDev, + ); + + it( + 'refuses when no config it finds owns the migration, and runs with --config', + async () => { + const root = createTempDir(); + writeProject(root, './migrations'); + const ctx = writeProject(join(root, 'inner'), './migrations', 'prisma.sqlite.config.ts'); + const migrationDir = await planFirstMigration(ctx, join(ctx.testDir, 'migrations/app')); + + const refused = runNode(['migration.ts'], migrationDir); + expect(refused, 'without --config').toMatchObject({ + exitCode: 1, + stdout: '', + stderr: expect.stringContaining('CLI.MIGRATION_CONFIG_NOT_FOUND'), }); + expect(refused.stderr, 'names the config it checked').toContain( + join(root, 'prisma.config.ts'), + ); + expect( + runNode( + ['migration.ts', '--config', relative(migrationDir, ctx.configPath)], + migrationDir, + ), + 'with --config', + ).toMatchObject(WROTE); }, timeouts.spinUpPpgDev, ); diff --git a/test/integration/test/utils/cli-test-helpers.ts b/test/integration/test/utils/cli-test-helpers.ts index 5541749d33fc..705496b16f14 100644 --- a/test/integration/test/utils/cli-test-helpers.ts +++ b/test/integration/test/utils/cli-test-helpers.ts @@ -494,9 +494,9 @@ class CapturingWritable extends Writable { * own tests use. * * Two process globals are saved and restored around the run, because the - * migration-file CLI is written for a process it owns: config discovery walks - * up from `process.cwd()` (so the helper chdirs to `cwd`, exactly where the - * old spawn pointed the child), and a failing run sets `process.exitCode` + * migration-file CLI is written for a process it owns: config discovery falls + * back to the config in `process.cwd()` (so the helper chdirs to `cwd`, + * exactly where the old spawn pointed the child), and a failing run sets `process.exitCode` * (which must not leak into the vitest worker's exit status when a test * asserts on a migration failure). Tests within a worker run sequentially * under the forks pool, so the temporary chdir cannot interleave with another diff --git a/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md b/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md index 866e529a26eb..1e72f1003225 100644 --- a/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md +++ b/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md @@ -25,7 +25,7 @@ changes: - '--delete loses its rows' - id: migration-ts-finds-its-config summary: | - Running a `migration.ts` with node loads the nearest `prisma.config.ts` above the migration's own directory, so `node migration.ts` works from any directory. It used to look only in the working directory, and failed with `CONFIG.FILE_NOT_FOUND` anywhere else unless `--config` named the file. + Running a `migration.ts` with node and no `--config` uses the config that owns the migration, whose migrations directory holds the file: the nearest `prisma.config.ts` above the migration's directory, else the one in the working directory. If neither owns it, it fails with `CLI.MIGRATION_CONFIG_NOT_FOUND`. It used to load only the working directory's config, whichever project it belonged to. --- # MongoDB carries out `--rename` and `--delete` @@ -67,4 +67,6 @@ A test that asserts the text of a data-loss question about data named by its sto ## `migration-ts-finds-its-config` -`node migration.ts`, run inside the migration's directory, now finds the project's `prisma.config.ts`, as does `node /migration.ts` run from any other directory. A command that runs the file from the project directory keeps working unchanged. A script that passed `--config` only because it ran the file from another directory can drop it; `--config` still overrides the lookup. Messages that tell you to run a `migration.ts` now say to run it with node, without a path or a working directory. +Without `--config`, `node migration.ts` now picks the config that owns the migration: a config owns it when the migration file is inside that config's `migrations.dir`. It tries the nearest `prisma.config.ts` above the migration's directory first, then the `prisma.config.ts` in the working directory. So the file runs from inside its own directory, from the project directory, or from any other directory, and with `migrations.dir` outside the project, such as `../shared-db/migrations`, when run from the project directory. If neither config owns the migration, it fails with `CLI.MIGRATION_CONFIG_NOT_FOUND`, naming the configs it checked; run it with `--config `. `--config` still overrides the lookup. + +A command that ran the file from the project directory keeps working. A script that ran it from a directory whose `prisma.config.ts` belongs to another project, and relied on that config, now fails with `CLI.MIGRATION_CONFIG_NOT_FOUND`; pass `--config`. `migration new` and `migration plan` given `--config` now repeat it in the `node` command they print. Messages that tell you to run a `migration.ts` say to run it with node, without a path or a working directory. From 63ef84c546e41b7b2ccdbcb1e4cd0aa3d91994d7 Mon Sep 17 00:00:00 2001 From: willbot Date: Tue, 13 Oct 2026 08:48:04 +0200 Subject: [PATCH 53/57] TML-3478: skills and docs give the self-emit command as node /migration.ts They named migrations/app/, which is wrong when migrations.dir is set. The prisma-8 skill and the CLI README also say how the file finds its config and to repeat --config. Signed-off-by: willbot Signed-off-by: Will Madden --- .../subsystems/7. Migration System.md | 6 +++--- examples/multi-extension-monorepo/README.md | 4 ++-- packages/1-framework/3-tooling/cli/README.md | 8 +++++--- .../02c-data-transform-placeholder.md | 2 +- skills/journey-tests/02e-hash-mismatch.md | 2 +- skills/journey-tests/02g-diamond-convergence.md | 2 +- skills/prisma-8/references/debug.md | 4 ++-- skills/prisma-8/references/migrations.md | 14 +++++++------- 8 files changed, 22 insertions(+), 20 deletions(-) diff --git a/docs/architecture docs/subsystems/7. Migration System.md b/docs/architecture docs/subsystems/7. Migration System.md index 25b064fe6883..39db64e2ac34 100644 --- a/docs/architecture docs/subsystems/7. Migration System.md +++ b/docs/architecture docs/subsystems/7. Migration System.md @@ -324,7 +324,7 @@ Each operation follows the same three-phase loop: evaluate prechecks → execute Per [ADR 192 — Apply-time verification](../adrs/ADR%20192%20-%20ops.json%20is%20the%20migration%20contract.md), `db migrate` performs two checks before executing operations: -1. **On-disk consistency.** For every loaded bundle, recompute `migrationHash` from the on-disk manifest + `ops.json` (`verifyMigrationBundle`) and compare against the stored `migrationHash`. A mismatch aborts apply with a structured runtime error pointing at the offending directory and asks the developer to re-emit the package (`node migrations//migration.ts`) or restore from version control. This catches FS corruption, partial writes, and post-emit hand edits before the runner touches the database. +1. **On-disk consistency.** For every loaded bundle, recompute `migrationHash` from the on-disk manifest + `ops.json` (`verifyMigrationBundle`) and compare against the stored `migrationHash`. A mismatch aborts apply with a structured runtime error pointing at the offending directory and asks the developer to re-emit the package (`node /migration.ts`) or restore from version control. This catches FS corruption, partial writes, and post-emit hand edits before the runner touches the database. 2. **Emit-drift detection.** When `migration.ts` is present, re-emit in-memory and compare against `ops.json`. *(Architecturally required; not yet implemented — tracked as a follow-up.)* ### Pre-DDL marker drift check @@ -565,7 +565,7 @@ A migration always exists or it doesn't — there is no on-disk "draft" state. E - Attested: `migrationHash` matches current content (the only on-disk state) - Applied: Ledger shows edge applied; DB marker equals `to` hash -The planner exposes one transient hint, `pendingPlaceholders`, in the `migration plan` result envelope. It is a CLI-only signal that the on-disk migration encodes zero ops because the user still needs to fill in `placeholder(...)` slots; it is **not** persisted to the manifest. After filling in the slots, the developer re-emits the package (`node migrations//migration.ts`) to compute the real ops and the corresponding `migrationHash`. Applying a placeholder-blocked migration will not advance the storage hash to the destination — the runner's destination-hash post-check surfaces this as a state mismatch. +The planner exposes one transient hint, `pendingPlaceholders`, in the `migration plan` result envelope. It is a CLI-only signal that the on-disk migration encodes zero ops because the user still needs to fill in `placeholder(...)` slots; it is **not** persisted to the manifest. After filling in the slots, the developer re-emits the package (`node /migration.ts`) to compute the real ops and the corresponding `migrationHash`. Applying a placeholder-blocked migration will not advance the storage hash to the destination — the runner's destination-hash post-check surfaces this as a state mismatch. ### Authoring intermediate state @@ -594,7 +594,7 @@ A placeholder-bearing migration is therefore visible to the runner. Applying it #### Transition -The intermediate state ends when the developer runs the package's `migration.ts` directly — either via its shebang (`./migrations//migration.ts`) or `node migrations//migration.ts`. Class-flow self-emission (see [ADR 196 — In-process emit for class-flow targets](../adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md)) detects that the file is the main module, runs `Migration.run(...)`, calls `buildMigrationArtifacts` against the now-real op list, and **rewrites both `ops.json` and `migration.json`** in place — including the `migrationHash`, which is recomputed over the new `(strippedManifest, ops)`. Any unfilled `placeholder(...)` still in the file at this point throws `PN-MIG-2001` and aborts the rewrite, leaving the package in its previous (still-intermediate) shape; see [ADR 200 — Placeholder utility for scaffolded migration slots](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md). There is no separate CLI command for this transition: the on-disk package is the source of truth, and running `migration.ts` is what materializes the filled `ops.json`. +The intermediate state ends when the developer runs the package's `migration.ts` directly — either via its shebang (`.//migration.ts`) or `node /migration.ts`. Class-flow self-emission (see [ADR 196 — In-process emit for class-flow targets](../adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md)) detects that the file is the main module, runs `Migration.run(...)`, calls `buildMigrationArtifacts` against the now-real op list, and **rewrites both `ops.json` and `migration.json`** in place — including the `migrationHash`, which is recomputed over the new `(strippedManifest, ops)`. Any unfilled `placeholder(...)` still in the file at this point throws `PN-MIG-2001` and aborts the rewrite, leaving the package in its previous (still-intermediate) shape; see [ADR 200 — Placeholder utility for scaffolded migration slots](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md). There is no separate CLI command for this transition: the on-disk package is the source of truth, and running `migration.ts` is what materializes the filled `ops.json`. #### Diagnostics diff --git a/examples/multi-extension-monorepo/README.md b/examples/multi-extension-monorepo/README.md index 46c9ac0d6f26..7319a8475981 100644 --- a/examples/multi-extension-monorepo/README.md +++ b/examples/multi-extension-monorepo/README.md @@ -78,9 +78,9 @@ Each internal "package" under `packages/` follows the **contract-space package l Then hand-edit the generated `migrations//migration.ts`'s `operations` getter so each op carries the package's stable `:-vN` `invariantId` (invariant IDs cannot be renamed once published). Re-emit `ops.json` + `migration.json`: ```sh - node migrations//migration.ts + node /migration.ts # or, on Node < 24: - pnpm exec tsx migrations//migration.ts + pnpm exec tsx /migration.ts ``` 4. Update `migrations/refs/head.json` to point at the new contract `storageHash` plus the union of `providedInvariants` across all migrations. diff --git a/packages/1-framework/3-tooling/cli/README.md b/packages/1-framework/3-tooling/cli/README.md index 3c876a8a9753..c9335b2fd40f 100644 --- a/packages/1-framework/3-tooling/cli/README.md +++ b/packages/1-framework/3-tooling/cli/README.md @@ -1036,7 +1036,7 @@ prisma migration plan [--config ] [--name ] [--from ] [--t 3. Determines the starting point: `--from ` if provided, otherwise the `db` ref. When the ref is absent, greenfield only on an empty graph; over existing migrations the command refuses (`MIGRATION.PLAN_ORIGIN_UNKNOWN`) unless `--from @empty` is passed 4. Diffs the starting contract against the destination using the target's migration planner 5. Scaffolds a new migration package: `migration.ts` (containing `placeholder(...)` lambdas for any data transforms), `migration.json` (with a content-addressed `migrationHash` over the planned ops, or over `[]` when the planner could not lower any calls because of placeholders), and `ops.json` (the planned ops, or `[]` in the placeholder-blocked case). The bookend contracts are written write-if-absent into the shared snapshot store at `migrations/snapshots//contract.{json,d.ts}`. The package is **always** fully attested — there is no draft state on disk. -6. If the plan has unfilled `placeholder(...)` slots, the command returns a successful `pendingPlaceholders` envelope (a warning, not a failure) asking the developer to fill in the slots before re-emitting. The output still lists the operations that resolved, and any `Statements applied`, so a rename and the drops beside it show; JSON `operations` lists the same operations, so the `operationIndexes` of `appliedStatements` point into it. The on-disk `ops.json` is `[]` and `migrationHash` is the hash of `(metadata, [])`, so applying the migration as-written will not advance the storage hash to the intended destination — the runner's destination-hash post-check surfaces this as a state mismatch. After filling in the placeholders, run `node migrations//migration.ts` to re-emit `ops.json` and the corresponding `migrationHash`. `PN-MIG-2001` is raised only at self-emit time when a slot is still unfilled. +6. If the plan has unfilled `placeholder(...)` slots, the command returns a successful `pendingPlaceholders` envelope (a warning, not a failure) asking the developer to fill in the slots before re-emitting. The output still lists the operations that resolved, and any `Statements applied`, so a rename and the drops beside it show; JSON `operations` lists the same operations, so the `operationIndexes` of `appliedStatements` point into it. The on-disk `ops.json` is `[]` and `migrationHash` is the hash of `(metadata, [])`, so applying the migration as-written will not advance the storage hash to the intended destination — the runner's destination-hash post-check surfaces this as a state mismatch. After filling in the placeholders, run `node /migration.ts` to re-emit `ops.json` and the corresponding `migrationHash`. `PN-MIG-2001` is raised only at self-emit time when a slot is still unfilled. **Outputs:** - `migrations//migration.ts` — editable migration source (with `placeholder(...)` slots when the planner inserted them) @@ -1127,7 +1127,7 @@ prisma db migrate [--db ] [--to ] [--advance-ref ] [--show] - `-v, --verbose`: Verbose output (debug info, timings) **What it does:** -1. Reads migration packages from `config.migrations.dir`. Every package is attested — there is no on-disk draft state. The loader (`readMigrationPackage` in `@internal/migration-tools/io`) rehashes `(metadata, ops)` for each `MigrationPackage` it returns and confirms the result matches the stored `migrationHash`. If a package has been hand-edited or partially written since emit, the load fails with `MIGRATION.HASH_MISMATCH` pointing at the offending directory and asks the developer to re-run `node migrations//migration.ts` (or restore from version control). +1. Reads migration packages from `config.migrations.dir`. Every package is attested — there is no on-disk draft state. The loader (`readMigrationPackage` in `@internal/migration-tools/io`) rehashes `(metadata, ops)` for each `MigrationPackage` it returns and confirms the result matches the stored `migrationHash`. If a package has been hand-edited or partially written since emit, the load fails with `MIGRATION.HASH_MISMATCH` pointing at the offending directory and asks the developer to re-run `node /migration.ts` (or restore from version control). 2. Reconstructs the migration graph from all loaded packages 3. Determines the destination hash and apply contract: from `--to`, or from `contract.json` when `--to` is omitted or `@contract` 4. Connects to the database and reads the current marker hash @@ -1150,9 +1150,11 @@ run `migration.ts` directly with Node to produce `ops.json` and attest `migration.json`: ```bash -node migrations//migration.ts +node /migration.ts ``` +`` is the directory `migration plan` or `migration new` printed, under the config's `migrations.dir`. The command works from any directory. Without `--config`, the file uses the config whose migrations directory holds it: the nearest `prisma.config.ts` above it, else the one in the working directory; if neither does, it fails with `CLI.MIGRATION_CONFIG_NOT_FOUND`. If you ran the CLI with `--config`, pass the same `--config ` to the file; the commands `migration plan` and `migration new` print already include it. + The scaffolded `migration.ts` calls `MigrationCLI.run(import.meta.url, ...)` from `@internal/cli/migration-cli` when invoked as the entrypoint. (Postgres and SQLite scaffolds re-export `MigrationCLI` through `@internal/postgres/migration` or `@internal/sqlite/migration` so a `migration.ts` only needs the single facade import; Mongo scaffolds still pull from `@internal/cli/migration-cli` directly.) The CLI entrypoint loads `prisma.config.ts`, assembles a `ControlStack`, instantiates the migration with that stack (so `dataTransform` and other adapter-aware helpers can materialize a real adapter), and serializes operations to `ops.json` while writing the content-addressed `migrationHash` into `migration.json`. If `migration.ts` contains unfilled `placeholder()` slots, the script exits with `PN-MIG-2001` and reports the slot to fill in. `MigrationCLI.run` accepts an optional third argument `{ argv?, stdout?, stderr? }` for in-process testability (default: `process.argv` / `process.stdout` / `process.stderr`) and returns the exit code as a `Promise`. The flag surface is `--help` / `--dry-run` / `--config `, parsed by [`clipanion`](https://github.com/arcanis/clipanion). The main multi-command surface (`contract emit`, `db verify`, etc.) runs on `@prisma/cli-engine`; the per-migration `MigrationCLI.run` entrypoint uses clipanion to keep authored migration files lightweight and in-process testable. diff --git a/skills/journey-tests/02c-data-transform-placeholder.md b/skills/journey-tests/02c-data-transform-placeholder.md index 07021b221a1d..ea9ba4961bbb 100644 --- a/skills/journey-tests/02c-data-transform-placeholder.md +++ b/skills/journey-tests/02c-data-transform-placeholder.md @@ -14,7 +14,7 @@ - [ ] Emits, plans, observes a `placeholder(...)` in `migration.ts`. - [ ] Replaces the placeholder with `UPDATE user SET displayName = email WHERE displayName IS NULL`. - [ ] Adds a follow-up step to ALTER COLUMN to NOT NULL. -- [ ] Self-emits the migration (`node migrations//migration.ts`). +- [ ] Self-emits the migration (`node /migration.ts`). - [ ] Applies. ## Success criteria diff --git a/skills/journey-tests/02e-hash-mismatch.md b/skills/journey-tests/02e-hash-mismatch.md index 97509dfafcec..f15e7d231679 100644 --- a/skills/journey-tests/02e-hash-mismatch.md +++ b/skills/journey-tests/02e-hash-mismatch.md @@ -23,7 +23,7 @@ Plan a migration. Edit its `migration.ts` (add a no-op comment). Do NOT self-emi - [ ] Reads the envelope, recognises `MIGRATION.HASH_MISMATCH`. - [ ] Names the cause: `migration.ts` edited after the initial emit. -- [ ] Runs `node migrations//migration.ts` (self-emit). +- [ ] Runs `node /migration.ts` (self-emit). - [ ] Re-runs `db migrate`. ## Success criteria diff --git a/skills/journey-tests/02g-diamond-convergence.md b/skills/journey-tests/02g-diamond-convergence.md index 7b0c2b92fd3d..11cd17c195b7 100644 --- a/skills/journey-tests/02g-diamond-convergence.md +++ b/skills/journey-tests/02g-diamond-convergence.md @@ -20,7 +20,7 @@ The 5-step diamond-convergence procedure: - [ ] **2.** Identify the topic-branch migration directory under `migrations/` (e.g. `migrations/app/20241112-add-tags/`) and treat it as the stale plan. Leave it on disk: it is still an edge any database that already applied it needs. No `rm -rf`. - [ ] **3.** Run `contract emit` then `migration plan --from --name ` to plan a fresh edge from the post-merge origin. The graph now has two edges off the old parent; that is a legal shape. If a database (e.g. the dev database) already applied the topic-branch migration, its marker sits at that tip and has no path to the new head: plan a second edge for it with `migration plan --from --name `, or reset that database. - [ ] **4.** Open the stale topic-branch `migration.ts`; port any custom data-transform logic into the new one. -- [ ] **5.** Self-emit (`node migrations/app//migration.ts`). +- [ ] **5.** Self-emit (`node /migration.ts`). ## Success criteria diff --git a/skills/prisma-8/references/debug.md b/skills/prisma-8/references/debug.md index bad5d318bbdd..241c97e13621 100644 --- a/skills/prisma-8/references/debug.md +++ b/skills/prisma-8/references/debug.md @@ -86,11 +86,11 @@ The single source of truth: read the envelope, find the row by `code`, follow th | `CONFIG.MISSING_EXTENSION_PACKS` | `contract emit` (e.g. contract uses `pgvector.Vector(...)` but config does not list the pgvector descriptor) | Add the descriptors named in `meta.missingExtensionPacks` to `extensions` in `prisma.config.ts`. See `references/contract.md`. | | `MIGRATION.PLANNING_FAILED` | `db init`, `db update`, `migration plan` | Inspect `meta.conflicts`. Recovery is per-conflict — chain to `references/migrations.md`. When `migration plan` raises it because the planner produced no operations, the conflict's `why` gives the next step (`migration new --from `, or `db sign` for a database managed with `db update`). | | `CLI.INIT_MISSING_FLAGS` / `CLI.INIT_INVALID_FLAG_VALUE` / `CLI.INIT_REINIT_NEEDS_FORCE` / `CLI.INIT_*` | `prisma orm init` | Re-run with the flags listed in `meta.missingFlags` or `meta.allowed`; a re-init needs `--confirm `. `CLI.INIT_INSTALL_FAILED` (exit 4) and `CLI.INIT_EMIT_FAILED` (exit 5) are findings on a completed scaffold — the files are on disk; fix and re-run the named step. | -| `MIGRATION.UNFILLED_PLACEHOLDER` | `node migrations/app//migration.ts` (self-emit) or `db migrate` | Edit `migration.ts`, replace the `placeholder("")` named by `meta.slot` with a real query closure, self-emit. See `references/migrations.md`. | +| `MIGRATION.UNFILLED_PLACEHOLDER` | `node /migration.ts` (self-emit) or `db migrate` | Edit `migration.ts`, replace the `placeholder("")` named by `meta.slot` with a real query closure, self-emit. See `references/migrations.md`. | | `MIGRATION.FILE_MISSING` | Reading a migration package | Restore from version control or scaffold a fresh package with `migration plan` / `migration new`. | | `MIGRATION.INVALID_DEFAULT_EXPORT` | Loading `migration.ts` | Use `export default class extends Migration { ... }` (or a factory returning `{ operations, targetId, destination }`). See `references/migrations.md`. | | `MIGRATION.DATA_TRANSFORM_CONTRACT_MISMATCH` | Building a data-transform query plan | Pass the same `endContract` reference to both `this.dataTransform(endContract, …)` and the query-builder context. | -| `MIGRATION.HASH_MISMATCH` | Any read of a migration package (`plan`, `list`, `db migrate`) | `ops.json` / `migration.json` were edited without self-emitting. Run `node migrations/app//migration.ts` to re-emit. | +| `MIGRATION.HASH_MISMATCH` | Any read of a migration package (`plan`, `list`, `db migrate`) | `ops.json` / `migration.json` were edited without self-emitting. Run `node /migration.ts` to re-emit. | | `CONTRACT.MARKER_MISSING` | `db verify` (`error` diagnostic, exit 4), runtime startup (warning) | DB has no marker yet. Run `prisma db init --db ` (baseline empty DB), `db update --db ` (apply contract directly), or `db sign --db ` if the schema already matches the contract. | | `CONTRACT.MARKER_MISMATCH` | `db verify` (exit 4), runtime startup (warning) | Marker disagrees with contract hash (`meta.expected` / `meta.actual`). Either migrate forward (`db migrate` / `db update`), or — if the DB is correct after a manual fix-up — `db sign`. See `references/migrations.md`. | | `CONTRACT.TARGET_MISMATCH` | `db verify`, runtime startup | Contract target ≠ config target; align them (see `meta.expected` / `meta.actual`). | diff --git a/skills/prisma-8/references/migrations.md b/skills/prisma-8/references/migrations.md index 896b2b68d209..d80701ab111a 100644 --- a/skills/prisma-8/references/migrations.md +++ b/skills/prisma-8/references/migrations.md @@ -38,7 +38,7 @@ Once the contract changes, you choose how the change reaches the database. This - `ops.json` — canonical operation list. Content-addressed; `migrationHash` is computed over this. - `migration.ts` — TypeScript authoring source, **framework-rendered** by `migration plan` (or `migration new`). You edit specific holes in it (see *Fill a placeholder* below) and re-emit `ops.json` / `migration.json` by running it. - **Contract snapshots.** `migration.ts` imports its bookend contracts from the shared, content-addressed store at `migrations/snapshots//contract.json` + `contract.d.ts` (`` is the contract's 64-hex storage hash) — not from files inside the migration package. -- **Self-emit.** Running `node migrations/app//migration.ts` regenerates `ops.json` and `migration.json` from the (possibly edited) TS source. This is the only supported way to update an existing migration package after edits. +- **Self-emit.** Running `node /migration.ts` regenerates `ops.json` and `migration.json` from the (possibly edited) TS source. This is the only supported way to update an existing migration package after edits. `` is the directory `migration plan` or `migration new` printed; it sits under the config's `migrations.dir` (by default `migrations/app/`), so the path depends on that setting. The command works from any directory. Without `--config`, the file uses the config whose migrations directory holds it: the nearest `prisma.config.ts` above it, else the one in the working directory; if neither does, it fails with `CLI.MIGRATION_CONFIG_NOT_FOUND`. If you gave the CLI `--config`, pass the same `--config ` to the file. - **`migration.ts` shape.** Framework-rendered. A class `M extends Migration` (from `@internal/postgres/migration` on Postgres, `@prisma/orm-mongo/target/migration` on Mongo — see the framing block below) that assigns the two snapshot imports to `startContractJson` / `endContractJson` (`Start` is `never` and there is no `startContractJson` on a baseline) and has an `operations` getter returning an array of operation values. **On Postgres the operation factories are methods on the base class** (`this.addColumn({...})`, `this.setNotNull({...})`, `this.dataTransform(...)`) taking one options object; free helpers like `col(...)` build the column descriptors they take. **On Mongo they are free factories** (`createIndex(...)`, `dataTransform(...)`) imported beside `Migration`. The file ends with `MigrationCLI.run(import.meta.url, M)` so executing it self-emits. - **`placeholder(slot)`.** A sentinel the planner emits into the rendered `migration.ts` (from the same `.../migration` import as `Migration`) wherever a data transform is needed. Calling `placeholder(...)` at emit time throws `MIGRATION.UNFILLED_PLACEHOLDER` with `meta.slot` naming the hole. The user replaces the `() => placeholder(...)` arrow with a real query-plan closure (Postgres) or fills `dataTransform({ check, run })` sources (Mongo — see *Fill a placeholder*), then self-emits. - **`this.dataTransform(endContract, name, { check, run })`.** The data-transform factory. `check` is a rowset query whose presence-of-any-row signals "work remains"; `run` is one or more mutation queries that perform the backfill. Both are lazy closures returning query-plans built against `endContract`. The runner wraps `check` as `EXISTS(...)` for precheck and `NOT EXISTS(...)` for postcheck, so the same closure asserts both "there is work" and "the work is done". @@ -70,7 +70,7 @@ Treat the rendered import lines as framework-managed on both targets: | `MIGRATION.FILE_MISSING` | Reading a migration package | `migration.ts`, `migration.json`, or `ops.json` is absent. Recover from version control, re-emit via `migration.ts`, or run `prisma migration new` for a fresh one. | | `MIGRATION.INVALID_DEFAULT_EXPORT` | Loading `migration.ts` | The file's default export is not a `Migration` subclass or factory function. Restore the planner-emitted scaffold from version control or re-run `migration plan` for a clean package. | | `MIGRATION.DATA_TRANSFORM_CONTRACT_MISMATCH` | Building a data-transform query plan | The query builder was instantiated with a contract reference different from the `endContract` passed to `this.dataTransform(...)`. Use the `endContract` imported at module scope for both. | -| `MIGRATION.HASH_MISMATCH` *Migration package is corrupt* | `db migrate` (or any read of the package) | `ops.json` / `migration.json` were edited without self-emitting. Run `node migrations/app//migration.ts` to re-emit, then re-run `db migrate`. | +| `MIGRATION.HASH_MISMATCH` *Migration package is corrupt* | `db migrate` (or any read of the package) | `ops.json` / `migration.json` were edited without self-emitting. Run `node /migration.ts` to re-emit, then re-run `db migrate`. | | `CLI.CONSENT_REQUIRED` | `migration plan` / `db update` run with nobody to answer (`--no-interactive`, CI, `--yes`) | The plan would lose data or (on `db update`) widen access. `meta.unanswered` and `nextActions` list each question with the flags that answer it: `--delete `, `--rename :`, `--allow `. Re-run with the ones you mean. `--confirm` and `--yes` answer none of them. | | `CLI.CONSENT_UNUSED` | `migration plan` / `db update` | Every question was answered, and a `--delete` or `--allow` was left over: a typo, a subject the plan does not touch, or a `--confirm` (which consents to nothing). Fix or drop the flag. A typo next to a question nobody answered is reported as `CLI.CONSENT_REQUIRED` instead, which names the flag that matched nothing. On `db update --dry-run` the same mistake is `MIGRATION.STATEMENT_ANSWERS_NO_QUESTION`. | | `MIGRATION.STATEMENT_DID_NOT_RESOLVE_LOSS` | A rename typed at the data-loss prompt | The plan was made again with the rename and still loses the subject's data. Answer with `delete`, or give a rename whose old name is the subject. | @@ -349,7 +349,7 @@ export default M; MigrationCLI.run(import.meta.url, M); ``` -Self-emit the same way: `node migrations/app//migration.ts`. +Self-emit the same way: `node /migration.ts`. ## Workflow — Author a migration by hand @@ -381,7 +381,7 @@ On Postgres the operations are **methods on the `Migration` base class**, each t - Collection options: `collMod`. - Data transforms: `dataTransform(name, { check, run })` (free factory; `check`/`run` use Mongo query-plan shapes). -Self-emit (`node migrations/app//migration.ts`) after each edit. +Self-emit (`node /migration.ts`) after each edit. ## Workflow — Inspect the live schema @@ -456,7 +456,7 @@ pnpm prisma db schema --db $DATABASE_URL --json Fix and re-run `db migrate`: ```bash -node migrations/app//migration.ts +node /migration.ts pnpm prisma db migrate --db $DATABASE_URL ``` @@ -467,7 +467,7 @@ If the failure was an out-of-band side-effect that left external systems half-ch The concept: `migrationHash` is content-addressed. A mismatch means `migration.json`'s stored hash disagrees with the hash recomputed from `ops.json` (and metadata). The cause is almost always: someone edited `migration.ts` and forgot to self-emit. The remediation is to self-emit the offending package. ```bash -node migrations/app//migration.ts +node /migration.ts pnpm prisma db migrate --db $DATABASE_URL ``` @@ -536,7 +536,7 @@ The CLI collects anonymous usage data by default. To opt out, set `PRISMA_DISABL - [ ] For `migration plan`: ran `migration show ` to review before `db migrate`. - [ ] Filled every `placeholder(...)` in `migration.ts` (if any), built against `endContract`. - [ ] `check` closures are rowset queries, not scalar aggregates. -- [ ] Self-emitted (`node migrations/app//migration.ts`) after editing the TS. +- [ ] Self-emitted (`node /migration.ts`) after editing the TS. - [ ] Ran `db migrate` (or `db update`) and saw it complete. - [ ] Used `db verify` only when diagnosing drift — not as a routine post-apply step. - [ ] Did NOT use `db update` against a shared or production database. From 18ff475fcbf004b68d674d99bd42cfac9f7ec62c Mon Sep 17 00:00:00 2001 From: willbot Date: Tue, 13 Oct 2026 09:15:11 +0200 Subject: [PATCH 54/57] TML-3478: tests that run a migration.ts next to their config give it a migrations directory that holds the file The round-trip tests put migration.ts beside prisma.config.ts, outside the default migrations directory, so the config no longer owns it. Their configs now set migrations.dir to the directory itself. The migration-file-config unit test resets modules before importing, so its mocks hold when the package runs its files in one process. Signed-off-by: willbot Signed-off-by: Will Madden --- .../3-tooling/cli/test/utils/migration-file-config.test.ts | 1 + .../3-mongo-target/1-mongo-target/test/migration-e2e.test.ts | 1 + .../postgres/test/migrations/render-typescript.roundtrip.test.ts | 1 + .../sqlite/test/migrations/render-typescript.roundtrip.test.ts | 1 + 4 files changed, 4 insertions(+) diff --git a/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts b/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts index c800f11e2443..f2d1c225741a 100644 --- a/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts +++ b/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts @@ -20,6 +20,7 @@ vi.mock('@internal/config-loader', async () => { }; }); +vi.resetModules(); const { loadMigrationFileConfig } = await import('../../src/utils/migration-file-config'); afterAll(() => { diff --git a/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts b/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts index 1bb38a5dc574..975c3ee6cae4 100644 --- a/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts @@ -51,6 +51,7 @@ const fixtureConfigSource = [ ' family: mongoFamilyDescriptor,', ' target: mongoTargetDescriptor,', ' adapter: mongoAdapter,', + " migrations: { dir: '.' },", '}) };', '', ].join('\n'); diff --git a/packages/3-targets/6-adapters/postgres/test/migrations/render-typescript.roundtrip.test.ts b/packages/3-targets/6-adapters/postgres/test/migrations/render-typescript.roundtrip.test.ts index ad7ac6a3606b..7df3c3738059 100644 --- a/packages/3-targets/6-adapters/postgres/test/migrations/render-typescript.roundtrip.test.ts +++ b/packages/3-targets/6-adapters/postgres/test/migrations/render-typescript.roundtrip.test.ts @@ -93,6 +93,7 @@ const fixtureConfigSource = [ ' family: sql,', ' target: postgres,', ' adapter: postgresAdapter,', + " migrations: { dir: '.' },", '}) };', '', ].join('\n'); diff --git a/packages/3-targets/6-adapters/sqlite/test/migrations/render-typescript.roundtrip.test.ts b/packages/3-targets/6-adapters/sqlite/test/migrations/render-typescript.roundtrip.test.ts index 4b581bf1233f..57bb43d3d5f9 100644 --- a/packages/3-targets/6-adapters/sqlite/test/migrations/render-typescript.roundtrip.test.ts +++ b/packages/3-targets/6-adapters/sqlite/test/migrations/render-typescript.roundtrip.test.ts @@ -75,6 +75,7 @@ const fixtureConfigSource = [ ' family: sql,', ' target: sqlite,', ' adapter: sqliteAdapter,', + " migrations: { dir: '.' },", '}) };', '', ].join('\n'); From f70973eb071030c4c3f1ba0f0398a40d2d320074 Mon Sep 17 00:00:00 2001 From: willbot Date: Tue, 13 Oct 2026 09:40:12 +0200 Subject: [PATCH 55/57] TML-3478: the migration-file config check and the CLI find the migrations directory with one function migration-file-config.ts and command-helpers.ts each repeated the rule for the default migrations directory. Both now call migrationsDirFor from orm/migration/paths.ts, which db migrate uses. A test checks that the config owns each migration the CLI writes. Signed-off-by: willbot Signed-off-by: Will Madden --- .../3-tooling/cli/src/orm/migration/paths.ts | 17 +++++++++---- .../cli/src/utils/command-helpers.ts | 3 ++- .../cli/src/utils/migration-file-config.ts | 25 +++++++++++-------- .../test/utils/migration-file-config.test.ts | 22 +++++++++++++++- 4 files changed, 50 insertions(+), 17 deletions(-) diff --git a/packages/1-framework/3-tooling/cli/src/orm/migration/paths.ts b/packages/1-framework/3-tooling/cli/src/orm/migration/paths.ts index 42602425248a..60aaf79851fe 100644 --- a/packages/1-framework/3-tooling/cli/src/orm/migration/paths.ts +++ b/packages/1-framework/3-tooling/cli/src/orm/migration/paths.ts @@ -9,20 +9,27 @@ import { relative, resolve } from 'pathe'; * engine validated it. Every command runs behind that validation, so its * absence here is a bug. */ -export function baseDirFor(config: PrismaNextConfig): string { +export function baseDirFor(config: Pick): string { if (config.baseDir === undefined) { throw new InternalError('the orm config section reached a command without baseDir'); } return config.baseDir; } -/** Where migrations live; the validated config carries it absolute and defaulted. */ -export function migrationsDirFor(config: PrismaNextConfig): string { - return config.migrations?.dir ?? resolve(baseDirFor(config), 'migrations'); +type MigrationsLocation = Pick; + +/** + * Where a config keeps its migrations: `migrations.dir`, default `migrations`, resolved against the config's `baseDir`. `base` stands in for `baseDir` when a config built in memory carries none. + */ +export function migrationsDirFor(config: MigrationsLocation, base?: string): string { + return resolve( + config.baseDir ?? base ?? baseDirFor(config), + config.migrations?.dir ?? 'migrations', + ); } /** The app subspace under {@link migrationsDirFor}. */ -export function appMigrationsDirFor(config: PrismaNextConfig): string { +export function appMigrationsDirFor(config: MigrationsLocation): string { return spaceMigrationDirectory(migrationsDirFor(config), APP_SPACE_ID); } diff --git a/packages/1-framework/3-tooling/cli/src/utils/command-helpers.ts b/packages/1-framework/3-tooling/cli/src/utils/command-helpers.ts index 10ed40c470ca..185da65d5b0f 100644 --- a/packages/1-framework/3-tooling/cli/src/utils/command-helpers.ts +++ b/packages/1-framework/3-tooling/cli/src/utils/command-helpers.ts @@ -8,6 +8,7 @@ import { ifDefined } from '@internal/utils/defined'; import type { NextAction } from '@internal/utils/structured-error'; import { relative, resolve } from 'pathe'; import type { ControlClient } from '../control-api/types'; +import { migrationsDirFor } from '../orm/migration/paths'; import { CliStructuredError, errorRuntime } from './cli-errors'; import { hasUrlScheme, @@ -66,7 +67,7 @@ export function resolveMigrationPaths( const configPath = 'prisma.config.ts'; // A validated config carries the directory absolute; a config handed in // raw by a programmatic caller is anchored on its baseDir, else on cwd. - const migrationsDir = resolve(config.baseDir ?? cwd, config.migrations?.dir ?? 'migrations'); + const migrationsDir = migrationsDirFor(config, cwd); const migrationsRelative = relative(cwd, migrationsDir); const appMigrationsDir = spaceMigrationDirectory(migrationsDir, APP_SPACE_ID); const appMigrationsRelative = relative(cwd, appMigrationsDir); diff --git a/packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts b/packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts index e679fa794f83..8b95f92ac619 100644 --- a/packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts +++ b/packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts @@ -14,7 +14,8 @@ import { errorMigrationCliConfigNotFound, } from '@internal/errors/control'; import { notOk, ok, type Result } from '@internal/utils/result'; -import { dirname, isAbsolute, join, relative, resolve } from 'pathe'; +import { dirname, isAbsolute, join, relative } from 'pathe'; +import { migrationsDirFor } from '../orm/migration/paths'; /** * The config a `migration.ts` runs with. `--config` names it outright. Without it, the config is the one that owns the migration, whose migrations directory holds the file: the nearest `prisma.config.ts` above the migration, else the one in the working directory. @@ -46,21 +47,25 @@ export async function loadMigrationFileConfig(options: { checked.push({ configPath: candidate, loadFailure: config.failure.message }); continue; } - const migrationsDir = migrationsDirOf(config.value, candidate); - if (contains(migrationsDir, migrationFile)) { + if (ownsMigration(config.value, candidate, migrationFile)) { return ok(config.value); } - checked.push({ configPath: candidate, migrationsDir }); + checked.push({ + configPath: candidate, + migrationsDir: migrationsDirFor(config.value, dirname(candidate)), + }); } return notOk(errorMigrationCliConfigNotFound({ migrationDir: dirname(migrationFile), checked })); } -function migrationsDirOf(config: PrismaNextConfig, configPath: string): string { - return resolve(config.baseDir ?? dirname(configPath), config.migrations?.dir ?? 'migrations'); -} - -function contains(dir: string, file: string): boolean { - const path = relative(realPath(dir), realPath(file)); +/** Whether `migrationFile` is inside the migrations directory of the config at `configPath`. */ +export function ownsMigration( + config: Pick, + configPath: string, + migrationFile: string, +): boolean { + const migrationsDir = migrationsDirFor(config, dirname(configPath)); + const path = relative(realPath(migrationsDir), realPath(migrationFile)); return path !== '' && !path.startsWith('..') && !isAbsolute(path); } diff --git a/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts b/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts index f2d1c225741a..9e4cbc1c26b6 100644 --- a/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts +++ b/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts @@ -21,7 +21,10 @@ vi.mock('@internal/config-loader', async () => { }); vi.resetModules(); -const { loadMigrationFileConfig } = await import('../../src/utils/migration-file-config'); +const { loadMigrationFileConfig, ownsMigration } = await import( + '../../src/utils/migration-file-config' +); +const { appMigrationsDirFor } = await import('../../src/orm/migration/paths'); afterAll(() => { vi.doUnmock('@internal/config-loader'); @@ -152,3 +155,20 @@ describe('loadMigrationFileConfig', () => { }); }); }); + +describe('ownsMigration', () => { + it.each([ + ['the default migrations directory', {}], + [ + 'a migrations directory outside the project', + { migrations: { dir: '../shared-db/migrations' } }, + ], + ])('owns a migration the CLI writes under %s', (_label, section) => { + const configPath = join(root, 'app/prisma.config.ts'); + const config = { ...section, baseDir: join(root, 'app') }; + const written = join(appMigrationsDirFor(config), '0001_initial/migration.ts'); + + expect(ownsMigration(config, configPath, written)).toBe(true); + expect(ownsMigration(config, configPath, join(root, 'other/migration.ts'))).toBe(false); + }); +}); From 1ef2c1df1502e47b9ac869154f555c54dac1d0d1 Mon Sep 17 00:00:00 2001 From: willbot Date: Tue, 13 Oct 2026 09:41:16 +0200 Subject: [PATCH 56/57] TML-3478: the migration plan placeholder summary builds its node command with selfEmitCommand It wrote its own command, quoted differently from the next action, and said "from this directory". It now uses the same command as the next action, including --config when given. Signed-off-by: willbot Signed-off-by: Will Madden --- .../cli/src/control-api/operations/migration-plan.ts | 6 +++--- .../3-tooling/cli/test/orm/migration-plan.test.ts | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts index 3a66f4e6a07d..d693ebb19c94 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts @@ -48,6 +48,7 @@ import { import { toExtensionInputs } from '../../utils/extension-pack-inputs'; import { assertFrameworkComponentsCompatible } from '../../utils/framework-components'; import { createProjectSpecifierResolver } from '../../utils/project-import-root'; +import { selfEmitCommand } from '../../utils/self-emit-command'; import { snapshotVerifierFor } from '../../utils/snapshot-content-verification'; import { type AnswerPlanQuestions, @@ -122,11 +123,10 @@ function placeholderSummary( dirs: readonly string[], configPath: string | undefined, ): string { - const config = configPath === undefined ? '' : ` --config "${configPath}"`; const commands = dirs - .map((dir) => `\`node ${join(dir, 'migration.ts')}${config}\``) + .map((dir) => `\`${selfEmitCommand(join(dir, 'migration.ts'), configPath)}\``) .join(' and '); - return `Planned ${planned} with placeholder(s) — edit migration.ts, then run ${commands} from this directory to self-emit`; + return `Planned ${planned} with placeholder(s) — edit migration.ts, then run ${commands} to self-emit`; } /** The origin of a plan from an empty database: no models, so no statement resolves. */ diff --git a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts index 149f5efcd0a7..eec9d659b749 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts @@ -814,7 +814,7 @@ describe('migration plan', () => { expect.stringMatching(/^migrations\/app\/[^/]+$/), ]); expect(data.summary).toBe( - `Planned baseline + migration with placeholder(s) — edit migration.ts, then run \`node ${data.baselineDir}/migration.ts\` and \`node ${data.dir}/migration.ts\` from this directory to self-emit`, + `Planned baseline + migration with placeholder(s) — edit migration.ts, then run \`node "${data.baselineDir}/migration.ts"\` and \`node "${data.dir}/migration.ts"\` to self-emit`, ); expect( data.appliedStatements.map(({ operationIndexes }) => @@ -846,7 +846,7 @@ describe('migration plan', () => { expect(data.dir).toMatch(/^migrations\/app\/[^/]+$/); expect(data).toMatchObject({ pendingPlaceholders: true, - summary: `Planned migration with placeholder(s) — edit migration.ts, then run \`node ${data.dir}/migration.ts\` from this directory to self-emit`, + summary: `Planned migration with placeholder(s) — edit migration.ts, then run \`node "${data.dir}/migration.ts"\` to self-emit`, }); }); @@ -867,7 +867,7 @@ describe('migration plan', () => { summary: data.summary, next: answered.presented?.presentation.next?.at(-1), }).toEqual({ - summary: `Planned migration with placeholder(s) — edit migration.ts, then run \`node ${data.dir}/migration.ts --config "prisma.config.ts"\` from this directory to self-emit`, + summary: `Planned migration with placeholder(s) — edit migration.ts, then run \`node "${data.dir}/migration.ts" --config "prisma.config.ts"\` to self-emit`, next: { kind: 'run-command', label: 'Run it to self-emit ops.json and attest the package', From ae4a7d48397abf12188646c5b4f40f98e14dd293 Mon Sep 17 00:00:00 2001 From: willbot Date: Tue, 13 Oct 2026 09:41:39 +0200 Subject: [PATCH 57/57] TML-3478: the multi-extension example names the migration directory in its self-emit command In that README "package" means the extension package the reader is already in, so was ambiguous. Step 3 now says migrations//migration.ts. Signed-off-by: willbot Signed-off-by: Will Madden --- examples/multi-extension-monorepo/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/multi-extension-monorepo/README.md b/examples/multi-extension-monorepo/README.md index 7319a8475981..46c9ac0d6f26 100644 --- a/examples/multi-extension-monorepo/README.md +++ b/examples/multi-extension-monorepo/README.md @@ -78,9 +78,9 @@ Each internal "package" under `packages/` follows the **contract-space package l Then hand-edit the generated `migrations//migration.ts`'s `operations` getter so each op carries the package's stable `:-vN` `invariantId` (invariant IDs cannot be renamed once published). Re-emit `ops.json` + `migration.json`: ```sh - node /migration.ts + node migrations//migration.ts # or, on Node < 24: - pnpm exec tsx /migration.ts + pnpm exec tsx migrations//migration.ts ``` 4. Update `migrations/refs/head.json` to point at the new contract `storageHash` plus the union of `providedInvariants` across all migrations.