diff --git a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md index 17457d3dba3d..831fbf40bf97 100644 --- a/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md +++ b/docs/architecture docs/adrs/ADR 188 - MongoDB migration operation model.md @@ -1,5 +1,7 @@ # ADR 188 — MongoDB migration operation model: data-driven commands and checks +> **Update — 2026-10:** document rewrites the planner writes for `--rename` and `--delete` statements are classed by their effect, not as `data`. See § DML operations and unified check shape. + ## Grounding example The migration planner has compared two contracts and determined that a unique ascending index on `users.email` needs to be created. It needs to produce an operation that the runner can execute. @@ -119,26 +121,26 @@ interface MongoMigrationCheck { This gives checks the same expressive power as MongoDB query filters, without inventing a separate vocabulary. -### Data transform operations and unified check shape +### DML operations and unified check shape -The envelope described above covers DDL operations — creating indexes, modifying collections. Data transform operations (backfills, renames, seed inserts) use the same three-phase envelope but carry different payloads. +The envelope described above covers DDL operations — creating indexes, modifying collections. DML operations, which read and write documents (backfills, field renames and removals, seed inserts), use the same three-phase envelope but carry different payloads. A data transform is a DML operation built with `dataTransform`. ```ts -interface MongoDataTransformOperation extends MigrationPlanOperation { - readonly operationClass: 'data'; +interface MongoDmlOperation extends MigrationPlanOperation { + readonly operationClass: MigrationOperationClass; readonly name: string; - readonly precheck: readonly MongoDataTransformCheck[]; + readonly precheck: readonly MongoDmlCheck[]; readonly run: readonly MongoQueryPlan[]; - readonly postcheck: readonly MongoDataTransformCheck[]; + readonly postcheck: readonly MongoDmlCheck[]; } ``` -The `operationClass` is always `'data'`. The `run` array contains `MongoQueryPlan` objects — aggregation pipelines or raw commands (`rawUpdateMany`, `rawInsertOne`, etc.) — rather than DDL commands. The `precheck` and `postcheck` arrays contain `MongoDataTransformCheck` objects instead of `MongoMigrationCheck`. +A transform built with `dataTransform`, whether a user writes it or the planner scaffolds it with a placeholder, has `operationClass: 'data'`. The `renameField` and `unsetField` rewrites carry the class of their effect, whoever writes them, as every other operation does: `renameField` is `widening` and `unsetField` is `destructive`. So `db update`, which does not allow `data`, can run a rename. The runner and the serializer tell a DML operation from a DDL operation by its shape (`run` against `execute`), not by its class. The `run` array contains `MongoQueryPlan` objects — aggregation pipelines or raw commands (`rawUpdateMany`, `rawInsertOne`, etc.) — rather than DDL commands. The `precheck` and `postcheck` arrays contain `MongoDmlCheck` objects instead of `MongoMigrationCheck`. -`MongoDataTransformCheck` has the same four-field shape as `MongoMigrationCheck`: +`MongoDmlCheck` has the same four-field shape as `MongoMigrationCheck`: ```ts -interface MongoDataTransformCheck { +interface MongoDmlCheck { readonly description: string; readonly source: MongoQueryPlan; readonly filter: MongoFilterExpr; @@ -148,7 +150,7 @@ interface MongoDataTransformCheck { The only difference is the `source` type: `MongoQueryPlan` (an aggregation or raw command) instead of `AnyMongoInspectionCommand`. Both use `MongoFilterExpr` for `filter` and `'exists' | 'notExists'` for `expect`. Both are evaluated by the same `FilterEvaluator` path in the runner — run the source query, apply the filter client-side, check against the expectation. -The two types are kept concrete rather than collapsed into a generic `MigrationCheck` because the source types need different executors — DDL checks dispatch through `MongoInspectionCommandVisitor`, data transform checks dispatch through `MongoAdapter.lower()` → `MongoDriver.execute()`. Two concrete types with a shared shape let the runner dispatch internally based on which operation type it's processing. +The two types are kept concrete rather than collapsed into a generic `MigrationCheck` because the source types need different executors — DDL checks dispatch through `MongoInspectionCommandVisitor`, DML checks dispatch through `MongoAdapter.lower()` → `MongoDriver.execute()`. Two concrete types with a shared shape let the runner dispatch internally based on which operation type it's processing. #### One factory derives both precheck and postcheck @@ -169,11 +171,11 @@ The factory resolves this into a precheck with `expect: 'exists'` (violations fo #### DML execution -Data transform `run` commands execute via `MongoAdapter.lower()` → `MongoDriver.execute()` — the same adapter+driver transport used for runtime queries. No bespoke DML executor is needed; the adapter lowers each `MongoQueryPlan` to a wire command, and the driver executes it against the database. +DML `run` commands execute via `MongoAdapter.lower()` → `MongoDriver.execute()` — the same adapter+driver transport used for runtime queries. No bespoke DML executor is needed; the adapter lowers each `MongoQueryPlan` to a wire command, and the driver executes it against the database. #### Relationship to the generic envelope -Both DDL and data transform operations are specializations of the generic three-phase migration operation envelope described in [ADR 191](ADR%20191%20-%20Generic%20three-phase%20migration%20operation%20envelope.md). The DDL variant fills the phases with inspection commands, DDL commands, and `MongoMigrationCheck`; the data transform variant fills them with query plans and `MongoDataTransformCheck`. The runner processes both through the same three-phase loop. +Both DDL and DML operations are specializations of the generic three-phase migration operation envelope described in [ADR 191](ADR%20191%20-%20Generic%20three-phase%20migration%20operation%20envelope.md). The DDL variant fills the phases with inspection commands, DDL commands, and `MongoMigrationCheck`; the DML variant fills them with query plans and `MongoDmlCheck`. The runner processes both through the same three-phase loop. ## Serialization diff --git a/docs/architecture docs/adrs/ADR 264 - A model names its storage verbatim, and a rename is an operation.md b/docs/architecture docs/adrs/ADR 264 - A model names its storage verbatim, and a rename is an operation.md index c749ad742877..0c276e9f5b0b 100644 --- a/docs/architecture docs/adrs/ADR 264 - A model names its storage verbatim, and a rename is an operation.md +++ b/docs/architecture docs/adrs/ADR 264 - A model names its storage verbatim, and a rename is an operation.md @@ -67,7 +67,7 @@ A schema that declares no `@@map` and whose storage was created under the loweri - A model with no `@@map` names storage that can be quoted exactly as written, which is what a database created by an earlier Prisma version holds. - `contract infer` round-trips: a table whose name already equals the model name infers with no `@@map` and verifies clean, and a snake_case table still infers with `@@map`. - A rename needs a hand-written migration. The user states one line per renamed table and the operations follow from the contracts. -- MongoDB has no rename operation. A collection is renamed outside Prisma, with `renameCollection`. +- MongoDB renames a collection with the `renameCollection` operation, which a `--rename` statement plans (see [Migration System § Statements](../subsystems/7.%20Migration%20System.md#statements)). ## Alternatives considered diff --git a/docs/architecture docs/subsystems/10. MongoDB Family.md b/docs/architecture docs/subsystems/10. MongoDB Family.md index a62db1c22326..ecec7ff2b0de 100644 --- a/docs/architecture docs/subsystems/10. MongoDB Family.md +++ b/docs/architecture docs/subsystems/10. MongoDB Family.md @@ -8,7 +8,7 @@ This document explains what makes MongoDB different from SQL, how each differenc **Responsibilities:** MongoDB-specific contract types (`MongoContract`), execution pipeline (`MongoQueryPlan`, `MongoDriver`, `MongoRuntime`), codec registry (BSON codecs), ORM client, and contract deserialization and validation (`MongoContractSerializer.deserializeContract()`). -**Migrations are supported.** MongoDB does have server-side schema state — indexes, JSON Schema validators, collection options (capped, time-series, collation) — and Mongo participates in the migration system on the same footing as SQL targets. `prisma migration new --target mongo` and `prisma migration plan --target mongo` scaffold a runnable `migration.ts` (extending `Migration` from `@internal/family-mongo/migration`). Two paths produce identical `ops.json` artifacts — the authoritative migration contract (see [ADR 192](../adrs/ADR%20192%20-%20ops.json%20is%20the%20migration%20contract.md)): inline emit at the end of `migration plan` and self-emit via `./migration.ts` (see [ADR 193](../adrs/ADR%20193%20-%20Class-flow%20as%20the%20canonical%20migration%20authoring%20strategy.md) for the authoring strategy). Data migrations are authored as `dataTransform({ check, run })` operations whose pipelines invariant-guard the transition (see [ADR 176](../adrs/ADR%20176%20-%20Data%20migrations%20as%20invariant-guarded%20transitions.md)). See the [Migration System](7.%20Migration%20System.md) subsystem doc for the full canonical pipeline. +**Migrations are supported.** MongoDB does have server-side schema state — indexes, JSON Schema validators, collection options (capped, time-series, collation) — and Mongo participates in the migration system on the same footing as SQL targets. `prisma migration new --target mongo` and `prisma migration plan --target mongo` scaffold a runnable `migration.ts` (extending `Migration` from `@internal/family-mongo/migration`). Two paths produce identical `ops.json` artifacts — the authoritative migration contract (see [ADR 192](../adrs/ADR%20192%20-%20ops.json%20is%20the%20migration%20contract.md)): inline emit at the end of `migration plan` and self-emit by running `node /migration.ts` from any directory (see [ADR 193](../adrs/ADR%20193%20-%20Class-flow%20as%20the%20canonical%20migration%20authoring%20strategy.md) for the authoring strategy). Data migrations are authored as `dataTransform({ check, run })` operations whose pipelines invariant-guard the transition (see [ADR 176](../adrs/ADR%20176%20-%20Data%20migrations%20as%20invariant-guarded%20transitions.md)). See the [Migration System](7.%20Migration%20System.md) subsystem doc for the full canonical pipeline. ## At a glance @@ -435,12 +435,14 @@ Both commands write into `migrations/_/`. The destination contr The Mongo *target* (not the framework) owns the scaffolding pipeline end-to-end. The framework defines the shape of a migration package and the runner contract; everything family-specific — what factories exist, how the planner IR renders into a runnable file, what the diff produces — lives under `packages/3-mongo-target/` and `packages/2-mongo-family/`. -Concretely, the Mongo planner produces an `OpFactoryCall[]` IR (a discriminated union of frozen AST classes — `CreateIndexCall`, `DropIndexCall`, `CreateCollectionCall`, `DropCollectionCall`, `CollModCall`) and feeds it to two renderers: an operation renderer that calls the matching factory functions to materialize `MongoMigrationPlanOperation[]` for `ops.json`, and a TypeScript renderer that emits a `migration.ts` calling the same factories. Both outputs derive from the same IR, so re-running `./migration.ts` after editing it produces an `ops.json` that matches the planner's output for the unedited regions. See [ADR 195 — Planner IR with two renderers](../adrs/ADR%20195%20-%20Planner%20IR%20with%20two%20renderers.md) and [ADR 194 — Plans carry their own authoring surface](../adrs/ADR%20194%20-%20Plans%20carry%20their%20own%20authoring%20surface.md). +Concretely, the Mongo planner produces an `OpFactoryCall[]` IR (a discriminated union of frozen AST classes — `CreateIndexCall`, `DropIndexCall`, `CreateCollectionCall`, `DropCollectionCall`, `RenameCollectionCall`, `RenameFieldCall`, `UnsetFieldCall`, `CollModCall`) and feeds it to two renderers: an operation renderer that calls the matching factory functions to materialize `MongoMigrationPlanOperation[]` for `ops.json`, and a TypeScript renderer that emits a `migration.ts` calling the same factories. Both outputs derive from the same IR, so running it again after editing it (`node /migration.ts`) produces an `ops.json` that matches the planner's output for the unedited regions. See [ADR 195 — Planner IR with two renderers](../adrs/ADR%20195%20-%20Planner%20IR%20with%20two%20renderers.md) and [ADR 194 — Plans carry their own authoring surface](../adrs/ADR%20194%20-%20Plans%20carry%20their%20own%20authoring%20surface.md). The factories the scaffold imports live in two layered modules: - `@internal/family-mongo/migration` — the `Migration` base class (a `MongoMigration` subclass that `implements MigrationPlan`) plus `Migration.run(...)` for the shebang self-emit path ([ADR 196](../adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md)). -- `@internal/target-mongo/migration` — the operation factories (`createCollection`, `dropCollection`, `createIndex`, `dropIndex`, `collMod`, `setValidation`, `validatedCollection`, `dataTransform`) and the `placeholder(slot)` utility for unfilled scaffold slots. +- `@internal/target-mongo/migration` — the operation factories (`createCollection`, `dropCollection`, `renameCollection`, `createIndex`, `dropIndex`, `collMod`, `setValidation`, `validatedCollection`, `renameField`, `unsetField`, `dataTransform`) and the `placeholder(slot)` utility for unfilled scaffold slots. + +`renameCollection(from, to)` renames a collection, keeping its documents, indexes and validator; it checks that the source exists and the target does not. `renameField(collection, from, to, { filter? })` and `unsetField(collection, field, { filter? })` rewrite the documents that hold the field with one `updateMany` (`$rename` or `$unset`) that skips validation; `filter` limits them to one variant's documents. They refuse an empty field name, `_id`, and a name containing `.` or starting with `$`. The planner emits all three for `--rename` and `--delete` statements; see [Subsystem 7 — Rename statements](7.%20Migration%20System.md#rename-statements). A scaffolded `migration.ts` therefore looks like: @@ -472,14 +474,7 @@ Non-nullable fields are added to `required`. Nullable fields are still listed in ### Polymorphic variants -For polymorphic models (`@@discriminator` on the base, `@@base` on each variant) using single-collection inheritance, the *intent* is that variants share the base model's collection and the validator becomes a discriminated `oneOf` over variant shapes (one branch per discriminator value, base fields hoisted to the top-level `properties`/`required`). This pattern is described in [Schema validation](#schema-validation-jsonschema) and reflects how the runtime interpreter already handles polymorphic indexes ([ADR 173](../adrs/ADR%20173%20-%20Polymorphism%20via%20discriminator%20and%20variants.md)). - -**Known gaps in the migration planner today** (tracked as [TML-2447](https://linear.app/prisma-company/issue/TML-2447)): - -- The planner emits standalone `createCollection` ops for variant models that should share the base's collection. Applying these would create unwanted empty collections; in example apps the migration artifacts are committed as-is and the variant ops have to be removed by hand before apply. -- Variant validators include only variant-specific fields and miss every base-model field (including `_id` and the discriminator). They are structurally wrong even when the variant collection is intentional. - -Until these are addressed, polymorphic schemas may need post-scaffold edits to `migration.ts` (or to `ops.json` after self-emit) before the migration is safe to apply. +For polymorphic models (`@@discriminator` on the base, `@@base` on each variant) using single-collection inheritance, variants share the base model's collection, and the planner creates that one collection. Its validator is a discriminated `oneOf` over variant shapes: base fields at the top-level `properties`/`required`, and one branch per discriminator value that repeats the base fields, adds the variant's own fields, pins the discriminator value and closes with `additionalProperties: false`. This pattern is described in [Schema validation](#schema-validation-jsonschema) and matches how the runtime handles polymorphic indexes ([ADR 173](../adrs/ADR%20173%20-%20Polymorphism%20via%20discriminator%20and%20variants.md)). Every variant document holds its discriminator value, so a variant's field rename or removal touches only that variant's documents. ### Author-editable seams @@ -487,12 +482,12 @@ Until these are addressed, polymorphic schemas may need post-scaffold edits to ` 1. Run `migration plan` (or `migration new`). The planner fills in everything it can derive from the contract diff. 2. Open `migration.ts`. Replace any `placeholder(slot)` calls with real query plans (typically inside a `dataTransform({ check, run })` operation, where the planner can scaffold the structure but not the body). Hand-author additional operations as needed. -3. Re-emit by running the file directly: `./migration.ts` or `node migration.ts`. `Migration.run(...)` detects it is the main module and rewrites `ops.json` and the `migrationId` in `migration.json`. Unfilled `placeholder(...)` calls fail here with the structured error `PN-MIG-2001` rather than producing silent nonsense ([ADR 200](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md)). +3. Re-emit by running the file with node from any directory: `node /migration.ts`. It loads the config `--config` names. Without `--config`, it uses the config that owns the migration, whose migrations directory holds the file: the nearest `prisma.config.ts` above the migration's directory if it owns it, else the `prisma.config.ts` in the working directory if that one does; otherwise it stops with `CLI.MIGRATION_CONFIG_NOT_FOUND`. `Migration.run(...)` detects it is the main module and rewrites `ops.json` and the `migrationId` in `migration.json`. Unfilled `placeholder(...)` calls fail here with the structured error `PN-MIG-2001` rather than producing silent nonsense ([ADR 200](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md)). 4. Commit both files. CI / `migrate` reads `ops.json` only — TypeScript is never executed at apply time. Apply-time hash verification rehashes `(manifest, ops)` and aborts on mismatch, so post-emit hand edits to `ops.json` (or any FS corruption) are caught before the runner touches the database. See [Subsystem 7 — Hash verification](7.%20Migration%20System.md#hash-verification). What the runner guarantees about the seam: -- **Round-trippable.** Re-running `./migration.ts` on an unedited scaffold produces byte-identical `ops.json` (the operation renderer and TypeScript renderer share the same IR). +- **Round-trippable.** Running `node /migration.ts` again on an unedited scaffold produces byte-identical `ops.json` (the operation renderer and TypeScript renderer share the same IR). - **No drift between source and contract.** `migrationId` is content-addressed over `(manifest, ops)`, so any edit to either side that changes the canonical operations updates `migrationId` and is caught at apply time. - **Hand-authored operations are first-class.** Operations added by the developer (e.g. extra `dataTransform` calls, custom `collMod` adjustments) flow through the same factories the planner uses; there is no "planner-only" vs "user-only" distinction in `ops.json`. diff --git a/docs/architecture docs/subsystems/7. Migration System.md b/docs/architecture docs/subsystems/7. Migration System.md index c738f2aa8a6e..39db64e2ac34 100644 --- a/docs/architecture docs/subsystems/7. Migration System.md +++ b/docs/architecture docs/subsystems/7. Migration System.md @@ -84,7 +84,7 @@ flowchart LR 3. **Scaffold.** The CLI calls `plan.renderTypeScript(resolveImportSpecifier)` and writes the result to `migration.ts`. The resolver comes from the consuming project's own `package.json` (`projectImportRoot` in `@internal/cli`), so the package names in the scaffold are ones that project can resolve — the same resolver `contract.d.ts` emission gets, for the same reason: both are files the application keeps. See [ADR 242 — Public npm surface](../adrs/ADR%20242%20-%20Public%20npm%20surface%20-%20single%20@prisma%20scope%20with%20consolidated%20publish%20packages.md). The destination contract is snapshotted into the migration directory so queries are typed against the schema at planning time. See [ADR 197 — Migration packages snapshot their own contract](../adrs/ADR%20197%20-%20Migration%20packages%20snapshot%20their%20own%20contract.md). Unfilled slots (data-transform bodies the planner can't derive) use `placeholder(slot)` — a `never`-returning function that throws a structured `PN-MIG-2001` error at emit time. See [ADR 200 — Placeholder utility for scaffolded migration slots](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md). -4. **Edit.** The developer opens `migration.ts`, replaces placeholders with real queries, and iterates by running the file directly (`./migration.ts` or `node migration.ts`) thanks to the shebang and `Migration.run(...)`. +4. **Edit.** The developer opens `migration.ts`, replaces placeholders with real queries, and iterates by running the file with node (`node /migration.ts`) thanks to the shebang and `Migration.run(...)`. The file loads the config `--config` names. Without `--config`, it uses the config that owns the migration, whose migrations directory holds the file: the nearest `prisma.config.ts` above the migration's directory if it owns it, else the `prisma.config.ts` in the working directory if that one does; otherwise it stops with `CLI.MIGRATION_CONFIG_NOT_FOUND`. 5. **Emit.** Every migration package is **fully attested on disk**: `migration plan` writes `migration.ts`, `migration.json` (with a content-addressed `migrationHash`), and `ops.json` (the planned ops, or `[]` if the planner could not lower any calls because of unfilled placeholders). After the developer fills any `placeholder(...)` slots, **self-emission** rewrites `ops.json` and the `migrationHash`: running the file via `./migration.ts` (shebang) or `node migration.ts` causes `Migration.run(...)` to detect it is the main module and re-emit artifacts in-process. See [ADR 196 — In-process emit for class-flow targets](../adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md). @@ -191,7 +191,7 @@ prisma migration plan --name tidy-users --rename Profile:User --rename User.name prisma db update --rename Profile:User --rename User.name:User.fullName --delete Legacy --allow User ``` -**The questions.** The planner's success result lists what the plan would lose (`dataLoss`) and whose access it would widen (`accessWidening`), each entry an operation's position and its subject (a `MigrationSubject`: a model, a field, or a storage name no model of the origin contract stores). Before `migration plan` writes anything, and before `db update` applies anything, the command asks one question per data-loss subject, answered by `delete` or, for a model or field the destination no longer has, `rename`. Where the target's migrations capability sets `renameStatements: { refused: true, keepDataByHand }`, as MongoDB's does, the question offers no rename and ends with the target's text on keeping the data by hand (renaming the collection in `mongosh` on each database before a plan that drops it is applied there; `db update` then drops nothing, and a migration written by `migration plan` still drops it, so remove that operation or do not apply it there). `db update` also asks one question per operation that changes who can read or write a model's rows, answered by `allow`: the question says the operation widens access when it does, such as disabling row-level security, and that it changes access otherwise, such as dropping a policy, since a permissive policy grants access. Two such operations on one model are two questions, each answered by its own `--allow`. The questions go through the CLI engine's statement prompt (`ctx.prompt.statements(questions, { last: true })`), which answers them from the flags first. In a terminal it asks the rest in turn, and the user types `delete`, `allow` or `rename :`; a typed rename is planned again with the other renames, and a plan that still has the operation asked about fails with `MIGRATION.STATEMENT_DID_NOT_RESOLVE_LOSS`; a loss the new plan has on the same subject, such as a type change on the renamed field, is asked next. Where nobody can answer (not interactive, or `--yes`), the command fails with `CLI.CONSENT_REQUIRED`, listing every unanswered question with the flags that answer it, and a `--delete` or `--allow` that answers no question fails with `CLI.CONSENT_UNUSED`; both happen before anything is written or applied. A `db update --dry-run` asks nothing; it checks the same flags against the subjects an apply would ask about and fails an unmatched one with `MIGRATION.STATEMENT_ANSWERS_NO_QUESTION`, so a script that checks the code handles both. `--confirm` answers none of these questions. The `--rename` values are taken from the run first (`ctx.statements.take('rename')`) and planned from the start, so the drop a rename replaces is never asked about. The shared question loop is `askPlanQuestions` in [`plan-questions.ts`](../../../packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts); the control API takes the answers through a callback, so it never depends on the engine's prompt. +**The questions.** The planner's success result lists what the plan would lose (`dataLoss`) and whose access it would widen (`accessWidening`), each entry an operation's position and its subject (a `MigrationSubject`: a model, a field, or a storage name no model of the origin contract stores). Before `migration plan` writes anything, and before `db update` applies anything, the command asks one question per data-loss subject, answered by `delete` or, for a model or field the destination no longer has, `rename`. A field whose model the destination names differently, with no statement renaming the model, is asked about with `delete` only, and the question says to pass the model rename first; the next question then offers the field rename. `db update` also asks one question per operation that changes who can read or write a model's rows, answered by `allow`: the question says the operation widens access when it does, such as disabling row-level security, and that it changes access otherwise, such as dropping a policy, since a permissive policy grants access. Two such operations on one model are two questions, each answered by its own `--allow`. The questions go through the CLI engine's statement prompt (`ctx.prompt.statements(questions, { last: true })`), which answers them from the flags first. In a terminal it asks the rest in turn, and the user types `delete`, `allow` or `rename :`; a typed rename is planned again with the other renames, and a plan that still has the operation asked about fails with `MIGRATION.STATEMENT_DID_NOT_RESOLVE_LOSS`; a loss the new plan has on the same subject, such as a type change on the renamed field, is asked next. Where nobody can answer (not interactive, or `--yes`), the command fails with `CLI.CONSENT_REQUIRED`, listing every unanswered question with the flags that answer it, and a `--delete` or `--allow` that answers no question fails with `CLI.CONSENT_UNUSED`; both happen before anything is written or applied. A `db update --dry-run` asks nothing; it checks the same flags against the subjects an apply would ask about and fails an unmatched one with `MIGRATION.STATEMENT_ANSWERS_NO_QUESTION`, so a script that checks the code handles both. `--confirm` answers none of these questions. The `--rename` values are taken from the run first (`ctx.statements.take('rename')`) and planned from the start, so the drop a rename replaces is never asked about. The shared question loop is `askPlanQuestions` in [`plan-questions.ts`](../../../packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts); the control API takes the answers through a callback, so it never depends on the engine's prompt. #### Rename statements @@ -207,6 +207,15 @@ prisma db update --rename Profile:User --rename User.name:User.fullName --delete Rename operations are `widening`, so neither command asks about them. The `migration.ts` that `migration plan` writes contains `...this.renameTable(...)` and `...this.renameColumn(...)` calls, as a user would write them by hand, and running it again writes the same `ops.json` and `migration.json`. +The MongoDB planner does the same with collections and documents ([`mongo-statement-planning.ts`](../../../packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts)). A statement names a field by its contract key, as on SQL; MongoDB's contract keys a field by its stored name, so `id @map("_id")` is `User._id` in a statement. + +- A model statement whose collection name changes renames the collection in the working copy and plans `renameCollection`. MongoDB keeps the documents, indexes and validator, so the diff then plans only what really changed. +- A field statement whose stored name changes plans `renameField`: one `updateMany` with `$rename` on the documents that hold the old name, skipping validation, since the old validator requires the old field. A variant's field is renamed only in that variant's documents, filtered by its discriminator value, so another variant that stores a field with the same name keeps it. A base field that a variant now stores itself is renamed only in the documents of the other variants. +- A field that an origin model stores and its destination model no longer stores, in a collection the plan keeps, is data loss. The plan unsets it with `unsetField` (`$unset`, skipping validation) from the documents that hold it, with the same variant filters, and asks about it as a `field` subject, so `--delete User.nickname` or a `--rename` answers it. Without an origin contract, a removed field is found from the origin validator's top-level properties and named `.`; a collection whose destination has no validator is skipped, and a variant's removed field is not found. +- Operations run in this order: collection renames, collection creates, index drops, document rewrites, index creates, validator changes, option changes, collection drops. Rewrites run after the index drops, so a unique index on the old field does not fail once documents lose it, and before the index creates, so a unique index on the new field sees the moved values. + +`renameCollection` and `renameField` are `widening` and `unsetField` is `destructive`, so `db update` runs them. Document rewrites are DML operations (`MongoDmlOperation`): the runner and the operation serializer tell a DML operation from a DDL operation by its shape (`run` against `execute`), with `isMongoDmlOperation`, not by its class; a hand-written `dataTransform` is a DML operation that keeps the class `data`. Each rewrite's after-check, that no matching document holds the old name, lets a run that stopped part way finish when run again. The `migration.ts` that `migration plan` writes calls `renameCollection(...)`, `renameField(...)` and `unsetField(...)` from the target's migration module. + **Result.** The planner's success result carries `appliedStatements` (`AppliedMigrationStatement`): one entry per statement, in order, with `operationIndexes`: the positions, in the plan's operations, of the operations the statement accounts for, empty when its storage did not change. Statements point at operations by position because operation ids are not unique within a plan (on Postgres, same-named tables in two schemas share them), and ids stay as they are so that re-emitting an existing migration does not change its hash. The framework's `describeMigrationStatement` writes the statement in domain names, such as `rename model "Profile" to "User"`, for every family's messages and for the CLI. `migration plan` and `db update` print each statement with its description and number of operations, and their JSON output carries the description and the operation positions, which index the output's `operations`. Messages name a model of the unbound namespace by its name alone, and JSON output leaves `namespaceId` out of a coordinate in the unbound namespace (`migrationStatementJson`), because that namespace's id is internal. **Refusals.** When the planner cannot carry out a resolved statement, planning fails with `MIGRATION.PLANNING_FAILED` and a conflict of kind `statementRefused` that carries the statement in its `refusedStatement` field. The [error reference](../../reference/error-reference.md#migrationplanning_failed) lists the cases. @@ -216,7 +225,8 @@ Rename operations are `widening`, so neither command asks about them. The `migra - A model move to another namespace resolves, but the planner refuses it with `statementRefused`. - A model that keeps its name but changes its table cannot be renamed by a statement, because the model has the same name on both sides. Rename the table by hand with `this.renameTable(...)` in its own `migration.ts`. A field statement on such a model is refused with the same advice. - Only models and fields can be renamed. Value objects, their fields, enum values and namespaces cannot. -- The MongoDB planner refuses every statement with `statementRefused`, so `migration plan` and `db update` with `--rename` fail with `MIGRATION.PLANNING_FAILED` and plan nothing, rather than dropping and creating the collection. +- On MongoDB, a statement cannot rename a variant whose discriminator value changes, since its documents would keep the old value; it cannot rename `_id`, a field whose stored name MongoDB reads as a path or an operator (it contains `.` or starts with `$`), a time-series collection or a field of one, or a field inside a value object. A field removed from a model on a time-series collection fails planning with a `fieldRemovalRefused` conflict, because MongoDB cannot remove it from those documents; keeping the field in the contract plans nothing. A collection rename onto a name the database already has, or onto a name MongoDB rejects, is refused too. +- On a MongoDB contract that gives a collection no validator, which today is every contract written with the TypeScript contract builder, field names are not part of the storage hash. A field rename or removal alone then leaves the hash unchanged, and `db migrate` refuses a migration that starts and ends at the same hash unless it runs a `dataTransform`. So `migration plan` refuses a field rename statement there before it writes anything (`sameHashFieldChangeRefusals` in [`same-hash-field-changes.ts`](../../../packages/3-mongo-target/1-mongo-target/src/core/migrations/same-hash-field-changes.ts)), and a field removal, which no statement names, leaves the hash unchanged, so the plan finds no change. `db update` writes no migration and carries out both. A model rename that changes the collection name still works, since the collection name is in the hash. A later release gives contracts written in TypeScript the same validator as PSL contracts, and this limit goes away. #### Delete and allow statements @@ -226,9 +236,9 @@ Rename operations are `widening`, so neither command asks about them. The `migra #### The origin of `db update` -`migration plan` already has an origin contract: the one `--from` resolves to. `db update` introspects the database instead, and on every run reads the contract for the marker's hash from the [snapshot store](#contract-resolution-through-the-snapshot-store). The planner gets it as `fromContract`, and names each subject of `dataLoss` and `accessWidening` through it. A missing or unreadable snapshot fails only a run with `--rename`; otherwise each subject is named by its storage name. The plan it hands the runner keeps `origin: null` either way (see [`db update`](#db-update-live-reconciliation)). The planner's operations do not depend on whether the snapshot was read: every strategy that emits a data transform needs the `data` operation class, which `db update` does not allow, and codec field-event hooks get the prior contract only when the plan has an origin, which a `db update` plan never has. `db update` stores a snapshot only when it advances a ref: the `db` ref by default, or with `--db ` only the ref named by `--advance-ref `. So a database last updated with `--db ` and no `--advance-ref` has no snapshot, and renames fail there with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`, which names the hash and the directory it looked in. Running `db update` again with the same statements fails with `MIGRATION.STATEMENT_UNRESOLVED`, because the origin is then a contract without the old names. +`migration plan` already has an origin contract: the one `--from` resolves to. `db update` introspects the database instead, and on every run reads the contract for the marker's hash from the [snapshot store](#contract-resolution-through-the-snapshot-store). The planner gets it as `fromContract`, and names each subject of `dataLoss` and `accessWidening` through it. A missing or unreadable snapshot fails only a run with `--rename`; otherwise each subject is named by its storage name. The plan it hands the runner keeps `origin: null` either way (see [`db update`](#db-update-live-reconciliation)). On SQL, the planner's operations do not depend on whether the snapshot was read: every strategy that emits a data transform needs the `data` operation class, which `db update` does not allow, and codec field-event hooks get the prior contract only when the plan has an origin, which a `db update` plan never has. On MongoDB they can: without the snapshot, a removed field is found only from the origin validator, as described under [Rename statements](#rename-statements). `db update` stores a snapshot only when it advances a ref: the `db` ref by default, or with `--db ` only the ref named by `--advance-ref `. So a database last updated with `--db ` and no `--advance-ref` has no snapshot, and renames fail there with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`, which names the hash and the directory it looked in. Running `db update` again with the same statements fails with `MIGRATION.STATEMENT_UNRESOLVED`, because the origin is then a contract without the old names. -The rest of the migration-statements work adds `--convert`, `--backfill`, the remaining renames and model moves across namespaces; and brings statements to MongoDB. +The rest of the migration-statements work adds `--convert`, `--backfill`, the remaining renames and model moves across namespaces. ## Authoring Surface @@ -314,7 +324,7 @@ Each operation follows the same three-phase loop: evaluate prechecks → execute Per [ADR 192 — Apply-time verification](../adrs/ADR%20192%20-%20ops.json%20is%20the%20migration%20contract.md), `db migrate` performs two checks before executing operations: -1. **On-disk consistency.** For every loaded bundle, recompute `migrationHash` from the on-disk manifest + `ops.json` (`verifyMigrationBundle`) and compare against the stored `migrationHash`. A mismatch aborts apply with a structured runtime error pointing at the offending directory and asks the developer to re-emit the package (`node migrations//migration.ts`) or restore from version control. This catches FS corruption, partial writes, and post-emit hand edits before the runner touches the database. +1. **On-disk consistency.** For every loaded bundle, recompute `migrationHash` from the on-disk manifest + `ops.json` (`verifyMigrationBundle`) and compare against the stored `migrationHash`. A mismatch aborts apply with a structured runtime error pointing at the offending directory and asks the developer to re-emit the package (`node /migration.ts`) or restore from version control. This catches FS corruption, partial writes, and post-emit hand edits before the runner touches the database. 2. **Emit-drift detection.** When `migration.ts` is present, re-emit in-memory and compare against `ops.json`. *(Architecturally required; not yet implemented — tracked as a follow-up.)* ### Pre-DDL marker drift check @@ -462,7 +472,7 @@ DDL operations carry commands like `CreateIndexCommand`, `CreateCollectionComman Data transforms carry `operationClass: 'data'` and use the **same three-phase envelope** as DDL operations. The runner has a single execution loop; there is no DT-specific dispatch path. The two families fill the envelope with their own query shapes: - **Postgres.** The `dataTransform` factory lowers the user-authored `check` / `run` query plans into the standard envelope at factory time. `precheck` is `[{ sql: 'SELECT EXISTS () AS ok', params: , … }]` (asserts "work to do"); `execute` carries the run plans as `SqlMigrationPlanOperationStep` entries with `params` threaded through `driver.query`; `postcheck` is `[{ sql: 'SELECT NOT EXISTS () AS ok', params: , … }]` (asserts "work done"). Idempotent re-apply uses the same pre-satisfied-skip path that DDL ops already use — when the postcheck is already satisfied, the runner records a `postcheck_pre_satisfied` skip without re-running execute. Routing the run-plan parameters through the driver's parameter binder (rather than inlining values into SQL text) is strictly safer than rolling a per-driver literal serializer. -- **Mongo.** The `run` array carries `MongoQueryPlan` objects. Checks use the same `{ description, source, filter, expect }` shape as DDL checks, with `MongoQueryPlan` as the source type. Both are evaluated by the same `FilterEvaluator` path. DML commands execute through `Adapter.lower()` → `Driver.execute()` — the same transport as runtime queries. +- **Mongo.** The `run` array carries `MongoQueryPlan` objects. Checks use the same `{ description, source, filter, expect }` shape as DDL checks, with `MongoQueryPlan` as the source type. Both are evaluated by the same `FilterEvaluator` path. DML commands execute through `Adapter.lower()` → `Driver.execute()` — the same transport as runtime queries. Such an operation is a DML operation (`MongoDmlOperation`), and the runner and the serializer recognise it by its `run` array, with `isMongoDmlOperation`, not by its class: the planner's own document rewrites (`renameField`, `unsetField`) have the same shape but carry the class of their effect, `widening` or `destructive`, so `db update` runs them. The shared envelope means policy gating on `operationClass: 'data'` (CI checks, audit, "this run may not perform data work") is uniform across families and decoupled from how each family describes its check/run payload. @@ -555,7 +565,7 @@ A migration always exists or it doesn't — there is no on-disk "draft" state. E - Attested: `migrationHash` matches current content (the only on-disk state) - Applied: Ledger shows edge applied; DB marker equals `to` hash -The planner exposes one transient hint, `pendingPlaceholders`, in the `migration plan` result envelope. It is a CLI-only signal that the on-disk migration encodes zero ops because the user still needs to fill in `placeholder(...)` slots; it is **not** persisted to the manifest. After filling in the slots, the developer re-emits the package (`node migrations//migration.ts`) to compute the real ops and the corresponding `migrationHash`. Applying a placeholder-blocked migration will not advance the storage hash to the destination — the runner's destination-hash post-check surfaces this as a state mismatch. +The planner exposes one transient hint, `pendingPlaceholders`, in the `migration plan` result envelope. It is a CLI-only signal that the on-disk migration encodes zero ops because the user still needs to fill in `placeholder(...)` slots; it is **not** persisted to the manifest. After filling in the slots, the developer re-emits the package (`node /migration.ts`) to compute the real ops and the corresponding `migrationHash`. Applying a placeholder-blocked migration will not advance the storage hash to the destination — the runner's destination-hash post-check surfaces this as a state mismatch. ### Authoring intermediate state @@ -584,13 +594,13 @@ A placeholder-bearing migration is therefore visible to the runner. Applying it #### Transition -The intermediate state ends when the developer runs the package's `migration.ts` directly — either via its shebang (`./migrations//migration.ts`) or `node migrations//migration.ts`. Class-flow self-emission (see [ADR 196 — In-process emit for class-flow targets](../adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md)) detects that the file is the main module, runs `Migration.run(...)`, calls `buildMigrationArtifacts` against the now-real op list, and **rewrites both `ops.json` and `migration.json`** in place — including the `migrationHash`, which is recomputed over the new `(strippedManifest, ops)`. Any unfilled `placeholder(...)` still in the file at this point throws `PN-MIG-2001` and aborts the rewrite, leaving the package in its previous (still-intermediate) shape; see [ADR 200 — Placeholder utility for scaffolded migration slots](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md). There is no separate CLI command for this transition: the on-disk package is the source of truth, and running `migration.ts` is what materializes the filled `ops.json`. +The intermediate state ends when the developer runs the package's `migration.ts` directly — either via its shebang (`.//migration.ts`) or `node /migration.ts`. Class-flow self-emission (see [ADR 196 — In-process emit for class-flow targets](../adrs/ADR%20196%20-%20In-process%20emit%20for%20class-flow%20targets.md)) detects that the file is the main module, runs `Migration.run(...)`, calls `buildMigrationArtifacts` against the now-real op list, and **rewrites both `ops.json` and `migration.json`** in place — including the `migrationHash`, which is recomputed over the new `(strippedManifest, ops)`. Any unfilled `placeholder(...)` still in the file at this point throws `PN-MIG-2001` and aborts the rewrite, leaving the package in its previous (still-intermediate) shape; see [ADR 200 — Placeholder utility for scaffolded migration slots](../adrs/ADR%20200%20-%20Placeholder%20utility%20for%20scaffolded%20migration%20slots.md). There is no separate CLI command for this transition: the on-disk package is the source of truth, and running `migration.ts` is what materializes the filled `ops.json`. #### Diagnostics A user who is unsure whether a migration is in the intermediate state has several signals: -- `prisma migration plan` returns `pendingPlaceholders: true` and prints a `⚠` line whose body says *"edit migration.ts then run `node migration.ts` to self-emit"*, followed by the operations that resolved and the statements applied. The same envelope is in the JSON output for tooling, with those operations in `operations`. +- `prisma migration plan` returns `pendingPlaceholders: true` and prints a `⚠` line whose body says to edit `migration.ts`, then run `node /migration.ts`, with the directory it wrote, to self-emit, followed by the operations that resolved and the statements applied. The same envelope is in the JSON output for tooling, with those operations in `operations`. - `ops.json` literally contains `[]`, and `migration.ts` mentions `placeholder(...)` (planner-led) or has no operation calls in its `plan()` body (custom flow). - Running `migration.ts` while a placeholder is still unfilled fails with `PN-MIG-2001` — the structured error names the slot. - `prisma db migrate` against a placeholder-blocked package will execute the lock + verify path but apply zero ops; the destination-hash post-check then surfaces `MIGRATION.POSTCHECK_FAILED` (state mismatch). This is the runner-side fallback diagnostic when an intermediate-state migration leaks past the CLI signals above. diff --git a/docs/reference/error-reference.md b/docs/reference/error-reference.md index a0dcb181103a..2a3cb4a5b67e 100644 --- a/docs/reference/error-reference.md +++ b/docs/reference/error-reference.md @@ -81,7 +81,7 @@ The migration-file CLI (`prisma migration`) received `--config` without a path a ### CLI.CONSENT_REQUIRED -Raised by the CLI engine, not by Prisma ORM's own code; the engine's entry is the reference: [CLI.CONSENT_REQUIRED](https://docs.prisma.io/docs/cli/error-reference#CLI.CONSENT_REQUIRED). `migration plan` and `db update` raise it when the plan would lose data, or for `db update` widen who can read or write rows, and the run has nobody to answer: it is not interactive, or `--yes` is set. The command asks one question per model, field or storage name whose data an operation would lose, and `db update` also one per model whose rows an operation would open up, such as disabling row-level security. The error lists every question still unanswered, with the flags that answer each: `--delete ` lets the plan lose that data, for a model or field `--rename :` keeps it under a new name, and `--allow ` lets `db update` widen access to that model's rows. Data no model of the origin contract stores can only be deleted. A field of a model the plan renames is named through the model's new name: after `--rename Profile:User`, the question about the dropped `nickname` field is answered with `--delete User.nickname` or `--rename User.nickname:User.handle`. Nothing is written or applied. `--confirm` does not answer these questions. Payload: `subject`, `verbs`, `unanswered` (each with its `subject` and `verbs`). +Raised by the CLI engine, not by Prisma ORM's own code; the engine's entry is the reference: [CLI.CONSENT_REQUIRED](https://docs.prisma.io/docs/cli/error-reference#CLI.CONSENT_REQUIRED). `migration plan` and `db update` raise it when the plan would lose data, or for `db update` widen who can read or write rows, and the run has nobody to answer: it is not interactive, or `--yes` is set. The command asks one question per model, field or storage name whose data an operation would lose, and `db update` also one per model whose rows an operation would open up, such as disabling row-level security. The error lists every question still unanswered, with the flags that answer each: `--delete ` lets the plan lose that data, for a model or field `--rename :` keeps it under a new name, and `--allow ` lets `db update` widen access to that model's rows. Data no model of the origin contract stores can only be deleted. A field of a model the plan renames is named through the model's new name: after `--rename Profile:User`, the question about the dropped `nickname` field is answered with `--delete User.nickname` or `--rename User.nickname:User.handle`. A field whose model the destination names differently, with no statement renaming the model, is offered only `--delete`, and the question says to pass `--rename :` first; the next question then offers the field rename. On MongoDB, removing a field from a model is asked about too, and `--delete` removes it from every document that holds it; with no snapshot of the origin contract the field is named `.`. Nothing is written or applied. `--confirm` does not answer these questions. Payload: `subject`, `verbs`, `unanswered` (each with its `subject` and `verbs`). ### CLI.CONSENT_UNUSED @@ -211,6 +211,10 @@ The main CLI received a `--format` value other than `pretty` or `json`. Raised d Reserved: a command was asked for a `--json` sub-format it does not support; the error lists the formats the command does accept. Declared in the shared error factories but not raised by any command today. Payload: `command`, `format`, `supportedFormats`. +### CLI.MIGRATION_CONFIG_NOT_FOUND + +Running a `migration.ts` with node and no `--config` found no config that owns the migration. Without `--config`, the file uses the config whose migrations directory holds it: the nearest `prisma.config.ts` above the migration's directory if it owns it, else the `prisma.config.ts` in the working directory if that one does. The message names each config it checked, with the migrations directory it keeps or the reason it failed to load, and says to run the file again with `--config `. Payload: `migrationDir`, `checked`. + ### CLI.OUTPUT_FORMAT_CONFLICT The main CLI received mutually exclusive output flags: `--format pretty` together with `--json`. Use `--format json` or `--json` alone for JSON output. Payload: none. @@ -1462,7 +1466,11 @@ The per-space migration planner received the same contract-space id more than on ### MIGRATION.EXECUTION_FAILED -A migration operation's SQL step failed while being executed against the database during apply (`db init`, `db update`, `db migrate`). The envelope carries the database error detail so you can see which statement failed and why. Payload: `operationId`, `stepDescription`, `sql`, `sqlState`, `constraint`, `table`, `column`, `detail`. +A migration operation's SQL step failed while being executed against the database during apply (`db init`, `db update`, `db migrate`). The envelope carries the database error detail so you can see which statement failed and why. Payload: `operationId`, `stepDescription`, `sql`, `sqlState`, `constraint`, `table`, `column`, `detail`. On MongoDB it is raised when a field rename or removal rewrites documents with `bypassDocumentValidation` and the database user lacks that action (MongoDB error 13, `Unauthorized`), as a `readWrite` user on a hosted cluster does; the message says to grant the `bypassDocumentValidation` action, for example with the `dbAdmin` or `restore` role, and to run the command again. Payload: `operationId`, `mongoCode`, `codeName`. + +### MIGRATION.FIELD_NAME_UNSUPPORTED + +A MongoDB `renameField` or `unsetField` operation names a field it cannot rewrite: an empty name, `_id` (MongoDB does not let a document change its `_id`), a name containing `.` (MongoDB reads it as a path into embedded documents), or a name starting with `$` (MongoDB reads it as an operator). Raised when the operation is built, before anything runs. Payload: `collection`, `field`, `reason`. ### MIGRATION.FILE_MISSING @@ -1506,7 +1514,7 @@ The migration name given to `migration new`/`migration plan --name` contains no ### MIGRATION.INVALID_OPERATION_ENTRY -An operation returned by an authored migration class failed schema validation during emit: each entry of `operations` must carry `id`, `label`, and an `operationClass` of `additive`, `widening`, `destructive`, or `data`. Also raised when deserializing a persisted Mongo migration plan hits a malformed or unknown entry (filter, pipeline stage, DML/DDL/inspection command). Payload: `index`, `reason` (emit validation) or `context`, `kind` (plan deserialization). +An operation returned by an authored migration class failed schema validation during emit: each entry of `operations` must carry `id`, `label`, and an `operationClass` of `additive`, `widening`, `destructive`, or `data`. Also raised when deserializing a persisted Mongo migration plan hits a malformed or unknown entry (filter, pipeline stage, DML/DDL/inspection command). A MongoDB `renameField` whose new name is its old name raises it when the operation is built, with `context: 'renameField'`, `collection`, `field` and `reason`. Payload: `index`, `reason` (emit validation) or `context`, `kind` (plan deserialization). ### MIGRATION.INVALID_REF_FILE @@ -1586,7 +1594,7 @@ An apply command (`db migrate`/`db update`) cannot find a path through the on-di ### MIGRATION.PLANNING_FAILED -Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); a rename that produces an operation the command's operation policy does not allow; or any statement on MongoDB, whose planner refuses statements in this release and plans nothing. `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. +Migration planning (typically during `db init`/`db update`) failed because of conflicts, e.g. the live database already contains objects that clash with the plan. The envelope aggregates each conflict's summary and suggested fix. Payload: `conflicts`; a conflict's `location` and `refusedStatement` leave out `namespaceId` for the unbound namespace, whose id is internal. A conflict of kind `statementRefused` refuses a `--rename` statement the planner cannot carry out, and carries that statement (`refusedStatement`, in namespace, model and field names): moving a model to another namespace (not supported in this release); renaming a table or column whose control policy is not `managed`; a model with no table; a table or column the starting schema does not have, or already has under the new name; a field that has a column on one side of the statement only; a field rename on a model whose table changes when no statement renames the table (the message gives the steps: change the contract so only the table name changes and emit it, create a migration with `prisma migration new --from `, add `...this.renameTable(...)` to its operations and run its `migration.ts` with `node`, then emit the final contract and plan the field rename with `--from` that migration; with `db update`: change the contract so only the table name changes and emit it, rename the table in the database yourself with the SQL the message gives, check that `db update --dry-run` plans no drop of a table or column and run `db update`, then emit the final contract and run `db update` with the field statement); or a rename that produces an operation the command's operation policy does not allow. On MongoDB the planner refuses: renaming `_id`; renaming a variant whose discriminator value changes (the message says to keep the value with `@@base(, "")`); a field whose stored name contains `.` or starts with `$` (renaming from such a name says to rename it in the database yourself; renaming to one says to choose another stored name); renaming a time-series collection (the message says to create the new collection with the same time-series options, copy the documents and leave out the statement); renaming a field of a time-series collection, since MongoDB updates its documents only through the metaField (the message says to keep the field's stored name); removing a field of a time-series collection, a conflict of kind `fieldRemovalRefused` (the message says to keep the field in the contract, optional if new documents leave it out); a collection the schema being planned from does not have, which includes a view; a collection rename onto a name the schema already has, or onto a name MongoDB rejects (empty, containing `$` or a null character, or starting with `system.`); a field stored in documents on one side of the statement only, or an embedded relation; a collection whose control policy is not `managed`; and a field rename whose collection changes with no statement renaming the model (when the model keeps its name, the message gives the steps: change only `@@map` and emit; with `db update`, rename the collection in `mongosh` and run `db update`; with `migration plan`, plan with `--delete `, make `renameCollection(...)` the first operation in its `migration.ts` and delete the drop, and the create if there is one, run its `migration.ts` with `node`, and apply it with `db migrate`; then rename the field). `db init` applies only additive operations, so when a conflict is an operation its policy refused (the conflict carries `refusedOperationClass`), such as adding a validator to a Mongo collection that already holds documents, the error's next action depends on the refused classes. When `db update` allows all of them (`widening` and `destructive`), the next action is `db update`, which applies them, asking what each operation that would lose data means. When one is `data`, which `db update` does not apply, the next action is `migration plan`, to plan a migration that `db migrate` then applies. `migration plan` raises it when the planner produces no operations. The conflict's kind names the case, and its `why` is the next action. With an earlier contract (`noDatabaseChange`), the contract changed in a way the database does not see, for example after a Prisma upgrade that changes how `contract.json` records values. If you deploy with migrations, run `prisma migration new --from `, with the earlier contract's hash from the message, to write a migration with no operations, then `prisma db migrate`. If you manage the database with `db init` or `db update`, run `prisma db sign` on each database instead. With no earlier contract (`nothingToCreate`), the contract describes nothing to create; if the database needs something the contract does not describe, run `prisma migration new` and write the operations in `migration.ts` by hand. `migration new` writes a migration only to the emitted contract, so with `--to` the message says to emit the target first, and for a database managed with `db update` it names `prisma db sign `. When the migrations directory is empty and the `db` ref points at a contract that describes nothing to create (`nothingToBaseline`), there is no baseline migration to plan, and no command writes one with no operations yet; keep using `db update`. @@ -1612,7 +1620,7 @@ A `PostgresMigration` operation (e.g. `createTable`, `dataTransform`) was invoke ### MIGRATION.PRECHECK_FAILED -Before executing a migration operation, one of its precheck steps (a query expected to return true) did not hold: the database is not in the state the operation requires, so the apply stops and rolls back. Payload: `operationId`, `phase`, `stepDescription`. +Before executing a migration operation, one of its precheck steps (a query expected to return true) did not hold: the database is not in the state the operation requires, so the apply stops and rolls back. Payload: `operationId`, `phase`, `stepDescription`. On MongoDB, where operations are not wrapped in a transaction, the operations before it stay applied, and a re-run skips them. A field rename (`renameField`) stops here when documents already hold the new name as well as the old one: the message names the collection, gives the `mongosh` query that finds those documents, and says to fix them and run the command again. A collection rename (`renameCollection`) stops here when the new collection already exists: the message names both collections, says nothing was renamed, and says to drop the new collection, or, when an earlier operation of the same migration created it, to make the rename the first operation of its `migration.ts`. Payload on MongoDB: `operationId`, `name` (a field rename only). ### MIGRATION.PROVIDED_INVARIANTS_MISMATCH diff --git a/packages/1-framework/1-core/errors/src/control.ts b/packages/1-framework/1-core/errors/src/control.ts index c51c516bba87..06704d637b30 100644 --- a/packages/1-framework/1-core/errors/src/control.ts +++ b/packages/1-framework/1-core/errors/src/control.ts @@ -443,6 +443,43 @@ export function errorTargetMigrationNotSupported(options?: { ); } +/** A config the migration-file CLI checked while looking for the one that owns the migration. */ +export interface CheckedMigrationConfig { + readonly configPath: string; + readonly migrationsDir?: string; + readonly loadFailure?: string; +} + +/** + * Without `--config`, the migration-file CLI found no config whose migrations directory holds the migration. + */ +export function errorMigrationCliConfigNotFound(options: { + readonly migrationDir: string; + readonly checked: readonly CheckedMigrationConfig[]; +}): CliStructuredError { + const rule = `Without --config, a migration.ts uses the prisma.config.ts whose migrations directory holds it: the nearest one above ${options.migrationDir}, else the one in the working directory.`; + const found = options.checked.map(({ configPath, migrationsDir, loadFailure }) => + loadFailure === undefined + ? `${configPath} keeps migrations in ${migrationsDir}` + : `${configPath} failed to load (${loadFailure})`, + ); + return new CliStructuredError( + 'CLI.MIGRATION_CONFIG_NOT_FOUND', + 'No config found for this migration', + { + why: + found.length === 0 + ? `${rule} There is neither.` + : `${rule} None of them does: ${found.join('; ')}.`, + fix: 'Run it again with `--config `.', + meta: { + migrationDir: options.migrationDir, + checked: options.checked.map(({ configPath }) => configPath), + }, + }, + ); +} + /** * The migration-file CLI received `--config` without a path argument (either * a bare trailing `--config`, or `--config` followed by another flag like diff --git a/packages/1-framework/1-core/errors/src/exports/control.ts b/packages/1-framework/1-core/errors/src/exports/control.ts index 4eb66fbbac67..67bb6d7d42f4 100644 --- a/packages/1-framework/1-core/errors/src/exports/control.ts +++ b/packages/1-framework/1-core/errors/src/exports/control.ts @@ -1,4 +1,4 @@ -export type { CliErrorConflict, CliErrorEnvelope } from '../control'; +export type { CheckedMigrationConfig, CliErrorConflict, CliErrorEnvelope } from '../control'; export { CliStructuredError, errorConfigEvaluationFailed, @@ -15,6 +15,7 @@ export { errorFileNotFound, errorInvalidOutputFormat, errorJsonFormatNotSupported, + errorMigrationCliConfigNotFound, errorMigrationCliInvalidConfigArg, errorMigrationCliUnknownFlag, errorMigrationPlanningFailed, diff --git a/packages/1-framework/1-core/errors/test/control.test.ts b/packages/1-framework/1-core/errors/test/control.test.ts index ddb9111dba89..c8ccd8d01d02 100644 --- a/packages/1-framework/1-core/errors/test/control.test.ts +++ b/packages/1-framework/1-core/errors/test/control.test.ts @@ -19,6 +19,7 @@ import { errorFileNotFound, errorInvalidOutputFormat, errorJsonFormatNotSupported, + errorMigrationCliConfigNotFound, errorMigrationCliInvalidConfigArg, errorMigrationCliUnknownFlag, errorMigrationPlanningFailed, @@ -523,6 +524,39 @@ describe('Config Errors', () => { expect(error.where).toBeUndefined(); }); + it('errorMigrationCliConfigNotFound names each config it checked and why it does not own the migration', () => { + const error = errorMigrationCliConfigNotFound({ + migrationDir: '/repo/inner/migrations/app/20260101T0000_initial', + checked: [ + { configPath: '/repo/prisma.config.ts', migrationsDir: '/repo/migrations' }, + { configPath: '/work/prisma.config.ts', loadFailure: 'Failed to evaluate config' }, + ], + }); + expect({ + code: error.code, + message: error.message, + why: error.why, + fix: error.fix, + meta: error.meta, + }).toEqual({ + code: 'CLI.MIGRATION_CONFIG_NOT_FOUND', + message: 'No config found for this migration', + why: 'Without --config, a migration.ts uses the prisma.config.ts whose migrations directory holds it: the nearest one above /repo/inner/migrations/app/20260101T0000_initial, else the one in the working directory. None of them does: /repo/prisma.config.ts keeps migrations in /repo/migrations; /work/prisma.config.ts failed to load (Failed to evaluate config).', + fix: 'Run it again with `--config `.', + meta: { + migrationDir: '/repo/inner/migrations/app/20260101T0000_initial', + checked: ['/repo/prisma.config.ts', '/work/prisma.config.ts'], + }, + }); + }); + + it('errorMigrationCliConfigNotFound says when it found no config at all', () => { + const error = errorMigrationCliConfigNotFound({ migrationDir: '/m', checked: [] }); + expect(error.why).toBe( + 'Without --config, a migration.ts uses the prisma.config.ts whose migrations directory holds it: the nearest one above /m, else the one in the working directory. There is neither.', + ); + }); + it('errorMigrationCliInvalidConfigArg creates correct error for missing path', () => { const error = errorMigrationCliInvalidConfigArg(); expect(error.code).toBe('CLI.CONFIG_ARG_MISSING_PATH'); diff --git a/packages/1-framework/1-core/framework-components/src/control/control-migration-types.ts b/packages/1-framework/1-core/framework-components/src/control/control-migration-types.ts index 8d1f0b85e152..21e778f3190a 100644 --- a/packages/1-framework/1-core/framework-components/src/control/control-migration-types.ts +++ b/packages/1-framework/1-core/framework-components/src/control/control-migration-types.ts @@ -9,7 +9,7 @@ * with additional fields for execution (precheck SQL, execute SQL, etc.). */ -import type { Contract, ContractWithDomain } from '@internal/contract/types'; +import type { Contract } from '@internal/contract/types'; import type { ImportRequirement } from '@internal/ts-render'; import type { Result } from '@internal/utils/result'; import type { TargetBoundComponentDescriptor } from '../shared/framework-components'; @@ -23,7 +23,6 @@ import type { OperationContext } from './control-operation-results'; import type { AppliedMigrationStatement, MigrationPlanSubjects, - MigrationSubject, ResolvedMigrationStatement, } from './migration-statements'; @@ -653,18 +652,6 @@ export interface TargetMigrationsCapability< adapter: ControlAdapterInstance, ): MigrationPlanner; createRunner(family: TFamilyInstance): MigrationRunner; - /** - * Set when the target's planner carries out no rename statement: a data-loss question then - * offers no rename, and says instead how to keep the data by hand. - */ - readonly renameStatements?: { - readonly refused: true; - /** How to keep the data of `subject` by hand, so that the plan made afterwards loses nothing. */ - readonly keepDataByHand: ( - subject: MigrationSubject, - fromContract: ContractWithDomain, - ) => string; - }; /** * Synthesizes a family-specific schema IR from a contract for offline planning. * The returned schema can be passed to `planner.plan({ schema })` as the "from" state. diff --git a/packages/1-framework/1-core/framework-components/src/control/migration-statements.ts b/packages/1-framework/1-core/framework-components/src/control/migration-statements.ts index d646cfd14a32..0802c673b292 100644 --- a/packages/1-framework/1-core/framework-components/src/control/migration-statements.ts +++ b/packages/1-framework/1-core/framework-components/src/control/migration-statements.ts @@ -68,6 +68,12 @@ export interface MigrationOperationSubject { readonly subject: MigrationSubject; } +/** An operation that loses the data of its subject. */ +export interface MigrationDataLoss extends MigrationOperationSubject { + /** How to keep the data instead, in the target's words, when no statement can say so. */ + readonly keepDataAdvice?: string; +} + /** An operation that changes who can read or write the rows of its subject. */ export interface MigrationAccessChange extends MigrationOperationSubject { /** @@ -79,7 +85,7 @@ export interface MigrationAccessChange extends MigrationOperationSubject { /** What the operations of a plan lose and whose access they change, each by its position in the plan. */ export interface MigrationPlanSubjects< - TEntry extends MigrationOperationSubject = MigrationOperationSubject, + TEntry extends MigrationOperationSubject = MigrationDataLoss, TAccess extends MigrationAccessChange = MigrationAccessChange, > { /** Each operation that loses data, in plan order, with what it loses. */ diff --git a/packages/1-framework/1-core/framework-components/src/exports/control.ts b/packages/1-framework/1-core/framework-components/src/exports/control.ts index d76f0c3756da..bdb2edc8dbfc 100644 --- a/packages/1-framework/1-core/framework-components/src/exports/control.ts +++ b/packages/1-framework/1-core/framework-components/src/exports/control.ts @@ -127,6 +127,7 @@ export type { FieldCoordinate, FieldCoordinateJson, MigrationAccessChange, + MigrationDataLoss, MigrationOperationSubject, MigrationPlanSubjects, MigrationStatementJson, diff --git a/packages/1-framework/3-tooling/cli/README.md b/packages/1-framework/3-tooling/cli/README.md index 45c3ae2797de..c9335b2fd40f 100644 --- a/packages/1-framework/3-tooling/cli/README.md +++ b/packages/1-framework/3-tooling/cli/README.md @@ -954,11 +954,11 @@ Update your database schema to match the currently emitted contract. prisma db update [--db ] [--to ] [--advance-ref ] [--config ] [--dry-run] [--rename ]... [--delete ]... [--allow ]... [--interactive|--no-interactive] [--json] [-v] [-q] [--color/--no-color] ``` -**Rename statements (`--rename :`, repeatable):** tell `db update` that a model or field was renamed, so it renames the table or column instead of dropping and creating it. The statements work exactly as they do for `migration plan` (see below), and produce the same operations. The origin contract they resolve against is the contract the database's marker names, read from the local snapshot store (`migrations/snapshots//`). The snapshot is read on every run, so the planner sees the prior contract whenever one is stored; a missing or unreadable snapshot fails only a run with `--rename`. Either way the plan applies from whatever state the database is in, as `db update` always has. `db update` keeps a snapshot of the contract it applies whenever it advances a ref: the `db` ref by default, or with `--db ` only the ref you name with `--advance-ref `. So to use renames against a database you update with `--db `, give that earlier run `--advance-ref `. Without a readable snapshot, a run with statements fails with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`, naming the hash and the directory it looked in, and gives the steps that store the missing snapshot: emit the contract the database is at, run `db update --advance-ref --dry-run` against it and check that it plans no operations, then run it without `--dry-run`, which changes nothing in the database and stores the snapshot, then emit the new contract and run the rename. A database with no marker has nothing to rename, and fails the same way. Running the same statements a second time fails with `MIGRATION.STATEMENT_UNRESOLVED`, because the database is now at a contract that no longer has the old names. The statements the plan applied are listed under `Statements applied` after the operations (dry run and apply), and as `appliedStatements` in `--json` output, each with its `description` and `operationIndexes`, the positions in `operations` of the operations it accounts for. +**Rename statements (`--rename :`, repeatable):** tell `db update` that a model or field was renamed, so it renames the table or column, or on MongoDB the collection or the field in every document, instead of dropping and creating it. The statements work exactly as they do for `migration plan` (see below), and produce the same operations. The origin contract they resolve against is the contract the database's marker names, read from the local snapshot store (`migrations/snapshots//`). The snapshot is read on every run, so the planner sees the prior contract whenever one is stored; a missing or unreadable snapshot fails only a run with `--rename`. Either way the plan applies from whatever state the database is in, as `db update` always has. `db update` keeps a snapshot of the contract it applies whenever it advances a ref: the `db` ref by default, or with `--db ` only the ref you name with `--advance-ref `. So to use renames against a database you update with `--db `, give that earlier run `--advance-ref `. Without a readable snapshot, a run with statements fails with `MIGRATION.STATEMENT_ORIGIN_UNKNOWN`, naming the hash and the directory it looked in, and gives the steps that store the missing snapshot: emit the contract the database is at, run `db update --advance-ref --dry-run` against it and check that it plans no operations, then run it without `--dry-run`, which changes nothing in the database and stores the snapshot, then emit the new contract and run the rename. A database with no marker has nothing to rename, and fails the same way. Running the same statements a second time fails with `MIGRATION.STATEMENT_UNRESOLVED`, because the database is now at a contract that no longer has the old names. The statements the plan applied are listed under `Statements applied` after the operations (dry run and apply), and as `appliedStatements` in `--json` output, each with its `description` and `operationIndexes`, the positions in `operations` of the operations it accounts for. **Questions before an apply:** an apply asks one question per model, field or storage name whose data an operation would lose (dropping a table or a column, or a type change that can change values), and one per model whose rows an operation would let more people read or write (disabling row-level security, for example). It asks before it applies anything. Subjects are named through the origin contract read from the snapshot store, as `migration plan` names them; without a readable snapshot each subject is its storage name, which can only be deleted or allowed, and the question says the origin contract is unknown. Recorded migrations of extension spaces are asked about for the data they would lose, never for widened access. Each question is answered with a statement: -- `--delete ` lets the update lose that data, for example `--delete Legacy` or `--delete User.nickname`. A field of a model the update renames is named through the model's new name. +- `--delete ` lets the update lose that data, for example `--delete Legacy` or `--delete User.nickname`. A field of a model the update renames is named through the model's new name. On MongoDB, a field removed from a model is asked about, and `--delete` removes it from every document that holds it; without a snapshot of the origin contract it is named `.`, for example `--delete profiles.nickname`. - `--rename :` keeps the data of a model or field under a new name instead. A field's new name keeps its model: `--rename User.nickname:User.handle`, which the question writes as `User.nickname:User.`. - `--allow ` lets the update change who can read or write that model's rows, for example `--allow User`. Each such operation is its own question: dropping a policy and disabling row-level security on one model take two `--allow User` flags. The question says disabling row-level security widens access, and that dropping a policy changes it, since a permissive policy grants access. @@ -1036,7 +1036,7 @@ prisma migration plan [--config ] [--name ] [--from ] [--t 3. Determines the starting point: `--from ` if provided, otherwise the `db` ref. When the ref is absent, greenfield only on an empty graph; over existing migrations the command refuses (`MIGRATION.PLAN_ORIGIN_UNKNOWN`) unless `--from @empty` is passed 4. Diffs the starting contract against the destination using the target's migration planner 5. Scaffolds a new migration package: `migration.ts` (containing `placeholder(...)` lambdas for any data transforms), `migration.json` (with a content-addressed `migrationHash` over the planned ops, or over `[]` when the planner could not lower any calls because of placeholders), and `ops.json` (the planned ops, or `[]` in the placeholder-blocked case). The bookend contracts are written write-if-absent into the shared snapshot store at `migrations/snapshots//contract.{json,d.ts}`. The package is **always** fully attested — there is no draft state on disk. -6. If the plan has unfilled `placeholder(...)` slots, the command returns a successful `pendingPlaceholders` envelope (a warning, not a failure) asking the developer to fill in the slots before re-emitting. The output still lists the operations that resolved, and any `Statements applied`, so a rename and the drops beside it show; JSON `operations` lists the same operations, so the `operationIndexes` of `appliedStatements` point into it. The on-disk `ops.json` is `[]` and `migrationHash` is the hash of `(metadata, [])`, so applying the migration as-written will not advance the storage hash to the intended destination — the runner's destination-hash post-check surfaces this as a state mismatch. After filling in the placeholders, run `node migrations//migration.ts` to re-emit `ops.json` and the corresponding `migrationHash`. `PN-MIG-2001` is raised only at self-emit time when a slot is still unfilled. +6. If the plan has unfilled `placeholder(...)` slots, the command returns a successful `pendingPlaceholders` envelope (a warning, not a failure) asking the developer to fill in the slots before re-emitting. The output still lists the operations that resolved, and any `Statements applied`, so a rename and the drops beside it show; JSON `operations` lists the same operations, so the `operationIndexes` of `appliedStatements` point into it. The on-disk `ops.json` is `[]` and `migrationHash` is the hash of `(metadata, [])`, so applying the migration as-written will not advance the storage hash to the intended destination — the runner's destination-hash post-check surfaces this as a state mismatch. After filling in the placeholders, run `node /migration.ts` to re-emit `ops.json` and the corresponding `migrationHash`. `PN-MIG-2001` is raised only at self-emit time when a slot is still unfilled. **Outputs:** - `migrations//migration.ts` — editable migration source (with `placeholder(...)` slots when the planner inserted them) @@ -1044,12 +1044,12 @@ prisma migration plan [--config ] [--name ] [--from ] [--t - `migrations//ops.json` — planned operations (empty list `[]` if placeholders blocked the planner) - `migrations/snapshots//contract.{json,d.ts}` — bookend contracts, written write-if-absent, keyed by each contract's storage hash (one entry for `from` when applicable, one for `to`) -**Rename statements:** each `--rename` names the old and the new name in contract vocabulary, never a table or a column. Each side is one of `Model`, `namespace.Model`, `Model.field` or `namespace.Model.field`; both sides name a model, or both name a field of the same model. Names match exactly, including case. A name without a namespace resolves when exactly one namespace declares the model. A field's model is named as the destination contract names it, so a model rename followed by a field rename on it is `--rename Profile:User --rename User.name:User.fullName`. The old name must exist in the starting contract and not in the destination; the new name must exist in the destination and not in the starting contract. Statements resolve before anything is written; a statement that is malformed fails with `MIGRATION.STATEMENT_INVALID`, and one that does not resolve fails with `MIGRATION.STATEMENT_UNRESOLVED`, naming what it searched and what it found. Each error's next step is the statement to type instead: the accepted forms, the statement written the right way round, the statements a model-and-field mix may have meant, or the order that works. A statement that has already been applied (the origin already has the new name and not the old one) says to leave it out. Statements cannot swap two names in one plan, and the error gives the three plans that do, through a temporary name. A plan from an empty database (`--from @empty`, or greenfield) has no models to rename, so every statement is unresolved. Moving a model to another namespace and renaming value objects are not supported in this release. On MongoDB the planner refuses every statement in this release: planning fails with `MIGRATION.PLANNING_FAILED`, carrying the statement, and nothing is planned. The planner renames the table or column, and the constraints and indexes named after it, ahead of the rest of the plan. On Postgres, a check, a row-level-security policy, or an expression or partial index whose SQL names a renamed column is dropped and created again, and adding a check back reads every row of the table under an exclusive lock, so on a large table such a rename takes time; `migration.ts` contains the same `...this.renameTable(...)` and `...this.renameColumn(...)` calls you could have written by hand. The statements are listed under `Statements applied` after the operations, each with its number of operations, and as `appliedStatements` in `--json` output, each with its `statement`, `description` and `operationIndexes`, the positions in `operations` of the operations it accounts for. A coordinate in a contract with no namespaces has no `namespaceId` in the JSON. A statement whose storage does not change, such as a model whose table name is kept with `@@map`, is listed with no operations. When the storage did not change at all and no statement needs an operation, no migration package is written and the output says so; when the storage changed but the planner planned nothing, planning fails with `MIGRATION.PLANNING_FAILED`, statements or not. A statement the planner cannot carry out, for example on a table whose control policy is not `managed`, fails planning with `MIGRATION.PLANNING_FAILED`, and the refused statement is carried in the error's conflicts. +**Rename statements:** each `--rename` names the old and the new name in contract vocabulary, never a table or a column. Each side is one of `Model`, `namespace.Model`, `Model.field` or `namespace.Model.field`; both sides name a model, or both name a field of the same model. Names match exactly, including case. A name without a namespace resolves when exactly one namespace declares the model. A field's model is named as the destination contract names it, so a model rename followed by a field rename on it is `--rename Profile:User --rename User.name:User.fullName`. The old name must exist in the starting contract and not in the destination; the new name must exist in the destination and not in the starting contract. Statements resolve before anything is written; a statement that is malformed fails with `MIGRATION.STATEMENT_INVALID`, and one that does not resolve fails with `MIGRATION.STATEMENT_UNRESOLVED`, naming what it searched and what it found. Each error's next step is the statement to type instead: the accepted forms, the statement written the right way round, the statements a model-and-field mix may have meant, or the order that works. A statement that has already been applied (the origin already has the new name and not the old one) says to leave it out. Statements cannot swap two names in one plan, and the error gives the three plans that do, through a temporary name. A plan from an empty database (`--from @empty`, or greenfield) has no models to rename, so every statement is unresolved. Moving a model to another namespace and renaming value objects are not supported in this release. The planner renames the table or column, and the constraints and indexes named after it, ahead of the rest of the plan. On Postgres, a check, a row-level-security policy, or an expression or partial index whose SQL names a renamed column is dropped and created again, and adding a check back reads every row of the table under an exclusive lock, so on a large table such a rename takes time; `migration.ts` contains the same `...this.renameTable(...)` and `...this.renameColumn(...)` calls you could have written by hand. On MongoDB a field is named by its stored name (`id @map("_id")` is `User._id`). A model rename renames the collection, which keeps its documents, indexes and validator (`renameCollection`); a field rename renames the field in every document that holds it (`renameField`), and only in that variant's documents for a variant's field; `migration.ts` calls these factories. The statements are listed under `Statements applied` after the operations, each with its number of operations, and as `appliedStatements` in `--json` output, each with its `statement`, `description` and `operationIndexes`, the positions in `operations` of the operations it accounts for. A coordinate in a contract with no namespaces has no `namespaceId` in the JSON. A statement whose storage does not change, such as a model whose table name is kept with `@@map`, is listed with no operations. When the storage did not change at all and no statement needs an operation, no migration package is written and the output says so; when the storage changed but the planner planned nothing, planning fails with `MIGRATION.PLANNING_FAILED`, statements or not. A statement the planner cannot carry out, for example on a table whose control policy is not `managed`, fails planning with `MIGRATION.PLANNING_FAILED`, and the refused statement is carried in the error's conflicts. **Data loss:** a plan that would lose data is written only once you say what each such operation means. The command asks one question per model, field or storage name whose data an operation would lose (dropping a table or a column, or a type change that can change values), before it writes anything, including the baseline package of an auto-baseline. Each question is answered with a statement: - `--delete ` lets the plan lose that data, for example `--delete Legacy` for a model or `--delete User.nickname` for a field. Data that no model of the starting contract stores is named by its storage name, as the question gives it, and can only be deleted. A field of a model the plan renames is named through the model's new name, as a field rename is written: after `--rename Profile:User`, a dropped `nickname` field is answered with `--delete User.nickname` or `--rename User.nickname:User.handle`. -- `--rename :` keeps the data of a model or field under a new name; a field's new name keeps its model (`--rename User.nickname:User.handle`). A rename on the command line is planned from the start, so the drop it replaces is never asked about. The question offers it only for a model or field the destination no longer has, so a field whose type changes is answered with `--delete`, and not on MongoDB, whose planner refuses renames in this release. There the question says how to keep the documents instead: rename the collection in `mongosh` on each database before a plan that drops it is applied; `db update` then drops nothing, and a migration written by `migration plan` still drops it. +- `--rename :` keeps the data of a model or field under a new name; a field's new name keeps its model (`--rename User.nickname:User.handle`). A rename on the command line is planned from the start, so the drop it replaces is never asked about. The question offers it only for a model or field the destination no longer has, so a field whose type changes is answered with `--delete`, and for a field whose model the destination names differently only after a `--rename` of the model, which the question then says to pass. On MongoDB, a field removed from a model is asked about too, and `--delete` removes it from every document that holds it (`unsetField`). Where nobody can answer (the run is not interactive, or `--yes` is set), the command fails with `CLI.CONSENT_REQUIRED` and lists every unanswered question with the flags that answer it. In a terminal it asks each question in turn: type `delete` to let the data go, or `rename :`; a rejected answer is asked again. A typed rename is planned again with the other renames, and when the plan still loses the subject's data the command fails with `MIGRATION.STATEMENT_DID_NOT_RESOLVE_LOSS`. A `--delete` that answers no question fails with `CLI.CONSENT_UNUSED` before anything is written. `--confirm` no longer consents to anything here. The deletes are listed under `Statements applied` after the renames, for example `delete model "Legacy" (1 operation)`, and in `--json` output each `appliedStatements` entry carries its `verb`, `rename` or `delete`; a delete's `statement` is `{ kind: 'delete', subject }`. The `⚠` marker stays on each destructive operation. @@ -1127,7 +1127,7 @@ prisma db migrate [--db ] [--to ] [--advance-ref ] [--show] - `-v, --verbose`: Verbose output (debug info, timings) **What it does:** -1. Reads migration packages from `config.migrations.dir`. Every package is attested — there is no on-disk draft state. The loader (`readMigrationPackage` in `@internal/migration-tools/io`) rehashes `(metadata, ops)` for each `MigrationPackage` it returns and confirms the result matches the stored `migrationHash`. If a package has been hand-edited or partially written since emit, the load fails with `MIGRATION.HASH_MISMATCH` pointing at the offending directory and asks the developer to re-run `node migrations//migration.ts` (or restore from version control). +1. Reads migration packages from `config.migrations.dir`. Every package is attested — there is no on-disk draft state. The loader (`readMigrationPackage` in `@internal/migration-tools/io`) rehashes `(metadata, ops)` for each `MigrationPackage` it returns and confirms the result matches the stored `migrationHash`. If a package has been hand-edited or partially written since emit, the load fails with `MIGRATION.HASH_MISMATCH` pointing at the offending directory and asks the developer to re-run `node /migration.ts` (or restore from version control). 2. Reconstructs the migration graph from all loaded packages 3. Determines the destination hash and apply contract: from `--to`, or from `contract.json` when `--to` is omitted or `@contract` 4. Connects to the database and reads the current marker hash @@ -1150,9 +1150,11 @@ run `migration.ts` directly with Node to produce `ops.json` and attest `migration.json`: ```bash -node migrations//migration.ts +node /migration.ts ``` +`` is the directory `migration plan` or `migration new` printed, under the config's `migrations.dir`. The command works from any directory. Without `--config`, the file uses the config whose migrations directory holds it: the nearest `prisma.config.ts` above it, else the one in the working directory; if neither does, it fails with `CLI.MIGRATION_CONFIG_NOT_FOUND`. If you ran the CLI with `--config`, pass the same `--config ` to the file; the commands `migration plan` and `migration new` print already include it. + The scaffolded `migration.ts` calls `MigrationCLI.run(import.meta.url, ...)` from `@internal/cli/migration-cli` when invoked as the entrypoint. (Postgres and SQLite scaffolds re-export `MigrationCLI` through `@internal/postgres/migration` or `@internal/sqlite/migration` so a `migration.ts` only needs the single facade import; Mongo scaffolds still pull from `@internal/cli/migration-cli` directly.) The CLI entrypoint loads `prisma.config.ts`, assembles a `ControlStack`, instantiates the migration with that stack (so `dataTransform` and other adapter-aware helpers can materialize a real adapter), and serializes operations to `ops.json` while writing the content-addressed `migrationHash` into `migration.json`. If `migration.ts` contains unfilled `placeholder()` slots, the script exits with `PN-MIG-2001` and reports the slot to fill in. `MigrationCLI.run` accepts an optional third argument `{ argv?, stdout?, stderr? }` for in-process testability (default: `process.argv` / `process.stdout` / `process.stderr`) and returns the exit code as a `Promise`. The flag surface is `--help` / `--dry-run` / `--config `, parsed by [`clipanion`](https://github.com/arcanis/clipanion). The main multi-command surface (`contract emit`, `db verify`, etc.) runs on `@prisma/cli-engine`; the per-migration `MigrationCLI.run` entrypoint uses clipanion to keep authored migration files lightweight and in-process testable. diff --git a/packages/1-framework/3-tooling/cli/src/control-api/operations/db-run.ts b/packages/1-framework/3-tooling/cli/src/control-api/operations/db-run.ts index c099b09e038c..ed173e3b0532 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/operations/db-run.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/operations/db-run.ts @@ -48,7 +48,6 @@ import { type AnswerPlanQuestions, askPlanQuestions, type ConsentedSubject, - keepDataByHandFor, type PlannedQuestions, refuseUnusedConsents, subjectText, @@ -326,7 +325,6 @@ export async function executeRun plan(resolveRenames(renames)), diff --git a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts index b58f25d10613..d693ebb19c94 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/operations/migration-plan.ts @@ -9,7 +9,6 @@ import { createControlStack, hasOperationPreview, type MigrationPlanOperation, - type MigrationSubject, migrationSubjectKey, type OperationPreview, planOriginOf, @@ -49,12 +48,12 @@ import { import { toExtensionInputs } from '../../utils/extension-pack-inputs'; import { assertFrameworkComponentsCompatible } from '../../utils/framework-components'; import { createProjectSpecifierResolver } from '../../utils/project-import-root'; +import { selfEmitCommand } from '../../utils/self-emit-command'; import { snapshotVerifierFor } from '../../utils/snapshot-content-verification'; import { type AnswerPlanQuestions, askPlanQuestions, - keepDataByHandFor, - type PlannedSubject, + type PlannedLoss, subjectText, } from '../statements/plan-questions'; import { @@ -87,6 +86,8 @@ export interface MigrationPlanOptions { readonly cwd: string; /** The project's directory, normally the validated config's `baseDir`; locates the project manifest. */ readonly projectDir?: string; + /** The `--config` the command was given, repeated in the command that runs a written `migration.ts`. */ + readonly configPath?: string; readonly name?: string; readonly from?: string; readonly to?: string; @@ -114,9 +115,20 @@ type PlannerSuccess = { readonly hasPlaceholders: boolean; readonly appliedStatements: readonly AppliedStatementReport[]; /** The operations that would lose data, positioned in the plan's operations. */ - readonly dataLoss: readonly PlannedSubject[]; + readonly dataLoss: readonly PlannedLoss[]; }; +function placeholderSummary( + planned: string, + dirs: readonly string[], + configPath: string | undefined, +): string { + const commands = dirs + .map((dir) => `\`${selfEmitCommand(join(dir, 'migration.ts'), configPath)}\``) + .join(' and '); + return `Planned ${planned} with placeholder(s) — edit migration.ts, then run ${commands} to self-emit`; +} + /** The origin of a plan from an empty database: no models, so no statement resolves. */ const EMPTY_ORIGIN: ContractWithDomain = { domain: { namespaces: {} } }; @@ -218,9 +230,10 @@ async function runPlannerLeg( plannedOps, migrationTsContent: plannerResult.plan.renderTypeScript(resolveImportSpecifier), hasPlaceholders, - dataLoss: plannerResult.dataLoss.map(({ operationIndex, subject }) => ({ + dataLoss: plannerResult.dataLoss.map(({ operationIndex, subject, keepDataAdvice }) => ({ operationIndex: position(operationIndex), subject, + ...ifDefined('keepDataAdvice', keepDataAdvice), label: resolved[operationIndex]?.label ?? subjectText(subject, { origin: fromContract ?? EMPTY_ORIGIN, renames: [] }), @@ -299,7 +312,6 @@ async function answerPlannedDataLoss(input: { readonly consents: readonly StatementText[]; readonly origin: ContractWithDomain; readonly originKnown: boolean; - readonly keepDataByHand: ((subject: MigrationSubject) => string) | undefined; readonly destination: ContractWithDomain; readonly answer: AnswerPlanQuestions; readonly planDelta: ( @@ -327,7 +339,6 @@ async function answerPlannedDataLoss(input: { consentAll: { delete: false, allow: false }, origin: input.origin, originKnown: input.originKnown, - keepDataByHand: input.keepDataByHand, destination: input.destination, answer: input.answer, replan: async (renames) => { @@ -752,7 +763,6 @@ async function executeMigrationPlanCommandInner( consents: statementTexts.filter((statement) => statement.verb !== 'rename'), origin: fromContract ?? EMPTY_ORIGIN, originKnown: true, - keepDataByHand: keepDataByHandFor(migrations, fromContract ?? EMPTY_ORIGIN), destination: toContract, answer: options.answerQuestions, planDelta: async (renames) => { @@ -868,8 +878,7 @@ async function executeMigrationPlanCommandInner( ...(warnings.length > 0 ? { warnings } : {}), pendingPlaceholders: true, appliedStatements: [...statementsWithoutOperations, ...deletes], - summary: - 'Planned baseline with placeholder(s) — edit migration.ts then run `node migration.ts` to self-emit', + summary: placeholderSummary('baseline', [baselineDir], options.configPath), timings: { total: Date.now() - startTime }, }; return ok(result); @@ -911,21 +920,26 @@ async function executeMigrationPlanCommandInner( const baselineOps = baselineLeg.value.hasPlaceholders ? [] : baselineLeg.value.plannedOps; const deltaOps = deltaLeg.value.hasPlaceholders ? [] : deltaLeg.value.plannedOps; if (baselineLeg.value.hasPlaceholders || deltaLeg.value.hasPlaceholders) { + const baselineDir = relative(cwd, baselinePackageDir); + const deltaDir = relative(cwd, deltaPackageDir); const result: MigrationPlanResult = { ok: true, noOp: false, from: fromHash, to: toStorageHash, - dir: relative(cwd, deltaPackageDir), - baselineDir: relative(cwd, baselinePackageDir), + dir: deltaDir, + baselineDir, operations: operationSummaries(deltaLeg.value.plannedOps), baselineOperations: operationSummaries(baselineLeg.value.plannedOps), emittedExtensionDirs, ...(warnings.length > 0 ? { warnings } : {}), pendingPlaceholders: true, appliedStatements: [...deltaLeg.value.appliedStatements, ...deletes], - summary: - 'Planned baseline + migration with placeholder(s) — edit migration.ts then run `node migration.ts` to self-emit', + summary: placeholderSummary( + 'baseline + migration', + [baselineDir, deltaDir], + options.configPath, + ), timings: { total: Date.now() - startTime }, }; return ok(result); @@ -1024,20 +1038,20 @@ async function executeMigrationPlanCommandInner( await writeDestinationSnapshot(toStorageHash); if (deltaLeg.value.hasPlaceholders) { + const dir = relative(cwd, packageDir); const result: MigrationPlanResult = { ok: true, noOp: false, from: fromHash, to: toStorageHash, - dir: relative(cwd, packageDir), + dir, operations: operationSummaries(deltaLeg.value.plannedOps), emittedExtensionDirs, ...(warnings.length > 0 ? { warnings } : {}), pendingPlaceholders: true, ...(fromDefaulted ? { fromDefaulted } : {}), appliedStatements: [...deltaLeg.value.appliedStatements, ...deletes], - summary: - 'Planned migration with placeholder(s) — edit migration.ts then run `node migration.ts` to self-emit', + summary: placeholderSummary('migration', [dir], options.configPath), timings: { total: Date.now() - startTime }, }; return ok(result); diff --git a/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts b/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts index d25ef495ae6e..18c2a43b7457 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/statements/plan-questions.ts @@ -6,7 +6,6 @@ import { migrationSubjectKey, modelDisplayName, type ResolvedMigrationStatement, - type TargetMigrationsCapability, } from '@internal/framework-components/control'; import { InternalError } from '@internal/utils/internal-error'; import { notOk, ok, type Result } from '@internal/utils/result'; @@ -60,6 +59,12 @@ export interface PlannedSubject { readonly subject: MigrationSubject; } +/** An operation of a plan that loses its subject's data. */ +export interface PlannedLoss extends PlannedSubject { + /** How to keep the data instead, in the target's words; the question ends with it. */ + readonly keepDataAdvice?: string; +} + /** An operation of a plan that changes who can read or write its subject's rows. */ export interface PlannedAccessChange extends PlannedSubject { readonly widens: boolean; @@ -67,7 +72,7 @@ export interface PlannedAccessChange extends PlannedSubject { /** What a plan would lose, and whose access it would change. */ export interface PlannedQuestions { - readonly dataLoss: readonly PlannedSubject[]; + readonly dataLoss: readonly PlannedLoss[]; readonly accessWidening: readonly PlannedAccessChange[]; } @@ -146,12 +151,12 @@ function lossText(subject: MigrationSubject, text: string, originKnown: boolean) case 'storage': return originKnown ? `would lose the data in "${text}", which no model of the origin contract stores` - : `would lose the data in "${text}", named by its storage name because the origin contract is unknown; --delete loses its rows`; + : `would lose the data in "${text}", named by its storage name because the origin contract is unknown; --delete loses that data`; } } /** - * How to keep the rows of a storage subject that was renamed, when the origin contract is unknown: + * How to keep the data of a storage subject that was renamed, when the origin contract is unknown: * store the snapshot of the contract the database is at, as `MIGRATION.STATEMENT_ORIGIN_UNKNOWN` * advises, then answer with a rename. The refusal gives it once, on its first storage question. The * engine does not put the CLI's name in a question's text, so the steps name `prisma`. @@ -159,7 +164,7 @@ function lossText(subject: MigrationSubject, text: string, originKnown: boolean) export const ORIGIN_SNAPSHOT_RECOVERY = [ 'If it was renamed, keep its data instead: store the snapshot of the contract the database is at, then run this command again and answer with --rename.', ...STORE_ORIGIN_SNAPSHOT_STEPS.map((step) => step.replaceAll('{bin}', 'prisma')), -].join(' '); +].join('\n'); function sameCoordinate(statement: ResolvedMigrationStatement, subject: MigrationSubject): boolean { if (subject.kind === 'storage') return false; @@ -175,16 +180,9 @@ function namesSubject(verb: ConsentVerb, subject: string) { text === subject ? undefined : `${verb} names "${subject}": --${verb} ${subject}.`; } -/** - * How the target says to keep a subject's data by hand when its planner carries out no rename, or - * `undefined` when it carries them out. - */ -export function keepDataByHandFor( - migrations: { readonly renameStatements?: TargetMigrationsCapability['renameStatements'] }, - origin: ContractWithDomain, -): ((subject: MigrationSubject) => string) | undefined { - const refused = migrations.renameStatements; - return refused === undefined ? undefined : (subject) => refused.keepDataByHand(subject, origin); +function modelRenameAdvice(model: ModelCoordinate, contracts: StatementContracts): string { + const name = modelText(contracts.origin, model); + return `The destination contract has no model "${name}"; to keep the values instead, also pass --rename ${name}:, naming the model that stores them now, and the question then offers to rename the field.`; } /** A rename as a statement is written: a field's new name keeps the model, `User.nickname:User.`. */ @@ -221,25 +219,42 @@ function inDestination( return subject.kind === 'model' || Object.hasOwn(destinationModel.fields, subject.field); } +/** + * The model of a field subject, under the name a rename of the plan gives it, when the destination + * has no model of that name: a rename of the field cannot be written until a statement renames the + * model. + */ +function missingFieldModel( + subject: MigrationSubject, + contracts: StatementContracts & { readonly destination: ContractWithDomain }, +): ModelCoordinate | undefined { + if (subject.kind !== 'field') return undefined; + const model = renamedModel(subject, contracts) ?? subject; + return contracts.destination.domain.namespaces[model.namespaceId]?.models[model.model] === + undefined + ? model + : undefined; +} + /** * The question for one operation that would lose data. A model or field the destination no longer * has, and no rename of the plan renamed, may be renamed instead of deleted; a rename is resolved * against the two contracts after the plan's other renames, and its old name must be the subject. * A subject the destination keeps, such as a field whose type changes, and data no model stores, - * can only be deleted. Where the target's planner carries out no rename, `keepDataByHand` says how - * to keep the data instead. + * can only be deleted. A field whose model the destination names differently can be renamed only + * after a statement renames the model, so the question says to pass one. */ export function dataLossQuestion( - loss: PlannedSubject, + loss: PlannedLoss, contracts: StatementContracts & { readonly destination: ContractWithDomain; readonly originKnown: boolean; - readonly keepDataByHand: ((subject: MigrationSubject) => string) | undefined; }, ): PlanQuestion { const subject = subjectText(loss.subject, contracts); + const missingModel = missingFieldModel(loss.subject, contracts); const renamable = - contracts.keepDataByHand === undefined && + missingModel === undefined && loss.subject.kind !== 'storage' && !inDestination(loss.subject, contracts) && !renamedByPlan(loss.subject, contracts); @@ -247,7 +262,8 @@ export function dataLossQuestion( return { question: [ `${loss.label} ${lossText(loss.subject, subject, contracts.originKnown)}.`, - ...(contracts.keepDataByHand === undefined ? [] : [contracts.keepDataByHand(loss.subject)]), + ...(missingModel === undefined ? [] : [modelRenameAdvice(missingModel, contracts)]), + ...(loss.keepDataAdvice === undefined ? [] : [loss.keepDataAdvice]), ].join(' '), subject, verbs: renamable ? ['rename', 'delete'] : ['delete'], @@ -353,7 +369,7 @@ function checkAnswers(questions: readonly PlanQuestion[], answers: readonly Plan } function withRecoveryOnFirstStorageQuestion( - asked: readonly { readonly loss: PlannedSubject; readonly question: PlanQuestion }[], + asked: readonly { readonly loss: PlannedLoss; readonly question: PlanQuestion }[], recoveryApplies: boolean, ): readonly PlanQuestion[] { const first = recoveryApplies @@ -361,7 +377,7 @@ function withRecoveryOnFirstStorageQuestion( : -1; return asked.map(({ question }, index) => index === first - ? { ...question, question: `${question.question} ${ORIGIN_SNAPSHOT_RECOVERY}` } + ? { ...question, question: `${question.question}\n${ORIGIN_SNAPSHOT_RECOVERY}` } : question, ); } @@ -383,7 +399,6 @@ export async function askPlanQuestions readonly consentAll: { readonly [verb in ConsentVerb]: boolean }; readonly origin: ContractWithDomain; readonly originKnown: boolean; - readonly keepDataByHand: ((subject: MigrationSubject) => string) | undefined; readonly destination: ContractWithDomain; readonly answer: AnswerPlanQuestions; readonly replan: (renames: readonly StatementText[]) => Promise>; @@ -405,7 +420,6 @@ export async function askPlanQuestions const contracts = { origin: input.origin, originKnown: input.originKnown, - keepDataByHand: input.keepDataByHand, destination: input.destination, renames, }; @@ -443,7 +457,7 @@ export async function askPlanQuestions const questions = [ ...withRecoveryOnFirstStorageQuestion( losses.map((loss) => ({ loss, question: dataLossQuestion(loss, contracts) })), - !input.originKnown && input.keepDataByHand === undefined, + !input.originKnown, ), ...widenings.map((widening) => accessWideningQuestion(widening, contracts)), ]; diff --git a/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts b/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts index 8105d8d9cc3b..111cf4225b3e 100644 --- a/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts +++ b/packages/1-framework/3-tooling/cli/src/control-api/statements/resolve-statements.ts @@ -356,7 +356,18 @@ function readSide( segments: StatementSide, ): Result { if (segments.length === 1) return readModel(state, side, undefined, segments[0]); - if (segments.length === 3) return readField(state, side, ...segments); + if (segments.length === 3) { + const [first, second, third] = segments; + const qualified = readField(state, side, first, second, third); + const dotted = readField(state, side, undefined, first, `${second}.${third}`); + if (qualified.ok && dotted.ok) { + return notOk({ + reason: `"${first}.${second}.${third}" resolves both as namespace "${first}" model "${second}" field "${third}" and as model "${first}" field "${second}.${third}"`, + fix: `Give the field a stored name without "." with @map, or rename namespace "${first}" or model "${first}", so that only one reading resolves.`, + }); + } + return qualified.ok || !dotted.ok ? qualified : dotted; + } const [first, second] = segments; const asModel = readModel(state, side, first, second); const asField = readField(state, side, undefined, first, second); diff --git a/packages/1-framework/3-tooling/cli/src/migration-cli.ts b/packages/1-framework/3-tooling/cli/src/migration-cli.ts index b3c43afb9c85..4d34bd47fbb8 100644 --- a/packages/1-framework/3-tooling/cli/src/migration-cli.ts +++ b/packages/1-framework/3-tooling/cli/src/migration-cli.ts @@ -17,9 +17,8 @@ * 1. Detects whether the file is the direct entrypoint (no-op when imported). * 2. Parses CLI args (`--help`, `--dry-run`, `--config `) via * [clipanion](https://github.com/arcanis/clipanion). - * 3. Loads the project's `prisma.config.ts` via the same - * `loadConfigForSections` the CLI commands use, walking up from the - * migration file's directory. + * 3. Loads the config `--config` names. Without it, loads the config that + * owns the migration (see `loadMigrationFileConfig`). * 4. Probe-instantiates the migration class without a stack so it can read * `targetId` and verify it matches `config.target.targetId` * (`MIGRATION.TARGET_MISMATCH` on mismatch) before any stack-driven adapter @@ -46,7 +45,6 @@ import { readFileSync, realpathSync, writeFileSync } from 'node:fs'; import type { Writable } from 'node:stream'; import { fileURLToPath } from 'node:url'; -import { loadConfigForSections } from '@internal/config-loader'; import { type CliStructuredError, errorMigrationCliInvalidConfigArg, @@ -61,6 +59,7 @@ import { isStructuredError } from '@internal/utils/structured-error'; import { Cli, Command, Option, UsageError } from 'clipanion'; import { dirname, join } from 'pathe'; import { errorFromCaught } from './control-api/operations/caught-errors'; +import { loadMigrationFileConfig } from './utils/migration-file-config'; /** * Constructor shape accepted by `MigrationCLI.run`. `Migration` subclasses @@ -549,13 +548,12 @@ async function runMigration( const migrationFile = fileURLToPath(importMetaUrl); const migrationDir = dirname(migrationFile); - const configResult = await loadConfigForSections(parsed.config, [ - 'family', - 'target', - 'adapter', - 'driver', - 'extensions', - ]); + const configResult = await loadMigrationFileConfig({ + configPath: parsed.config, + migrationFile, + cwd: process.cwd(), + sections: ['family', 'target', 'adapter', 'driver', 'extensions'], + }); if (!configResult.ok) { throw configResult.failure; } diff --git a/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts b/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts index ff44ce7a5ad4..aa5ea17ede34 100644 --- a/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts +++ b/packages/1-framework/3-tooling/cli/src/orm/migration/new.ts @@ -10,6 +10,7 @@ import type { MigrationNewResult } from '../../control-api/operations/migration- import { executeMigrationNewCommand } from '../../control-api/operations/migration-new'; import type { CreateControlClient } from '../../control-api/types'; import { runCommandAction } from '../../utils/next-actions'; +import { selfEmitCommand } from '../../utils/self-emit-command'; import { defineOrmCommand } from '../define-command'; import { normalizeError } from '../normalize-error'; import { appMigrationsDirFor, baseDirFor, contractPathFor, displayPath } from './paths'; @@ -18,6 +19,7 @@ function newPresentations(inputs: { readonly document: MigrationNewResult; readonly contractPath: string; readonly appMigrationsRelative: string; + readonly configPath: string | undefined; }): Presentations { const { document } = inputs; const migrationTs = join(document.dir, 'migration.ts'); @@ -52,7 +54,7 @@ function newPresentations(inputs: { { kind: 'edit-file', label: `Write the migration body in ${migrationTs}` }, runCommandAction( 'Run it to self-emit ops.json and attest the package', - `node "${migrationTs}"`, + selfEmitCommand(migrationTs, inputs.configPath), ), ], }; @@ -64,8 +66,8 @@ export function createMigrationNewCommand(createClient: CreateControlClient) { summary: 'Scaffold a new migration for manual authoring', description: 'Creates a migration package with a migration.ts file for manual authoring.\n' + - 'Write the migration body in migration.ts, then run the file with Node\n' + - '(`node migration.ts`) to self-emit ops.json and attest the package.\n' + + 'Write the migration body in migration.ts, then run the file with Node, as this\n' + + 'command prints, to self-emit ops.json and attest the package.\n' + 'Offline — does not consult the database.', examples: [ 'migration new --name split-name', @@ -114,6 +116,7 @@ export function createMigrationNewCommand(createClient: CreateControlClient) { contractPath: contractPath === undefined ? '(unset)' : displayPath(contractPath, ctx.cwd), appMigrationsRelative: displayPath(appMigrationsDirFor(ctx.config), ctx.cwd), + configPath: ctx.configPath, }), ), ); diff --git a/packages/1-framework/3-tooling/cli/src/orm/migration/paths.ts b/packages/1-framework/3-tooling/cli/src/orm/migration/paths.ts index 42602425248a..60aaf79851fe 100644 --- a/packages/1-framework/3-tooling/cli/src/orm/migration/paths.ts +++ b/packages/1-framework/3-tooling/cli/src/orm/migration/paths.ts @@ -9,20 +9,27 @@ import { relative, resolve } from 'pathe'; * engine validated it. Every command runs behind that validation, so its * absence here is a bug. */ -export function baseDirFor(config: PrismaNextConfig): string { +export function baseDirFor(config: Pick): string { if (config.baseDir === undefined) { throw new InternalError('the orm config section reached a command without baseDir'); } return config.baseDir; } -/** Where migrations live; the validated config carries it absolute and defaulted. */ -export function migrationsDirFor(config: PrismaNextConfig): string { - return config.migrations?.dir ?? resolve(baseDirFor(config), 'migrations'); +type MigrationsLocation = Pick; + +/** + * Where a config keeps its migrations: `migrations.dir`, default `migrations`, resolved against the config's `baseDir`. `base` stands in for `baseDir` when a config built in memory carries none. + */ +export function migrationsDirFor(config: MigrationsLocation, base?: string): string { + return resolve( + config.baseDir ?? base ?? baseDirFor(config), + config.migrations?.dir ?? 'migrations', + ); } /** The app subspace under {@link migrationsDirFor}. */ -export function appMigrationsDirFor(config: PrismaNextConfig): string { +export function appMigrationsDirFor(config: MigrationsLocation): string { return spaceMigrationDirectory(migrationsDirFor(config), APP_SPACE_ID); } diff --git a/packages/1-framework/3-tooling/cli/src/orm/migration/plan.ts b/packages/1-framework/3-tooling/cli/src/orm/migration/plan.ts index abd2bc4afd54..5116aa75569d 100644 --- a/packages/1-framework/3-tooling/cli/src/orm/migration/plan.ts +++ b/packages/1-framework/3-tooling/cli/src/orm/migration/plan.ts @@ -16,6 +16,7 @@ import { } from '../../utils/contract-ref-forms'; import { previewBlockHeader } from '../../utils/formatters/migrations'; import { runCommandAction } from '../../utils/next-actions'; +import { selfEmitCommand } from '../../utils/self-emit-command'; import { defineOrmCommand } from '../define-command'; import { normalizeError } from '../normalize-error'; import { promptPlanQuestions } from '../plan-question-prompt'; @@ -190,6 +191,7 @@ function planBlocks(result: MigrationPlanResult, migrationsRelative: string): re function planNextActions( result: MigrationPlanResult, migrationsRelative: string, + configPath: string | undefined, ): readonly NextAction[] { if (result.pendingPlaceholders === true) { const stubFiles = [ @@ -206,7 +208,7 @@ function planNextActions( }, runCommandAction( 'Run it to self-emit ops.json and attest the package', - `node "${migrationTs}"`, + selfEmitCommand(migrationTs, configPath), ), ]; } @@ -234,6 +236,7 @@ function planPresentations(inputs: { readonly from: string | undefined; readonly to: string | undefined; readonly name: string | undefined; + readonly configPath: string | undefined; }): Presentations { return { stdout: () => [], @@ -252,7 +255,7 @@ function planPresentations(inputs: { ...planBlocks(inputs.document, inputs.migrationsRelative), ], json: () => inputs.document, - next: () => planNextActions(inputs.document, inputs.migrationsRelative), + next: () => planNextActions(inputs.document, inputs.migrationsRelative, inputs.configPath), }; } @@ -312,6 +315,7 @@ export function createMigrationPlanCommand(createClient: CreateControlClient) { config: ctx.config, cwd: ctx.cwd, projectDir: baseDirFor(ctx.config), + ...ifDefined('configPath', ctx.configPath), ...ifDefined('name', args.flags.name), ...ifDefined('from', args.flags.from), ...ifDefined('to', args.flags.to), @@ -353,6 +357,7 @@ export function createMigrationPlanCommand(createClient: CreateControlClient) { from: args.flags.from, to: args.flags.to, name: args.flags.name, + configPath: ctx.configPath, }), ), ); diff --git a/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts b/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts index 76a0603b3b50..93a3b9dc780b 100644 --- a/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts +++ b/packages/1-framework/3-tooling/cli/src/utils/cli-errors.ts @@ -779,8 +779,8 @@ function noOperationsConflict( summary: 'This contract describes nothing migration plan can create, so there is no first migration to plan.', why: destination.isEmitted - ? 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then run `node migration.ts` in that directory to write `ops.json`.' - : '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then run `node migration.ts` in that directory to write `ops.json`.', + ? 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then run that `migration.ts` with `node` to write `ops.json`.' + : '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then run that `migration.ts` with `node` to write `ops.json`.', }; case 'baseline': return { diff --git a/packages/1-framework/3-tooling/cli/src/utils/command-helpers.ts b/packages/1-framework/3-tooling/cli/src/utils/command-helpers.ts index 10ed40c470ca..185da65d5b0f 100644 --- a/packages/1-framework/3-tooling/cli/src/utils/command-helpers.ts +++ b/packages/1-framework/3-tooling/cli/src/utils/command-helpers.ts @@ -8,6 +8,7 @@ import { ifDefined } from '@internal/utils/defined'; import type { NextAction } from '@internal/utils/structured-error'; import { relative, resolve } from 'pathe'; import type { ControlClient } from '../control-api/types'; +import { migrationsDirFor } from '../orm/migration/paths'; import { CliStructuredError, errorRuntime } from './cli-errors'; import { hasUrlScheme, @@ -66,7 +67,7 @@ export function resolveMigrationPaths( const configPath = 'prisma.config.ts'; // A validated config carries the directory absolute; a config handed in // raw by a programmatic caller is anchored on its baseDir, else on cwd. - const migrationsDir = resolve(config.baseDir ?? cwd, config.migrations?.dir ?? 'migrations'); + const migrationsDir = migrationsDirFor(config, cwd); const migrationsRelative = relative(cwd, migrationsDir); const appMigrationsDir = spaceMigrationDirectory(migrationsDir, APP_SPACE_ID); const appMigrationsRelative = relative(cwd, appMigrationsDir); diff --git a/packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts b/packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts new file mode 100644 index 000000000000..8b95f92ac619 --- /dev/null +++ b/packages/1-framework/3-tooling/cli/src/utils/migration-file-config.ts @@ -0,0 +1,74 @@ +import { existsSync, realpathSync } from 'node:fs'; +import type { PrismaNextConfig } from '@internal/config/config-types'; +import { + CONFIG_FILENAME, + type ConfigSection, + findNearestConfigPathForFile, + loadConfig, + loadConfigForSections, + requireConfigSections, +} from '@internal/config-loader'; +import { + type CheckedMigrationConfig, + type CliStructuredError, + errorMigrationCliConfigNotFound, +} from '@internal/errors/control'; +import { notOk, ok, type Result } from '@internal/utils/result'; +import { dirname, isAbsolute, join, relative } from 'pathe'; +import { migrationsDirFor } from '../orm/migration/paths'; + +/** + * The config a `migration.ts` runs with. `--config` names it outright. Without it, the config is the one that owns the migration, whose migrations directory holds the file: the nearest `prisma.config.ts` above the migration, else the one in the working directory. + */ +export async function loadMigrationFileConfig(options: { + readonly configPath: string | undefined; + readonly migrationFile: string; + readonly cwd: string; + readonly sections: readonly ConfigSection[]; +}): Promise> { + const { configPath, migrationFile, cwd, sections } = options; + if (configPath !== undefined) { + return loadConfigForSections(configPath, sections); + } + const workingDirectoryConfig = join(cwd, CONFIG_FILENAME); + const candidates = [ + ...new Set([ + await findNearestConfigPathForFile(migrationFile), + existsSync(workingDirectoryConfig) ? workingDirectoryConfig : undefined, + ]), + ].filter((candidate) => candidate !== undefined); + const checked: CheckedMigrationConfig[] = []; + for (const candidate of candidates) { + const loaded = await loadConfig(candidate); + const config = loaded.ok + ? requireConfigSections(loaded.value, [...sections, 'migrations']) + : loaded; + if (!config.ok) { + checked.push({ configPath: candidate, loadFailure: config.failure.message }); + continue; + } + if (ownsMigration(config.value, candidate, migrationFile)) { + return ok(config.value); + } + checked.push({ + configPath: candidate, + migrationsDir: migrationsDirFor(config.value, dirname(candidate)), + }); + } + return notOk(errorMigrationCliConfigNotFound({ migrationDir: dirname(migrationFile), checked })); +} + +/** Whether `migrationFile` is inside the migrations directory of the config at `configPath`. */ +export function ownsMigration( + config: Pick, + configPath: string, + migrationFile: string, +): boolean { + const migrationsDir = migrationsDirFor(config, dirname(configPath)); + const path = relative(realPath(migrationsDir), realPath(migrationFile)); + return path !== '' && !path.startsWith('..') && !isAbsolute(path); +} + +function realPath(path: string): string { + return existsSync(path) ? realpathSync(path) : path; +} diff --git a/packages/1-framework/3-tooling/cli/src/utils/self-emit-command.ts b/packages/1-framework/3-tooling/cli/src/utils/self-emit-command.ts new file mode 100644 index 000000000000..7cd7f26921ab --- /dev/null +++ b/packages/1-framework/3-tooling/cli/src/utils/self-emit-command.ts @@ -0,0 +1,6 @@ +/** The command that runs a `migration.ts`, repeating the `--config` the CLI was given so it loads the same config. */ +export function selfEmitCommand(migrationTs: string, configPath: string | undefined): string { + return configPath === undefined + ? `node "${migrationTs}"` + : `node "${migrationTs}" --config "${configPath}"`; +} diff --git a/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts index b82f805b80e7..c48b7bf7585a 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/db-update.test.ts @@ -579,7 +579,7 @@ describe('executeDbUpdate', () => { expect(asked).toEqual([ { - question: `Drop column nickname from user would lose the data in "user.nickname", named by its storage name because the origin contract is unknown; --delete loses its rows. ${ORIGIN_SNAPSHOT_RECOVERY}`, + question: `Drop column nickname from user would lose the data in "user.nickname", named by its storage name because the origin contract is unknown; --delete loses that data.\n${ORIGIN_SNAPSHOT_RECOVERY}`, subject: 'user.nickname', verbs: ['delete'], }, diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts index 46dda0df1d95..636cba080fdb 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/ask-plan-questions.test.ts @@ -44,7 +44,6 @@ function run(replanned: PlannedQuestions, typed: readonly (readonly string[])[]) consentAll: { delete: false, allow: false }, origin, originKnown: true, - keepDataByHand: undefined, destination, answer: async (questions: readonly PlanQuestion[]) => { asked.push(questions.map(({ question }) => question)); @@ -101,7 +100,6 @@ describe('questions about storage when the origin contract is unknown', () => { consentAll: { delete: false, allow: false }, origin, originKnown: false, - keepDataByHand: undefined, destination, answer: async (questions) => { asked.push(...questions.map(({ question }) => question)); @@ -111,8 +109,14 @@ describe('questions about storage when the origin contract is unknown', () => { }); expect(asked).toEqual([ - `Drop table Legacy would lose the data in "Legacy", named by its storage name because the origin contract is unknown; --delete loses its rows. ${ORIGIN_SNAPSHOT_RECOVERY}`, - 'Drop table Profile would lose the data in "Profile", named by its storage name because the origin contract is unknown; --delete loses its rows.', + `Drop table Legacy would lose the data in "Legacy", named by its storage name because the origin contract is unknown; --delete loses that data.\n${ORIGIN_SNAPSHOT_RECOVERY}`, + 'Drop table Profile would lose the data in "Profile", named by its storage name because the origin contract is unknown; --delete loses that data.', + ]); + expect(ORIGIN_SNAPSHOT_RECOVERY.split('\n').map((line) => line.slice(0, 2))).toEqual([ + 'If', + '1.', + '2.', + '3.', ]); expect(ORIGIN_SNAPSHOT_RECOVERY).toContain('--dry-run'); expect(ORIGIN_SNAPSHOT_RECOVERY).toContain('`prisma db update --advance-ref --dry-run`'); @@ -150,7 +154,6 @@ describe('questions about access', () => { consentAll: { delete: false, allow: false }, origin, originKnown: true, - keepDataByHand: undefined, destination, answer: async (questions) => { asked.push(...questions.map(({ question }) => question)); @@ -196,7 +199,6 @@ describe('allow statements given as pre-answers', () => { consentAll: { delete: false, allow: false }, origin, originKnown: true, - keepDataByHand: undefined, destination, answer: async (questions) => { asked.push(...questions.map(({ question }) => question)); @@ -256,7 +258,6 @@ describe('an access change consented before a typed rename', () => { consentAll: { delete: false, allow: false }, origin, originKnown: true, - keepDataByHand: undefined, destination, answer: async (questions) => { rounds += 1; @@ -277,38 +278,3 @@ describe('an access change consented before a typed rename', () => { ).toEqual([{ verb: 'allow', operationIndex: 3 }]); }); }); - -describe('questions about storage on a target that carries out no rename', () => { - it('say how to keep the data by hand, without the steps that make a rename possible', async () => { - const asked: string[] = []; - await askPlanQuestions({ - plan: { - dataLoss: [ - { - operationIndex: 0, - label: 'Drop collection events', - subject: { kind: 'storage', name: 'events' }, - }, - ], - accessWidening: [], - }, - askAccess: false, - renames: [], - preAnswers: [], - consentAll: { delete: false, allow: false }, - origin, - originKnown: false, - keepDataByHand: () => 'Rename the collection by hand first.', - destination, - answer: async (questions) => { - asked.push(...questions.map(({ question }) => question)); - return questions.map(({ subject }) => ({ verb: 'delete', text: subject })); - }, - replan: async () => ok({ dataLoss: [], accessWidening: [] }), - }); - - expect(asked).toEqual([ - 'Drop collection events would lose the data in "events", named by its storage name because the origin contract is unknown; --delete loses its rows. Rename the collection by hand first.', - ]); - }); -}); diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts index 5231178ba9a7..56acf775cb40 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/data-loss-question.test.ts @@ -18,7 +18,6 @@ describe('dataLossQuestion for data no model names', () => { destination, renames: [], originKnown: true, - keepDataByHand: undefined, }); expect({ question: question.question, verbs: question.verbs }).toEqual({ @@ -34,11 +33,10 @@ describe('dataLossQuestion for data no model names', () => { destination, renames: [], originKnown: false, - keepDataByHand: undefined, }); expect(question.question).toBe( - 'Drop column age from user would lose the data in "public.user.age", named by its storage name because the origin contract is unknown; --delete loses its rows.', + 'Drop column age from user would lose the data in "public.user.age", named by its storage name because the origin contract is unknown; --delete loses that data.', ); }); }); @@ -50,33 +48,27 @@ describe('dataLossQuestion verbs', () => { }); const withNickname = contractOf({ app: { models: { User: { fields: ['email', 'nickname'] } } } }); - const keepByHand = () => 'Rename it by hand first.'; - - function questionFor( - field: string, - destination: typeof origin, - keepDataByHand: (() => string) | undefined, - ) { + function questionFor(field: string, destination: typeof origin) { return dataLossQuestion( { operationIndex: 0, label: 'An operation', subject: { kind: 'field', namespaceId: unbound, model: 'User', field }, }, - { origin: withNickname, destination, renames: [], originKnown: true, keepDataByHand }, + { origin: withNickname, destination, renames: [], originKnown: true }, ); } - function verbsFor(field: string, destination: typeof origin, renamesPlannable: boolean) { - return questionFor(field, destination, renamesPlannable ? undefined : keepByHand).verbs; + function verbsFor(field: string, destination: typeof origin) { + return questionFor(field, destination).verbs; } it('offers rename for a field the destination no longer has', () => { - expect(verbsFor('nickname', destinationWithoutNickname, true)).toEqual(['rename', 'delete']); + expect(verbsFor('nickname', destinationWithoutNickname)).toEqual(['rename', 'delete']); }); it('offers only delete for a field that keeps its name, such as one whose type changes', () => { - expect(verbsFor('email', destinationWithoutNickname, true)).toEqual(['delete']); + expect(verbsFor('email', destinationWithoutNickname)).toEqual(['delete']); }); it('offers only delete for a field a rename statement of the plan already renamed', () => { @@ -92,24 +84,81 @@ describe('dataLossQuestion verbs', () => { destination, renames: [{ verb: 'rename', text: 'User.nickname:User.handle' }], originKnown: true, - keepDataByHand: undefined, }, ); expect(question.verbs).toEqual(['delete']); }); - it('offers only delete where the planner cannot carry out a rename, and says how to keep the data by hand', () => { - const question = questionFor('nickname', destinationWithoutNickname, keepByHand); + it('writes the rename form of a field with its model, as a field rename is written', () => { + expect(questionFor('nickname', destinationWithoutNickname).forms).toEqual({ + rename: 'User.nickname:User.', + }); + }); +}); + +describe('dataLossQuestion for a field whose model the destination names differently', () => { + const app = asNamespaceId('app'); + const profile = contractOf({ app: { models: { Profile: { fields: ['email', 'nickname'] } } } }); + const person = contractOf({ app: { models: { Person: { fields: ['email', 'alias'] } } } }); + const loss = { + operationIndex: 0, + label: 'Remove field nickname from collection profiles', + subject: { kind: 'field', namespaceId: app, model: 'Profile', field: 'nickname' }, + } as const; + + it('offers only delete, and says to rename the model first', () => { + const question = dataLossQuestion(loss, { + origin: profile, + destination: person, + renames: [], + originKnown: true, + }); expect({ verbs: question.verbs, question: question.question }).toEqual({ verbs: ['delete'], question: - 'An operation would lose the values of field "User.nickname". Rename it by hand first.', + 'Remove field nickname from collection profiles would lose the values of field "Profile.nickname". The destination contract has no model "Profile"; to keep the values instead, also pass --rename Profile:, naming the model that stores them now, and the question then offers to rename the field.', }); }); - it('writes the rename form of a field with its model, as a field rename is written', () => { - expect(questionFor('nickname', destinationWithoutNickname, undefined).forms).toEqual({ - rename: 'User.nickname:User.', + it('offers the field rename once the printed model rename is passed, and accepts it', () => { + const question = dataLossQuestion(loss, { + origin: profile, + destination: person, + renames: [{ verb: 'rename', text: 'Profile:Person' }], + originKnown: true, + }); + expect({ + subject: question.subject, + verbs: question.verbs, + forms: question.forms, + accepted: question.validate('rename', 'Person.nickname:Person.alias'), + }).toEqual({ + subject: 'Person.nickname', + verbs: ['rename', 'delete'], + forms: { rename: 'Person.nickname:Person.' }, + accepted: undefined, + }); + }); +}); + +describe('dataLossQuestion with advice from the target', () => { + it('ends the question with how to keep the data, and still offers only delete', () => { + const app = asNamespaceId('app'); + const users = contractOf({ app: { models: { User: { fields: ['email'] } } } }); + const question = dataLossQuestion( + { + operationIndex: 0, + label: 'Drop storage users', + subject: { kind: 'model', namespaceId: app, model: 'User' }, + keepDataAdvice: 'To keep its data, move it by hand.', + }, + { origin: users, destination: users, renames: [], originKnown: true }, + ); + + expect({ question: question.question, verbs: question.verbs }).toEqual({ + question: + 'Drop storage users would lose the data of model "User". To keep its data, move it by hand.', + verbs: ['delete'], }); }); }); diff --git a/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts b/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts index e5df619d2571..3aeb3bcd26a2 100644 --- a/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts +++ b/packages/1-framework/3-tooling/cli/test/control-api/statements/resolve-field-statements.test.ts @@ -274,3 +274,38 @@ describe('resolveStatements, field renames', () => { }); }); }); + +describe('a field whose stored name holds a dot', () => { + it('reads Model.a.b as the field "a.b" when no namespace is named "Model"', () => { + const result = resolve( + ['User.o.d:User.od'], + contractOf({ __unbound__: { models: { User: { fields: ['_id', 'o.d'] } } } }), + contractOf({ __unbound__: { models: { User: { fields: ['_id', 'od'] } } } }), + ); + expect(expectValue(result)).toEqual([ + { + kind: 'rename', + entity: 'field', + from: { namespaceId: '__unbound__', model: 'User', field: 'o.d' }, + to: { namespaceId: '__unbound__', model: 'User', field: 'od' }, + }, + ]); + }); + + it('refuses Model.a.b when it also resolves as namespace.Model.field', () => { + const origin = contractOf({ + a: { models: { b: { fields: ['c'] } } }, + app: { models: { a: { fields: ['b.c'] } } }, + }); + const destination = contractOf({ + a: { models: { b: { fields: ['d'] } } }, + app: { models: { a: { fields: ['b.d'] } } }, + }); + expectFailure( + resolve(['a.b.c:a.b.d'], origin, destination), + UNRESOLVED, + 'namespace "a" model "b" field "c"', + 'model "a" field "b.c"', + ); + }); +}); diff --git a/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts b/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts index 5f0753d6955f..6a51e2add92f 100644 --- a/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts +++ b/packages/1-framework/3-tooling/cli/test/migration-cli.test.ts @@ -24,8 +24,8 @@ const { loadConfigMock, createControlStackMock } = vi.hoisted(() => ({ createControlStackMock: vi.fn(), })); -vi.mock('@internal/config-loader', () => ({ - loadConfigForSections: loadConfigMock, +vi.mock('../src/utils/migration-file-config', () => ({ + loadMigrationFileConfig: loadConfigMock, })); vi.mock('@internal/framework-components/control', async () => { @@ -39,6 +39,7 @@ vi.resetModules(); const { MigrationCLI } = await import('../src/migration-cli'); afterAll(() => { + vi.doUnmock('../src/utils/migration-file-config'); vi.resetModules(); }); @@ -349,7 +350,27 @@ describe('MigrationCLI.run', () => { }); expect(exitCode).toBe(0); - expect(loadConfigMock).toHaveBeenCalledWith('/explicit/config.ts', expect.any(Array)); + expect(loadConfigMock).toHaveBeenCalledWith( + expect.objectContaining({ configPath: '/explicit/config.ts' }), + ); + }); + + it('loads the config that owns the migration file when --config is not given', async () => { + loadConfigMock.mockResolvedValue(ok(okConfig)); + createControlStackMock.mockReturnValue({ adapter: { create: () => ({}) } }); + + await MigrationCLI.run(pathToFileURL(migrationFile).href, FakeMigration, { + argv: entrypointArgv(), + stdout: new BufferStream(), + stderr: new BufferStream(), + }); + + expect(loadConfigMock).toHaveBeenCalledWith({ + configPath: undefined, + migrationFile, + cwd: process.cwd(), + sections: expect.any(Array), + }); }); it('forwards --config= (equals form) to loadConfig', async () => { @@ -365,7 +386,9 @@ describe('MigrationCLI.run', () => { }); expect(exitCode).toBe(0); - expect(loadConfigMock).toHaveBeenCalledWith('/equals/config.ts', expect.any(Array)); + expect(loadConfigMock).toHaveBeenCalledWith( + expect.objectContaining({ configPath: '/equals/config.ts' }), + ); }); it('preserves createdAt from a previously-scaffolded migration.json', async () => { diff --git a/packages/1-framework/3-tooling/cli/test/orm/db-update-consent.test.ts b/packages/1-framework/3-tooling/cli/test/orm/db-update-consent.test.ts index 350daa383e85..f12e10bac12f 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/db-update-consent.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/db-update-consent.test.ts @@ -70,7 +70,6 @@ const contracts = { destination, renames: [], originKnown: true, - keepDataByHand: undefined, }; /** One question of each kind, across two spaces: an extension's recorded drop is a storage subject. */ diff --git a/packages/1-framework/3-tooling/cli/test/orm/migration-new.test.ts b/packages/1-framework/3-tooling/cli/test/orm/migration-new.test.ts index 0dc4c233d8ea..c02fcfdb1a1b 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/migration-new.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/migration-new.test.ts @@ -142,6 +142,23 @@ describe('migration new', () => { ]); }); + it('repeats the --config it was given in the command that runs the stub', async () => { + const project = await createOfflineProject({ storageHash: HASH_TO }); + + const run = await harness(project).run(['migration', 'new', '--config', 'prisma.config.ts'], { + cwd: project.dir, + isTty: { stdout: true }, + }); + const [dirName] = await scaffoldedDirs(project); + const migrationTs = join('migrations', 'app', dirName ?? '', 'migration.ts'); + + expect(run.presented?.presentation.next?.at(-1)).toEqual({ + kind: 'run-command', + label: 'Run it to self-emit ops.json and attest the package', + command: `node "${migrationTs}" --config "prisma.config.ts"`, + }); + }); + it('writes nothing to stdout in human mode', async () => { const project = await createOfflineProject({ storageHash: HASH_TO }); diff --git a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts index 404374aaeab0..05eee6b097cf 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.no-operations.test.ts @@ -191,7 +191,7 @@ describe('migration plan when the planner produces no operations', () => { planningFailed({ kind: 'nothingToCreate', summary: NOTHING_TO_CREATE, - why: 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then run `node migration.ts` in that directory to write `ops.json`.', + why: 'If the database needs something the contract does not describe, run `prisma migration new`, add its operations to the new `migration.ts`, then run that `migration.ts` with `node` to write `ops.json`.', }), ); expect({ exitCode: scaffold.exitCode, migrations: await migrationsOf(project) }).toEqual({ @@ -208,7 +208,7 @@ describe('migration plan when the planner produces no operations', () => { planningFailed({ kind: 'nothingToCreate', summary: NOTHING_TO_CREATE, - why: '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then run `node migration.ts` in that directory to write `ops.json`.', + why: '`prisma migration new` writes a migration only to the emitted contract. If the database needs something this contract does not describe, emit it, run `prisma migration new`, add its operations to the new `migration.ts`, then run that `migration.ts` with `node` to write `ops.json`.', }), ); }); diff --git a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts index 231205172693..eec9d659b749 100644 --- a/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts +++ b/packages/1-framework/3-tooling/cli/test/orm/migration-plan.test.ts @@ -802,10 +802,20 @@ describe('migration plan', () => { expect(run.exitCode).toBe(0); const data = run.presented?.data as { readonly pendingPlaceholders: boolean; + readonly dir: string; + readonly baselineDir: string; + readonly summary: string; readonly operations: readonly { readonly id: string }[]; readonly appliedStatements: readonly { readonly operationIndexes: readonly number[] }[]; }; expect(data.pendingPlaceholders).toBe(true); + expect([data.baselineDir, data.dir]).toEqual([ + expect.stringMatching(/^migrations\/app\/[^/]+$/), + expect.stringMatching(/^migrations\/app\/[^/]+$/), + ]); + expect(data.summary).toBe( + `Planned baseline + migration with placeholder(s) — edit migration.ts, then run \`node "${data.baselineDir}/migration.ts"\` and \`node "${data.dir}/migration.ts"\` to self-emit`, + ); expect( data.appliedStatements.map(({ operationIndexes }) => operationIndexes.map((index) => data.operations[index]?.id), @@ -832,7 +842,38 @@ describe('migration plan', () => { { cwd: project.dir }, ); expect(answered.exitCode).toBe(0); - expect(answered.presented?.data).toMatchObject({ pendingPlaceholders: true }); + const data = answered.presented?.data as { readonly dir: string }; + expect(data.dir).toMatch(/^migrations\/app\/[^/]+$/); + expect(data).toMatchObject({ + pendingPlaceholders: true, + summary: `Planned migration with placeholder(s) — edit migration.ts, then run \`node "${data.dir}/migration.ts"\` to self-emit`, + }); + }); + + it('repeats the --config it was given in the commands that run the stub', async () => { + const project = await losingProject(); + const script: FakePlannerScript = { + operations: [ADDITIVE_OP, DESTRUCTIVE_OP], + operationsAccessorThrows: errorUnfilledPlaceholder('backfill'), + dataLoss: [LEGACY_LOSS], + }; + + const answered = await harness(project, { script }).run( + ['migration', 'plan', '--delete', 'Legacy', '--config', 'prisma.config.ts', '--json'], + { cwd: project.dir }, + ); + const data = answered.presented?.data as { readonly dir: string; readonly summary: string }; + expect({ + summary: data.summary, + next: answered.presented?.presentation.next?.at(-1), + }).toEqual({ + summary: `Planned migration with placeholder(s) — edit migration.ts, then run \`node "${data.dir}/migration.ts" --config "prisma.config.ts"\` to self-emit`, + next: { + kind: 'run-command', + label: 'Run it to self-emit ops.json and attest the package', + command: `node "${data.dir}/migration.ts" --config "prisma.config.ts"`, + }, + }); }); it('asks a question a re-plan brings up in a second batch', async () => { diff --git a/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts b/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts new file mode 100644 index 000000000000..9e4cbc1c26b6 --- /dev/null +++ b/packages/1-framework/3-tooling/cli/test/utils/migration-file-config.test.ts @@ -0,0 +1,174 @@ +import { mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { errorConfigFileNotFound } from '@internal/errors/control'; +import { notOk, ok } from '@internal/utils/result'; +import { join } from 'pathe'; +import { afterAll, afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +const { loadConfigMock, loadConfigForSectionsMock } = vi.hoisted(() => ({ + loadConfigMock: vi.fn(), + loadConfigForSectionsMock: vi.fn(), +})); + +vi.mock('@internal/config-loader', async () => { + const actual = + await vi.importActual('@internal/config-loader'); + return { + ...actual, + loadConfig: loadConfigMock, + loadConfigForSections: loadConfigForSectionsMock, + }; +}); + +vi.resetModules(); +const { loadMigrationFileConfig, ownsMigration } = await import( + '../../src/utils/migration-file-config' +); +const { appMigrationsDirFor } = await import('../../src/orm/migration/paths'); + +afterAll(() => { + vi.doUnmock('@internal/config-loader'); + vi.resetModules(); +}); + +const SECTIONS = ['family', 'target'] as const; + +let root: string; + +beforeEach(() => { + root = realpathSync(mkdtempSync(join(tmpdir(), 'migration-file-config-'))); + loadConfigMock.mockReset(); + loadConfigForSectionsMock.mockReset(); +}); + +afterEach(() => { + rmSync(root, { recursive: true, force: true }); +}); + +function writeConfig(dir: string): string { + mkdirSync(dir, { recursive: true }); + const path = join(dir, 'prisma.config.ts'); + writeFileSync(path, ''); + return path; +} + +function writeMigration(dir: string): string { + mkdirSync(dir, { recursive: true }); + const path = join(dir, 'migration.ts'); + writeFileSync(path, ''); + return path; +} + +/** Each config path loads a config that keeps its migrations in the given directory. */ +function configsKeepingMigrationsIn(dirs: Record): void { + loadConfigMock.mockImplementation(async (configPath: string) => { + const dir = dirs[configPath]; + return dir === undefined + ? notOk(errorConfigFileNotFound(configPath)) + : ok({ config: { configPath, migrations: { dir } }, diagnostics: [] }); + }); +} + +describe('loadMigrationFileConfig', () => { + it('loads the config --config names, without checking that it owns the migration', async () => { + loadConfigForSectionsMock.mockResolvedValue(ok({ named: true })); + + const loaded = await loadMigrationFileConfig({ + configPath: 'custom.config.ts', + migrationFile: join(root, 'elsewhere/migration.ts'), + cwd: root, + sections: SECTIONS, + }); + + expect(loaded.assertOk()).toEqual({ named: true }); + expect(loadConfigForSectionsMock).toHaveBeenCalledWith('custom.config.ts', SECTIONS); + expect(loadConfigMock).not.toHaveBeenCalled(); + }); + + it('uses the nearest config above the migration when its migrations directory holds it', async () => { + const config = writeConfig(root); + const migrationFile = writeMigration(join(root, 'db/migrations/app/0001_initial')); + configsKeepingMigrationsIn({ [config]: join(root, 'db/migrations') }); + + const loaded = await loadMigrationFileConfig({ + configPath: undefined, + migrationFile, + cwd: join(root, 'db'), + sections: SECTIONS, + }); + + expect(loaded.assertOk()).toMatchObject({ configPath: config }); + }); + + it('falls back to the working directory config when the nearest does not own the migration', async () => { + const outer = writeConfig(root); + const project = writeConfig(join(root, 'app')); + const migrationFile = writeMigration(join(root, 'shared-db/migrations/app/0001_initial')); + configsKeepingMigrationsIn({ + [outer]: join(root, 'migrations'), + [project]: join(root, 'shared-db/migrations'), + }); + + const loaded = await loadMigrationFileConfig({ + configPath: undefined, + migrationFile, + cwd: join(root, 'app'), + sections: SECTIONS, + }); + + expect(loaded.assertOk()).toMatchObject({ configPath: project }); + }); + + it('falls back to the working directory config when the nearest fails to load', async () => { + const outer = writeConfig(root); + const project = writeConfig(join(root, 'app')); + const migrationFile = writeMigration(join(root, 'shared-db/migrations/app/0001_initial')); + configsKeepingMigrationsIn({ [project]: join(root, 'shared-db/migrations') }); + + const loaded = await loadMigrationFileConfig({ + configPath: undefined, + migrationFile, + cwd: join(root, 'app'), + sections: SECTIONS, + }); + + expect(loaded.assertOk()).toMatchObject({ configPath: project }); + expect(loadConfigMock).toHaveBeenCalledWith(outer); + }); + + it('refuses, naming each config it checked, when none owns the migration', async () => { + const outer = writeConfig(root); + const migrationDir = join(root, 'inner/migrations/app/0001_initial'); + const migrationFile = writeMigration(migrationDir); + configsKeepingMigrationsIn({ [outer]: join(root, 'migrations') }); + + const loaded = await loadMigrationFileConfig({ + configPath: undefined, + migrationFile, + cwd: migrationDir, + sections: SECTIONS, + }); + + expect(loaded.ok ? loaded.value : loaded.failure.toEnvelope()).toMatchObject({ + code: 'CLI.MIGRATION_CONFIG_NOT_FOUND', + meta: { migrationDir, checked: [outer] }, + }); + }); +}); + +describe('ownsMigration', () => { + it.each([ + ['the default migrations directory', {}], + [ + 'a migrations directory outside the project', + { migrations: { dir: '../shared-db/migrations' } }, + ], + ])('owns a migration the CLI writes under %s', (_label, section) => { + const configPath = join(root, 'app/prisma.config.ts'); + const config = { ...section, baseDir: join(root, 'app') }; + const written = join(appMigrationsDirFor(config), '0001_initial/migration.ts'); + + expect(ownsMigration(config, configPath, written)).toBe(true); + expect(ownsMigration(config, configPath, join(root, 'other/migration.ts'))).toBe(false); + }); +}); diff --git a/packages/1-framework/3-tooling/config-loader/src/exports/index.ts b/packages/1-framework/3-tooling/config-loader/src/exports/index.ts index a45ee4656385..973a41994af3 100644 --- a/packages/1-framework/3-tooling/config-loader/src/exports/index.ts +++ b/packages/1-framework/3-tooling/config-loader/src/exports/index.ts @@ -3,6 +3,7 @@ export { expandContractInputs, globContractInputMatching } from '../expand-contr export type { ConfigFile, ConfigFiles, LoadedConfig } from '../load'; export { buildLoadedConfig, + CONFIG_FILENAME, evaluateConfigModule, findNearestConfigPathForFile, loadConfig, diff --git a/packages/1-framework/3-tooling/config-loader/src/load.ts b/packages/1-framework/3-tooling/config-loader/src/load.ts index 458dd99e8930..0b2ec8abc281 100644 --- a/packages/1-framework/3-tooling/config-loader/src/load.ts +++ b/packages/1-framework/3-tooling/config-loader/src/load.ts @@ -25,7 +25,7 @@ import { validateOrmSectionExcept, } from './orm-section'; -const CONFIG_FILENAME = 'prisma.config.ts'; +export const CONFIG_FILENAME = 'prisma.config.ts'; export type { ConfigSection }; diff --git a/packages/1-framework/3-tooling/migration/src/errors.ts b/packages/1-framework/3-tooling/migration/src/errors.ts index b2201406e4ae..2ceac23dcd84 100644 --- a/packages/1-framework/3-tooling/migration/src/errors.ts +++ b/packages/1-framework/3-tooling/migration/src/errors.ts @@ -16,7 +16,7 @@ import type { MigrationGraph } from './graph'; */ function reemitHint(dir: string, fallback?: string): string { const relativeDir = relative(process.cwd(), dir); - const reemit = `Re-emit the package by running \`node "${relativeDir}/migration.ts"\``; + const reemit = `Re-emit the package by running \`node "${relativeDir}/migration.ts"\`, adding the \`--config \` you gave this command, if any`; return fallback ? `${reemit}, ${fallback}` : `${reemit}.`; } diff --git a/packages/1-framework/3-tooling/migration/test/errors.test.ts b/packages/1-framework/3-tooling/migration/test/errors.test.ts index 9bfee2759e06..ac1e87705f8d 100644 --- a/packages/1-framework/3-tooling/migration/test/errors.test.ts +++ b/packages/1-framework/3-tooling/migration/test/errors.test.ts @@ -1,4 +1,5 @@ import { CliStructuredError } from '@internal/errors/control'; +import { join } from 'pathe'; import { describe, expect, it } from 'vitest'; import { errorBundleNotFoundForGraphNode, @@ -252,6 +253,13 @@ describe('MigrationToolsError base type', () => { expect(MigrationToolsError.is(err)).toBe(true); }); + it('tells the user to repeat --config when re-emitting a package', () => { + const err = errorMissingFile('ops.json', join(process.cwd(), 'db/migrations/app/0001_init')); + expect(err.fix).toBe( + 'Re-emit the package by running `node "db/migrations/app/0001_init/migration.ts"`, adding the `--config ` you gave this command, if any, or delete the directory if the migration is unwanted and the source TypeScript is gone.', + ); + }); + it('exposes toEnvelope() with code, summary, why, fix, nextActions, and meta', () => { const err = errorDirectoryExists('/tmp/m/20260101_init'); expect(err.toEnvelope()).toEqual({ diff --git a/packages/2-mongo-family/4-query/query-ast/src/ddl-commands.ts b/packages/2-mongo-family/4-query/query-ast/src/ddl-commands.ts index 2c39038759df..feac8cfd7d9b 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/ddl-commands.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/ddl-commands.ts @@ -162,6 +162,23 @@ export class DropCollectionCommand extends MongoAstNode { } } +export class RenameCollectionCommand extends MongoAstNode { + readonly kind = 'renameCollection' as const; + readonly collection: string; + readonly to: string; + + constructor(collection: string, to: string) { + super(); + this.collection = collection; + this.to = to; + this.freeze(); + } + + accept(visitor: MongoDdlCommandVisitor): R { + return visitor.renameCollection(this); + } +} + export interface CollModOptions { readonly validator?: Record; readonly validationLevel?: 'strict' | 'moderate'; @@ -198,4 +215,5 @@ export type AnyMongoDdlCommand = | DropIndexCommand | CreateCollectionCommand | DropCollectionCommand + | RenameCollectionCommand | CollModCommand; diff --git a/packages/2-mongo-family/4-query/query-ast/src/ddl-visitors.ts b/packages/2-mongo-family/4-query/query-ast/src/ddl-visitors.ts index a53730f93f14..5beb098b9b78 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/ddl-visitors.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/ddl-visitors.ts @@ -4,6 +4,7 @@ import type { CreateIndexCommand, DropCollectionCommand, DropIndexCommand, + RenameCollectionCommand, } from './ddl-commands'; import type { ListCollectionsCommand, ListIndexesCommand } from './inspection-commands'; @@ -12,6 +13,7 @@ export interface MongoDdlCommandVisitor { dropIndex(command: DropIndexCommand): R; createCollection(command: CreateCollectionCommand): R; dropCollection(command: DropCollectionCommand): R; + renameCollection(command: RenameCollectionCommand): R; collMod(command: CollModCommand): R; } diff --git a/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts b/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts index 3e043827f515..f29e84e2b188 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/exports/control.ts @@ -11,6 +11,7 @@ export { CreateIndexCommand, DropCollectionCommand, DropIndexCommand, + RenameCollectionCommand, } from '../ddl-commands'; export type { MongoDdlCommandVisitor, MongoInspectionCommandVisitor } from '../ddl-visitors'; export type { MongoFilterExpr } from '../filter-expressions'; @@ -27,10 +28,11 @@ export { ListCollectionsCommand, ListIndexesCommand } from '../inspection-comman export { buildIndexOpId, defaultMongoIndexName, keysToKeySpec } from '../migration-helpers'; export type { AnyMongoMigrationOperation, - MongoDataTransformCheck, - MongoDataTransformOperation, + MongoDmlCheck, + MongoDmlOperation, MongoMigrationCheck, MongoMigrationPlanOperation, MongoMigrationStep, } from '../migration-operation-types'; +export { isMongoDmlOperation } from '../migration-operation-types'; export type { MongoFilterVisitor } from '../visitors'; diff --git a/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts b/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts index 485f6116878c..90c368e23903 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/migration-operation-types.ts @@ -25,30 +25,38 @@ export interface MongoMigrationPlanOperation extends MigrationPlanOperation { readonly postcheck: readonly MongoMigrationCheck[]; } -export interface MongoDataTransformCheck { +export interface MongoDmlCheck { readonly description: string; readonly source: MongoQueryPlan; readonly filter: MongoFilterExpr; readonly expect: 'exists' | 'notExists'; } -export interface MongoDataTransformOperation extends MigrationPlanOperation { - readonly operationClass: 'data'; - /** - * Human-readable label for this data transform. - */ +/** + * An operation that runs queries (`run`) where a DDL operation executes commands (`execute`): a + * `dataTransform`, with class `data`, or a document rewrite the planner plans, with the class of + * its effect. + */ +export interface MongoDmlOperation extends MigrationPlanOperation { + readonly operationClass: MigrationOperationClass; readonly name: string; /** - * Optional opt-in routing identity. Presence opts the transform into - * invariant-aware routing; absence means the transform is - * path-dependent and not referenceable from refs. + * Optional opt-in routing identity. Presence opts the operation into + * invariant-aware routing; absence means it is path-dependent and not + * referenceable from refs. */ readonly invariantId?: string; - readonly precheck: readonly MongoDataTransformCheck[]; + readonly precheck: readonly MongoDmlCheck[]; readonly run: readonly MongoQueryPlan[]; - readonly postcheck: readonly MongoDataTransformCheck[]; + readonly postcheck: readonly MongoDmlCheck[]; } -export type AnyMongoMigrationOperation = MongoMigrationPlanOperation | MongoDataTransformOperation; +export type AnyMongoMigrationOperation = MongoMigrationPlanOperation | MongoDmlOperation; + +export function isMongoDmlOperation( + operation: AnyMongoMigrationOperation, +): operation is MongoDmlOperation { + return 'run' in operation; +} export type { MigrationOperationClass, MigrationPlanOperation }; diff --git a/packages/2-mongo-family/4-query/query-ast/src/raw-commands.ts b/packages/2-mongo-family/4-query/query-ast/src/raw-commands.ts index 4d403103f0d7..050ac2139c5f 100644 --- a/packages/2-mongo-family/4-query/query-ast/src/raw-commands.ts +++ b/packages/2-mongo-family/4-query/query-ast/src/raw-commands.ts @@ -55,17 +55,29 @@ export class RawUpdateOneCommand extends MongoAstNode { } } +export interface RawUpdateManyOptions { + readonly bypassDocumentValidation?: boolean; +} + export class RawUpdateManyCommand extends MongoAstNode { readonly kind = 'rawUpdateMany' as const; readonly collection: string; readonly filter: Document; readonly update: Document | ReadonlyArray; + declare readonly bypassDocumentValidation?: boolean; - constructor(collection: string, filter: Document, update: Document | ReadonlyArray) { + constructor( + collection: string, + filter: Document, + update: Document | ReadonlyArray, + options?: RawUpdateManyOptions, + ) { super(); this.collection = collection; this.filter = filter; this.update = update; + if (options?.bypassDocumentValidation !== undefined) + this.bypassDocumentValidation = options.bypassDocumentValidation; this.freeze(); } } diff --git a/packages/2-mongo-family/4-query/query-ast/test/ddl-commands.test.ts b/packages/2-mongo-family/4-query/query-ast/test/ddl-commands.test.ts index 6e15e7108e78..2b75e4234f19 100644 --- a/packages/2-mongo-family/4-query/query-ast/test/ddl-commands.test.ts +++ b/packages/2-mongo-family/4-query/query-ast/test/ddl-commands.test.ts @@ -5,6 +5,7 @@ import { CreateIndexCommand, DropCollectionCommand, DropIndexCommand, + RenameCollectionCommand, } from '../src/ddl-commands'; import type { MongoDdlCommandVisitor, MongoInspectionCommandVisitor } from '../src/ddl-visitors'; import { ListCollectionsCommand, ListIndexesCommand } from '../src/inspection-commands'; @@ -17,6 +18,7 @@ function makeDdlVisitor( dropIndex: () => 'dropIndex', createCollection: () => 'createCollection', dropCollection: () => 'dropCollection', + renameCollection: () => 'renameCollection', collMod: () => 'collMod', ...overrides, }; @@ -252,6 +254,29 @@ describe('DropCollectionCommand', () => { }); }); +describe('RenameCollectionCommand', () => { + it('constructs with the source collection and the new name', () => { + const cmd = new RenameCollectionCommand('profiles', 'users'); + expect({ kind: cmd.kind, collection: cmd.collection, to: cmd.to }).toEqual({ + kind: 'renameCollection', + collection: 'profiles', + to: 'users', + }); + }); + + it('is frozen', () => { + expect(Object.isFrozen(new RenameCollectionCommand('profiles', 'users'))).toBe(true); + }); + + it('dispatches via DDL visitor', () => { + const cmd = new RenameCollectionCommand('profiles', 'users'); + const visitor = makeDdlVisitor({ + renameCollection: (c) => `rename:${c.collection}->${c.to}`, + }); + expect(cmd.accept(visitor)).toBe('rename:profiles->users'); + }); +}); + describe('CollModCommand', () => { it('constructs with validator', () => { const cmd = new CollModCommand('users', { diff --git a/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts b/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts index 2562d5283488..329bfaf6aa7c 100644 --- a/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts +++ b/packages/2-mongo-family/4-query/query-ast/test/migration-helpers.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from 'vitest'; import { buildIndexOpId, defaultMongoIndexName, keysToKeySpec } from '../src/migration-helpers'; +import { isMongoDmlOperation } from '../src/migration-operation-types'; describe('buildIndexOpId', () => { it('produces deterministic create ID for single key', () => { @@ -67,3 +68,19 @@ describe('keysToKeySpec', () => { expect(keysToKeySpec([{ field: 'content', direction: 'text' }])).toEqual({ content: 'text' }); }); }); + +describe('isMongoDmlOperation', () => { + it('tells an operation that runs queries from one that executes DDL, whatever its class', () => { + const common = { id: 'op', label: 'op', precheck: [], postcheck: [] }; + expect([ + isMongoDmlOperation({ ...common, operationClass: 'widening', execute: [] }), + isMongoDmlOperation({ + ...common, + operationClass: 'widening', + name: 'rewrite', + run: [], + }), + isMongoDmlOperation({ ...common, operationClass: 'data', name: 'data', run: [] }), + ]).toEqual([false, true, true]); + }); +}); diff --git a/packages/2-mongo-family/4-query/query-ast/test/raw-commands.test.ts b/packages/2-mongo-family/4-query/query-ast/test/raw-commands.test.ts index 7859f62c3bf9..e912739b368f 100644 --- a/packages/2-mongo-family/4-query/query-ast/test/raw-commands.test.ts +++ b/packages/2-mongo-family/4-query/query-ast/test/raw-commands.test.ts @@ -74,6 +74,17 @@ describe('RawUpdateOneCommand', () => { expect(cmd.update).toEqual(update); }); + it('carries bypassDocumentValidation only when given', () => { + const plain = new RawUpdateManyCommand('users', filter, update); + const bypassing = new RawUpdateManyCommand('users', filter, update, { + bypassDocumentValidation: true, + }); + expect({ + plainHasKey: 'bypassDocumentValidation' in plain, + bypassing: bypassing.bypassDocumentValidation, + }).toEqual({ plainHasKey: false, bypassing: true }); + }); + it('accepts pipeline-style update (array)', () => { const pipelineUpdate = [{ $set: { fullName: { $concat: ['$first', ' ', '$last'] } } }]; const cmd = new RawUpdateOneCommand('users', filter, pipelineUpdate); diff --git a/packages/2-mongo-family/6-transport/mongo-lowering/src/adapter-types.ts b/packages/2-mongo-family/6-transport/mongo-lowering/src/adapter-types.ts index afc676a7e306..8a9bb7a6a917 100644 --- a/packages/2-mongo-family/6-transport/mongo-lowering/src/adapter-types.ts +++ b/packages/2-mongo-family/6-transport/mongo-lowering/src/adapter-types.ts @@ -94,6 +94,7 @@ export type MongoLoweredDraft = readonly collection: string; readonly filter: Record; readonly update: Record | ReadonlyArray>; + readonly bypassDocumentValidation: boolean; } | { readonly kind: 'rawDeleteOne'; diff --git a/packages/2-mongo-family/6-transport/mongo-wire/src/exports/index.ts b/packages/2-mongo-family/6-transport/mongo-wire/src/exports/index.ts index 9290c92a15e2..698515eacaad 100644 --- a/packages/2-mongo-family/6-transport/mongo-wire/src/exports/index.ts +++ b/packages/2-mongo-family/6-transport/mongo-wire/src/exports/index.ts @@ -25,6 +25,7 @@ export { InsertManyWireCommand, InsertOneWireCommand, isDdlWireCommand, + RenameCollectionWireCommand, UpdateManyWireCommand, UpdateOneWireCommand, } from '../wire-commands'; diff --git a/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts b/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts index 36ffe626bf7d..bc70571d2a2c 100644 --- a/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts +++ b/packages/2-mongo-family/6-transport/mongo-wire/src/wire-commands.ts @@ -82,17 +82,20 @@ export class UpdateManyWireCommand extends MongoWireCommand { readonly filter: Document; readonly update: Document | ReadonlyArray; readonly upsert: boolean; + readonly bypassDocumentValidation: boolean; constructor( collection: string, filter: Document, update: Document | ReadonlyArray, upsert = false, + bypassDocumentValidation = false, ) { super(collection); this.filter = filter; this.update = update; this.upsert = upsert; + this.bypassDocumentValidation = bypassDocumentValidation; this.freeze(); } } @@ -250,6 +253,17 @@ export class DropCollectionWireCommand extends MongoWireCommand { } } +export class RenameCollectionWireCommand extends MongoWireCommand { + readonly kind = 'renameCollection' as const; + readonly to: string; + + constructor(collection: string, to: string) { + super(collection); + this.to = to; + this.freeze(); + } +} + export class DropIndexWireCommand extends MongoWireCommand { readonly kind = 'dropIndex' as const; readonly name: string; @@ -294,6 +308,7 @@ export type AnyMongoDdlWireCommand = | CreateCollectionWireCommand | CreateIndexWireCommand | DropCollectionWireCommand + | RenameCollectionWireCommand | DropIndexWireCommand | CollModWireCommand; @@ -303,6 +318,7 @@ const DDL_KINDS: ReadonlySet = new Set([ 'createCollection', 'createIndex', 'dropCollection', + 'renameCollection', 'dropIndex', 'collMod', ]); diff --git a/packages/2-mongo-family/7-runtime/src/param-ref-mutator.ts b/packages/2-mongo-family/7-runtime/src/param-ref-mutator.ts index 9c84ef500826..468cc648ac69 100644 --- a/packages/2-mongo-family/7-runtime/src/param-ref-mutator.ts +++ b/packages/2-mongo-family/7-runtime/src/param-ref-mutator.ts @@ -285,6 +285,7 @@ function buildMutatedDraft( collection: draft.collection, filter: substituteDoc(draft.filter, overrides), update: substituteUpdate(draft.update, overrides), + bypassDocumentValidation: draft.bypassDocumentValidation, }; case 'deleteOne': return { diff --git a/packages/2-mongo-family/7-runtime/test/param-ref-mutator.test.ts b/packages/2-mongo-family/7-runtime/test/param-ref-mutator.test.ts index acd9b3ef381e..2d6686001db0 100644 --- a/packages/2-mongo-family/7-runtime/test/param-ref-mutator.test.ts +++ b/packages/2-mongo-family/7-runtime/test/param-ref-mutator.test.ts @@ -158,6 +158,7 @@ describe('flattenMongoParamRefs', () => { collection: 'users', filter: { id: new MongoParamRef(1) }, update: { $set: { v: new MongoParamRef(2) } }, + bypassDocumentValidation: false, } satisfies MongoLoweredDraft, expectedLength: 2, }, @@ -647,6 +648,7 @@ describe('createMongoParamRefMutator', () => { collection: 'users', filter: { id: new MongoParamRef(1) }, update: { $set: { v: new MongoParamRef(0) } }, + bypassDocumentValidation: false, } satisfies MongoLoweredDraft, read: (updated: MongoLoweredDraft) => { if (updated.kind !== 'rawUpdateMany') throw new Error('wrong kind'); diff --git a/packages/2-mongo-family/9-family/README.md b/packages/2-mongo-family/9-family/README.md index cae5f029e46b..70d34744ee34 100644 --- a/packages/2-mongo-family/9-family/README.md +++ b/packages/2-mongo-family/9-family/README.md @@ -130,7 +130,7 @@ export default AddUsersCollection; MigrationCLI.run(import.meta.url, AddUsersCollection); ``` -Run `node migration.ts` to produce `ops.json` and `migration.json`. Use `--dry-run` to preview without writing. +Run `node /migration.ts` from any directory to produce `ops.json` and `migration.json`. Use `--dry-run` to preview without writing. ## Package Structure diff --git a/packages/2-mongo-family/9-family/src/core/field-rewrite.ts b/packages/2-mongo-family/9-family/src/core/field-rewrite.ts new file mode 100644 index 000000000000..a831f56dfb51 --- /dev/null +++ b/packages/2-mongo-family/9-family/src/core/field-rewrite.ts @@ -0,0 +1,38 @@ +import type { MigrationPlanOperation } from '@internal/framework-components/control'; +import { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; + +/** A document rewrite the planner plans: `renameField` or `unsetField`. */ +export interface FieldRewrite { + readonly collection: string; + /** The field the rewrite renames or removes, by its old name. */ + readonly field: string; + readonly command: RawUpdateManyCommand; +} + +const FIELD_OPERATORS: ReadonlySet = new Set(['$rename', '$unset']); + +function onlyKey(value: unknown): string | undefined { + if (typeof value !== 'object' || value === null || Array.isArray(value)) return undefined; + const keys = Object.keys(value); + const [key] = keys; + return keys.length === 1 ? key : undefined; +} + +/** + * The planner's field rewrite an operation is, if it is one: a single `updateMany` that skips + * validation and renames or unsets one field. + */ +export function fieldRewriteOf(operation: MigrationPlanOperation): FieldRewrite | undefined { + const run: unknown = Reflect.get(operation, 'run'); + if (!Array.isArray(run) || run.length !== 1) return undefined; + const [step]: readonly unknown[] = run; + const command: unknown = + typeof step === 'object' && step !== null ? Reflect.get(step, 'command') : undefined; + if (!(command instanceof RawUpdateManyCommand) || command.bypassDocumentValidation !== true) { + return undefined; + } + const operator = onlyKey(command.update); + if (operator === undefined || !FIELD_OPERATORS.has(operator)) return undefined; + const field = onlyKey(Reflect.get(command.update, operator)); + return field === undefined ? undefined : { collection: command.collection, field, command }; +} diff --git a/packages/2-mongo-family/9-family/src/core/mongo-migration.ts b/packages/2-mongo-family/9-family/src/core/mongo-migration.ts index f774663b0b4d..fa7ed2aaa0f9 100644 --- a/packages/2-mongo-family/9-family/src/core/mongo-migration.ts +++ b/packages/2-mongo-family/9-family/src/core/mongo-migration.ts @@ -12,9 +12,9 @@ import { MongoContractView } from './ir/mongo-contract-view'; * the abstract `targetId` member from `Migration`. * * The operation type parameter is `AnyMongoMigrationOperation` — the union - * of DDL-shaped `MongoMigrationPlanOperation` and `MongoDataTransformOperation` — + * of DDL-shaped `MongoMigrationPlanOperation` and `MongoDmlOperation` — * so subclasses can return a mix of schema operations (e.g. `createIndex`, - * `setValidation`) and data-transform operations (e.g. `dataTransform`). + * `setValidation`) and DML operations (e.g. `dataTransform`, `renameField`). * Mirrors the generic parameter used by `PlannerProducedMongoMigration`. * * Binds the framework base's `Start` / `End` contract generics so a subclass diff --git a/packages/2-mongo-family/9-family/src/core/operation-preview.ts b/packages/2-mongo-family/9-family/src/core/operation-preview.ts index 0303816b0853..fbd3ee5b508b 100644 --- a/packages/2-mongo-family/9-family/src/core/operation-preview.ts +++ b/packages/2-mongo-family/9-family/src/core/operation-preview.ts @@ -3,6 +3,7 @@ import type { OperationPreview, } from '@internal/framework-components/control'; import type { + AnyMongoDdlCommand, CollModCommand, CreateCollectionCommand, CreateIndexCommand, @@ -10,7 +11,10 @@ import type { DropIndexCommand, MongoDdlCommandVisitor, MongoIndexKey, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; +import type { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; +import { fieldRewriteOf } from './field-rewrite'; function formatKeySpec(keys: ReadonlyArray): string { const entries = keys.map((k) => `${JSON.stringify(k.field)}: ${JSON.stringify(k.direction)}`); @@ -35,17 +39,21 @@ function formatCreateCollectionOptions(cmd: CreateCollectionCommand): string | u return formatOptionEntries(cmd); } +function collectionRef(name: string): string { + return `db.getCollection(${JSON.stringify(name)})`; +} + class MongoDdlCommandFormatter implements MongoDdlCommandVisitor { createIndex(cmd: CreateIndexCommand): string { const keySpec = formatKeySpec(cmd.keys); const opts = formatOptions(cmd); return opts - ? `db.${cmd.collection}.createIndex(${keySpec}, ${opts})` - : `db.${cmd.collection}.createIndex(${keySpec})`; + ? `${collectionRef(cmd.collection)}.createIndex(${keySpec}, ${opts})` + : `${collectionRef(cmd.collection)}.createIndex(${keySpec})`; } dropIndex(cmd: DropIndexCommand): string { - return `db.${cmd.collection}.dropIndex(${JSON.stringify(cmd.name)})`; + return `${collectionRef(cmd.collection)}.dropIndex(${JSON.stringify(cmd.name)})`; } createCollection(cmd: CreateCollectionCommand): string { @@ -56,7 +64,11 @@ class MongoDdlCommandFormatter implements MongoDdlCommandVisitor { } dropCollection(cmd: DropCollectionCommand): string { - return `db.${cmd.collection}.drop()`; + return `${collectionRef(cmd.collection)}.drop()`; + } + + renameCollection(cmd: RenameCollectionCommand): string { + return `${collectionRef(cmd.collection)}.renameCollection(${JSON.stringify(cmd.to)})`; } collMod(cmd: CollModCommand): string { @@ -71,22 +83,36 @@ class MongoDdlCommandFormatter implements MongoDdlCommandVisitor { const formatter = new MongoDdlCommandFormatter(); -interface MongoExecuteStep { - readonly command: { readonly accept: (visitor: MongoDdlCommandVisitor) => R }; +function formatUpdateMany(cmd: RawUpdateManyCommand): string { + const args = [JSON.stringify(cmd.filter), JSON.stringify(cmd.update)]; + if (cmd.bypassDocumentValidation) args.push('{ bypassDocumentValidation: true }'); + return `${collectionRef(cmd.collection)}.updateMany(${args.join(', ')})`; +} + +function arrayAt(value: object, key: string): readonly unknown[] { + const entry: unknown = Reflect.get(value, key); + return Array.isArray(entry) ? entry : []; +} + +function objectAt(value: unknown, key: string): object | undefined { + if (typeof value !== 'object' || value === null) return undefined; + const entry: unknown = Reflect.get(value, key); + return typeof entry === 'object' && entry !== null ? entry : undefined; +} + +function isDdlCommand(value: object | undefined): value is AnyMongoDdlCommand { + return value !== undefined && typeof Reflect.get(value, 'accept') === 'function'; } export function formatMongoOperations(operations: readonly MigrationPlanOperation[]): string[] { const statements: string[] = []; for (const operation of operations) { - const candidate = operation as unknown as Record; - if (!('execute' in candidate) || !Array.isArray(candidate['execute'])) { - continue; - } - for (const step of candidate['execute'] as MongoExecuteStep[]) { - if (step.command && typeof step.command.accept === 'function') { - statements.push(step.command.accept(formatter)); - } + for (const step of arrayAt(operation, 'execute')) { + const command = objectAt(step, 'command'); + if (isDdlCommand(command)) statements.push(command.accept(formatter)); } + const rewrite = fieldRewriteOf(operation); + if (rewrite !== undefined) statements.push(formatUpdateMany(rewrite.command)); } return statements; } diff --git a/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts b/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts index 2c2157a04e07..36025cbd9121 100644 --- a/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts +++ b/packages/2-mongo-family/9-family/src/core/operation-storage-name.ts @@ -1,4 +1,5 @@ import type { MigrationPlanOperation } from '@internal/framework-components/control'; +import { fieldRewriteOf } from './field-rewrite'; function collectionOfStep(step: unknown): string | undefined { if (typeof step !== 'object' || step === null) return undefined; @@ -10,8 +11,8 @@ function collectionOfStep(step: unknown): string | undefined { /** * The collection an operation's command names, as the database knows it: the name the Mongo - * planner gives a dropped collection. An operation whose commands name no collection is named by - * its id. + * planner gives a dropped collection. A planner field rewrite is named `.`. An operation + * whose commands name no collection is named by its id. */ export function mongoStorageNameOf(operation: MigrationPlanOperation): string { const execute: unknown = Reflect.get(operation, 'execute'); @@ -20,5 +21,12 @@ export function mongoStorageNameOf(operation: MigrationPlanOperation): string { const collection = collectionOfStep(step); if (collection !== undefined) return collection; } + const rewrite = fieldRewriteOf(operation); + if (rewrite !== undefined) return `${rewrite.collection}.${rewrite.field}`; + const run: unknown = Reflect.get(operation, 'run'); + for (const step of Array.isArray(run) ? run : []) { + const collection = collectionOfStep(step); + if (collection !== undefined) return collection; + } return operation.id; } diff --git a/packages/2-mongo-family/9-family/src/exports/control.ts b/packages/2-mongo-family/9-family/src/exports/control.ts index 8386ea3617a7..9943492436af 100644 --- a/packages/2-mongo-family/9-family/src/exports/control.ts +++ b/packages/2-mongo-family/9-family/src/exports/control.ts @@ -6,6 +6,7 @@ export { } from '../core/control-instance'; export type { MongoControlTargetDescriptor } from '../core/control-target-descriptor'; export type { MongoControlExtensionDescriptor } from '../core/control-types'; +export { type FieldRewrite, fieldRewriteOf } from '../core/field-rewrite'; export { formatMongoOperations, mongoOperationsToPreview, diff --git a/packages/2-mongo-family/9-family/test/field-rewrite.test.ts b/packages/2-mongo-family/9-family/test/field-rewrite.test.ts new file mode 100644 index 000000000000..2a9d3aa22662 --- /dev/null +++ b/packages/2-mongo-family/9-family/test/field-rewrite.test.ts @@ -0,0 +1,40 @@ +import type { MigrationPlanOperation } from '@internal/framework-components/control'; +import { RawInsertOneCommand, RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; +import { describe, expect, it } from 'vitest'; +import { fieldRewriteOf } from '../src/core/field-rewrite'; + +const meta = { target: 'mongo', storageHash: '', lane: 'mongo-raw' }; + +function dmlOperation(commands: readonly object[]): MigrationPlanOperation { + const run = commands.map((command) => ({ collection: 'users', command, meta })); + return { id: 'op', label: 'op', operationClass: 'destructive', run } as MigrationPlanOperation; +} + +const bypass = { bypassDocumentValidation: true }; + +describe('fieldRewriteOf', () => { + it('reads the collection and field of the planner rewrites', () => { + const unset = new RawUpdateManyCommand('users', {}, { $unset: { nickname: '' } }, bypass); + const rename = new RawUpdateManyCommand('users', {}, { $rename: { name: 'fullName' } }, bypass); + expect([fieldRewriteOf(dmlOperation([unset])), fieldRewriteOf(dmlOperation([rename]))]).toEqual( + [ + { collection: 'users', field: 'nickname', command: unset }, + { collection: 'users', field: 'name', command: rename }, + ], + ); + }); + + it('does not read a hand-written transform that does not skip validation, or that does more', () => { + const unset = new RawUpdateManyCommand('users', {}, { $unset: { nickname: '' } }); + const set = new RawUpdateManyCommand('users', {}, { $set: { status: 'active' } }, bypass); + const two = new RawUpdateManyCommand('users', {}, { $unset: { a: '', b: '' } }, bypass); + const rewrite = new RawUpdateManyCommand('users', {}, { $unset: { nickname: '' } }, bypass); + expect([ + fieldRewriteOf(dmlOperation([unset])), + fieldRewriteOf(dmlOperation([set])), + fieldRewriteOf(dmlOperation([two])), + fieldRewriteOf(dmlOperation([rewrite, new RawInsertOneCommand('users', { a: 1 })])), + fieldRewriteOf({ id: 'ddl', label: 'ddl', operationClass: 'additive' }), + ]).toEqual([undefined, undefined, undefined, undefined, undefined]); + }); +}); diff --git a/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts b/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts index 3ea37b482383..8445e4e0159a 100644 --- a/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts +++ b/packages/2-mongo-family/9-family/test/operation-preview-format.test.ts @@ -5,7 +5,9 @@ import { DropCollectionCommand, DropIndexCommand, type MongoMigrationPlanOperation, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; +import { RawInsertOneCommand, RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; import { describe, expect, it } from 'vitest'; import { formatMongoOperations } from '../src/core/operation-preview'; @@ -29,7 +31,7 @@ describe('formatMongoOperations', () => { }; const result = formatMongoOperations([op]); expect(result).toEqual([ - 'db.users.createIndex({ "email": 1 }, { unique: true, name: "email_1" })', + 'db.getCollection("users").createIndex({ "email": 1 }, { unique: true, name: "email_1" })', ]); }); @@ -48,7 +50,7 @@ describe('formatMongoOperations', () => { postcheck: [], }; const result = formatMongoOperations([op]); - expect(result).toEqual(['db.posts.createIndex({ "title": 1 })']); + expect(result).toEqual(['db.getCollection("posts").createIndex({ "title": 1 })']); }); it('formats dropIndex', () => { @@ -66,7 +68,7 @@ describe('formatMongoOperations', () => { postcheck: [], }; const result = formatMongoOperations([op]); - expect(result).toEqual(['db.users.dropIndex("email_1")']); + expect(result).toEqual(['db.getCollection("users").dropIndex("email_1")']); }); it('formats compound index', () => { @@ -92,7 +94,7 @@ describe('formatMongoOperations', () => { }; const result = formatMongoOperations([op]); expect(result).toEqual([ - 'db.users.createIndex({ "email": 1, "tenantId": -1 }, { unique: true })', + 'db.getCollection("users").createIndex({ "email": 1, "tenantId": -1 }, { unique: true })', ]); }); @@ -131,6 +133,80 @@ describe('formatMongoOperations', () => { expect(result[1]).toContain('dropIndex'); }); + it('writes every collection name as an escaped string', () => { + const step = (command: MongoMigrationPlanOperation['execute'][number]['command']) => ({ + id: 'test', + label: 'test', + operationClass: 'widening' as const, + precheck: [], + execute: [{ description: 'step', command }], + postcheck: [], + }); + expect( + formatMongoOperations([ + step(new DropCollectionCommand('my-coll')), + step(new DropIndexCommand('a"b', 'x_1')), + step(new CreateIndexCommand('my-coll', [{ field: 'x', direction: 1 }])), + step(new RenameCollectionCommand('my-coll', 'a"b')), + ]), + ).toEqual([ + 'db.getCollection("my-coll").drop()', + 'db.getCollection("a\\"b").dropIndex("x_1")', + 'db.getCollection("my-coll").createIndex({ "x": 1 })', + 'db.getCollection("my-coll").renameCollection("a\\"b")', + ]); + }); + + it('formats a document rewrite as updateMany', () => { + const plan = { + collection: 'my-coll', + command: new RawUpdateManyCommand( + 'my-coll', + { name: { $exists: true } }, + { $rename: { name: 'fullName' } }, + { bypassDocumentValidation: true }, + ), + meta: { target: 'mongo', storageHash: '', lane: 'mongo-raw' }, + }; + const rewrite = { + id: 'field.my-coll.name.rename(fullName)', + label: 'Rename field', + operationClass: 'widening' as const, + name: 'rename', + precheck: [], + run: [plan], + postcheck: [], + }; + expect(formatMongoOperations([rewrite])).toEqual([ + 'db.getCollection("my-coll").updateMany({"name":{"$exists":true}}, {"$rename":{"name":"fullName"}}, { bypassDocumentValidation: true })', + ]); + }); + + it('renders nothing for a user data transform, even one with an updateMany step', () => { + const meta = { target: 'mongo', storageHash: 'h', lane: 'mongo-raw' }; + const transform = { + id: 'data_transform.backfill', + label: 'Data transform: backfill', + operationClass: 'data' as const, + name: 'backfill', + precheck: [], + run: [ + { + collection: 'users', + command: new RawUpdateManyCommand('users', {}, { $set: { status: 'active' } }), + meta, + }, + { + collection: 'users', + command: new RawInsertOneCommand('users', { name: 'seed' }), + meta, + }, + ], + postcheck: [], + }; + expect(formatMongoOperations([transform])).toEqual([]); + }); + it('skips operations without execute steps', () => { const ops = [{ id: 'test', label: 'test', operationClass: 'additive' as const }]; const result = formatMongoOperations(ops); @@ -156,7 +232,7 @@ describe('formatMongoOperations', () => { }; const result = formatMongoOperations([op]); expect(result).toEqual([ - 'db.sessions.createIndex({ "expiresAt": 1 }, { sparse: true, expireAfterSeconds: 3600 })', + 'db.getCollection("sessions").createIndex({ "expiresAt": 1 }, { sparse: true, expireAfterSeconds: 3600 })', ]); }); @@ -243,7 +319,26 @@ describe('formatMongoOperations', () => { postcheck: [], }; const result = formatMongoOperations([op]); - expect(result).toEqual(['db.events.drop()']); + expect(result).toEqual(['db.getCollection("events").drop()']); + }); + + it('formats renameCollection', () => { + const op: MongoMigrationPlanOperation = { + id: 'collection.profiles.rename', + label: 'Rename collection', + operationClass: 'widening', + precheck: [], + execute: [ + { + description: 'rename collection', + command: new RenameCollectionCommand('profiles', 'users'), + }, + ], + postcheck: [], + }; + expect(formatMongoOperations([op])).toEqual([ + 'db.getCollection("profiles").renameCollection("users")', + ]); }); it('formats collMod with validator', () => { diff --git a/packages/2-mongo-family/9-family/test/operation-preview.test.ts b/packages/2-mongo-family/9-family/test/operation-preview.test.ts index c94afb8b645e..0abeccf537b5 100644 --- a/packages/2-mongo-family/9-family/test/operation-preview.test.ts +++ b/packages/2-mongo-family/9-family/test/operation-preview.test.ts @@ -39,7 +39,7 @@ describe('MongoControlFamilyInstance OperationPreviewCapable', () => { const preview = instantiate().toOperationPreview([op]); expect(preview.statements).toEqual([ { - text: 'db.users.createIndex({ "email": 1 }, { unique: true, name: "email_1" })', + text: 'db.getCollection("users").createIndex({ "email": 1 }, { unique: true, name: "email_1" })', language: 'mongodb-shell', }, ]); @@ -61,7 +61,7 @@ describe('MongoControlFamilyInstance OperationPreviewCapable', () => { }; const preview = instantiate().toOperationPreview([op]); expect(preview.statements).toEqual([ - { text: 'db.users.dropIndex("email_1")', language: 'mongodb-shell' }, + { text: 'db.getCollection("users").dropIndex("email_1")', language: 'mongodb-shell' }, ]); }); diff --git a/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts b/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts index d92698fbea8b..f8fe6665c110 100644 --- a/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts +++ b/packages/2-mongo-family/9-family/test/operation-storage-name.test.ts @@ -1,5 +1,6 @@ import type { MigrationPlanOperation } from '@internal/framework-components/control'; import { DropCollectionCommand } from '@internal/mongo-query-ast/control'; +import { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; import { describe, expect, it } from 'vitest'; import { mongoStorageNameOf } from '../src/core/operation-storage-name'; @@ -22,6 +23,26 @@ describe('mongoStorageNameOf', () => { ]).toEqual(['events', 'logs']); }); + it('names a field rewrite by its collection and the field it removes or renames', () => { + const rewrite = (update: Record): MigrationPlanOperation => { + const command = new RawUpdateManyCommand('users', {}, update, { + bypassDocumentValidation: true, + }); + const run = [{ collection: 'users', command, meta: {} }]; + return { + id: 'rewrite', + label: 'rewrite', + operationClass: 'destructive', + run, + } as MigrationPlanOperation; + }; + expect([ + mongoStorageNameOf(rewrite({ $unset: { nickname: '' } })), + mongoStorageNameOf(rewrite({ $rename: { name: 'fullName' } })), + mongoStorageNameOf(rewrite({ $set: { status: 'active' } })), + ]).toEqual(['users.nickname', 'users.name', 'users']); + }); + it('names an operation whose commands name no collection by its id, never its label', () => { expect(mongoStorageNameOf(operation([]))).toBe('collection.events.drop'); }); diff --git a/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts b/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts index e6c77bcc9696..86d3025604d7 100644 --- a/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts +++ b/packages/2-sql/9-family/src/core/migrations/operation-subjects.ts @@ -1,7 +1,7 @@ import { asNamespaceId, type ContractWithDomain } from '@internal/contract/types'; import type { MigrationAccessChange, - MigrationOperationSubject, + MigrationDataLoss, MigrationPlanSubjects, MigrationSubject, ModelCoordinate, @@ -13,6 +13,7 @@ import { fieldRenameStorageEffect, isSqlModelStorage, modelRenameStorageEffect, + type TableRenameByHand, } from './statement-planning'; /** @@ -78,6 +79,7 @@ interface SubjectContext { readonly contract: ContractWithDomain; /** The rename statements the plan applied, whose renames a target names the new way. */ readonly statements: readonly ResolvedMigrationStatement[]; + readonly byHand: TableRenameByHand; } function originTable( @@ -165,6 +167,29 @@ function subjectOf(target: SubjectStorage, context: SubjectContext): MigrationSu return storage; } +/** + * How to keep the rows of a dropped table whose model the destination keeps under the same name in + * another table of the same namespace: no statement can rename that table, so the user renames it + * by hand. None when the origin contract already stores a model in that table. + */ +function keepTableAdvice( + target: SubjectStorage, + subject: MigrationSubject, + context: SubjectContext, +): string | undefined { + if (subject.kind !== 'model' || target.table === undefined || context.fromContract === null) { + return undefined; + } + const storage = + context.contract.domain.namespaces[subject.namespaceId]?.models[subject.model]?.storage; + if (storage === undefined || !isSqlModelStorage(storage)) return undefined; + const { namespaceId, table: from } = target.table; + if (storage.namespaceId !== namespaceId || storage.table === from) return undefined; + if (modelsStoring(context.fromContract, namespaceId, storage.table).length > 0) return undefined; + const rename = { namespaceId, from, to: storage.table }; + return `To keep its rows, rename table "${from}" to "${storage.table}" instead; a statement cannot rename the table of a model that keeps its name. With db update, rename the table in the database yourself with ${context.byHand.tableRenameByHand(rename).join('; ')}, then run prisma db update. With migration plan, write the rename as a migration of its own instead: run prisma migration new --name --from , add ${context.byHand.renderTableRename(rename)} to the operations of its migration.ts, run node on that migration.ts so it writes ops.json, then run prisma db migrate.`; +} + /** * The subject of each operation that loses data and each that widens access, at its position in * the plan. A subject is a model or a field of the origin contract when one stores it, found under @@ -174,12 +199,18 @@ export function subjectsOfCalls( calls: readonly CallSubjects[], context: SubjectContext, ): MigrationPlanSubjects { - const dataLoss: MigrationOperationSubject[] = []; + const dataLoss: MigrationDataLoss[] = []; const accessWidening: MigrationAccessChange[] = []; let operationIndex = 0; for (const call of calls) { for (const target of call.dataLoss) { - dataLoss.push({ operationIndex, subject: subjectOf(target, context) }); + const subject = subjectOf(target, context); + const keepDataAdvice = keepTableAdvice(target, subject, context); + dataLoss.push( + keepDataAdvice === undefined + ? { operationIndex, subject } + : { operationIndex, subject, keepDataAdvice }, + ); } for (const target of call.accessWidening) { accessWidening.push({ diff --git a/packages/2-sql/9-family/src/core/migrations/statement-planning.ts b/packages/2-sql/9-family/src/core/migrations/statement-planning.ts index 5ca1f6a0b836..4148b868af89 100644 --- a/packages/2-sql/9-family/src/core/migrations/statement-planning.ts +++ b/packages/2-sql/9-family/src/core/migrations/statement-planning.ts @@ -156,8 +156,17 @@ export interface CallWithCompanions extends SingleOperationCall { readonly companions: readonly SingleOperationCall[]; } +/** How a user renames a table by hand, written for the target. */ +export interface TableRenameByHand { + /** The `renameTable` call a user writes in `migration.ts` to make `rename` by hand. */ + renderTableRename(rename: TableRename): string; + /** The SQL statements a user runs against the database to make `rename` by hand. */ + tableRenameByHand(rename: TableRename): readonly string[]; +} + /** What a target supplies for planning statements against its working schema. */ -export interface StatementPlanningTarget { +export interface StatementPlanningTarget + extends TableRenameByHand { /** The tables of the working schema as earlier statements have left it. */ tables(): SchemaTables; /** The call that renames a table, with its companions, computed against the working schema. */ @@ -166,10 +175,6 @@ export interface StatementPlanningTarget { renameColumnCall(rename: ColumnRename): TCall; /** Applies a call to the working schema. */ apply(call: TCall): void; - /** The `renameTable` call a user writes in `migration.ts` to make `rename` by hand. */ - renderTableRename(rename: TableRename): string; - /** The SQL statements a user runs against the database to make `rename` by hand. */ - tableRenameByHand(rename: TableRename): readonly string[]; } function operationsOf(call: CallWithCompanions): readonly SingleOperationCall[] { diff --git a/packages/2-sql/9-family/src/exports/control.ts b/packages/2-sql/9-family/src/exports/control.ts index 97e9729bd33d..10f3a275bed7 100644 --- a/packages/2-sql/9-family/src/exports/control.ts +++ b/packages/2-sql/9-family/src/exports/control.ts @@ -101,6 +101,7 @@ export type { PlannedStatements, SingleOperationCall, StatementPlanningTarget, + TableRenameByHand, } from '../core/migrations/statement-planning'; export { fieldRenameStorageEffect, diff --git a/packages/2-sql/9-family/test/operation-subjects.test.ts b/packages/2-sql/9-family/test/operation-subjects.test.ts index b8383d007b47..b9853cbb219a 100644 --- a/packages/2-sql/9-family/test/operation-subjects.test.ts +++ b/packages/2-sql/9-family/test/operation-subjects.test.ts @@ -8,10 +8,16 @@ import { subjectsOfCalls, unknownCallNames, } from '../src/core/migrations/operation-subjects'; +import type { TableRenameByHand } from '../src/core/migrations/statement-planning'; import { contractOf, renameField, renameModel } from './statement-fixtures'; const app = asNamespaceId('app'); +const byHand: TableRenameByHand = { + renderTableRename: ({ from, to }) => `this.renameTable("${from}", "${to}")`, + tableRenameByHand: ({ from, to }) => [`ALTER TABLE "${from}" RENAME TO "${to}"`], +}; + function table(name: string, storageName = name): SubjectStorage { return { storageName, table: { namespaceId: 'app', table: name, column: undefined } }; } @@ -40,6 +46,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([unchanged, losing(table('legacy')), losing(column('user', 'full_name'))], { fromContract: origin, contract: contractOf({ User: { table: 'user', fields: { id: 'id' } } }), + byHand, statements: [], }), ).toEqual({ @@ -58,7 +65,7 @@ describe('subjectsOfCalls', () => { expect( subjectsOfCalls( [{ operationCount: 3, dataLoss: [], accessWidening: [] }, losing(table('legacy'))], - { fromContract: origin, contract: origin, statements: [] }, + { fromContract: origin, contract: origin, byHand, statements: [] }, ).dataLoss, ).toEqual([ { operationIndex: 3, subject: { kind: 'model', namespaceId: app, model: 'Legacy' } }, @@ -76,6 +83,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([losing(column('account', 'display_name'))], { fromContract: from, contract: destination, + byHand, statements: [ renameModel('User', 'Account'), renameField('User', 'name', 'displayName', 'Account'), @@ -96,6 +104,7 @@ describe('subjectsOfCalls', () => { { fromContract: origin, contract: origin, + byHand, statements: [], }, ).dataLoss, @@ -110,6 +119,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([losing(table('legacy'), column('user', 'full_name'))], { fromContract: null, contract: origin, + byHand, statements: [], }).dataLoss, ).toEqual([ @@ -123,6 +133,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([losing({ storageName: 'public.mood', table: undefined })], { fromContract: origin, contract: origin, + byHand, statements: [], }).dataLoss, ).toEqual([{ operationIndex: 0, subject: { kind: 'storage', name: 'public.mood' } }]); @@ -139,7 +150,7 @@ describe('subjectsOfCalls', () => { accessWidening: [{ ...table('user'), widens: false }], }, ], - { fromContract: origin, contract: origin, statements: [] }, + { fromContract: origin, contract: origin, byHand, statements: [] }, ), ).toEqual({ dataLoss: [], @@ -165,6 +176,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([losing(table('users')), losing(column('users', 'email'))], { fromContract: sharing, contract: sharing, + byHand, statements: [], }).dataLoss.map(({ subject }) => subject), ).toEqual([ @@ -178,6 +190,7 @@ describe('subjectsOfCalls', () => { subjectsOfCalls([losing(column('users', 'level'))], { fromContract: sharing, contract: sharing, + byHand, statements: [], }).dataLoss.map(({ subject }) => subject), ).toEqual([{ kind: 'field', namespaceId: app, model: 'Admin', field: 'level' }]); @@ -209,7 +222,7 @@ describe('fieldEventStorage', () => { fieldEventStorage({ namespaceId: 'app', tableName: 'user', columnName: 'nickname' }), ), ], - { fromContract: origin, contract: origin, statements: [] }, + { fromContract: origin, contract: origin, byHand, statements: [] }, ); expect(subjects.dataLoss.map(({ subject }) => subject)).toEqual([ { kind: 'field', namespaceId: app, model: 'User', field: 'name' }, @@ -221,3 +234,59 @@ describe('fieldEventStorage', () => { ).toBe('user.bio'); }); }); + +describe('subjectsOfCalls for a table whose model keeps its name', () => { + const from = contractOf({ Profile: { table: 'profiles', fields: { id: 'id' } } }); + + it('says how to rename the table by hand to keep its rows', () => { + const to = contractOf({ Profile: { table: 'people', fields: { id: 'id' } } }); + + expect( + subjectsOfCalls([losing(table('profiles'))], { + fromContract: from, + contract: to, + statements: [], + byHand, + }).dataLoss, + ).toEqual([ + { + operationIndex: 0, + subject: { kind: 'model', namespaceId: app, model: 'Profile' }, + keepDataAdvice: + 'To keep its rows, rename table "profiles" to "people" instead; a statement cannot rename the table of a model that keeps its name. With db update, rename the table in the database yourself with ALTER TABLE "profiles" RENAME TO "people", then run prisma db update. With migration plan, write the rename as a migration of its own instead: run prisma migration new --name --from , add this.renameTable("profiles", "people") to the operations of its migration.ts, run node on that migration.ts so it writes ops.json, then run prisma db migrate.', + }, + ]); + }); + + it('gives no advice when the destination has no model of that name', () => { + const to = contractOf({ Person: { table: 'people', fields: { id: 'id' } } }); + + expect( + subjectsOfCalls([losing(table('profiles'))], { + fromContract: from, + contract: to, + statements: [], + byHand, + }).dataLoss, + ).toEqual([ + { operationIndex: 0, subject: { kind: 'model', namespaceId: app, model: 'Profile' } }, + ]); + }); + + it('gives no advice when the new table already holds another model of the origin', () => { + const withPeople = contractOf({ + Profile: { table: 'profiles', fields: { id: 'id' } }, + Person: { table: 'people', fields: { id: 'id' } }, + }); + const to = contractOf({ Profile: { table: 'people', fields: { id: 'id' } } }); + + expect( + subjectsOfCalls([losing(table('profiles'))], { + fromContract: withPeople, + contract: to, + statements: [], + byHand, + }).dataLoss.map(({ keepDataAdvice }) => keepDataAdvice), + ).toEqual([undefined]); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/control-target.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/control-target.ts index 21479ec048f3..d8e1460cc605 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/control-target.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/control-target.ts @@ -20,7 +20,7 @@ import { notOk, ok } from '@internal/utils/result'; import { mongoTargetDescriptorMeta } from '../descriptor-meta'; import type { MongoTargetContract } from '../mongo-target-contract'; import { MongoTargetContractSerializer } from '../mongo-target-contract-serializer'; -import { keepDataByHand, MongoMigrationPlanner } from './mongo-planner'; +import { MongoMigrationPlanner } from './mongo-planner'; import { MongoMigrationRunner, type MongoMigrationRunnerExecuteOptions } from './mongo-runner'; import { MongoTargetSchemaVerifier } from './mongo-target-schema-verifier'; import { entityNamesDeclaredBy, scopeVerifyResultToSpace } from './scope-verify-result'; @@ -42,7 +42,6 @@ export const mongoTargetDescriptor: MongoControlTargetDescriptor) { return new MongoMigrationPlanner(); }, diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts index 87c5dcf53702..487d9c19fb8b 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/migration-factories.ts @@ -1,6 +1,6 @@ import type { - MongoDataTransformCheck, - MongoDataTransformOperation, + MongoDmlCheck, + MongoDmlOperation, MongoFilterExpr, MongoIndexKey, } from '@internal/mongo-query-ast/control'; @@ -20,13 +20,19 @@ import { MongoExistsExpr, MongoFieldFilter, type MongoMigrationPlanOperation, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; -import type { MongoQueryPlan } from '@internal/mongo-query-ast/execution'; +import { + type MongoQueryPlan, + RawAggregateCommand, + RawUpdateManyCommand, +} from '@internal/mongo-query-ast/execution'; import { createFieldAccessor } from '@internal/mongo-query-builder'; import { collection } from '@internal/mongo-query-builder/contract-free'; -import type { MongoValue } from '@internal/mongo-value'; +import type { Document, MongoValue } from '@internal/mongo-value'; import { blindCast } from '@internal/utils/casts'; import { ifDefined } from '@internal/utils/defined'; +import { mongoTargetError } from '../mongo-target-errors'; import type { CollModMeta } from './op-factory-call'; type StringField = { readonly codecId: 'mongo/string@1'; readonly nullable: false }; @@ -77,9 +83,9 @@ export function dataTransform( }; run: () => MongoQueryPlan | Buildable; }, -): MongoDataTransformOperation { - let precheck: readonly MongoDataTransformCheck[] = []; - let postcheck: readonly MongoDataTransformCheck[] = []; +): MongoDmlOperation { + let precheck: readonly MongoDmlCheck[] = []; + let postcheck: readonly MongoDmlCheck[] = []; if (options.check) { const source = resolveQuery(options.check.source()); @@ -107,6 +113,177 @@ export function dataTransform( }; } +export interface FieldRewriteOptions { + readonly filter?: Document; +} + +const REWRITE_PLAN_META = { target: 'mongo', storageHash: '', lane: 'mongo-raw' } as const; + +function documentsHolding(fields: readonly string[], filter: Document | undefined): Document { + const conditions: Document[] = [ + ...(filter === undefined ? [] : [filter]), + ...fields.map((field) => ({ [field]: { $exists: true } })), + ]; + const [only] = conditions; + return conditions.length === 1 && only !== undefined ? only : { $and: conditions }; +} + +const UNREWRITABLE_FIELD_NAMES: ReadonlyArray<{ + readonly matches: (field: string) => boolean; + readonly reason: string; +}> = [ + { matches: (field) => field === '', reason: 'it is empty' }, + { matches: (field) => field === '_id', reason: 'MongoDB does not let a document change its _id' }, + { + matches: (field) => field.includes('.'), + reason: 'MongoDB reads a name containing "." as a path into embedded documents', + }, + { + matches: (field) => field.startsWith('$'), + reason: 'MongoDB reads a name starting with "$" as an operator', + }, +]; + +/** Why `renameField` and `unsetField` cannot rewrite a field of this name, if they cannot. */ +export function unrewritableFieldNameReason(field: string): string | undefined { + return UNREWRITABLE_FIELD_NAMES.find((rule) => rule.matches(field))?.reason; +} + +function assertRewritableField(collectionName: string, field: string): void { + const reason = unrewritableFieldNameReason(field); + if (reason === undefined) return; + throw mongoTargetError( + 'MIGRATION.FIELD_NAME_UNSUPPORTED', + `Field "${field}" in collection ${collectionName} cannot be rewritten: ${reason}.`, + { meta: { collection: collectionName, field, reason } }, + ); +} + +function rewritePlan(collectionName: string, command: RawAggregateCommand | RawUpdateManyCommand) { + return { collection: collectionName, command, meta: { ...REWRITE_PLAN_META } }; +} + +function noDocumentHolds( + collectionName: string, + fields: readonly string[], + filter: Document | undefined, +): MongoDmlCheck { + return { + description: `no document in ${collectionName} holds ${fields.length === 1 ? fields[0] : `both ${fields.join(' and ')}`}`, + source: rewritePlan( + collectionName, + new RawAggregateCommand(collectionName, [ + { $match: documentsHolding(fields, filter) }, + { $limit: 1 }, + ]), + ), + filter: MATCH_ALL_FILTER, + expect: 'notExists', + }; +} + +function rewriteUpdate( + collectionName: string, + field: string, + filter: Document | undefined, + update: Document, +): MongoQueryPlan { + return rewritePlan( + collectionName, + new RawUpdateManyCommand(collectionName, documentsHolding([field], filter), update, { + bypassDocumentValidation: true, + }), + ); +} + +function idSuffix(filter: Document | undefined): string { + return filter === undefined ? '' : `.where(${JSON.stringify(filter)})`; +} + +function valueList(values: readonly unknown[]): string { + return values.map((value) => JSON.stringify(value)).join(', '); +} + +/** A field condition in words: `type is "bug"`, `type is not "bug"`, `type is none of "bug", "task"`. */ +function conditionInWords(field: string, condition: unknown): string | undefined { + const excluded: unknown = + typeof condition === 'object' && condition !== null && !Array.isArray(condition) + ? Reflect.get(condition, '$nin') + : undefined; + if (Array.isArray(excluded)) { + return excluded.length === 1 + ? `${field} is not ${valueList(excluded)}` + : `${field} is none of ${valueList(excluded)}`; + } + return typeof condition === 'object' && condition !== null + ? undefined + : `${field} is ${JSON.stringify(condition)}`; +} + +/** The documents a filter selects, in words where the planner wrote it, for an operation's label. */ +function labelSuffix(filter: Document | undefined): string { + if (filter === undefined) return ''; + const entries = Object.entries(filter); + const words = entries.map(([field, condition]) => conditionInWords(field, condition)); + return words.every((word) => word !== undefined) && words.length > 0 + ? ` in documents whose ${words.join(' and ')}` + : ` in documents matching ${JSON.stringify(filter)}`; +} + +export function renameField( + collectionName: string, + from: string, + to: string, + options?: FieldRewriteOptions, +): MongoDmlOperation { + assertRewritableField(collectionName, from); + assertRewritableField(collectionName, to); + if (from === to) { + throw mongoTargetError( + 'MIGRATION.INVALID_OPERATION_ENTRY', + `renameField renames field "${from}" in collection ${collectionName} to its own name.`, + { + meta: { + context: 'renameField', + collection: collectionName, + field: from, + reason: 'the new name is the old name', + }, + }, + ); + } + const filter = options?.filter; + const id = `field.${collectionName}.${from}.rename(${to})${idSuffix(filter)}`; + return { + id, + label: `Rename field ${from} to ${to} in collection ${collectionName}${labelSuffix(filter)}`, + operationClass: 'widening', + name: id, + precheck: [noDocumentHolds(collectionName, [from, to], filter)], + run: [rewriteUpdate(collectionName, from, filter, { $rename: { [from]: to } })], + postcheck: [noDocumentHolds(collectionName, [from], filter)], + }; +} + +export function unsetField( + collectionName: string, + field: string, + options?: FieldRewriteOptions, +): MongoDmlOperation { + assertRewritableField(collectionName, field); + const filter = options?.filter; + const id = `field.${collectionName}.${field}.unset${idSuffix(filter)}`; + return { + id, + label: `Remove field ${field} from collection ${collectionName}${labelSuffix(filter)}`, + operationClass: 'destructive', + name: id, + precheck: [], + run: [rewriteUpdate(collectionName, field, filter, { $unset: { [field]: '' } })], + postcheck: [noDocumentHolds(collectionName, [field], filter)], + }; +} + function formatKeys(keys: ReadonlyArray): string { return keys.map((k) => `${k.field}:${k.direction}`).join(', '); } @@ -249,6 +426,32 @@ export function dropCollection(collectionName: string): MongoMigrationPlanOperat }; } +function collectionCheck(name: string, expect: 'exists' | 'notExists') { + return { + description: + expect === 'exists' ? `collection ${name} exists` : `collection ${name} does not exist`, + source: new ListCollectionsCommand(), + filter: MongoFieldFilter.eq('name', name), + expect, + }; +} + +export function renameCollection(from: string, to: string): MongoMigrationPlanOperation { + return { + id: `collection.${from}.rename(${to})`, + label: `Rename collection ${from} to ${to}`, + operationClass: 'widening', + precheck: [collectionCheck(from, 'exists'), collectionCheck(to, 'notExists')], + execute: [ + { + description: `rename collection ${from} to ${to}`, + command: new RenameCollectionCommand(from, to), + }, + ], + postcheck: [collectionCheck(to, 'exists'), collectionCheck(from, 'notExists')], + }; +} + export function setValidation( collectionName: string, schema: Record, diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts index f4e005e11475..f2a3f2c9922d 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-ops-serializer.ts @@ -11,8 +11,8 @@ import { ListCollectionsCommand, ListIndexesCommand, MongoAndExpr, - type MongoDataTransformCheck, - type MongoDataTransformOperation, + type MongoDmlCheck, + type MongoDmlOperation, MongoExistsExpr, MongoFieldFilter, type MongoFilterExpr, @@ -21,6 +21,7 @@ import { type MongoMigrationStep, MongoNotExpr, MongoOrExpr, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import { AggregateCommand, @@ -115,6 +116,12 @@ const DropCollectionJson = type({ collection: 'string', }); +const RenameCollectionJson = type({ + kind: '"renameCollection"', + collection: 'string', + to: 'string', +}); + const CollModJson = type({ kind: '"collMod"', collection: 'string', @@ -189,6 +196,7 @@ const RawUpdateManyJson = type({ collection: 'string', filter: 'Record', update: 'Record | Record[]', + 'bypassDocumentValidation?': 'boolean', }); const RawDeleteOneJson = type({ @@ -304,17 +312,17 @@ const DdlOperationJson = type({ postcheck: 'Record[]', }); -const DataTransformCheckJson = type({ +const DmlCheckJson = type({ description: 'string', source: 'Record', filter: 'Record', expect: '"exists" | "notExists"', }); -const DataTransformOperationJson = type({ +const DmlOperationJson = type({ id: 'string', label: 'string', - operationClass: '"data"', + operationClass: '"additive" | "widening" | "destructive" | "data"', name: 'string', precheck: 'Record[]', run: 'Record[]', @@ -578,7 +586,12 @@ export function deserializeDmlCommand(json: unknown): AnyMongoCommand { } case 'rawUpdateMany': { const data = validate(RawUpdateManyJson, json, 'rawUpdateMany command'); - return new RawUpdateManyCommand(data.collection, data.filter, data.update); + return new RawUpdateManyCommand( + data.collection, + data.filter, + data.update, + ifDefined('bypassDocumentValidation', data.bypassDocumentValidation), + ); } case 'rawDeleteOne': { const data = validate(RawDeleteOneJson, json, 'rawDeleteOne command'); @@ -678,6 +691,10 @@ function deserializeDdlCommand(json: unknown): AnyMongoDdlCommand { const data = validate(DropCollectionJson, json, 'dropCollection command'); return new DropCollectionCommand(data.collection); } + case 'renameCollection': { + const data = validate(RenameCollectionJson, json, 'renameCollection command'); + return new RenameCollectionCommand(data.collection, data.to); + } case 'collMod': { const data = validate(CollModJson, json, 'collMod command'); return new CollModCommand(data.collection, { @@ -734,8 +751,8 @@ function deserializeStep(json: unknown): MongoMigrationStep { }; } -function isDataTransformJson(json: unknown): boolean { - return recordValue(json, 'operationClass', 'migration operation') === 'data'; +function isDmlOperationJson(json: unknown): boolean { + return Array.isArray(recordValue(json, 'run', 'migration operation')); } function deserializeDdlOp(json: unknown): MongoMigrationPlanOperation { @@ -750,8 +767,8 @@ function deserializeDdlOp(json: unknown): MongoMigrationPlanOperation { }; } -function deserializeDataTransformCheck(json: unknown): MongoDataTransformCheck { - const data = validate(DataTransformCheckJson, json, 'data transform check'); +function deserializeDmlCheck(json: unknown): MongoDmlCheck { + const data = validate(DmlCheckJson, json, 'DML operation check'); return { description: data.description, source: deserializeMongoQueryPlan(data.source), @@ -760,22 +777,22 @@ function deserializeDataTransformCheck(json: unknown): MongoDataTransformCheck { }; } -function deserializeDataTransformOp(json: unknown): MongoDataTransformOperation { - const data = validate(DataTransformOperationJson, json, 'data transform operation'); +function deserializeDmlOp(json: unknown): MongoDmlOperation { + const data = validate(DmlOperationJson, json, 'DML operation'); return { id: data.id, label: data.label, - operationClass: 'data', + operationClass: data.operationClass, name: data.name, - precheck: data.precheck.map(deserializeDataTransformCheck), + precheck: data.precheck.map(deserializeDmlCheck), run: data.run.map(deserializeMongoQueryPlan), - postcheck: data.postcheck.map(deserializeDataTransformCheck), + postcheck: data.postcheck.map(deserializeDmlCheck), }; } export function deserializeMongoOp(json: unknown): AnyMongoMigrationOperation { - if (isDataTransformJson(json)) { - return deserializeDataTransformOp(json); + if (isDmlOperationJson(json)) { + return deserializeDmlOp(json); } return deserializeDdlOp(json); } diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts index 308f72a53f45..35eb603ee7ab 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-planner.ts @@ -1,19 +1,19 @@ import { asNamespaceId, type Contract, type ContractWithDomain } from '@internal/contract/types'; import { contractToMongoSchemaIR } from '@internal/family-mongo/control'; import type { TargetBoundComponentDescriptor } from '@internal/framework-components/components'; -import { - describeMigrationStatement, - type MigrationOperationPolicy, - type MigrationOperationSubject, - type MigrationPlanner, - type MigrationPlannerConflict, - type MigrationPlannerResult, - type MigrationPlanWithAuthoringSurface, - type MigrationScaffoldContext, - type MigrationSubject, - type ModelCoordinate, - type PlanOrigin, - type ResolvedMigrationStatement, +import type { + MigrationDataLoss, + MigrationOperationPolicy, + MigrationOperationSubject, + MigrationPlanner, + MigrationPlannerConflict, + MigrationPlannerResult, + MigrationPlanWithAuthoringSurface, + MigrationScaffoldContext, + MigrationSubject, + ModelCoordinate, + PlanOrigin, + ResolvedMigrationStatement, } from '@internal/framework-components/control'; import type { MongoContract } from '@internal/mongo-contract'; import type { @@ -25,6 +25,16 @@ import type { } from '@internal/mongo-schema-ir'; import { canonicalize, deepEqual } from '@internal/mongo-schema-ir'; import { blindCast } from '@internal/utils/casts'; +import { + collectionOf, + type FieldRemoval, + keepCollectionByHandAdvice, + planFieldRemovals, + planStatements, + planValidatorFieldRemovals, + type StatementCalls, + timeSeriesRemovalRefusals, +} from './mongo-statement-planning'; import type { OpFactoryCall } from './op-factory-call'; import { CollModCall, @@ -36,6 +46,7 @@ import { schemaIndexToCreateIndexOptions, } from './op-factory-call'; import { PlannerProducedMongoMigration } from './planner-produced-migration'; +import { sameHashFieldChangeRefusals } from './same-hash-field-changes'; function buildIndexLookupKey(index: MongoSchemaIndex): string { const keys = index.keys.map((k) => `${k.field}:${k.direction}`).join(','); @@ -103,14 +114,60 @@ function validatorUpdateLabel( : `Update validator on ${collName} (${parts.join('; ')})`; } +/** + * The keys of an option MongoDB fills in when a collection is created without them, each with a + * test for the value it fills in. `listCollections` reports a time-series collection's granularity + * as "seconds" and its bucketMaxSpanSeconds when they were not given, and bucketRoundingSeconds only + * for custom bucketing, which a contract cannot state; and a clustered index's v, key and unique, and + * its name as "_id_". + */ +const SERVER_FILLED: Readonly< + Record<'timeseries' | 'clusteredIndex', Readonly boolean>>> +> = { + timeseries: { + granularity: (value) => value === 'seconds', + bucketMaxSpanSeconds: () => true, + bucketRoundingSeconds: () => true, + }, + clusteredIndex: { + v: () => true, + key: () => true, + unique: () => true, + name: (value) => value === '_id_', + }, +}; + +/** The origin's value of an option without the keys the server filled in and the destination leaves out. */ +function withoutServerFilled( + option: keyof typeof SERVER_FILLED, + origin: unknown, + dest: unknown, +): unknown { + if (!isPlainObject(origin)) return origin; + const filled = SERVER_FILLED[option]; + return Object.fromEntries( + Object.entries(origin).filter( + ([key, value]) => (isPlainObject(dest) && Object.hasOwn(dest, key)) || !filled[key]?.(value), + ), + ); +} + function hasImmutableOptionChange( origin: MongoSchemaCollectionOptions | undefined, dest: MongoSchemaCollectionOptions | undefined, ): string | undefined { if (canonicalize(origin?.capped) !== canonicalize(dest?.capped)) return 'capped'; - if (canonicalize(origin?.timeseries) !== canonicalize(dest?.timeseries)) return 'timeseries'; + if ( + canonicalize(withoutServerFilled('timeseries', origin?.timeseries, dest?.timeseries)) !== + canonicalize(dest?.timeseries) + ) + return 'timeseries'; if (canonicalize(origin?.collation) !== canonicalize(dest?.collation)) return 'collation'; - if (canonicalize(origin?.clusteredIndex) !== canonicalize(dest?.clusteredIndex)) + if ( + canonicalize( + withoutServerFilled('clusteredIndex', origin?.clusteredIndex, dest?.clusteredIndex), + ) !== canonicalize(dest?.clusteredIndex) + ) return 'clusteredIndex'; return undefined; } @@ -123,15 +180,6 @@ export type PlanCallsResult = | { readonly kind: 'success'; readonly calls: OpFactoryCall[] } | { readonly kind: 'failure'; readonly conflicts: MigrationPlannerConflict[] }; -/** The collection a model stores its documents in; a model without `@@map` names it verbatim. */ -function collectionOf(contract: ContractWithDomain | null, coordinate: ModelCoordinate): string { - const collection = - contract?.domain.namespaces[coordinate.namespaceId]?.models[coordinate.model]?.storage[ - 'collection' - ]; - return typeof collection === 'string' ? collection : coordinate.model; -} - /** * The model of `fromContract` that stores its documents in `collection`, if any: the root model * when variants share the collection with it. @@ -152,79 +200,53 @@ function modelStoredIn( return (storing.find(({ isRoot }) => isRoot) ?? storing[0])?.coordinate; } -/** Each collection drop of the plan, with the model whose documents it loses. */ +/** + * Each collection drop of the plan, with the model whose documents it loses. When that model keeps + * its name and the plan brings its new collection into existence, the drop says how to rename the + * collection by hand instead. + */ function collectionDrops( calls: readonly OpFactoryCall[], fromContract: ContractWithDomain | null, -): readonly MigrationOperationSubject[] { - return calls.flatMap((call, operationIndex) => { + contract: ContractWithDomain, + collections: { readonly origin: MongoSchemaIR; readonly destination: MongoSchemaIR }, +): readonly MigrationDataLoss[] { + const created = (collection: string) => + collections.origin.collection(collection) === undefined && + collections.destination.collection(collection) !== undefined; + return calls.flatMap((call, operationIndex): readonly MigrationDataLoss[] => { if (!(call instanceof DropCollectionCall)) return []; const model = fromContract === null ? undefined : modelStoredIn(fromContract, call.collection); - const subject: MigrationSubject = - model === undefined - ? { kind: 'storage', name: call.collection } - : { kind: 'model', ...model }; - return [{ operationIndex, subject }]; + if (model === undefined) { + return [{ operationIndex, subject: { kind: 'storage', name: call.collection } }]; + } + const subject: MigrationSubject = { kind: 'model', ...model }; + const destinationCollection = modelStoredInDestination(contract, model); + return [ + destinationCollection !== undefined && created(destinationCollection) + ? { + operationIndex, + subject, + keepDataAdvice: keepCollectionByHandAdvice( + model, + call.collection, + destinationCollection, + ), + } + : { operationIndex, subject }, + ]; }); } -const NOT_IN_THIS_RELEASE = 'MongoDB cannot carry out rename statements in this release.'; - -function shellString(name: string): string { - return JSON.stringify(name); -} - -/** - * How to keep the documents of a subject a plan would drop, since the planner carries out no - * rename: rename the collection by hand before a plan that drops it is applied. Right for both - * `db update`, which then finds nothing to drop, and `migration plan`, whose written migration - * still drops it. - */ -export function keepDataByHand( - subject: MigrationSubject, - fromContract: ContractWithDomain, -): string { - const collection = - subject.kind === 'storage' ? subject.name : collectionOf(fromContract, subject); - return `If it was renamed, keep its documents instead: rename collection "${collection}" by hand on each database before a plan that drops it is applied there, for example with db.getCollection(${shellString(collection)}).renameCollection("") in mongosh. db update then drops nothing; a migration written by migration plan still drops "${collection}", so remove that operation from its migration.ts, or do not apply it where the collection was renamed.`; -} - -/** - * What a plan made without the statement does to the data, and how to keep it. Right for both - * `db update` and `migration plan`, since the planner does not know which command it serves. - * Mongo contracts key a model's fields by their stored names, so the statement's field names are - * the names a `$rename` needs. - */ -function keepTheData( - statement: ResolvedMigrationStatement, - fromContract: ContractWithDomain | null, +/** The collection the destination stores `model` in, when the destination still has that model. */ +function modelStoredInDestination( contract: ContractWithDomain, -): string { - const from = collectionOf(fromContract, statement.from); - if (statement.entity === 'field') { - const { field } = statement.from; - const newField = statement.to.field; - const collection = `db.getCollection(${shellString(from)})`; - return `${NOT_IN_THIS_RELEASE} Without the statement, the documents in collection "${from}" keep their values under "${field}", and nothing moves them to "${newField}". To move them, run these in mongosh on each database before a plan made without the statement is applied there, using the field names as they are stored. First turn off the collection's validator, which still requires "${field}": db.runCommand({ collMod: ${shellString(from)}, validationLevel: "off" }). Then drop each unique index that includes "${field}", or the move fails once two documents have lost it, for example ${collection}.dropIndex(${shellString(`${field}_1`)}). Then move the values: ${collection}.updateMany({}, { $rename: { ${shellString(field)}: ${shellString(newField)} } }). Applying the plan then creates the indexes on "${newField}" and turns the validator back on.`; - } - const to = collectionOf(contract, statement.to); - if (from === to) { - return `${NOT_IN_THIS_RELEASE} Both models store their documents in collection "${from}", so a plan made without the statement keeps them.`; - } - return `${NOT_IN_THIS_RELEASE} Without the statement, a plan drops collection "${from}" with its documents and creates collection "${to}". To keep the documents, rename the collection by hand on each database before a plan made without the statement is applied there, for example with db.getCollection(${shellString(from)}).renameCollection(${shellString(to)}) in mongosh. A migration written by migration plan without the statement still drops "${from}" wherever it is applied, so check its operations first.`; -} - -function statementNotApplied( - statement: ResolvedMigrationStatement, - fromContract: ContractWithDomain | null, - contract: ContractWithDomain, -): MigrationPlannerConflict { - return { - kind: 'statementRefused', - summary: `MongoDB does not apply rename statements in this release, so nothing was planned: ${describeMigrationStatement(statement, fromContract ?? contract, contract)}`, - why: keepTheData(statement, fromContract, contract), - refusedStatement: statement, - }; + model: ModelCoordinate, +): string | undefined { + const namespace = contract.domain.namespaces[model.namespaceId]; + return namespace !== undefined && Object.hasOwn(namespace.models, model.model) + ? collectionOf(contract, model) + : undefined; } export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> { @@ -233,15 +255,19 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> readonly schema: unknown; readonly policy: MigrationOperationPolicy; readonly frameworkComponents: ReadonlyArray>; + /** What the statements and removed fields plan; its `schema` is diffed in place of `schema`. */ + readonly statementCalls?: StatementCalls; }): PlanCallsResult { const contract = blindCast< MongoContract, 'framework planner passes the Mongo contract selected for the mongo target' >(options.contract); - const originIR = blindCast< - MongoSchemaIR, - 'framework planner passes the inspected Mongo schema IR selected for the mongo target' - >(options.schema); + const originIR = + options.statementCalls?.schema ?? + blindCast< + MongoSchemaIR, + 'framework planner passes the inspected Mongo schema IR selected for the mongo target' + >(options.schema); const destinationIR = contractToMongoSchemaIR(contract); const collCreates: OpFactoryCall[] = []; @@ -342,8 +368,10 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> } const allCalls = [ + ...(options.statementCalls?.collectionRenames ?? []), ...collCreates, ...drops, + ...(options.statementCalls?.rewrites ?? []), ...creates, ...validatorOps, ...mutableOptionOps, @@ -376,7 +404,7 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> readonly fromContract: Contract | null; /** The origin the produced plan asserts: its `describe().from` and `origin`. */ readonly origin: PlanOrigin | null; - /** The `--rename` statements, resolved; MongoDB refuses any statement in this release. */ + /** The `--rename` statements, resolved. */ readonly statements: readonly ResolvedMigrationStatement[]; readonly frameworkComponents: ReadonlyArray>; /** @@ -390,27 +418,80 @@ export class MongoMigrationPlanner implements MigrationPlanner<'mongo', 'mongo'> MongoContract, 'framework planner passes the Mongo contract selected for the mongo target' >(options.contract); - const [statement] = options.statements; - if (statement !== undefined) { - return { - kind: 'failure', - conflicts: [statementNotApplied(statement, options.fromContract, contract)], - }; - } - const result = this.planCalls(options); + const originIR = blindCast< + MongoSchemaIR, + 'framework planner passes the inspected Mongo schema IR selected for the mongo target' + >(options.schema); + const fromContract = options.fromContract; + const statementPlan = planStatements({ + statements: options.statements, + fromContract: fromContract ?? contract, + contract, + schema: originIR, + }); + if (!statementPlan.ok) return { kind: 'failure', conflicts: statementPlan.failure }; + const { schema, collectionRenames, fieldRenames, callOfStatement } = statementPlan.value; + const destinationIR = contractToMongoSchemaIR(contract); + const renamedTo = new Map(collectionRenames.map((call) => [call.from, call.to])); + const removals: readonly FieldRemoval[] = + fromContract === null + ? planValidatorFieldRemovals({ origin: schema, destination: destinationIR, contract }) + : planFieldRemovals({ + statements: options.statements, + fromContract, + contract, + collectionAfterRenames: (collection) => renamedTo.get(collection) ?? collection, + keptCollections: new Set( + schema.collectionNames.filter((name) => destinationIR.collection(name) !== undefined), + ), + }); + const removalRefusals = [ + ...timeSeriesRemovalRefusals(removals, schema), + ...sameHashFieldChangeRefusals({ + origin: options.origin, + fromContract: fromContract ?? contract, + contract, + statements: options.statements, + callOfStatement, + removals, + }), + ]; + if (removalRefusals.length > 0) return { kind: 'failure', conflicts: removalRefusals }; + const result = this.planCalls({ + ...options, + statementCalls: { + schema, + collectionRenames, + rewrites: [...fieldRenames, ...removals.map((removal) => removal.call)], + }, + }); if (result.kind === 'failure') return result; + const { calls } = result; + const fieldLoss: readonly MigrationOperationSubject[] = removals.map((removal) => ({ + operationIndex: calls.indexOf(removal.call), + subject: removal.subject, + })); return { kind: 'success', plan: new PlannerProducedMongoMigration( - result.calls, + calls, { from: options.origin?.storageHash ?? null, to: contract.storage.storageHash, }, options.snapshotsImportPath, ), - appliedStatements: [], - dataLoss: collectionDrops(result.calls, options.fromContract), + appliedStatements: options.statements.map((statement, index) => { + const call = callOfStatement[index]; + return { statement, operationIndexes: call === undefined ? [] : [calls.indexOf(call)] }; + }), + dataLoss: [ + ...collectionDrops(calls, fromContract, contract, { + origin: schema, + destination: destinationIR, + }), + ...fieldLoss, + ].sort((a, b) => a.operationIndex - b.operationIndex), accessWidening: [], }; } diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts index 4dc8d0632be5..0f08afcb938e 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-runner.ts @@ -1,5 +1,6 @@ import type { ContractMarkerRecord } from '@internal/contract/types'; import { errorRunnerFailed } from '@internal/errors/execution'; +import { fieldRewriteOf } from '@internal/family-mongo/control'; import type { MarkerOperations, MongoRunnerDependencies, @@ -22,12 +23,18 @@ import type { MongoContract } from '@internal/mongo-contract'; import type { MongoAdapter, MongoDriver } from '@internal/mongo-lowering'; import type { AnyMongoMigrationOperation, - MongoDataTransformCheck, - MongoDataTransformOperation, + MongoDmlCheck, + MongoDmlOperation, MongoInspectionCommandVisitor, MongoMigrationCheck, MongoMigrationPlanOperation, } from '@internal/mongo-query-ast/control'; +import { isMongoDmlOperation, RenameCollectionCommand } from '@internal/mongo-query-ast/control'; +import { + type MongoQueryPlan, + RawAggregateCommand, + RawUpdateManyCommand, +} from '@internal/mongo-query-ast/execution'; import { blindCast } from '@internal/utils/casts'; import { InternalError } from '@internal/utils/internal-error'; import { notOk, ok, type Result } from '@internal/utils/result'; @@ -80,6 +87,91 @@ function runnerFailure( }); } +/** + * The failure when a field rename's before-check finds documents that already hold the new name, + * which `$rename` would overwrite: what to look for and fix, and that the plan resumes. + */ +function renameOverwriteRefusal(op: MongoDmlOperation): MongoMigrationRunnerResult | undefined { + const rewrite = fieldRewriteOf(op); + if (rewrite === undefined) return undefined; + const renames: unknown = Reflect.get(rewrite.command.update, '$rename'); + const to: unknown = + typeof renames === 'object' && renames !== null + ? Reflect.get(renames, rewrite.field) + : undefined; + if (typeof to !== 'string') return undefined; + const [check] = op.precheck; + const source = check?.source.command; + const match: unknown = + source instanceof RawAggregateCommand + ? Reflect.get(source.pipeline[0] ?? {}, '$match') + : undefined; + const find = `db.getCollection(${JSON.stringify(rewrite.collection)}).find(${JSON.stringify(match ?? {})})`; + return runnerFailure( + 'MIGRATION.PRECHECK_FAILED', + `Operation ${op.id} stopped: some documents in collection ${rewrite.collection} already hold ${to} as well as ${rewrite.field}`, + { + why: `Renaming ${rewrite.field} to ${to} would overwrite their ${to}. Find them with ${find} in mongosh, and move or remove one of the two values in each. Nothing was renamed. The operations before this one in the plan already ran, such as dropping an index on ${rewrite.field}; once the documents are fixed, run the command again and the plan finishes from here.`, + meta: { operationId: op.id, name: op.name }, + }, + ); +} + +/** + * The failure when a collection rename's before-check finds that the new collection already exists. + */ +function renameTargetExistsRefusal( + op: MongoMigrationPlanOperation, + failed: MongoMigrationCheck, +): MongoMigrationRunnerResult | undefined { + const rename = op.execute.find( + (step) => step.command instanceof RenameCollectionCommand, + )?.command; + if (!(rename instanceof RenameCollectionCommand) || failed.expect !== 'notExists') { + return undefined; + } + const { collection: from, to } = rename; + return runnerFailure( + 'MIGRATION.PRECHECK_FAILED', + `Operation ${op.id} stopped: collection ${to} already exists, so collection ${from} cannot be renamed to it`, + { + why: `Nothing was renamed. If ${to} holds nothing you need, drop it, then run the command again; operations that already ran are skipped. If an earlier operation of the same migration created ${to}, as a createIndex on ${to} does, make renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}) the first operation of its migration.ts, run node on that migration.ts, and then drop ${to} and run the command again.`, + meta: { operationId: op.id }, + }, + ); +} + +const MONGO_UNAUTHORIZED = 13; + +/** + * The failure for a rewrite that skips validation when the database user lacks the + * `bypassDocumentValidation` action, as a `readWrite` user on a hosted cluster does. + */ +function bypassValidationRefusal( + op: MongoDmlOperation, + plan: MongoQueryPlan, + error: unknown, +): MongoMigrationRunnerResult | undefined { + if (!(plan.command instanceof RawUpdateManyCommand) || !plan.command.bypassDocumentValidation) { + return undefined; + } + if (typeof error !== 'object' || error === null) return undefined; + if (Reflect.get(error, 'code') !== MONGO_UNAUTHORIZED) return undefined; + const codeName: unknown = Reflect.get(error, 'codeName'); + return runnerFailure( + 'MIGRATION.EXECUTION_FAILED', + `Operation ${op.id} failed: the database user may not skip document validation`, + { + why: `Renaming or removing a field rewrites documents with bypassDocumentValidation, because the old validator requires the old field. MongoDB refused it with error ${MONGO_UNAUTHORIZED} (${String(codeName)}). Grant the user the bypassDocumentValidation action on the database, for example with the dbAdmin or restore role, then run the command again; operations that already ran are skipped.`, + meta: { + operationId: op.id, + mongoCode: MONGO_UNAUTHORIZED, + codeName, + }, + }, + ); +} + export class MongoMigrationRunner { constructor(private readonly deps: MongoRunnerDependencies) {} @@ -116,12 +208,9 @@ export class MongoMigrationRunner { for (const operation of operations) { options.callbacks?.onOperationStart?.(operation); try { - if (operation.operationClass === 'data') { - const result = await this.executeDataTransform( - blindCast< - MongoDataTransformOperation, - 'operationClass data discriminates Mongo data transform operations' - >(operation), + if (isMongoDmlOperation(operation)) { + const result = await this.executeDmlOperation( + operation, adapter, driver, filterEvaluator, @@ -136,14 +225,9 @@ export class MongoMigrationRunner { continue; } - const ddlOp = blindCast< - MongoMigrationPlanOperation, - 'non-data Mongo operations are DDL migration plan operations' - >(operation); - if (runPostchecks && runIdempotency) { const allSatisfied = await this.allChecksSatisfied( - ddlOp.postcheck, + operation.postcheck, inspectionExecutor, filterEvaluator, ); @@ -151,27 +235,30 @@ export class MongoMigrationRunner { } if (runPrechecks) { - const precheckResult = await this.evaluateChecks( - ddlOp.precheck, + const failed = await this.firstFailingCheck( + operation.precheck, inspectionExecutor, filterEvaluator, ); - if (!precheckResult) { - return runnerFailure( - 'MIGRATION.PRECHECK_FAILED', - `Operation ${operation.id} failed during precheck`, - { meta: { operationId: operation.id } }, + if (failed !== undefined) { + return ( + renameTargetExistsRefusal(operation, failed) ?? + runnerFailure( + 'MIGRATION.PRECHECK_FAILED', + `Operation ${operation.id} failed during precheck`, + { meta: { operationId: operation.id } }, + ) ); } } - for (const step of ddlOp.execute) { + for (const step of operation.execute) { await executeDdl(step.command); } if (runPostchecks) { const postcheckResult = await this.evaluateChecks( - ddlOp.postcheck, + operation.postcheck, inspectionExecutor, filterEvaluator, ); @@ -306,8 +393,8 @@ export class MongoMigrationRunner { } } - private async executeDataTransform( - op: MongoDataTransformOperation, + private async executeDmlOperation( + op: MongoDmlOperation, adapter: MongoAdapter, driver: MongoDriver, filterEvaluator: FilterEvaluator, @@ -316,7 +403,7 @@ export class MongoMigrationRunner { runPostchecks: boolean, ): Promise<{ executed: boolean; failure?: MongoMigrationRunnerResult }> { if (runPostchecks && runIdempotency && op.postcheck.length > 0) { - const allSatisfied = await this.evaluateDataTransformChecks( + const allSatisfied = await this.evaluateDmlChecks( op.postcheck, adapter, driver, @@ -326,40 +413,38 @@ export class MongoMigrationRunner { } if (runPrechecks && op.precheck.length > 0) { - const passed = await this.evaluateDataTransformChecks( - op.precheck, - adapter, - driver, - filterEvaluator, - ); + const passed = await this.evaluateDmlChecks(op.precheck, adapter, driver, filterEvaluator); if (!passed) { return { executed: false, - failure: runnerFailure( - 'MIGRATION.PRECHECK_FAILED', - `Operation ${op.id} failed during precheck`, - { - meta: { operationId: op.id, name: op.name }, - }, - ), + failure: + renameOverwriteRefusal(op) ?? + runnerFailure( + 'MIGRATION.PRECHECK_FAILED', + `Operation ${op.id} failed during precheck`, + { + meta: { operationId: op.id, name: op.name }, + }, + ), }; } } for (const plan of op.run) { const wireCommand = await adapter.lower(plan, {}); - for await (const _ of driver.execute(wireCommand)) { - /* consume */ + try { + for await (const _ of driver.execute(wireCommand)) { + /* consume */ + } + } catch (error) { + const refusal = bypassValidationRefusal(op, plan, error); + if (refusal === undefined) throw error; + return { executed: false, failure: refusal }; } } if (runPostchecks && op.postcheck.length > 0) { - const passed = await this.evaluateDataTransformChecks( - op.postcheck, - adapter, - driver, - filterEvaluator, - ); + const passed = await this.evaluateDmlChecks(op.postcheck, adapter, driver, filterEvaluator); if (!passed) { return { executed: false, @@ -377,8 +462,8 @@ export class MongoMigrationRunner { return { executed: true }; } - private async evaluateDataTransformChecks( - checks: readonly MongoDataTransformCheck[], + private async evaluateDmlChecks( + checks: readonly MongoDmlCheck[], adapter: MongoAdapter, driver: MongoDriver, filterEvaluator: FilterEvaluator, @@ -418,13 +503,23 @@ export class MongoMigrationRunner { inspectionExecutor: MongoInspectionCommandVisitor[]>>, filterEvaluator: FilterEvaluator, ): Promise { + return ( + (await this.firstFailingCheck(checks, inspectionExecutor, filterEvaluator)) === undefined + ); + } + + private async firstFailingCheck( + checks: readonly MongoMigrationCheck[], + inspectionExecutor: MongoInspectionCommandVisitor[]>>, + filterEvaluator: FilterEvaluator, + ): Promise { for (const check of checks) { const documents = await check.source.accept(inspectionExecutor); const matchFound = documents.some((doc) => filterEvaluator.evaluate(check.filter, doc)); const passed = check.expect === 'exists' ? matchFound : !matchFound; - if (!passed) return false; + if (!passed) return check; } - return true; + return undefined; } private async allChecksSatisfied( diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts new file mode 100644 index 000000000000..5667143d6dea --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/mongo-statement-planning.ts @@ -0,0 +1,596 @@ +import { + asNamespaceId, + type ContractModelBase, + type ContractRelation, + type ContractWithDomain, + effectiveControlPolicy, +} from '@internal/contract/types'; +import { + describeMigrationStatement, + type FieldCoordinate, + type MigrationPlannerConflict, + type MigrationSubject, + type ModelCoordinate, + modelDisplayName, + type ResolvedFieldRenameStatement, + type ResolvedMigrationStatement, + type ResolvedModelRenameStatement, +} from '@internal/framework-components/control'; +import type { MongoContract } from '@internal/mongo-contract'; +import { MongoSchemaCollection, MongoSchemaIR } from '@internal/mongo-schema-ir'; +import { notOk, ok, type Result } from '@internal/utils/result'; +import { unrewritableFieldNameReason } from './migration-factories'; +import { + type OpFactoryCall, + RenameCollectionCall, + RenameFieldCall, + UnsetFieldCall, +} from './op-factory-call'; + +type Document = Record; + +/** The collection a model stores its documents in; a model without `@@map` names it verbatim. */ +export function collectionOf( + contract: ContractWithDomain | null, + coordinate: ModelCoordinate, +): string { + const collection = + contract?.domain.namespaces[coordinate.namespaceId]?.models[coordinate.model]?.storage[ + 'collection' + ]; + return typeof collection === 'string' ? collection : coordinate.model; +} + +function modelAt( + contract: ContractWithDomain, + coordinate: ModelCoordinate, +): ContractModelBase | undefined { + const namespace = contract.domain.namespaces[coordinate.namespaceId]; + return namespace !== undefined && Object.hasOwn(namespace.models, coordinate.model) + ? namespace.models[coordinate.model] + : undefined; +} + +/** + * The condition that selects a variant's documents in the collection it shares with its base: the + * base's discriminator field holding the variant's value. A model that is not a variant has none. + */ +function variantFilter( + contract: ContractWithDomain, + coordinate: ModelCoordinate, +): Document | undefined { + const base = modelAt(contract, coordinate)?.base; + if (base === undefined) return undefined; + const baseModel = contract.domain.namespaces[base.namespace]?.models[base.model]; + const field = baseModel?.discriminator?.field; + const value = baseModel?.variants?.[coordinate.model]?.value; + return field === undefined || value === undefined ? undefined : { [field]: value }; +} + +function collectionControlPolicy(contract: MongoContract, collection: string) { + const declared = Object.values(contract.storage.namespaces) + .map((namespace) => namespace?.entries.collection?.[collection]) + .find((entry) => entry !== undefined); + return effectiveControlPolicy(declared?.control, contract.defaultControlPolicy); +} + +/** Why MongoDB would refuse a collection name, if it would. */ +function invalidCollectionNameReason(name: string): string | undefined { + if (name === '') return 'it is empty'; + if (name.includes('$')) return 'it contains "$"'; + if (name.includes('\0')) return 'it contains a null character'; + if (name.startsWith('system.')) return 'it starts with "system.", which MongoDB reserves'; + return undefined; +} + +function isEmbedRelation(relation: ContractRelation): boolean { + return !('on' in relation) && !('through' in relation); +} + +type StoredField = + | { readonly kind: 'field' } + | { readonly kind: 'reference' } + | { readonly kind: 'embed' } + | { readonly kind: 'absent' }; + +function storedField(model: ContractModelBase | undefined, field: string): StoredField { + if (model === undefined) return { kind: 'absent' }; + if (Object.hasOwn(model.fields, field)) return { kind: 'field' }; + const relation = Object.hasOwn(model.relations, field) ? model.relations[field] : undefined; + if (relation === undefined) return { kind: 'absent' }; + return isEmbedRelation(relation) ? { kind: 'embed' } : { kind: 'reference' }; +} + +/** + * The calls statements and removed fields add to a plan, with the schema the diff starts from: + * the origin with each renamed collection under its new name, so the diff does not drop and + * create it again. + */ +export interface StatementCalls { + readonly schema: MongoSchemaIR; + /** Run first. */ + readonly collectionRenames: readonly RenameCollectionCall[]; + /** Run after index drops and before index creates. */ + readonly rewrites: readonly (RenameFieldCall | UnsetFieldCall)[]; +} + +export interface StatementPlan { + /** The origin schema with each renamed collection under its new name, for the diff to compare. */ + readonly schema: MongoSchemaIR; + readonly collectionRenames: readonly RenameCollectionCall[]; + readonly fieldRenames: readonly RenameFieldCall[]; + /** For each statement, in order, the call that carries it out, or `undefined` when storage does not change. */ + readonly callOfStatement: readonly (OpFactoryCall | undefined)[]; +} + +function refused( + statement: ResolvedMigrationStatement, + label: string, + reason: string, + why: string, +): MigrationPlannerConflict { + return { + kind: 'statementRefused', + summary: `${label}: ${reason}`, + why, + refusedStatement: statement, + }; +} + +const LEAVE_OUT = 'Leave out this statement.'; + +const TIME_SERIES_UPDATES = + 'MongoDB updates the documents of a time-series collection only through its metaField'; + +/** + * A removal the plan cannot carry out because its collection is a time-series collection, whose + * documents MongoDB does not let an update change except through the metaField. + */ +export function timeSeriesRemovalRefusals( + removals: readonly FieldRemoval[], + schema: MongoSchemaIR, +): readonly MigrationPlannerConflict[] { + return removals + .filter( + (removal) => schema.collection(removal.call.collection)?.options?.timeseries !== undefined, + ) + .map((removal) => ({ + kind: 'fieldRemovalRefused', + summary: `Cannot remove field ${JSON.stringify(removal.call.field)} from collection ${JSON.stringify(removal.call.collection)}: it is a time-series collection`, + why: `${TIME_SERIES_UPDATES}, so the field cannot be removed from the documents that hold it. Keep the field in the contract, as an optional field if new documents leave it out, then run this command again.`, + })); +} + +/** + * Plans the rename each statement states, in order: a model statement whose collection changes renames the + * collection in the schema the diff then starts from, and a field statement whose stored name + * changes renames the field in the documents that hold it. + */ +export function planStatements(input: { + readonly statements: readonly ResolvedMigrationStatement[]; + readonly fromContract: ContractWithDomain; + readonly contract: MongoContract; + readonly schema: MongoSchemaIR; +}): Result { + let schema = input.schema; + const collectionRenames: RenameCollectionCall[] = []; + const fieldRenames: RenameFieldCall[] = []; + const callOfStatement: (OpFactoryCall | undefined)[] = []; + const conflicts: MigrationPlannerConflict[] = []; + const renamedCollections = new Map(); + + for (const statement of input.statements) { + const label = `Cannot ${describeMigrationStatement(statement, input.fromContract, input.contract)}`; + const outcome = + statement.entity === 'model' + ? planModelStatement(statement, label, schema, input) + : planFieldStatement(statement, label, renamedCollections, schema, input); + if (outcome.kind === 'refused') { + conflicts.push(outcome.conflict); + callOfStatement.push(undefined); + continue; + } + if (outcome.kind === 'renameCollection') { + schema = renameInSchema(schema, outcome.call.from, outcome.call.to); + renamedCollections.set(outcome.call.from, outcome.call.to); + collectionRenames.push(outcome.call); + } + if (outcome.kind === 'renameField') fieldRenames.push(outcome.call); + callOfStatement.push(outcome.kind === 'unchanged' ? undefined : outcome.call); + } + + return conflicts.length > 0 + ? notOk(conflicts) + : ok({ schema, collectionRenames, fieldRenames, callOfStatement }); +} + +type StatementOutcome = + | { readonly kind: 'unchanged' } + | { readonly kind: 'renameCollection'; readonly call: RenameCollectionCall } + | { readonly kind: 'renameField'; readonly call: RenameFieldCall } + | { readonly kind: 'refused'; readonly conflict: MigrationPlannerConflict }; + +function renameInSchema(schema: MongoSchemaIR, from: string, to: string): MongoSchemaIR { + return new MongoSchemaIR( + schema.collections.map((collection) => + collection.name === from + ? new MongoSchemaCollection({ + name: to, + indexes: collection.indexes, + ...(collection.validator === undefined ? {} : { validator: collection.validator }), + ...(collection.options === undefined ? {} : { options: collection.options }), + }) + : collection, + ), + ); +} + +function controlPolicyRefusal( + statement: ResolvedMigrationStatement, + label: string, + contract: MongoContract, + collection: string, +): StatementOutcome | undefined { + const policy = collectionControlPolicy(contract, collection); + if (policy === 'managed') return undefined; + return { + kind: 'refused', + conflict: refused( + statement, + label, + `collection "${collection}" has control policy "${policy}"`, + `Statements rename only collections, and fields of collections, whose control policy is "managed". Make the change in the database yourself and leave out this statement.`, + ), + }; +} + +/** + * The reason and advice when a statement renames a variant whose discriminator value changes: its + * documents keep the old value, which the new validator does not accept. + */ +function discriminatorValueChange( + statement: ResolvedModelRenameStatement, + input: { readonly fromContract: ContractWithDomain; readonly contract: MongoContract }, +): readonly [reason: string, why: string] | undefined { + const before = variantFilter(input.fromContract, statement.from); + const after = variantFilter(input.contract, statement.to); + const base = modelAt(input.contract, statement.to)?.base; + if (before === undefined || after === undefined || base === undefined) return undefined; + const [field, oldValue] = Object.entries(before)[0] ?? []; + const newValue = Object.values(after)[0]; + if (field === undefined || oldValue === newValue) return undefined; + const fromName = modelDisplayName(statement.from); + const toName = modelDisplayName(statement.to); + return [ + `its discriminator value changes from ${JSON.stringify(oldValue)} to ${JSON.stringify(newValue)}`, + `The documents of "${fromName}" hold ${field} ${JSON.stringify(oldValue)}, and this release does not rewrite a discriminator value, so they would no longer match "${toName}" and would fail validation. Keep the value: declare "${toName}" with @@base(${base.model}, ${JSON.stringify(oldValue)}). A later release changes a stored value with --convert.`, + ]; +} + +function planModelStatement( + statement: ResolvedModelRenameStatement, + label: string, + schema: MongoSchemaIR, + input: { readonly fromContract: ContractWithDomain; readonly contract: MongoContract }, +): StatementOutcome { + const discriminatorChange = discriminatorValueChange(statement, input); + if (discriminatorChange !== undefined) { + return { kind: 'refused', conflict: refused(statement, label, ...discriminatorChange) }; + } + const from = collectionOf(input.fromContract, statement.from); + const to = collectionOf(input.contract, statement.to); + if (from === to) return { kind: 'unchanged' }; + const policyRefusal = controlPolicyRefusal(statement, label, input.contract, to); + if (policyRefusal !== undefined) return policyRefusal; + const refuse = (reason: string, why: string): StatementOutcome => ({ + kind: 'refused', + conflict: refused(statement, label, reason, why), + }); + const source = schema.collection(from); + if (source === undefined) { + return refuse( + `the schema being planned from has no collection "${from}"`, + `The database has no collection "${from}", or "${from}" is a view, which MongoDB cannot rename. Inspect it with prisma db schema, or ${LEAVE_OUT.toLowerCase()}`, + ); + } + if (source.options?.timeseries !== undefined) { + return refuse( + `collection "${from}" is a time-series collection`, + `MongoDB cannot rename a time-series collection. Leave out this statement. To keep the documents, first create collection "${to}" yourself with the same time-series options and copy them into it; the plan then drops "${from}" once --delete names the model.`, + ); + } + const invalid = invalidCollectionNameReason(to); + if (invalid !== undefined) { + return refuse( + `MongoDB does not accept the collection name "${to}": ${invalid}`, + 'Give the model a collection name MongoDB accepts with @@map, then run this command again.', + ); + } + if (schema.collection(to) !== undefined) { + return refuse( + `the schema being planned from already has a collection "${to}"`, + `A rename cannot replace a collection that already exists. Rename or drop collection "${to}" first, or ${LEAVE_OUT.toLowerCase()}`, + ); + } + return { kind: 'renameCollection', call: new RenameCollectionCall(from, to) }; +} + +function planFieldStatement( + statement: ResolvedFieldRenameStatement, + label: string, + renamedCollections: ReadonlyMap, + schema: MongoSchemaIR, + input: { readonly fromContract: ContractWithDomain; readonly contract: MongoContract }, +): StatementOutcome { + const refuse = (reason: string, why: string): StatementOutcome => ({ + kind: 'refused', + conflict: refused(statement, label, reason, why), + }); + const fromField = storedField(modelAt(input.fromContract, statement.from), statement.from.field); + const toField = storedField(modelAt(input.contract, statement.to), statement.to.field); + if (fromField.kind === 'embed' || toField.kind === 'embed') { + return refuse( + 'the field is an embedded relation', + `Renaming an embedded relation is not supported in this release. ${LEAVE_OUT}`, + ); + } + if (fromField.kind !== 'field' && toField.kind !== 'field') return { kind: 'unchanged' }; + if (fromField.kind !== 'field' || toField.kind !== 'field') { + return refuse( + 'the field is stored in documents on one side of the statement only', + `A relation is not stored in documents, so a statement cannot rename a field to or from one. ${LEAVE_OUT}`, + ); + } + if (statement.from.field === '_id' || statement.to.field === '_id') { + return refuse('it renames _id', `MongoDB does not let a document change its _id. ${LEAVE_OUT}`); + } + const unsafeFrom = unrewritableFieldNameReason(statement.from.field); + if (unsafeFrom !== undefined) { + return refuse( + `the stored field name "${statement.from.field}" cannot be rewritten`, + `${unsafeFrom}, so a statement cannot move its values. Rename it in the database yourself, or ${LEAVE_OUT.toLowerCase()}`, + ); + } + const unsafeTo = unrewritableFieldNameReason(statement.to.field); + if (unsafeTo !== undefined) { + return refuse( + `the stored field name "${statement.to.field}" cannot be written`, + `${unsafeTo}. Give the field a stored name without "." that does not start with "$" (with @map), or ${LEAVE_OUT.toLowerCase()}`, + ); + } + const originCollection = collectionOf(input.fromContract, statement.from); + const collection = renamedCollections.get(originCollection) ?? originCollection; + const destinationCollection = collectionOf(input.contract, statement.to); + if (collection !== destinationCollection) { + return refuse( + `the model's collection changes from "${originCollection}" to "${destinationCollection}", and no statement renames it`, + collectionChangeAdvice(statement, originCollection, destinationCollection), + ); + } + if (statement.from.field === statement.to.field) return { kind: 'unchanged' }; + if (schema.collection(collection)?.options?.timeseries !== undefined) { + return refuse( + `collection "${collection}" is a time-series collection`, + `${TIME_SERIES_UPDATES}, so a statement cannot move the values of a field in them. Keep the stored name: name the field ${JSON.stringify(statement.from.field)} in the contract (with @map(${JSON.stringify(statement.from.field)}) in PSL), then run this command again without this statement.`, + ); + } + const policyRefusal = controlPolicyRefusal(statement, label, input.contract, collection); + if (policyRefusal !== undefined) return policyRefusal; + const filter = + variantFilter(input.fromContract, statement.from) ?? + rootDocumentsLosing(input.contract, statement.to, statement.from.field)?.filter; + return { + kind: 'renameField', + call: new RenameFieldCall(collection, statement.from.field, statement.to.field, filter), + }; +} + +function collectionChangeAdvice( + statement: ResolvedFieldRenameStatement, + from: string, + to: string, +): string { + if (!sameModel(statement.from, statement.to)) { + return `Add --rename ${modelDisplayName(statement.from)}:${modelDisplayName(statement.to)} before this statement, so the collection is renamed with its documents.`; + } + return `A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. ${renameCollectionByHandSteps(statement.from, from, to)} Then change the field and run this statement again.`; +} + +function renameCollectionByHandSteps(model: ModelCoordinate, from: string, to: string): string { + return `With db update, rename the collection in the database yourself, for example db.getCollection(${JSON.stringify(from)}).renameCollection(${JSON.stringify(to)}) in mongosh, then run db update. With migration plan, plan it, passing --delete ${modelDisplayName(model)} to answer its question; in its migration.ts make renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)}) the first operation, imported with the other operations, delete the drop of "${from}", and delete the create of "${to}" if there is one; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate.`; +} + +/** + * How to keep the documents of a model that keeps its name while its collection changes from + * `from` to `to`: no statement can rename that collection, so the user renames it by hand. + */ +export function keepCollectionByHandAdvice( + model: ModelCoordinate, + from: string, + to: string, +): string { + return `To keep its documents, rename collection "${from}" to "${to}" instead; a statement cannot rename the collection of a model that keeps its name. ${renameCollectionByHandSteps(model, from, to)}`; +} + +export interface FieldRemoval { + readonly call: UnsetFieldCall; + readonly subject: MigrationSubject; +} + +function modelCoordinates(contract: ContractWithDomain): readonly ModelCoordinate[] { + return Object.entries(contract.domain?.namespaces ?? {}).flatMap(([namespaceId, namespace]) => + Object.keys(namespace.models).map((model) => ({ + namespaceId: asNamespaceId(namespaceId), + model, + })), + ); +} + +function sameModel(a: ModelCoordinate, b: ModelCoordinate): boolean { + return a.namespaceId === b.namespaceId && a.model === b.model; +} + +/** + * The destination model an origin model became: the one a statement renamed it to, else the one of + * the same name, else the only one stored the same way (same collection, same variant value). + */ +function destinationModelOf( + origin: ModelCoordinate, + input: FieldRemovalInput, +): ModelCoordinate | undefined { + const renamed = input.statements.find( + (statement): statement is ResolvedModelRenameStatement => + statement.entity === 'model' && sameModel(statement.from, origin), + ); + if (renamed !== undefined) return renamed.to; + if (modelAt(input.contract, origin) !== undefined) return origin; + const collection = input.collectionAfterRenames(collectionOf(input.fromContract, origin)); + const filter = JSON.stringify(variantFilter(input.fromContract, origin) ?? null); + const stored = modelCoordinates(input.contract).filter( + (candidate) => + collectionOf(input.contract, candidate) === collection && + JSON.stringify(variantFilter(input.contract, candidate) ?? null) === filter, + ); + const [only] = stored; + return stored.length === 1 ? only : undefined; +} + +interface FieldRemovalInput { + readonly statements: readonly ResolvedMigrationStatement[]; + readonly fromContract: ContractWithDomain; + readonly contract: MongoContract; + readonly collectionAfterRenames: (collection: string) => string; + readonly keptCollections: ReadonlySet; +} + +function renamedByStatement(input: FieldRemovalInput, field: FieldCoordinate): boolean { + return input.statements.some( + (statement) => + statement.entity === 'field' && + sameModel(statement.from, field) && + statement.from.field === field.field, + ); +} + +function storedFieldsOf(contract: ContractWithDomain, coordinate: ModelCoordinate): Set { + const model = modelAt(contract, coordinate); + if (model === undefined) return new Set(); + const base = + model.base === undefined + ? undefined + : modelAt(contract, { + namespaceId: asNamespaceId(model.base.namespace), + model: model.base.model, + }); + return new Set([...Object.keys(model.fields), ...Object.keys(base?.fields ?? {})]); +} + +/** + * The documents of a root model that lose `field`: all of them, or, when some of its destination + * variants now store the field themselves, those whose discriminator value is none of theirs. + * `undefined` when the root model itself still stores it. + */ +function rootDocumentsLosing( + contract: ContractWithDomain, + root: ModelCoordinate, + field: string, +): { readonly filter: Document | undefined } | undefined { + const model = modelAt(contract, root); + if (model === undefined || Object.hasOwn(model.fields, field)) return undefined; + const discriminator = model.discriminator?.field; + const keeping = Object.entries(model.variants ?? {}) + .filter(([variant]) => + Object.hasOwn(modelAt(contract, { ...root, model: variant })?.fields ?? {}, field), + ) + .map(([, entry]) => entry.value) + .sort(); + if (discriminator === undefined || keeping.length === 0) return { filter: undefined }; + return { filter: { [discriminator]: { $nin: keeping } } }; +} + +/** + * Each field an origin model stores that its destination model no longer stores, in a collection + * the plan keeps: removing it is data loss, carried out by unsetting it from the documents that + * hold it, in a collection whose control policy is `managed`. A variant's field is unset only from that variant's documents, unless its destination + * base now stores it. A base field is unset from the documents of every variant that does not now + * store it itself. + */ +export function planFieldRemovals(input: FieldRemovalInput): readonly FieldRemoval[] { + return modelCoordinates(input.fromContract).flatMap((origin) => { + const collection = input.collectionAfterRenames(collectionOf(input.fromContract, origin)); + if (!input.keptCollections.has(collection)) return []; + if (collectionControlPolicy(input.contract, collection) !== 'managed') return []; + const destination = destinationModelOf(origin, input); + if (destination === undefined) return []; + const model = modelAt(input.fromContract, origin); + if (model === undefined) return []; + const variant = variantFilter(input.fromContract, origin); + const losing = (field: string): { readonly filter: Document | undefined } | undefined => { + if (variant !== undefined) { + return storedFieldsOf(input.contract, destination).has(field) + ? undefined + : { filter: variant }; + } + return rootDocumentsLosing(input.contract, destination, field); + }; + return Object.keys(model.fields) + .filter( + (field) => + unrewritableFieldNameReason(field) === undefined && + !renamedByStatement(input, { ...origin, field }), + ) + .sort() + .flatMap((field) => { + const lost = losing(field); + return lost === undefined + ? [] + : [ + { + call: new UnsetFieldCall(collection, field, lost.filter), + subject: { kind: 'field' as const, ...origin, field }, + }, + ]; + }); + }); +} + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function propertyNames(schema: unknown): readonly string[] { + const properties = isRecord(schema) ? schema['properties'] : undefined; + return isRecord(properties) ? Object.keys(properties) : []; +} + +function branchPropertyNames(schema: unknown): readonly string[] { + const branches = isRecord(schema) ? schema['oneOf'] : undefined; + return Array.isArray(branches) ? branches.flatMap(propertyNames) : []; +} + +/** + * Without an origin contract, a field is known removed when the origin validator declares it at the + * top level and the destination validator declares it nowhere. It is named by its storage name. + */ +export function planValidatorFieldRemovals(input: { + readonly origin: MongoSchemaIR; + readonly destination: MongoSchemaIR; + readonly contract: MongoContract; +}): readonly FieldRemoval[] { + return input.origin.collections.flatMap((originCollection) => { + if (collectionControlPolicy(input.contract, originCollection.name) !== 'managed') return []; + const destination = input.destination.collection(originCollection.name); + if (destination?.validator === undefined || originCollection.validator === undefined) return []; + const kept = new Set([ + ...propertyNames(destination.validator?.jsonSchema), + ...branchPropertyNames(destination.validator?.jsonSchema), + ]); + return propertyNames(originCollection.validator.jsonSchema) + .filter((field) => unrewritableFieldNameReason(field) === undefined && !kept.has(field)) + .sort() + .map((field) => ({ + call: new UnsetFieldCall(originCollection.name, field), + subject: { kind: 'storage' as const, name: `${originCollection.name}.${field}` }, + })); + }); +} diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts index 49e30382823a..b7c5e326794d 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/op-factory-call.ts @@ -22,6 +22,7 @@ import type { MigrationOperationClass, } from '@internal/framework-components/control'; import type { + AnyMongoMigrationOperation, CollModOptions, CreateCollectionOptions, CreateIndexOptions, @@ -34,6 +35,7 @@ import type { MongoSchemaIndex, MongoSchemaValidator, } from '@internal/mongo-schema-ir'; +import type { Document } from '@internal/mongo-value'; import { type ImportRequirement, jsonToTsSource, TsExpression } from '@internal/ts-render'; import { ifDefined } from '@internal/utils/defined'; import { @@ -42,6 +44,9 @@ import { createIndex, dropCollection, dropIndex, + renameCollection, + renameField, + unsetField, } from './migration-factories'; export interface CollModMeta { @@ -63,7 +68,7 @@ abstract class OpFactoryCallNode extends TsExpression implements FrameworkOpFact abstract readonly factoryName: string; abstract readonly operationClass: MigrationOperationClass; abstract readonly label: string; - abstract toOp(): MongoMigrationPlanOperation; + abstract toOp(): AnyMongoMigrationOperation; importRequirements(): readonly ImportRequirement[] { return [{ moduleSpecifier: TARGET_MIGRATION_MODULE, symbol: this.factoryName }]; @@ -182,6 +187,88 @@ export class DropCollectionCall extends OpFactoryCallNode { } } +export class RenameCollectionCall extends OpFactoryCallNode { + readonly factoryName = 'renameCollection' as const; + readonly operationClass = 'widening' as const; + readonly from: string; + readonly to: string; + readonly label: string; + + constructor(from: string, to: string) { + super(); + this.from = from; + this.to = to; + this.label = `Rename collection ${from} to ${to}`; + this.freeze(); + } + + toOp(): MongoMigrationPlanOperation { + return renameCollection(this.from, this.to); + } + + renderTypeScript(): string { + return `renameCollection(${jsonToTsSource(this.from)}, ${jsonToTsSource(this.to)})`; + } +} + +function renderFilterOption(filter: Document | undefined): string { + return filter === undefined ? '' : `, ${jsonToTsSource({ filter })}`; +} + +export class RenameFieldCall extends OpFactoryCallNode { + readonly factoryName = 'renameField' as const; + readonly operationClass = 'widening' as const; + readonly collection: string; + readonly from: string; + readonly to: string; + readonly filter: Document | undefined; + readonly label: string; + + constructor(collection: string, from: string, to: string, filter?: Document) { + super(); + this.collection = collection; + this.from = from; + this.to = to; + this.filter = filter; + this.label = this.toOp().label; + this.freeze(); + } + + toOp(): AnyMongoMigrationOperation { + return renameField(this.collection, this.from, this.to, ifDefined('filter', this.filter)); + } + + renderTypeScript(): string { + return `renameField(${jsonToTsSource(this.collection)}, ${jsonToTsSource(this.from)}, ${jsonToTsSource(this.to)}${renderFilterOption(this.filter)})`; + } +} + +export class UnsetFieldCall extends OpFactoryCallNode { + readonly factoryName = 'unsetField' as const; + readonly operationClass = 'destructive' as const; + readonly collection: string; + readonly field: string; + readonly filter: Document | undefined; + readonly label: string; + + constructor(collection: string, field: string, filter?: Document) { + super(); + this.collection = collection; + this.field = field; + this.filter = filter; + this.label = this.toOp().label; + this.freeze(); + } + + toOp(): AnyMongoMigrationOperation { + return unsetField(this.collection, this.field, ifDefined('filter', this.filter)); + } + + renderTypeScript(): string { + return `unsetField(${jsonToTsSource(this.collection)}, ${jsonToTsSource(this.field)}${renderFilterOption(this.filter)})`; + } +} + export class CollModCall extends OpFactoryCallNode { readonly factoryName = 'collMod' as const; readonly collection: string; @@ -216,6 +303,9 @@ export type OpFactoryCall = | DropIndexCall | CreateCollectionCall | DropCollectionCall + | RenameCollectionCall + | RenameFieldCall + | UnsetFieldCall | CollModCall; export function schemaIndexToCreateIndexOptions(index: MongoSchemaIndex): CreateIndexOptions { diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/render-ops.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/render-ops.ts index 49df5ff696f4..2a81affc22b0 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/migrations/render-ops.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/render-ops.ts @@ -1,6 +1,6 @@ -import type { MongoMigrationPlanOperation } from '@internal/mongo-query-ast/control'; +import type { AnyMongoMigrationOperation } from '@internal/mongo-query-ast/control'; import type { OpFactoryCall } from './op-factory-call'; -export function renderOps(calls: ReadonlyArray): MongoMigrationPlanOperation[] { +export function renderOps(calls: ReadonlyArray): AnyMongoMigrationOperation[] { return calls.map((call) => call.toOp()); } diff --git a/packages/3-mongo-target/1-mongo-target/src/core/migrations/same-hash-field-changes.ts b/packages/3-mongo-target/1-mongo-target/src/core/migrations/same-hash-field-changes.ts new file mode 100644 index 000000000000..cad2d5f25a42 --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/src/core/migrations/same-hash-field-changes.ts @@ -0,0 +1,55 @@ +import type { ContractWithDomain } from '@internal/contract/types'; +import { + describeMigrationStatement, + type MigrationPlannerConflict, + type PlanOrigin, + type ResolvedMigrationStatement, +} from '@internal/framework-components/control'; +import type { MongoContract } from '@internal/mongo-contract'; +import type { FieldRemoval } from './mongo-statement-planning'; +import { type OpFactoryCall, RenameFieldCall } from './op-factory-call'; + +const SAME_HASH = 'the migration would start and end at the same contract hash'; + +function why(collection: string): string { + return `The contract gives collection ${JSON.stringify(collection)} no validator, so the names of its fields are not part of the contract's storage hash; a contract written in TypeScript has none yet. A migration that only renames or removes a field would start and end at the same hash, and db migrate refuses such a migration. Use prisma db update for this change instead: it carries the change out with the same --rename or --delete and writes no migration. Or write the migration by hand: run prisma migration new --from , and carry the change out in a dataTransform operation of its migration.ts, since db migrate accepts a migration that starts and ends at the same hash when it runs a dataTransform.`; +} + +/** + * Refuses, for a migration package, each field rename and field removal of a plan whose contract + * storage hash does not change: the collection has no validator in the contract, so its field + * names are not in the hash, and `db migrate` refuses a migration that starts and ends at the same + * hash. `db update` writes no migration and plans without an origin, so it is not refused. + */ +export function sameHashFieldChangeRefusals(input: { + readonly origin: PlanOrigin | null; + readonly fromContract: ContractWithDomain; + readonly contract: MongoContract; + readonly statements: readonly ResolvedMigrationStatement[]; + readonly callOfStatement: readonly (OpFactoryCall | undefined)[]; + readonly removals: readonly FieldRemoval[]; +}): readonly MigrationPlannerConflict[] { + if (input.origin === null || input.origin.storageHash !== input.contract.storage.storageHash) { + return []; + } + const renames = input.statements.flatMap((statement, index): MigrationPlannerConflict[] => { + const call = input.callOfStatement[index]; + if (!(call instanceof RenameFieldCall)) return []; + return [ + { + kind: 'statementRefused', + summary: `Cannot ${describeMigrationStatement(statement, input.fromContract, input.contract)}: ${SAME_HASH}`, + why: why(call.collection), + refusedStatement: statement, + }, + ]; + }); + const removals = input.removals.map( + ({ call }): MigrationPlannerConflict => ({ + kind: 'fieldRemovalRefused', + summary: `Cannot remove field ${JSON.stringify(call.field)} from collection ${JSON.stringify(call.collection)}: ${SAME_HASH}`, + why: why(call.collection), + }), + ); + return [...renames, ...removals]; +} diff --git a/packages/3-mongo-target/1-mongo-target/src/core/mongo-target-errors.ts b/packages/3-mongo-target/1-mongo-target/src/core/mongo-target-errors.ts index 97c6ac4260a6..23fd6000ca22 100644 --- a/packages/3-mongo-target/1-mongo-target/src/core/mongo-target-errors.ts +++ b/packages/3-mongo-target/1-mongo-target/src/core/mongo-target-errors.ts @@ -7,7 +7,10 @@ export type MongoTargetErrorCode = | 'RUNTIME.ENCODE_FAILED' | 'RUNTIME.TYPE_PARAMS_INVALID'; -type MigrationSubcode = 'INVALID_OPERATION_ENTRY' | 'OPERATION_UNSUPPORTED'; +type MigrationSubcode = + | 'FIELD_NAME_UNSUPPORTED' + | 'INVALID_OPERATION_ENTRY' + | 'OPERATION_UNSUPPORTED'; export function mongoTargetError( code: MongoTargetErrorCode, diff --git a/packages/3-mongo-target/1-mongo-target/src/exports/control.ts b/packages/3-mongo-target/1-mongo-target/src/exports/control.ts index ee1b7b1e2f33..a3705b349bd3 100644 --- a/packages/3-mongo-target/1-mongo-target/src/exports/control.ts +++ b/packages/3-mongo-target/1-mongo-target/src/exports/control.ts @@ -19,8 +19,11 @@ export { CreateIndexCall, DropCollectionCall, DropIndexCall, + RenameCollectionCall, + RenameFieldCall, schemaCollectionToCreateCollectionOptions, schemaIndexToCreateIndexOptions, + UnsetFieldCall, } from '../core/migrations/op-factory-call'; export { PlannerProducedMongoMigration } from '../core/migrations/planner-produced-migration'; export { renderOps } from '../core/migrations/render-ops'; diff --git a/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts b/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts index 363dd0a6c152..2550a3681c6d 100644 --- a/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts +++ b/packages/3-mongo-target/1-mongo-target/src/exports/migration.ts @@ -19,6 +19,9 @@ export { dataTransform, dropCollection, dropIndex, + renameCollection, + renameField, setValidation, + unsetField, validatedCollection, } from '../core/migrations/migration-factories'; diff --git a/packages/3-mongo-target/1-mongo-target/test/data-transform.test.ts b/packages/3-mongo-target/1-mongo-target/test/data-transform.test.ts index 39d5e093a8a9..5a9514385c8f 100644 --- a/packages/3-mongo-target/1-mongo-target/test/data-transform.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/data-transform.test.ts @@ -2,7 +2,7 @@ import { CliStructuredError } from '@internal/errors/control'; import { placeholder } from '@internal/errors/migration'; import type { AnyMongoMigrationOperation, - MongoDataTransformOperation, + MongoDmlOperation, } from '@internal/mongo-query-ast/control'; import { AggregateCommand, @@ -20,11 +20,11 @@ import { serializeMongoOps, } from '../src/core/migrations/mongo-ops-serializer'; -function asDataTransformOp(op: AnyMongoMigrationOperation): MongoDataTransformOperation { +function asDataTransformOp(op: AnyMongoMigrationOperation): MongoDmlOperation { if (op.operationClass !== 'data') { throw new Error('Expected data transform operation'); } - return op as MongoDataTransformOperation; + return op as MongoDmlOperation; } function makePlanMeta() { diff --git a/packages/3-mongo-target/1-mongo-target/test/db-update-statements.test.ts b/packages/3-mongo-target/1-mongo-target/test/db-update-statements.test.ts index 06e6bbbd8c9f..db695cd0ac91 100644 --- a/packages/3-mongo-target/1-mongo-target/test/db-update-statements.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/db-update-statements.test.ts @@ -9,6 +9,7 @@ import type { TargetMigrationsCapability, } from '@internal/framework-components/control'; import { writeContractSnapshot } from '@internal/migration-tools/contract-snapshot-store'; +import { MongoCollection } from '@internal/mongo-contract'; import { MongoSchemaCollection, MongoSchemaIR } from '@internal/mongo-schema-ir'; import { join } from 'pathe'; import { afterEach, describe, expect, it } from 'vitest'; @@ -21,7 +22,16 @@ function contractWithModel(storageHash: string, model: string): Contract { return { target: 'mongo', targetFamily: 'mongo', - storage: { storageHash, namespaces: {} }, + storage: { + storageHash, + namespaces: { + app: { + id: 'app', + kind: 'mongo-namespace', + entries: { collection: { [model]: new MongoCollection({}) } }, + }, + }, + }, domain: { namespaces: { app: { models: { [model]: { fields: {}, relations: {}, storage: {} } } } }, }, @@ -56,7 +66,7 @@ afterEach(async () => { }); describe('db update with statements on MongoDB', () => { - it('fails planning with the statement the Mongo planner refuses', async () => { + it('plans the collection rename the statement states, and nothing that loses data', async () => { const dir = await mkdtemp(join(tmpdir(), 'mongo-db-update-statements-')); tempDirs.push(dir); const migrationsDir = join(dir, 'migrations'); @@ -91,21 +101,21 @@ describe('db update with statements on MongoDB', () => { }); expect(result).toMatchObject({ - ok: false, - failure: { - code: 'PLANNING_FAILED', - conflicts: [ - { - kind: 'statementRefused', - summary: expect.stringContaining('MongoDB does not apply rename statements'), - refusedStatement: { - kind: 'rename', - entity: 'model', - from: { namespaceId: 'app', model: 'Profile' }, - to: { namespaceId: 'app', model: 'User' }, - }, - }, + ok: true, + value: { + appliedStatements: [ + { statement: { entity: 'model', from: { model: 'Profile' } }, operationIndexes: [0] }, ], + dataLoss: [], + plan: { + operations: [ + { + id: 'collection.Profile.rename(User)', + label: 'Rename collection Profile to User', + operationClass: 'widening', + }, + ], + }, }, }); }); diff --git a/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts b/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts new file mode 100644 index 000000000000..1041d1244e88 --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/test/field-rewrites.test.ts @@ -0,0 +1,245 @@ +import { MongoExistsExpr } from '@internal/mongo-query-ast/control'; +import { RawAggregateCommand, RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; +import { describe, expect, it } from 'vitest'; +import { renameField, unsetField } from '../src/core/migrations/migration-factories'; +import { + deserializeMongoOps, + serializeMongoOps, +} from '../src/core/migrations/mongo-ops-serializer'; +import { RenameFieldCall, UnsetFieldCall } from '../src/core/migrations/op-factory-call'; + +const meta = { target: 'mongo', storageHash: '', lane: 'mongo-raw' }; + +function noDocumentHolds(collection: string, match: Record, description: string) { + return { + description, + source: { + collection, + command: new RawAggregateCommand(collection, [{ $match: match }, { $limit: 1 }]), + meta, + }, + filter: MongoExistsExpr.exists('_id'), + expect: 'notExists' as const, + }; +} + +describe('renameField', () => { + it('renames the field on every document that holds it, skipping validation', () => { + const op = renameField('users', 'name', 'fullName'); + + expect(op).toEqual({ + id: 'field.users.name.rename(fullName)', + label: 'Rename field name to fullName in collection users', + operationClass: 'widening', + name: 'field.users.name.rename(fullName)', + precheck: [ + noDocumentHolds( + 'users', + { $and: [{ name: { $exists: true } }, { fullName: { $exists: true } }] }, + 'no document in users holds both name and fullName', + ), + ], + run: [ + { + collection: 'users', + command: new RawUpdateManyCommand( + 'users', + { name: { $exists: true } }, + { $rename: { name: 'fullName' } }, + { bypassDocumentValidation: true }, + ), + meta, + }, + ], + postcheck: [ + noDocumentHolds('users', { name: { $exists: true } }, 'no document in users holds name'), + ], + }); + }); + + it('limits the rename and its checks to documents that match the filter', () => { + const op = renameField('posts', 'title', 'headline', { filter: { type: 'bug' } }); + + expect({ + id: op.id, + label: op.label, + precheck: op.precheck[0]?.source.command, + run: op.run[0]?.command, + postcheck: op.postcheck[0]?.source.command, + }).toEqual({ + id: 'field.posts.title.rename(headline).where({"type":"bug"})', + label: 'Rename field title to headline in collection posts in documents whose type is "bug"', + precheck: new RawAggregateCommand('posts', [ + { + $match: { + $and: [{ type: 'bug' }, { title: { $exists: true } }, { headline: { $exists: true } }], + }, + }, + { $limit: 1 }, + ]), + run: new RawUpdateManyCommand( + 'posts', + { $and: [{ type: 'bug' }, { title: { $exists: true } }] }, + { $rename: { title: 'headline' } }, + { bypassDocumentValidation: true }, + ), + postcheck: new RawAggregateCommand('posts', [ + { $match: { $and: [{ type: 'bug' }, { title: { $exists: true } }] } }, + { $limit: 1 }, + ]), + }); + }); +}); + +describe('unsetField', () => { + it('removes the field from every document that holds it, skipping validation', () => { + expect(unsetField('users', 'nickname')).toEqual({ + id: 'field.users.nickname.unset', + label: 'Remove field nickname from collection users', + operationClass: 'destructive', + name: 'field.users.nickname.unset', + precheck: [], + run: [ + { + collection: 'users', + command: new RawUpdateManyCommand( + 'users', + { nickname: { $exists: true } }, + { $unset: { nickname: '' } }, + { bypassDocumentValidation: true }, + ), + meta, + }, + ], + postcheck: [ + noDocumentHolds( + 'users', + { nickname: { $exists: true } }, + 'no document in users holds nickname', + ), + ], + }); + }); + + it('limits the removal to documents that match the filter', () => { + const op = unsetField('posts', 'score', { filter: { type: 'bug' } }); + + expect({ id: op.id, run: op.run[0]?.command }).toEqual({ + id: 'field.posts.score.unset.where({"type":"bug"})', + run: new RawUpdateManyCommand( + 'posts', + { $and: [{ type: 'bug' }, { score: { $exists: true } }] }, + { $unset: { score: '' } }, + { bypassDocumentValidation: true }, + ), + }); + }); +}); + +describe('field rewrite labels', () => { + it('describe the documents a filter selects in words', () => { + expect([ + unsetField('tasks', 'note', { filter: { type: { $nin: ['bug', 'task'] } } }).label, + unsetField('tasks', 'note', { filter: { type: { $in: ['bug'] } } }).label, + ]).toEqual([ + 'Remove field note from collection tasks in documents whose type is none of "bug", "task"', + 'Remove field note from collection tasks in documents matching {"type":{"$in":["bug"]}}', + ]); + }); +}); + +describe('field names a rewrite refuses', () => { + const refusals = [ + { field: 'a.b', reason: 'contains "."' }, + { field: '$a', reason: 'starts with "$"' }, + { field: '_id', reason: 'is _id' }, + { field: '', reason: 'is empty' }, + ]; + + for (const { field, reason } of refusals) { + it(`refuses a field name that ${reason}`, () => { + const expected = expect.objectContaining({ + code: 'MIGRATION.FIELD_NAME_UNSUPPORTED', + meta: expect.objectContaining({ collection: 'users', field }), + }); + expect(() => renameField('users', field, 'x')).toThrow(expected); + expect(() => renameField('users', 'x', field)).toThrow(expected); + expect(() => unsetField('users', field)).toThrow(expected); + expect(() => new RenameFieldCall('users', field, 'x')).toThrow(expected); + expect(() => new UnsetFieldCall('users', field)).toThrow(expected); + }); + } + + it('refuses renaming a field to its own name as an invalid operation, not an unsupported name', () => { + expect(() => renameField('users', 'name', 'name')).toThrow( + expect.objectContaining({ + code: 'MIGRATION.INVALID_OPERATION_ENTRY', + meta: expect.objectContaining({ + context: 'renameField', + collection: 'users', + field: 'name', + reason: 'the new name is the old name', + }), + }), + ); + }); +}); + +describe('field rewrite calls', () => { + it('RenameFieldCall is widening, renders renameField(...) and lowers to renameField', () => { + const plain = new RenameFieldCall('users', 'name', 'fullName'); + const filtered = new RenameFieldCall('posts', 'title', 'headline', { type: 'bug' }); + + expect({ + frozen: Object.isFrozen(plain), + operationClass: plain.operationClass, + label: plain.label, + plain: plain.renderTypeScript(), + filtered: filtered.renderTypeScript(), + imports: plain.importRequirements(), + }).toEqual({ + frozen: true, + operationClass: 'widening', + label: 'Rename field name to fullName in collection users', + plain: 'renameField("users", "name", "fullName")', + filtered: 'renameField("posts", "title", "headline", { filter: { type: "bug" } })', + imports: [{ moduleSpecifier: '@internal/target-mongo/migration', symbol: 'renameField' }], + }); + expect(filtered.toOp()).toEqual( + renameField('posts', 'title', 'headline', { filter: { type: 'bug' } }), + ); + }); + + it('UnsetFieldCall is destructive, renders unsetField(...) and lowers to unsetField', () => { + const plain = new UnsetFieldCall('users', 'nickname'); + const filtered = new UnsetFieldCall('posts', 'score', { type: 'bug' }); + + expect({ + frozen: Object.isFrozen(plain), + operationClass: plain.operationClass, + label: plain.label, + plain: plain.renderTypeScript(), + filtered: filtered.renderTypeScript(), + imports: plain.importRequirements(), + }).toEqual({ + frozen: true, + operationClass: 'destructive', + label: 'Remove field nickname from collection users', + plain: 'unsetField("users", "nickname")', + filtered: 'unsetField("posts", "score", { filter: { type: "bug" } })', + imports: [{ moduleSpecifier: '@internal/target-mongo/migration', symbol: 'unsetField' }], + }); + expect(filtered.toOp()).toEqual(unsetField('posts', 'score', { filter: { type: 'bug' } })); + }); +}); + +describe('field rewrite serialization', () => { + it('round-trips both rewrites with their classes, told apart from DDL by their shape', () => { + const ops = [ + renameField('posts', 'title', 'headline', { filter: { type: 'bug' } }), + unsetField('users', 'nickname'), + ]; + + expect(deserializeMongoOps(JSON.parse(serializeMongoOps(ops)))).toEqual(ops); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts b/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts index cfdfa6f9cdbb..975c3ee6cae4 100644 --- a/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/migration-e2e.test.ts @@ -51,6 +51,7 @@ const fixtureConfigSource = [ ' family: mongoFamilyDescriptor,', ' target: mongoTargetDescriptor,', ' adapter: mongoAdapter,', + " migrations: { dir: '.' },", '}) };', '', ].join('\n'); @@ -310,6 +311,60 @@ describe('migration file E2E', () => { expect(ops[0].execute[0].command.kind).toBe('collMod'); }); + it('reproduces the renameCollection operation in ops.json', async () => { + const { renderCallsToTypeScript } = await import('../src/core/migrations/render-typescript'); + const { RenameCollectionCall } = await import('../src/core/migrations/op-factory-call'); + const { deserializeMongoOps } = await import('../src/core/migrations/mongo-ops-serializer'); + const call = new RenameCollectionCall('profiles', 'users'); + + const tsSource = renderCallsToTypeScript([call], defaultMeta); + expect(tsSource).toContain('renameCollection("profiles", "users")'); + const resolvedSource = tsSource.replace( + "'@internal/target-mongo/migration'", + `'${factoryExport}'`, + ); + await writeFile(join(tmpDir, 'migration.ts'), resolvedSource); + await writeContractFixtures(defaultMeta); + + const result = await runFile('migration.ts'); + expect({ exitCode: result.exitCode, stderr: result.stderr }).toEqual({ + exitCode: 0, + stderr: '', + }); + + const ops = JSON.parse(await readFile(join(tmpDir, 'ops.json'), 'utf-8')); + expect(deserializeMongoOps(ops)).toEqual([call.toOp()]); + }); + + it('reproduces renameField and unsetField operations in ops.json', async () => { + const { renderCallsToTypeScript } = await import('../src/core/migrations/render-typescript'); + const { RenameFieldCall, UnsetFieldCall } = await import( + '../src/core/migrations/op-factory-call' + ); + const { deserializeMongoOps } = await import('../src/core/migrations/mongo-ops-serializer'); + const calls = [ + new RenameFieldCall('users', 'name', 'fullName'), + new UnsetFieldCall('posts', 'score', { type: 'bug' }), + ]; + + const tsSource = renderCallsToTypeScript(calls, defaultMeta); + const resolvedSource = tsSource.replace( + "'@internal/target-mongo/migration'", + `'${factoryExport}'`, + ); + await writeFile(join(tmpDir, 'migration.ts'), resolvedSource); + await writeContractFixtures(defaultMeta); + + const result = await runFile('migration.ts'); + expect({ exitCode: result.exitCode, stderr: result.stderr }).toEqual({ + exitCode: 0, + stderr: '', + }); + + const ops = JSON.parse(await readFile(join(tmpDir, 'ops.json'), 'utf-8')); + expect(deserializeMongoOps(ops)).toEqual(calls.map((call) => call.toOp())); + }); + it('round-trips describe() meta through TypeScript execution', async () => { const { renderCallsToTypeScript } = await import('../src/core/migrations/render-typescript'); const { DropCollectionCall } = await import('../src/core/migrations/op-factory-call'); diff --git a/packages/3-mongo-target/1-mongo-target/test/migration-factories.test.ts b/packages/3-mongo-target/1-mongo-target/test/migration-factories.test.ts index 88d001d20a0a..38e02af6a911 100644 --- a/packages/3-mongo-target/1-mongo-target/test/migration-factories.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/migration-factories.test.ts @@ -12,6 +12,7 @@ import { MongoAndExpr, MongoFieldFilter, type MongoMigrationPlanOperation, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import { MongoSchemaCollection, MongoSchemaCollectionOptions } from '@internal/mongo-schema-ir'; import { describe, expect, it } from 'vitest'; @@ -21,6 +22,7 @@ import { createIndex, dropCollection, dropIndex, + renameCollection, setValidation, validatedCollection, } from '../src/core/migrations/migration-factories'; @@ -159,6 +161,43 @@ describe('dropIndex', () => { }); }); +describe('renameCollection', () => { + it('renames when the source exists and the target does not, and skips once the target exists without the source', () => { + const sourceIs = (expect: 'exists' | 'notExists') => ({ + description: 'collection profiles exists', + source: new ListCollectionsCommand(), + filter: MongoFieldFilter.eq('name', 'profiles'), + expect, + }); + const targetIs = (expect: 'exists' | 'notExists') => ({ + description: 'collection users exists', + source: new ListCollectionsCommand(), + filter: MongoFieldFilter.eq('name', 'users'), + expect, + }); + + expect(renameCollection('profiles', 'users')).toEqual({ + id: 'collection.profiles.rename(users)', + label: 'Rename collection profiles to users', + operationClass: 'widening', + precheck: [ + sourceIs('exists'), + { ...targetIs('notExists'), description: 'collection users does not exist' }, + ], + execute: [ + { + description: 'rename collection profiles to users', + command: new RenameCollectionCommand('profiles', 'users'), + }, + ], + postcheck: [ + targetIs('exists'), + { ...sourceIs('notExists'), description: 'collection profiles does not exist' }, + ], + }); + }); +}); + describe('createCollection', () => { it('produces correct operation structure', () => { const op = createCollection('users'); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-ops-serializer.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-ops-serializer.test.ts index c127697b9a0a..305bd6ff9993 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-ops-serializer.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-ops-serializer.test.ts @@ -15,6 +15,7 @@ import { } from '@internal/mongo-query-ast/control'; import { isStructuredError } from '@internal/utils/structured-error'; import { describe, expect, it } from 'vitest'; +import { renameCollection } from '../src/core/migrations/migration-factories'; import { deserializeMongoOps, serializeMongoOps, @@ -634,6 +635,14 @@ describe('serializeMongoOps / deserializeMongoOps', () => { expect(cmd.collection).toBe('events'); }); + it('round-trips a renameCollection operation', () => { + const op = renameCollection('profiles', 'users'); + + const deserialized = deserializeMongoOps(JSON.parse(serializeMongoOps([op])) as unknown[]); + + expect(deserialized).toEqual([op]); + }); + it('round-trips collMod command', () => { const op: MongoMigrationPlanOperation = { id: 'validator.users.update', diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts index 46c548a6ad51..82f019d6cb15 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.data-loss.test.ts @@ -4,8 +4,14 @@ import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; import { MongoSchemaCollection, MongoSchemaIndex, MongoSchemaIR } from '@internal/mongo-schema-ir'; import { expectDataLossMatchesDestructive } from '@repo/test-utils/data-loss-expectations'; import { describe, expect, it } from 'vitest'; -import { keepDataByHand, MongoMigrationPlanner } from '../src/core/migrations/mongo-planner'; +import { MongoMigrationPlanner } from '../src/core/migrations/mongo-planner'; import type { PlannerProducedMongoMigration } from '../src/core/migrations/planner-produced-migration'; +import { + contract, + planned, + plan as planStatementsFixture, + schemaCollection, +} from './mongo-statement-fixtures'; const ALL_CLASSES_POLICY: MigrationOperationPolicy = { allowedOperationClasses: ['additive', 'widening', 'destructive', 'data'], @@ -61,9 +67,12 @@ function contractWith( } as unknown as MongoContract; } -function plan(fromContract: MongoContract | null) { +function plan( + fromContract: MongoContract | null, + contract: MongoContract = contractWith({ User: 'users' }, 'to'), +) { const result = new MongoMigrationPlanner().plan({ - contract: contractWith({ User: 'users' }, 'to'), + contract, schema: new MongoSchemaIR([ new MongoSchemaCollection({ name: 'users' }), new MongoSchemaCollection({ @@ -142,23 +151,58 @@ describe('MongoDB planner, data loss', () => { }, ]); }); -}); -describe('keepDataByHand', () => { - it('says to rename the collection a dropped model stores its documents in', () => { - expect( - keepDataByHand( - { kind: 'model', namespaceId: asNamespaceId('__unbound__'), model: 'Event' }, - contractWith({ Event: 'events' }, 'from'), - ), - ).toBe( - 'If it was renamed, keep its documents instead: rename collection "events" by hand on each database before a plan that drops it is applied there, for example with db.getCollection("events").renameCollection("") in mongosh. db update then drops nothing; a migration written by migration plan still drops "events", so remove that operation from its migration.ts, or do not apply it where the collection was renamed.', + it('advises renaming the collection by hand when only the collection of a model that keeps its name changes', () => { + const planned = plan( + contractWith({ User: 'users', Event: 'events' }, 'from'), + contractWith({ User: 'users', Event: 'occasions' }, 'to'), ); + + expect( + planned.dataLoss.map(({ operationIndex, subject, keepDataAdvice }) => ({ + operation: planned.operations[operationIndex], + subject, + keepDataAdvice, + })), + ).toEqual([ + { + operation: 'Drop collection events', + subject: { kind: 'model', namespaceId: asNamespaceId('__unbound__'), model: 'Event' }, + keepDataAdvice: + 'To keep its documents, rename collection "events" to "occasions" instead; a statement cannot rename the collection of a model that keeps its name. With db update, rename the collection in the database yourself, for example db.getCollection("events").renameCollection("occasions") in mongosh, then run db update. With migration plan, plan it, passing --delete Event to answer its question; in its migration.ts make renameCollection("events", "occasions") the first operation, imported with the other operations, delete the drop of "events", and delete the create of "occasions" if there is one; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate.', + }, + ]); }); - it('names a collection no model stores by its name', () => { - expect(keepDataByHand({ kind: 'storage', name: 'audit' }, contractWith({}, 'from'))).toBe( - 'If it was renamed, keep its documents instead: rename collection "audit" by hand on each database before a plan that drops it is applied there, for example with db.getCollection("audit").renameCollection("") in mongosh. db update then drops nothing; a migration written by migration plan still drops "audit", so remove that operation from its migration.ts, or do not apply it where the collection was renamed.', - ); + it('gives no advice when the model of the dropped collection is gone', () => { + const planned = plan(contractWith({ User: 'users', Event: 'events' }, 'from')); + expect(planned.dataLoss.map(({ keepDataAdvice }) => keepDataAdvice)).toEqual([undefined]); + }); + + it('advises renaming the collection by hand when the plan creates the new collection only with its indexes', () => { + const result = planStatementsFixture({ + from: contract( + 'from', + { Event: { collection: 'events', fields: ['_id', 'name'] } }, + { events: { unique: ['name'] } }, + ), + to: contract( + 'to', + { Event: { collection: 'occasions', fields: ['_id', 'name'] } }, + { occasions: { unique: ['name'] } }, + ), + schema: [schemaCollection('events', { unique: ['name'] })], + }); + const { labels, dataLoss } = planned(result); + + expect({ + creates: labels.filter((label) => label.startsWith('Create')), + advice: dataLoss.map(({ keepDataAdvice }) => keepDataAdvice), + }).toEqual({ + creates: ['Create index on occasions (name:1)'], + advice: [ + 'To keep its documents, rename collection "events" to "occasions" instead; a statement cannot rename the collection of a model that keeps its name. With db update, rename the collection in the database yourself, for example db.getCollection("events").renameCollection("occasions") in mongosh, then run db update. With migration plan, plan it, passing --delete Event to answer its question; in its migration.ts make renameCollection("events", "occasions") the first operation, imported with the other operations, delete the drop of "events", and delete the create of "occasions" if there is one; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate.', + ], + }); }); }); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.same-hash-field-changes.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.same-hash-field-changes.test.ts new file mode 100644 index 000000000000..3c5645006321 --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.same-hash-field-changes.test.ts @@ -0,0 +1,95 @@ +import { describe, expect, it } from 'vitest'; +import { + contract, + plan, + planned, + refusal, + renameField, + renameModel, + schemaCollection, +} from './mongo-statement-fixtures'; + +const WHY = + 'The contract gives collection "profiles" no validator, so the names of its fields are not part of the contract\'s storage hash; a contract written in TypeScript has none yet. A migration that only renames or removes a field would start and end at the same hash, and db migrate refuses such a migration. Use prisma db update for this change instead: it carries the change out with the same --rename or --delete and writes no migration. Or write the migration by hand: run prisma migration new --from , and carry the change out in a dataTransform operation of its migration.ts, since db migrate accepts a migration that starts and ends at the same hash when it runs a dataTransform.'; + +const withName = contract( + 'same', + { Profile: { collection: 'profiles', fields: ['_id', 'name', 'nickname'] } }, + { profiles: {} }, +); +const schema = [schemaCollection('profiles')]; +const migrationOrigin = { storageHash: withName.storage.storageHash }; + +describe('MongoMigrationPlanner, field changes that leave the storage hash unchanged', () => { + it('refuses a field rename for a migration package', () => { + const to = contract( + 'same', + { Profile: { collection: 'profiles', fields: ['_id', 'fullName', 'nickname'] } }, + { profiles: {} }, + ); + expect( + refusal( + plan({ + from: withName, + to, + schema, + origin: migrationOrigin, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: + 'Cannot rename field "Profile.name" to "Profile.fullName": the migration would start and end at the same contract hash', + why: WHY, + }, + ]); + }); + + it('refuses a field removal for a migration package', () => { + const to = contract( + 'same', + { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: {} }, + ); + expect(refusal(plan({ from: withName, to, schema, origin: migrationOrigin }))).toEqual([ + { + kind: 'fieldRemovalRefused', + summary: + 'Cannot remove field "nickname" from collection "profiles": the migration would start and end at the same contract hash', + why: WHY, + }, + ]); + }); + + it('plans the same change for db update, which plans without an origin', () => { + const to = contract( + 'same', + { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: {} }, + ); + expect(planned(plan({ from: withName, to, schema })).labels).toEqual([ + 'Remove field nickname from collection profiles', + ]); + }); + + it('plans a model rename that changes the collection, whose name is in the hash', () => { + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'name', 'nickname'] } }, + { users: {} }, + ); + expect( + planned( + plan({ + from: withName, + to, + schema, + origin: migrationOrigin, + statements: [renameModel('Profile', 'User')], + }), + ).labels, + ).toEqual(['Rename collection profiles to users']); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts new file mode 100644 index 000000000000..33f92d3c73fc --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.fields.test.ts @@ -0,0 +1,318 @@ +import type { ControlPolicy } from '@internal/contract/types'; +import { describe, expect, it } from 'vitest'; +import { + contract, + NS, + plan, + planned, + profiles, + profilesSchema, + renameField, + renameModel, + schemaCollection, +} from './mongo-statement-fixtures'; + +describe('MongoMigrationPlanner, field statements', () => { + const fullName = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'fullName'] } }, + { profiles: { unique: ['fullName'], required: ['fullName'] } }, + ); + + it('renames the field between the index drop and the index create, before the validator update', () => { + expect( + planned( + plan({ + from: profiles, + to: fullName, + schema: profilesSchema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ), + ).toEqual({ + labels: [ + 'Drop index on profiles (name:1)', + 'Rename field name to fullName in collection profiles', + 'Create index on profiles (fullName:1)', + 'Update validator on profiles (added: fullName; removed: name)', + ], + classes: ['widening', 'widening', 'additive', 'widening'], + applied: [[1]], + dataLoss: [], + }); + }); + + it('renames the collection first, then the field in the renamed collection', () => { + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'fullName'] } }, + { users: { unique: ['fullName'], required: ['fullName'] } }, + ); + expect( + planned( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [ + renameModel('Profile', 'User'), + renameField('Profile', 'name', 'User', 'fullName'), + ], + }), + ), + ).toMatchObject({ + labels: [ + 'Rename collection profiles to users', + 'Drop index on users (name:1)', + 'Rename field name to fullName in collection users', + 'Create index on users (fullName:1)', + 'Update validator on users (added: fullName; removed: name)', + ], + applied: [[0], [2]], + dataLoss: [], + }); + }); + + it("renames a variant's field only in that variant's documents", () => { + const tasks = (bugField: string) => + contract( + bugField, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type'], + discriminator: 'type', + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: [bugField], base: 'Task' }, + Feature: { collection: 'tasks', fields: ['score'], base: 'Task' }, + }, + { tasks: {} }, + ); + expect( + planned( + plan({ + from: tasks('score'), + to: tasks('severity'), + schema: [schemaCollection('tasks')], + statements: [renameField('Bug', 'score', 'Bug', 'severity')], + }), + ).labels, + ).toEqual([ + 'Rename field score to severity in collection tasks in documents whose type is "bug"', + ]); + }); + + it('plans nothing for a relation renamed to another relation, which documents do not store', () => { + const withRelation = (relation: string) => + contract( + relation, + { Profile: { collection: 'profiles', fields: ['_id', 'name'], relations: [relation] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + planned( + plan({ + from: withRelation('owner'), + to: withRelation('author'), + schema: profilesSchema, + statements: [renameField('Profile', 'owner', 'Profile', 'author')], + }), + ), + ).toEqual({ labels: [], classes: [], applied: [[]], dataLoss: [] }); + }); +}); + +describe('MongoMigrationPlanner, removed fields', () => { + const withoutName = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'email'] } }, + { profiles: { required: ['email'] } }, + ); + const from = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'email', 'name'] } }, + { profiles: { required: ['email', 'name'] } }, + ); + const schema = [schemaCollection('profiles', { required: ['email', 'name'] })]; + + it('unsets a removed field from every document and reports the field as lost', () => { + expect(planned(plan({ from, to: withoutName, schema }))).toEqual({ + labels: [ + 'Remove field name from collection profiles', + 'Update validator on profiles (removed: name)', + ], + classes: ['destructive', 'widening'], + applied: [], + dataLoss: [ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Profile', field: 'name' }, + }, + ], + }); + }); + + it('names the field by its storage name when the origin contract is unknown', () => { + expect(planned(plan({ from: null, to: withoutName, schema })).dataLoss).toEqual([ + { operationIndex: 0, subject: { kind: 'storage', name: 'profiles.name' } }, + ]); + }); + + it('unsets nothing when the collection is dropped', () => { + const none = contract('to', {}, {}); + expect(planned(plan({ from, to: none, schema })).labels).toEqual(['Drop collection profiles']); + }); + + const tasks = ( + seed: string, + baseFields: readonly string[], + bugFields: readonly string[], + featureFields: readonly string[], + ) => + contract( + seed, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type', ...baseFields], + discriminator: 'type', + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: bugFields, base: 'Task' }, + Feature: { collection: 'tasks', fields: featureFields, base: 'Task' }, + }, + { tasks: {} }, + ); + + it("unsets a variant's removed field only from that variant's documents", () => { + expect( + planned( + plan({ + from: tasks('from', [], ['score', 'severity'], ['score']), + to: tasks('to', [], ['severity'], ['score']), + schema: [schemaCollection('tasks')], + }), + ), + ).toMatchObject({ + labels: ['Remove field score from collection tasks in documents whose type is "bug"'], + dataLoss: [ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Bug', field: 'score' }, + }, + ], + }); + }); + + it('unsets a base field moved to one variant from the documents of every other variant', () => { + expect( + planned( + plan({ + from: tasks('from', ['note'], [], []), + to: tasks('to', [], ['note'], []), + schema: [schemaCollection('tasks')], + }), + ), + ).toMatchObject({ + labels: ['Remove field note from collection tasks in documents whose type is not "bug"'], + dataLoss: [ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Task', field: 'note' }, + }, + ], + }); + }); + + it('renames a base field moved to one variant only in the documents of the other variants', () => { + expect( + planned( + plan({ + from: tasks('from', ['note'], [], []), + to: tasks('to', ['remark'], ['note'], []), + schema: [schemaCollection('tasks')], + statements: [renameField('Task', 'note', 'Task', 'remark')], + }), + ), + ).toMatchObject({ + labels: [ + 'Rename field note to remark in collection tasks in documents whose type is not "bug"', + ], + applied: [[0]], + dataLoss: [], + }); + }); + + it('keeps a variant field its base now stores', () => { + expect( + planned( + plan({ + from: tasks('from', [], ['note'], []), + to: tasks('to', ['note'], [], []), + schema: [schemaCollection('tasks')], + }), + ).labels, + ).toEqual([]); + }); + + it('unsets nothing without an origin contract when the destination collection has no validator', () => { + const unvalidated = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'email'] } }, + { profiles: {} }, + ); + expect(planned(plan({ from: null, to: unvalidated, schema })).labels).toEqual([ + 'Remove validator on profiles', + ]); + }); + + it('unsets nothing for a field a statement renames', () => { + const fullName = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'email', 'fullName'] } }, + { profiles: { required: ['email', 'fullName'] } }, + ); + expect( + planned( + plan({ + from, + to: fullName, + schema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ).dataLoss, + ).toEqual([]); + }); +}); + +describe('MongoMigrationPlanner, removed fields in a collection that is not managed', () => { + const schema = [schemaCollection('auth_users', { required: ['email', 'legacy'] })]; + const authUsers = (seed: string, fields: readonly string[], control: ControlPolicy) => + contract( + seed, + { AuthUser: { collection: 'auth_users', fields: ['_id', ...fields] } }, + { auth_users: { required: fields, control } }, + ); + + for (const control of ['external', 'observed', 'tolerated'] as const) { + it(`unsets nothing and asks nothing when the collection is ${control}`, () => { + const result = planned( + plan({ + from: authUsers('from', ['email', 'legacy'], control), + to: authUsers('to', ['email'], control), + schema, + }), + ); + const noOrigin = planned( + plan({ from: null, to: authUsers('to', ['email'], control), schema }), + ); + expect({ + unsets: [...result.labels, ...noOrigin.labels].filter((label) => + label.startsWith('Remove field'), + ), + dataLoss: [...result.dataLoss, ...noOrigin.dataLoss], + }).toEqual({ unsets: [], dataLoss: [] }); + }); + } +}); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts new file mode 100644 index 000000000000..6c0ee354068e --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.refusals.test.ts @@ -0,0 +1,305 @@ +import { describe, expect, it } from 'vitest'; +import { + contract, + plan, + profiles, + profilesSchema, + refusal, + renameField, + renameModel, + schemaCollection, + users, +} from './mongo-statement-fixtures'; + +describe('MongoMigrationPlanner, statements it refuses', () => { + it('refuses renaming _id', () => { + const to = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['key', 'name'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameField('Profile', '_id', 'Profile', 'key')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: 'Cannot rename field "Profile._id" to "Profile.key": it renames _id', + why: 'MongoDB does not let a document change its _id. Leave out this statement.', + }, + ]); + }); + + it('refuses renaming a stored field whose name MongoDB reads as a path, and says to rename it by hand', () => { + const named = (field: string) => + contract( + field, + { Profile: { collection: 'profiles', fields: ['_id', field] } }, + { profiles: {} }, + ); + expect( + refusal( + plan({ + from: named('o.d'), + to: named('od'), + schema: [schemaCollection('profiles')], + statements: [renameField('Profile', 'o.d', 'Profile', 'od')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: + 'Cannot rename field "Profile.o.d" to "Profile.od": the stored field name "o.d" cannot be rewritten', + why: 'MongoDB reads a name containing "." as a path into embedded documents, so a statement cannot move its values. Rename it in the database yourself, or leave out this statement.', + }, + ]); + }); + + it('refuses a stored field name MongoDB reads as a path or an operator', () => { + const to = (field: string) => + contract( + field, + { Profile: { collection: 'profiles', fields: ['_id', field] } }, + { profiles: {} }, + ); + expect( + ['a.b', '$a'].flatMap((field) => + refusal( + plan({ + from: to('name'), + to: to(field), + schema: [schemaCollection('profiles')], + statements: [renameField('Profile', 'name', 'Profile', field)], + }), + ).map(({ summary }) => summary), + ), + ).toEqual([ + 'Cannot rename field "Profile.name" to "Profile.a.b": the stored field name "a.b" cannot be written', + 'Cannot rename field "Profile.name" to "Profile.$a": the stored field name "$a" cannot be written', + ]); + }); + + it('refuses renaming a time-series collection, a collection the schema does not have, and onto a name it already has', () => { + const statements = [renameModel('Profile', 'User')]; + expect( + [ + [schemaCollection('profiles', { unique: ['name'], required: ['name'], timeseries: true })], + [], + [...profilesSchema, schemaCollection('users')], + ].flatMap((schema) => + refusal(plan({ from: profiles, to: users, schema, statements })).map( + ({ summary }) => summary, + ), + ), + ).toEqual([ + 'Cannot rename model "Profile" to "User": collection "profiles" is a time-series collection', + 'Cannot rename model "Profile" to "User": the schema being planned from has no collection "profiles"', + 'Cannot rename model "Profile" to "User": the schema being planned from already has a collection "users"', + ]); + }); + + it('refuses a collection name MongoDB rejects', () => { + const to = (collection: string) => + contract('to', { User: { collection, fields: ['_id', 'name'] } }, { [collection]: {} }); + expect( + ['a$b', 'system.users', 'a\0b'].flatMap((collection) => + refusal( + plan({ + from: profiles, + to: to(collection), + schema: profilesSchema, + statements: [renameModel('Profile', 'User')], + }), + ).map(({ summary }) => summary), + ), + ).toEqual([ + 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "a$b": it contains "$"', + 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "system.users": it starts with "system.", which MongoDB reserves', + 'Cannot rename model "Profile" to "User": MongoDB does not accept the collection name "a\0b": it contains a null character', + ]); + }); + + it('refuses a field stored on one side of the statement only', () => { + const to = contract( + 'to', + { Profile: { collection: 'profiles', fields: ['_id', 'name'], relations: ['owner'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + const from = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'name', 'ownerName'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + refusal( + plan({ + from, + to, + schema: profilesSchema, + statements: [renameField('Profile', 'ownerName', 'Profile', 'owner')], + }), + ).map(({ summary }) => summary), + ).toEqual([ + 'Cannot rename field "Profile.ownerName" to "Profile.owner": the field is stored in documents on one side of the statement only', + ]); + }); + + it('refuses a statement on a collection whose control policy is not managed', () => { + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'name'] } }, + { users: { unique: ['name'], required: ['name'], control: 'external' } }, + ); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameModel('Profile', 'User')], + }), + ).map(({ summary }) => summary), + ).toEqual([ + 'Cannot rename model "Profile" to "User": collection "users" has control policy "external"', + ]); + }); + + it("refuses a field rename when the model's collection changes and no statement renames it", () => { + const to = contract( + 'to', + { Profile: { collection: 'users', fields: ['_id', 'fullName'] } }, + { users: {} }, + ); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + ).map(({ summary }) => summary), + ).toEqual([ + 'Cannot rename field "Profile.name" to "Profile.fullName": the model\'s collection changes from "profiles" to "users", and no statement renames it', + ]); + expect( + refusal( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameField('Profile', 'name', 'Profile', 'fullName')], + }), + )[0]?.why, + ).toBe( + 'A statement cannot rename the collection of a model that keeps its name. Rename the collection in a step of its own first: change only @@map and emit the contract. With db update, rename the collection in the database yourself, for example db.getCollection("profiles").renameCollection("users") in mongosh, then run db update. With migration plan, plan it, passing --delete Profile to answer its question; in its migration.ts make renameCollection("profiles", "users") the first operation, imported with the other operations, delete the drop of "profiles", and delete the create of "users" if there is one; run node on that migration.ts so it writes ops.json again; then apply it with prisma db migrate. Then change the field and run this statement again.', + ); + }); +}); + +describe('MongoMigrationPlanner, a variant rename whose discriminator value changes', () => { + const tasks = (seed: string, variant: string, value: string) => + contract( + seed, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type'], + discriminator: 'type', + variants: { [variant]: value, Feature: 'feature' }, + }, + [variant]: { collection: 'tasks', fields: ['score'], base: 'Task' }, + Feature: { collection: 'tasks', fields: ['priority'], base: 'Task' }, + }, + { tasks: {} }, + ); + + it('is refused, and says to keep the value', () => { + expect( + refusal( + plan({ + from: tasks('from', 'Bug', 'bug'), + to: tasks('to', 'Defect', 'defect'), + schema: [schemaCollection('tasks')], + statements: [renameModel('Bug', 'Defect')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: + 'Cannot rename model "Bug" to "Defect": its discriminator value changes from "bug" to "defect"', + why: 'The documents of "Bug" hold type "bug", and this release does not rewrite a discriminator value, so they would no longer match "Defect" and would fail validation. Keep the value: declare "Defect" with @@base(Task, "bug"). A later release changes a stored value with --convert.', + }, + ]); + }); + + it('plans nothing when the value is kept', () => { + const keep = tasks('to', 'Defect', 'bug'); + const result = plan({ + from: tasks('from', 'Bug', 'bug'), + to: keep, + schema: [schemaCollection('tasks')], + statements: [renameModel('Bug', 'Defect')], + }); + expect( + result.kind === 'success' ? result.appliedStatements[0]?.operationIndexes : result, + ).toEqual([]); + }); +}); + +describe('MongoMigrationPlanner, a field of a time-series collection', () => { + const readings = (seed: string, fields: readonly string[]) => + contract( + seed, + { Reading: { collection: 'readings', fields: ['_id', 'at', ...fields] } }, + { readings: { timeField: 'at' } }, + ); + const schema = [schemaCollection('readings', { timeseries: true })]; + + it('cannot be renamed, and the refusal says to keep its stored name', () => { + expect( + refusal( + plan({ + from: readings('from', ['value']), + to: readings('to', ['reading']), + schema, + statements: [renameField('Reading', 'value', 'Reading', 'reading')], + }), + ), + ).toEqual([ + { + kind: 'statementRefused', + summary: + 'Cannot rename field "Reading.value" to "Reading.reading": collection "readings" is a time-series collection', + why: 'MongoDB updates the documents of a time-series collection only through its metaField, so a statement cannot move the values of a field in them. Keep the stored name: name the field "value" in the contract (with @map("value") in PSL), then run this command again without this statement.', + }, + ]); + }); + + it('plans nothing once the stored name is kept, as the refusal says', () => { + expect( + plan({ from: readings('from', ['value']), to: readings('to', ['value']), schema }).kind, + ).toBe('success'); + }); + + it('cannot be removed, and the refusal says to keep it', () => { + expect( + refusal(plan({ from: readings('from', ['value']), to: readings('to', []), schema })), + ).toEqual([ + { + kind: 'fieldRemovalRefused', + summary: + 'Cannot remove field "value" from collection "readings": it is a time-series collection', + why: 'MongoDB updates the documents of a time-series collection only through its metaField, so the field cannot be removed from the documents that hold it. Keep the field in the contract, as an optional field if new documents leave it out, then run this command again.', + }, + ]); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts index 387c18e25a46..f90594f4ff6a 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.statements.test.ts @@ -1,128 +1,124 @@ -import { asNamespaceId } from '@internal/contract/types'; -import type { - MigrationOperationPolicy, - ResolvedMigrationStatement, -} from '@internal/framework-components/control'; -import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; -import { MongoSchemaCollection, MongoSchemaIR } from '@internal/mongo-schema-ir'; +import { MongoSchemaCollection, MongoSchemaCollectionOptions } from '@internal/mongo-schema-ir'; import { describe, expect, it } from 'vitest'; -import { MongoMigrationPlanner } from '../src/core/migrations/mongo-planner'; +import { + contract, + plan, + planned, + profiles, + profilesSchema, + renameModel, + users, +} from './mongo-statement-fixtures'; -const ALL_CLASSES_POLICY: MigrationOperationPolicy = { - allowedOperationClasses: ['additive', 'widening', 'destructive', 'data'], -}; - -const NAMESPACE = asNamespaceId('__unbound__'); - -function contractWithModel(model: string, collection: string): MongoContract { - return { - target: 'mongo', - targetFamily: 'mongo', - profileHash: 'test-profile', - capabilities: {}, - extensions: {}, - meta: {}, - roots: {}, - domain: { - namespaces: { - __unbound__: { - models: { [model]: { fields: {}, relations: {}, storage: { collection } } }, - }, - }, - }, - storage: { - storageHash: `storage-${collection}`, - namespaces: { - __unbound__: { - id: '__unbound__', - kind: 'mongo-namespace', - entries: { collection: { [collection]: new MongoCollection({}) } }, - }, - }, - }, - } as unknown as MongoContract; -} - -const modelRename: ResolvedMigrationStatement = { - kind: 'rename', - entity: 'model', - from: { namespaceId: NAMESPACE, model: 'Profile' }, - to: { namespaceId: NAMESPACE, model: 'User' }, -}; +describe('MongoMigrationPlanner, model statements', () => { + it('without a statement, drops the old collection and creates the new one', () => { + expect(planned(plan({ from: profiles, to: users, schema: profilesSchema })).labels).toEqual([ + 'Create collection users', + 'Drop index on profiles (name:1)', + 'Create index on users (name:1)', + 'Drop collection profiles', + ]); + }); -const fieldRename: ResolvedMigrationStatement = { - kind: 'rename', - entity: 'field', - from: { namespaceId: NAMESPACE, model: 'Profile', field: 'name' }, - to: { namespaceId: NAMESPACE, model: 'User', field: 'fullName' }, -}; + it('renames the collection, which keeps its indexes and validator, and reports the rename applied', () => { + expect( + planned( + plan({ + from: profiles, + to: users, + schema: profilesSchema, + statements: [renameModel('Profile', 'User')], + }), + ), + ).toEqual({ + labels: ['Rename collection profiles to users'], + classes: ['widening'], + applied: [[0]], + dataLoss: [], + }); + }); -function plan( - statements: readonly ResolvedMigrationStatement[], - collections: { readonly from: string; readonly to: string } = { from: 'profiles', to: 'users' }, -) { - return new MongoMigrationPlanner().plan({ - contract: contractWithModel('User', collections.to), - schema: new MongoSchemaIR([new MongoSchemaCollection({ name: collections.from })]), - policy: ALL_CLASSES_POLICY, - fromContract: contractWithModel('Profile', collections.from), - origin: null, - statements, - frameworkComponents: [], - snapshotsImportPath: '../../snapshots', + it('plans nothing for a model whose collection keeps its name', () => { + const to = contract( + 'to', + { User: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, + ); + expect( + planned( + plan({ + from: profiles, + to, + schema: profilesSchema, + statements: [renameModel('Profile', 'User')], + }), + ), + ).toEqual({ labels: [], classes: [], applied: [[]], dataLoss: [] }); }); -} +}); -describe('MongoMigrationPlanner with statements', () => { - it('plans the rename as a collection drop and create when no statement is given', async () => { - const result = plan([]); - expect(result.kind).toBe('success'); - if (result.kind !== 'success') throw new Error('Expected success'); - const operations = await Promise.all(result.plan.operations); - expect(operations.map((operation) => operation.id)).toEqual([ - 'collection.users.create', - 'collection.profiles.drop', - ]); - expect(result.appliedStatements).toEqual([]); +describe('MongoMigrationPlanner, time-series options the server fills in', () => { + it('plans nothing when the live collection holds only server defaults the contract leaves out', () => { + const readings = contract( + 'to', + { Reading: { collection: 'readings', fields: ['_id', 'at'] } }, + { readings: { timeField: 'at' } }, + ); + const serverFilled = { + timeField: 'at', + granularity: 'seconds' as const, + bucketMaxSpanSeconds: 3600, + }; + const live = new MongoSchemaCollection({ + name: 'readings', + options: new MongoSchemaCollectionOptions({ timeseries: serverFilled }), + }); + expect(planned(plan({ from: readings, to: readings, schema: [live] })).labels).toEqual([]); }); - it('refuses the first statement, plans nothing, and says how to keep the documents', () => { - expect(plan([modelRename, fieldRename])).toEqual({ - kind: 'failure', - conflicts: [ - { - kind: 'statementRefused', - summary: - 'MongoDB does not apply rename statements in this release, so nothing was planned: rename model "Profile" to "User"', - why: 'MongoDB cannot carry out rename statements in this release. Without the statement, a plan drops collection "profiles" with its documents and creates collection "users". To keep the documents, rename the collection by hand on each database before a plan made without the statement is applied there, for example with db.getCollection("profiles").renameCollection("users") in mongosh. A migration written by migration plan without the statement still drops "profiles" wherever it is applied, so check its operations first.', - refusedStatement: modelRename, - }, - ], + const readings = contract( + 'to', + { Reading: { collection: 'readings', fields: ['_id', 'at'] } }, + { readings: { timeField: 'at' } }, + ); + + function liveReadings(extra: Record) { + const timeseries = { timeField: 'at', bucketMaxSpanSeconds: 3600, ...extra }; + return new MongoSchemaCollection({ + name: 'readings', + options: new MongoSchemaCollectionOptions({ timeseries }), + }); + } + + it('reports a conflict when the contract removes a declared metaField', () => { + const result = plan({ + from: readings, + to: readings, + schema: [liveReadings({ metaField: 'meta', granularity: 'seconds' })], }); + expect(result.kind === 'failure' ? result.conflicts.map(({ summary }) => summary) : []).toEqual( + [expect.stringContaining('timeseries')], + ); }); - it('names a field through the model the destination contract names, and says how to move its values', () => { - const result = plan([fieldRename]); - if (result.kind !== 'failure') throw new Error('Expected failure'); - expect(result.conflicts).toMatchObject([ - { - kind: 'statementRefused', - summary: expect.stringContaining('rename field "User.name" to "User.fullName"'), - why: 'MongoDB cannot carry out rename statements in this release. Without the statement, the documents in collection "profiles" keep their values under "name", and nothing moves them to "fullName". To move them, run these in mongosh on each database before a plan made without the statement is applied there, using the field names as they are stored. First turn off the collection\'s validator, which still requires "name": db.runCommand({ collMod: "profiles", validationLevel: "off" }). Then drop each unique index that includes "name", or the move fails once two documents have lost it, for example db.getCollection("profiles").dropIndex("name_1"). Then move the values: db.getCollection("profiles").updateMany({}, { $rename: { "name": "fullName" } }). Applying the plan then creates the indexes on "fullName" and turns the validator back on.', - refusedStatement: fieldRename, - }, - ]); + it('reports a conflict when the contract removes a granularity other than the server default', () => { + const result = plan({ + from: readings, + to: readings, + schema: [liveReadings({ granularity: 'hours' })], + }); + expect(result.kind).toBe('failure'); }); - it('says a plan without the statement keeps the documents when both models use one collection', () => { - const result = plan([modelRename], { from: 'posts', to: 'posts' }); - if (result.kind !== 'failure') throw new Error('Expected failure'); - expect(result.conflicts).toMatchObject([ - { - kind: 'statementRefused', - why: 'MongoDB cannot carry out rename statements in this release. Both models store their documents in collection "posts", so a plan made without the statement keeps them.', - refusedStatement: modelRename, - }, - ]); + it('plans nothing for custom bucketing set outside the contract', () => { + expect( + planned( + plan({ + from: readings, + to: readings, + schema: [liveReadings({ bucketRoundingSeconds: 3600 })], + }), + ).labels, + ).toEqual([]); }); }); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts index 6f2c1c167d05..397ec529ab10 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-planner.test.ts @@ -782,10 +782,13 @@ describe('MongoMigrationPlanner', () => { }), }), ]); - const [operation] = planSuccess(planner, contract, origin).operations; - expect(operation?.label).toBe( - 'Update validator on items (changed: name; added: note; removed: legacy)', + const labels = planSuccess(planner, contract, origin).operations.map( + (operation) => operation.label, ); + expect(labels).toEqual([ + 'Remove field legacy from collection items', + 'Update validator on items (changed: name; added: note; removed: legacy)', + ]); }); }); @@ -1344,6 +1347,59 @@ describe('MongoMigrationPlanner', () => { expect(result.conflicts.some((c) => c.summary.includes('clusteredIndex'))).toBe(true); }); + function liveClusteredIndex(name: string) { + const clusteredIndex = { v: 2, key: { _id: 1 }, unique: true, name }; + return clusteredIndex; + } + + it('plans nothing for the clustered index fields the server fills in', () => { + const contract = makeContract({ events: { options: { clusteredIndex: {} } } }); + const origin = new MongoSchemaIR([ + new MongoSchemaCollection({ + name: 'events', + options: new MongoSchemaCollectionOptions({ + clusteredIndex: liveClusteredIndex('_id_'), + }), + }), + ]); + const result = planner.plan({ + contract, + schema: origin, + policy: ALL_CLASSES_POLICY, + fromContract: null, + origin: null, + statements: [], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + expect(result.kind).toBe('success'); + }); + + it('reports conflict when the contract removes a declared clustered index name', () => { + const contract = makeContract({ events: { options: { clusteredIndex: {} } } }); + const origin = new MongoSchemaIR([ + new MongoSchemaCollection({ + name: 'events', + options: new MongoSchemaCollectionOptions({ + clusteredIndex: liveClusteredIndex('byId'), + }), + }), + ]); + const result = planner.plan({ + contract, + schema: origin, + policy: ALL_CLASSES_POLICY, + fromContract: null, + origin: null, + statements: [], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + expect( + result.kind === 'failure' ? result.conflicts.map(({ summary }) => summary) : [], + ).toEqual([expect.stringContaining('clusteredIndex')]); + }); + it('reports conflict when timeseries changes', () => { const contract = makeContract({ events: { diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts index d1a6473f3223..6cda4c426540 100644 --- a/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-runner.test.ts @@ -39,8 +39,15 @@ import type { AnyMongoDmlWireCommand, AnyMongoWireCommand, } from '@internal/mongo-wire'; +import { MongoServerError } from 'mongodb'; import { describe, expect, it } from 'vitest'; -import { createCollection, dataTransform } from '../src/core/migrations/migration-factories'; +import { + createCollection, + dataTransform, + renameCollection, + renameField, + unsetField, +} from '../src/core/migrations/migration-factories'; import { serializeMongoOps } from '../src/core/migrations/mongo-ops-serializer'; import { MongoMigrationRunner } from '../src/core/migrations/mongo-runner'; @@ -68,9 +75,14 @@ class EventLog { class StubMongoDriver implements MongoDriver { readonly executeCalls: WireCommand[] = []; private readonly responses: Row[][] = []; + private failure: unknown; constructor(private readonly log?: EventLog) {} + failUpdateManyWith(error: unknown): void { + this.failure = error; + } + queueResponse(rows: Row[]): void { this.responses.push(rows); } @@ -78,8 +90,10 @@ class StubMongoDriver implements MongoDriver { execute(wireCommand: WireCommand): AsyncIterable { this.executeCalls.push(wireCommand); this.log?.record(`dml:${wireCommand.kind}:${wireCommand.collection}`); - const rows = this.responses.shift() ?? []; + const failure = wireCommand.kind === 'updateMany' ? this.failure : undefined; + const rows = failure === undefined ? (this.responses.shift() ?? []) : []; return (async function* () { + if (failure !== undefined) throw failure; for (const row of rows) yield row as R; })(); } @@ -131,6 +145,7 @@ class StubMongoAdapter implements MongoAdapter { class StubInspectionExecutor implements MongoInspectionCommandVisitor> { readonly calls: AnyMongoInspectionCommand[] = []; + collections: Row[] = []; async listIndexes(command: AnyMongoInspectionCommand): Promise { this.calls.push(command); @@ -138,7 +153,7 @@ class StubInspectionExecutor implements MongoInspectionCommandVisitor { this.calls.push(command); - return []; + return this.collections; } } @@ -280,7 +295,7 @@ async function execute( }); } -describe('MongoMigrationRunner.executeDataTransform', () => { +describe('MongoMigrationRunner.executeDmlOperation', () => { it('runs the DML wire command once when there are no checks to gate it', async () => { const harness = makeHarness(); const op = dataTransform('backfill-status', { run: () => makeRunPlan() }); @@ -779,3 +794,83 @@ describe('MongoMigrationRunner - per-edge ledger', () => { expect((ledgerEntries[0]!.operations as unknown[]).length).toBe(1); }); }); + +const COMMAND_ECHO = + 'not authorized on app to execute command { update: "users", ordered: true, bypassDocumentValidation: true, lsid: { id: UUID("0b2c8a4e-6f0d-4a7e-9d6f-2f1d5c3e8a91") }, txnNumber: 1, $clusterTime: { clusterTime: Timestamp(1791552000, 3), signature: { hash: BinData(0, 7F3C1D9E2A4B6C8D0E1F2A3B4C5D6E7F8A9B0C1D), keyId: 7421987654321098753 } }, $db: "app" }'; + +describe('MongoMigrationRunner, a document rewrite without the privilege to skip validation', () => { + const unauthorized = () => + new MongoServerError({ + ok: 0, + code: 13, + codeName: 'Unauthorized', + errmsg: COMMAND_ECHO, + }); + + it('names the bypassDocumentValidation action and the roles that grant it', async () => { + const harness = makeHarness(); + harness.driver.queueResponse([{ _id: 1 }]); + harness.driver.failUpdateManyWith(unauthorized()); + + const result = await execute(harness, [unsetField('users', 'nickname')]); + + expect(JSON.stringify(result.assertNotOk())).not.toContain('clusterTime'); + expect(result.assertNotOk()).toEqual({ + code: 'MIGRATION.EXECUTION_FAILED', + summary: + 'Operation field.users.nickname.unset failed: the database user may not skip document validation', + why: 'Renaming or removing a field rewrites documents with bypassDocumentValidation, because the old validator requires the old field. MongoDB refused it with error 13 (Unauthorized). Grant the user the bypassDocumentValidation action on the database, for example with the dbAdmin or restore role, then run the command again; operations that already ran are skipped.', + meta: { operationId: 'field.users.nickname.unset', mongoCode: 13, codeName: 'Unauthorized' }, + }); + }); + + it('lets any other failure of a data transform through unchanged', async () => { + const harness = makeHarness(); + harness.driver.failUpdateManyWith(unauthorized()); + + await expect( + execute(harness, [dataTransform('backfill-status', { run: () => makeRunPlan() })]), + ).rejects.toBeInstanceOf(MongoServerError); + }); +}); + +describe('MongoMigrationRunner, a field rename stopped because documents already hold the new name', () => { + it('says which collection, how to find the documents, and that the operations before it already ran', async () => { + const harness = makeHarness(); + harness.driver.queueResponse([{ _id: 1 }]); + harness.driver.queueResponse([{ _id: 2 }]); + + const result = await execute(harness, [ + renameField('posts', 'title', 'headline', { filter: { type: 'bug' } }), + ]); + + expect(result.assertNotOk()).toEqual({ + code: 'MIGRATION.PRECHECK_FAILED', + summary: + 'Operation field.posts.title.rename(headline).where({"type":"bug"}) stopped: some documents in collection posts already hold headline as well as title', + why: 'Renaming title to headline would overwrite their headline. Find them with db.getCollection("posts").find({"$and":[{"type":"bug"},{"title":{"$exists":true}},{"headline":{"$exists":true}}]}) in mongosh, and move or remove one of the two values in each. Nothing was renamed. The operations before this one in the plan already ran, such as dropping an index on title; once the documents are fixed, run the command again and the plan finishes from here.', + meta: { + operationId: 'field.posts.title.rename(headline).where({"type":"bug"})', + name: 'field.posts.title.rename(headline).where({"type":"bug"})', + }, + }); + }); +}); + +describe('MongoMigrationRunner, a collection rename stopped because the new collection exists', () => { + it('says that the new collection exists, names both collections, and that nothing was renamed', async () => { + const harness = makeHarness(); + harness.inspectionExecutor.collections = [{ name: 'events' }, { name: 'occasions' }]; + + const result = await execute(harness, [renameCollection('events', 'occasions')]); + + expect(harness.ddlCalls).toEqual([]); + expect(result.assertNotOk()).toEqual({ + code: 'MIGRATION.PRECHECK_FAILED', + summary: + 'Operation collection.events.rename(occasions) stopped: collection occasions already exists, so collection events cannot be renamed to it', + why: 'Nothing was renamed. If occasions holds nothing you need, drop it, then run the command again; operations that already ran are skipped. If an earlier operation of the same migration created occasions, as a createIndex on occasions does, make renameCollection("events", "occasions") the first operation of its migration.ts, run node on that migration.ts, and then drop occasions and run the command again.', + meta: { operationId: 'collection.events.rename(occasions)' }, + }); + }); +}); diff --git a/packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts b/packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts new file mode 100644 index 000000000000..405b7899ac17 --- /dev/null +++ b/packages/3-mongo-target/1-mongo-target/test/mongo-statement-fixtures.ts @@ -0,0 +1,234 @@ +import { asNamespaceId, type ControlPolicy } from '@internal/contract/types'; +import type { + MigrationOperationPolicy, + PlanOrigin, + ResolvedMigrationStatement, +} from '@internal/framework-components/control'; +import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; +import { + MongoSchemaCollection, + MongoSchemaCollectionOptions, + MongoSchemaIndex, + MongoSchemaIR, + MongoSchemaValidator, +} from '@internal/mongo-schema-ir'; +import { MongoMigrationPlanner } from '../src/core/migrations/mongo-planner'; +import type { PlannerProducedMongoMigration } from '../src/core/migrations/planner-produced-migration'; + +export const ALL_CLASSES_POLICY: MigrationOperationPolicy = { + allowedOperationClasses: ['additive', 'widening', 'destructive', 'data'], +}; + +export const NS = asNamespaceId('__unbound__'); + +export interface ModelSpec { + readonly collection: string; + readonly fields: readonly string[]; + readonly relations?: readonly string[]; + readonly base?: string; + readonly discriminator?: string; + readonly variants?: Readonly>; +} + +export interface CollectionSpec { + readonly unique?: readonly string[]; + readonly required?: readonly string[]; + readonly control?: ControlPolicy; + readonly timeField?: string; +} + +export function jsonSchema(fields: readonly string[]) { + return { + bsonType: 'object', + required: ['_id', ...fields], + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ...fields.map((field) => [field, { bsonType: 'string' }]), + ]), + additionalProperties: false, + }; +} + +export function contract( + seed: string, + models: Readonly>, + collections: Readonly>, +): MongoContract { + return { + target: 'mongo', + targetFamily: 'mongo', + profileHash: 'test-profile', + capabilities: {}, + extensions: {}, + meta: {}, + roots: {}, + domain: { + namespaces: { + __unbound__: { + models: Object.fromEntries( + Object.entries(models).map(([name, spec]) => [ + name, + { + fields: Object.fromEntries(spec.fields.map((field) => [field, {}])), + relations: Object.fromEntries( + (spec.relations ?? []).map((relation) => [ + relation, + { + to: { model: 'Other', namespace: '__unbound__' }, + cardinality: 'N:1', + on: { localFields: ['otherId'], targetFields: ['_id'] }, + }, + ]), + ), + storage: { collection: spec.collection }, + ...(spec.base === undefined + ? {} + : { base: { namespace: '__unbound__', model: spec.base } }), + ...(spec.discriminator === undefined + ? {} + : { + discriminator: { field: spec.discriminator }, + variants: Object.fromEntries( + Object.entries(spec.variants ?? {}).map(([variant, value]) => [ + variant, + { value }, + ]), + ), + }), + }, + ]), + ), + }, + }, + }, + storage: { + storageHash: `storage-${seed}`, + namespaces: { + __unbound__: { + id: '__unbound__', + kind: 'mongo-namespace', + entries: { + collection: Object.fromEntries( + Object.entries(collections).map(([name, spec]) => [ + name, + new MongoCollection({ + indexes: (spec.unique ?? []).map((field) => ({ + keys: [{ field, direction: 1 as const }], + unique: true, + })), + ...(spec.required === undefined + ? {} + : { + validator: { + jsonSchema: jsonSchema(spec.required), + validationLevel: 'strict' as const, + validationAction: 'error' as const, + }, + }), + ...(spec.control === undefined ? {} : { control: spec.control }), + ...(spec.timeField === undefined + ? {} + : { options: { timeseries: { timeField: spec.timeField } } }), + }), + ]), + ), + }, + }, + }, + }, + } as unknown as MongoContract; +} + +export function schemaCollection( + name: string, + spec: CollectionSpec & { readonly timeseries?: boolean } = {}, +): MongoSchemaCollection { + return new MongoSchemaCollection({ + name, + indexes: (spec.unique ?? []).map( + (field) => new MongoSchemaIndex({ keys: [{ field, direction: 1 }], unique: true }), + ), + ...(spec.required === undefined + ? {} + : { + validator: new MongoSchemaValidator({ + jsonSchema: jsonSchema(spec.required), + validationLevel: 'strict', + validationAction: 'error', + }), + }), + ...(spec.timeseries + ? { options: new MongoSchemaCollectionOptions({ timeseries: { timeField: 'at' } }) } + : {}), + }); +} + +export function plan(input: { + readonly from: MongoContract | null; + readonly to: MongoContract; + readonly schema: readonly MongoSchemaCollection[]; + readonly statements?: readonly ResolvedMigrationStatement[]; + readonly policy?: MigrationOperationPolicy; + /** The origin a migration package asserts; `db update` plans without one. */ + readonly origin?: PlanOrigin | null; +}) { + return new MongoMigrationPlanner().plan({ + contract: input.to, + schema: new MongoSchemaIR(input.schema), + policy: input.policy ?? ALL_CLASSES_POLICY, + fromContract: input.from, + origin: input.origin ?? null, + statements: input.statements ?? [], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); +} + +export function planned(result: ReturnType) { + if (result.kind !== 'success') throw new Error(JSON.stringify(result.conflicts, null, 2)); + const operations = (result.plan as PlannerProducedMongoMigration).operations; + return { + labels: operations.map((operation) => operation.label), + classes: operations.map((operation) => operation.operationClass), + applied: result.appliedStatements.map((applied) => applied.operationIndexes), + dataLoss: result.dataLoss, + }; +} + +export function refusal(result: ReturnType) { + if (result.kind !== 'failure') throw new Error('Expected the plan to be refused'); + return result.conflicts.map(({ kind, summary, why }) => ({ kind, summary, why })); +} + +export const renameModel = (from: string, to: string): ResolvedMigrationStatement => ({ + kind: 'rename', + entity: 'model', + from: { namespaceId: NS, model: from }, + to: { namespaceId: NS, model: to }, +}); + +export const renameField = ( + fromModel: string, + from: string, + toModel: string, + to: string, +): ResolvedMigrationStatement => ({ + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: fromModel, field: from }, + to: { namespaceId: NS, model: toModel, field: to }, +}); + +export const profiles = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: { unique: ['name'], required: ['name'] } }, +); +export const users = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'name'] } }, + { users: { unique: ['name'], required: ['name'] } }, +); +export const profilesSchema = [ + schemaCollection('profiles', { unique: ['name'], required: ['name'] }), +]; diff --git a/packages/3-mongo-target/1-mongo-target/test/op-factory-call.test.ts b/packages/3-mongo-target/1-mongo-target/test/op-factory-call.test.ts index db2f8cc600ca..3c375d7f9623 100644 --- a/packages/3-mongo-target/1-mongo-target/test/op-factory-call.test.ts +++ b/packages/3-mongo-target/1-mongo-target/test/op-factory-call.test.ts @@ -20,6 +20,7 @@ import { createIndex, dropCollection, dropIndex, + renameCollection, } from '../src/core/migrations/migration-factories'; import { CollModCall, @@ -27,6 +28,7 @@ import { CreateIndexCall, DropCollectionCall, DropIndexCall, + RenameCollectionCall, } from '../src/core/migrations/op-factory-call'; describe('Mongo call classes', () => { @@ -87,6 +89,19 @@ describe('Mongo call classes', () => { expect(call.toOp()).toEqual(dropCollection('users')); }); + it('RenameCollectionCall freezes, labels as widening, and lowers to renameCollection(...)', () => { + const call = new RenameCollectionCall('profiles', 'users'); + + expect(Object.isFrozen(call)).toBe(true); + expect(call).toMatchObject({ + factoryName: 'renameCollection', + operationClass: 'widening', + label: 'Rename collection profiles to users', + }); + + expect(call.toOp()).toEqual(renameCollection('profiles', 'users')); + }); + it('CollModCall defaults operationClass to widening, uses a default label, and lowers to collMod(...)', () => { const call = new CollModCall('users', { validator: { $jsonSchema: { type: 'object' } } }); @@ -183,6 +198,15 @@ describe('Mongo call classes', () => { ]); }); + it('RenameCollectionCall emits the source and target names and imports renameCollection only', () => { + const call = new RenameCollectionCall('profiles', 'users'); + + expect(call.renderTypeScript()).toBe('renameCollection("profiles", "users")'); + expect(call.importRequirements()).toEqual([ + { moduleSpecifier: '@internal/target-mongo/migration', symbol: 'renameCollection' }, + ]); + }); + it('CollModCall omits the trailing meta argument when no meta supplied', () => { const call = new CollModCall('users', { validator: { $jsonSchema: { type: 'object' } } }); diff --git a/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts b/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts index 6d5fee67aab5..a9d5240e7144 100644 --- a/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts +++ b/packages/3-mongo-target/2-mongo-adapter/src/mongo-adapter.ts @@ -26,6 +26,7 @@ import { FindOneAndUpdateWireCommand, InsertManyWireCommand, InsertOneWireCommand, + RenameCollectionWireCommand, UpdateManyWireCommand, UpdateOneWireCommand, } from '@internal/mongo-wire'; @@ -66,6 +67,8 @@ function lowerDdlCommand(command: AnyMongoDdlCommand): AnyMongoDdlWireCommand { return new CreateIndexWireCommand(command.collection, keysToKeySpec(command.keys), command); case 'dropCollection': return new DropCollectionWireCommand(command.collection); + case 'renameCollection': + return new RenameCollectionWireCommand(command.collection, command.to); case 'dropIndex': return new DropIndexWireCommand(command.collection, command.name); case 'collMod': @@ -183,6 +186,7 @@ class MongoAdapterImpl implements MongoAdapter { collection: command.collection, filter: command.filter, update: command.update, + bypassDocumentValidation: command.bypassDocumentValidation ?? false, }; case 'rawDeleteOne': return { kind: 'rawDeleteOne', collection: command.collection, filter: command.filter }; @@ -291,7 +295,13 @@ class MongoAdapterImpl implements MongoAdapter { case 'rawUpdateOne': return new UpdateOneWireCommand(draft.collection, draft.filter, draft.update); case 'rawUpdateMany': - return new UpdateManyWireCommand(draft.collection, draft.filter, draft.update); + return new UpdateManyWireCommand( + draft.collection, + draft.filter, + draft.update, + false, + draft.bypassDocumentValidation, + ); case 'rawDeleteOne': return new DeleteOneWireCommand(draft.collection, draft.filter); case 'rawDeleteMany': diff --git a/packages/3-mongo-target/2-mongo-adapter/test/command-executor.test.ts b/packages/3-mongo-target/2-mongo-adapter/test/command-executor.test.ts index e8df92781691..da158e65a5a1 100644 --- a/packages/3-mongo-target/2-mongo-adapter/test/command-executor.test.ts +++ b/packages/3-mongo-target/2-mongo-adapter/test/command-executor.test.ts @@ -2,7 +2,7 @@ * Behavioral tests for DDL execution via the adapter-owned seam * (`controlAdapter.executeDdl(driver, command)` → `driver.run(wire)`). * - * Each test exercises one of the five DDL command kinds against mongodb-memory-server + * Each test exercises one of the DDL command kinds against mongodb-memory-server * and asserts the resulting database state. */ import { MongoDriverImpl } from '@internal/driver-mongo'; @@ -14,6 +14,7 @@ import { DropIndexCommand, ListCollectionsCommand, ListIndexesCommand, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import { type Db, MongoClient } from 'mongodb'; import { MongoMemoryReplSet } from 'mongodb-memory-server'; @@ -55,6 +56,7 @@ async function execDdl( | CreateIndexCommand | DropCollectionCommand | DropIndexCommand + | RenameCollectionCommand | CollModCommand, ): Promise { const controlAdapter = new MongoControlAdapterImpl(); @@ -294,6 +296,18 @@ describe('dropCollection via executeDdl', () => { }); }); +describe('renameCollection via executeDdl', () => { + it('renames an existing collection', async () => { + await db.createCollection('old_name'); + await execDdl(new RenameCollectionCommand('old_name', 'new_name')); + + const names = ( + await db.listCollections({ name: { $in: ['old_name', 'new_name'] } }).toArray() + ).map((c) => c['name']); + expect(names).toEqual(['new_name']); + }); +}); + describe('collMod via executeDdl', () => { it('applies validator to an existing collection', async () => { await db.createCollection('docs'); diff --git a/packages/3-mongo-target/2-mongo-adapter/test/ddl-lowering-oracle.test.ts b/packages/3-mongo-target/2-mongo-adapter/test/ddl-lowering-oracle.test.ts index 7cb4e03b3b4c..93b17d54d6a5 100644 --- a/packages/3-mongo-target/2-mongo-adapter/test/ddl-lowering-oracle.test.ts +++ b/packages/3-mongo-target/2-mongo-adapter/test/ddl-lowering-oracle.test.ts @@ -4,6 +4,7 @@ import { CreateIndexCommand, DropCollectionCommand, DropIndexCommand, + RenameCollectionCommand, } from '@internal/mongo-query-ast/control'; import { describe, expect, it } from 'vitest'; import { createMongoAdapter } from '../src/mongo-adapter'; @@ -16,6 +17,7 @@ async function lowerCmd( | CreateIndexCommand | DropCollectionCommand | DropIndexCommand + | RenameCollectionCommand | CollModCommand, ) { return adapter.lower({ command }, {}); @@ -232,6 +234,17 @@ describe('DDL lowering oracle — dropCollection', () => { }); }); +describe('DDL lowering oracle — renameCollection', () => { + it('produces the source collection and the new name', async () => { + const wire = await lowerCmd(new RenameCollectionCommand('profiles', 'users')); + expect({ ...wire }).toEqual({ + kind: 'renameCollection', + collection: 'profiles', + to: 'users', + }); + }); +}); + describe('DDL lowering oracle — dropIndex', () => { it('produces {dropIndexes: collection, index: name}', async () => { const wire = await lowerCmd(new DropIndexCommand('users', 'email_1')); diff --git a/packages/3-mongo-target/2-mongo-adapter/test/mongo-adapter.test.ts b/packages/3-mongo-target/2-mongo-adapter/test/mongo-adapter.test.ts index 4f88a3376d68..ea59ee793344 100644 --- a/packages/3-mongo-target/2-mongo-adapter/test/mongo-adapter.test.ts +++ b/packages/3-mongo-target/2-mongo-adapter/test/mongo-adapter.test.ts @@ -297,6 +297,17 @@ describe('MongoAdapter', () => { expect(wire.update).toEqual(update); }); + it('passes bypassDocumentValidation through to the wire command', async () => { + const command = new RawUpdateManyCommand( + 'users', + {}, + { $rename: { name: 'fullName' } }, + { bypassDocumentValidation: true }, + ); + const wire = narrowWire(await adapter.lower(plan('users', command), {}), 'updateMany'); + expect(wire.bypassDocumentValidation).toBe(true); + }); + it('passes pipeline-style update (array) through unchanged', async () => { const filter = { firstName: { $exists: true } }; const update = [{ $set: { fullName: { $concat: ['$firstName', ' ', '$lastName'] } } }]; diff --git a/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts b/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts index f30cc9be13cc..762fc937b6e8 100644 --- a/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts +++ b/packages/3-mongo-target/3-mongo-driver/src/mongo-driver.ts @@ -18,6 +18,7 @@ import type { InsertManyWireCommand, InsertOneResult, InsertOneWireCommand, + RenameCollectionWireCommand, UpdateManyResult, UpdateManyWireCommand, UpdateOneResult, @@ -98,6 +99,8 @@ export class MongoDriverImpl implements MongoDriver { return this.executeCreateIndexCommand(wireCommand); case 'dropCollection': return this.executeDropCollectionCommand(wireCommand); + case 'renameCollection': + return this.executeRenameCollectionCommand(wireCommand); case 'dropIndex': return this.executeDropIndexCommand(wireCommand); case 'collMod': @@ -166,7 +169,10 @@ export class MongoDriverImpl implements MongoDriver { cmd: UpdateManyWireCommand, ): AsyncIterable { const collection = this.db.collection(cmd.collection); - const result = await collection.updateMany(cmd.filter, cmd.update, { upsert: cmd.upsert }); + const result = await collection.updateMany(cmd.filter, cmd.update, { + upsert: cmd.upsert, + ...(cmd.bypassDocumentValidation ? { bypassDocumentValidation: true } : {}), + }); yield { matchedCount: result.matchedCount, modifiedCount: result.modifiedCount, @@ -281,6 +287,10 @@ export class MongoDriverImpl implements MongoDriver { await this.db.collection(cmd.collection).drop(); } + protected async executeRenameCollectionCommand(cmd: RenameCollectionWireCommand): Promise { + await this.db.renameCollection(cmd.collection, cmd.to); + } + protected async executeDropIndexCommand(cmd: DropIndexWireCommand): Promise { await this.db.collection(cmd.collection).dropIndex(cmd.name); } diff --git a/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts b/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts index a25d78a1c54e..c8dc6a39834f 100644 --- a/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts +++ b/packages/3-mongo-target/3-mongo-driver/test/mongo-driver.test.ts @@ -11,6 +11,7 @@ import { FindOneAndUpdateWireCommand, InsertManyWireCommand, InsertOneWireCommand, + RenameCollectionWireCommand, UpdateManyWireCommand, UpdateOneWireCommand, } from '@internal/mongo-wire'; @@ -172,6 +173,38 @@ describe('MongoDriver', () => { describe('updateMany', () => { const col = 'driver_update_many'; + it('skips validation when bypassDocumentValidation is set', async () => { + const db = seedClient.db(dbName); + const driver = MongoDriverImpl.fromDb(db); + const closed = 'driver_update_many_bypass'; + await db.dropCollection(closed).catch(() => undefined); + await db.createCollection(closed, { + validator: { + $jsonSchema: { + bsonType: 'object', + required: ['name'], + properties: { _id: { bsonType: 'objectId' }, name: { bsonType: 'string' } }, + additionalProperties: false, + }, + }, + validationLevel: 'strict', + validationAction: 'error', + }); + await db.collection(closed).insertOne({ name: 'Ada' }); + const rename = { $rename: { name: 'fullName' } }; + + await expect( + collect(driver.execute(new UpdateManyWireCommand(closed, {}, rename))), + ).rejects.toThrow(/Document failed validation/); + await collect(driver.execute(new UpdateManyWireCommand(closed, {}, rename, false, true))); + + const docs = await db + .collection(closed) + .find({}, { projection: { _id: 0 } }) + .toArray(); + expect(docs).toEqual([{ fullName: 'Ada' }]); + }); + it('updates multiple documents and returns counts', async () => { const driver = await createMongoDriver(connectionUri, dbName); try { @@ -457,6 +490,30 @@ describe('MongoDriver', () => { }); }); + describe('renameCollection', () => { + it('renames the collection and keeps its documents', async () => { + const db = seedClient.db(dbName); + const driver = MongoDriverImpl.fromDb(db); + const from = 'driver_rename_from'; + const to = 'driver_rename_to'; + + await db.dropCollection(from).catch(() => undefined); + await db.dropCollection(to).catch(() => undefined); + await db.collection(from).insertOne({ name: 'Ada' }); + + await driver.run(new RenameCollectionWireCommand(from, to)); + + const names = (await db.listCollections({ name: { $in: [from, to] } }).toArray()).map( + (c) => c.name, + ); + const docs = await db + .collection(to) + .find({}, { projection: { _id: 0 } }) + .toArray(); + expect({ names, docs }).toEqual({ names: [to], docs: [{ name: 'Ada' }] }); + }); + }); + describe('dropIndex', () => { it('drops a named index', async () => { const db = seedClient.db(dbName); diff --git a/packages/3-targets/3-targets/postgres/src/core/migrations/planner.ts b/packages/3-targets/3-targets/postgres/src/core/migrations/planner.ts index e68e81317db8..77fcb7e4d882 100644 --- a/packages/3-targets/3-targets/postgres/src/core/migrations/planner.ts +++ b/packages/3-targets/3-targets/postgres/src/core/migrations/planner.ts @@ -86,6 +86,7 @@ import { resolveDdlSchemaForNamespaceStorage } from './resolve-ddl-schema'; import { postgresSchemaTables } from './schema-tables'; import { postgresColumnRenameCall, + postgresTableRenameByHand, postgresTableRenameCall, renameTableByHandStatements, renderRenameTableCall, @@ -474,6 +475,7 @@ export class PostgresMigrationPlanner implements MigrationPlanner<'sql', 'postgr fromContract: options.fromContract, contract: options.contract, statements: options.statements, + byHand: postgresTableRenameByHand(options.contract), }, ), ...(warnings.length > 0 ? { warnings: Object.freeze(warnings) } : {}), @@ -742,8 +744,7 @@ export class PostgresMigrationPlanner implements MigrationPlanner<'sql', 'postgr frameworkComponents: options.frameworkComponents, }), apply: (call) => working.apply(call), - renderTableRename: renderRenameTableCall, - tableRenameByHand: (rename) => renameTableByHandStatements(options.contract, rename), + ...postgresTableRenameByHand(options.contract), }, }); if (!planned.ok) return planned; diff --git a/packages/3-targets/3-targets/postgres/src/core/migrations/table-rename-calls.ts b/packages/3-targets/3-targets/postgres/src/core/migrations/table-rename-calls.ts index 638def676022..2cd3d9c2457f 100644 --- a/packages/3-targets/3-targets/postgres/src/core/migrations/table-rename-calls.ts +++ b/packages/3-targets/3-targets/postgres/src/core/migrations/table-rename-calls.ts @@ -3,6 +3,7 @@ import type { ColumnRename, MigrationOperationPolicy, TableRename, + TableRenameByHand, TableRenameRequest, } from '@internal/family-sql/control'; import type { TargetBoundComponentDescriptor } from '@internal/framework-components/components'; @@ -58,6 +59,14 @@ export function renameTableByHandStatements( ]; } +/** How a user renames a table of `contract` by hand on Postgres. */ +export function postgresTableRenameByHand(contract: Contract): TableRenameByHand { + return { + renderTableRename: renderRenameTableCall, + tableRenameByHand: (rename) => renameTableByHandStatements(contract, rename), + }; +} + function tableNode( schema: PostgresDatabaseSchemaNode, ddlSchema: string, diff --git a/packages/3-targets/3-targets/postgres/test/migrations/operation-subjects.test.ts b/packages/3-targets/3-targets/postgres/test/migrations/operation-subjects.test.ts index edaa5399aceb..dc6c8c955655 100644 --- a/packages/3-targets/3-targets/postgres/test/migrations/operation-subjects.test.ts +++ b/packages/3-targets/3-targets/postgres/test/migrations/operation-subjects.test.ts @@ -5,6 +5,7 @@ import { SqlStorage, StorageTable } from '@internal/sql-contract/types'; import { applicationDomainOf } from '@repo/test-utils'; import { describe, expect, it } from 'vitest'; import { postgresCallSubjects } from '../../src/core/migrations/operation-subjects'; +import { postgresTableRenameByHand } from '../../src/core/migrations/table-rename-calls'; import { PostgresSchema } from '../../src/core/postgres-schema'; const text = { dataType: 'pg/text', codecId: 'pg/text@1', nullable: false }; @@ -70,7 +71,12 @@ function subjectsOf(calls: readonly OpFactoryCall[], fieldEvents: readonly Field contract, fieldEvents: new Map(fieldEvents.map((fieldEvent) => [fieldEvent.call, fieldEvent])), }), - { fromContract: contract, contract, statements: [] }, + { + fromContract: contract, + contract, + byHand: postgresTableRenameByHand(contract), + statements: [], + }, ).dataLoss; } diff --git a/packages/3-targets/3-targets/sqlite/src/core/migrations/planner.ts b/packages/3-targets/3-targets/sqlite/src/core/migrations/planner.ts index 3c8b12fe76c6..5d9ba18a93e3 100644 --- a/packages/3-targets/3-targets/sqlite/src/core/migrations/planner.ts +++ b/packages/3-targets/3-targets/sqlite/src/core/migrations/planner.ts @@ -57,6 +57,7 @@ import { renameTableByHandStatements, renderRenameTableCall, sqliteColumnRenameCall, + sqliteTableRenameByHand, sqliteTableRenameCall, } from './table-rename-calls'; import { createWorkingSchema } from './working-schema'; @@ -264,6 +265,7 @@ export class SqliteMigrationPlanner fromContract: options.fromContract, contract: options.contract, statements: options.statements, + byHand: sqliteTableRenameByHand, }, ), }; @@ -303,8 +305,7 @@ export class SqliteMigrationPlanner frameworkComponents: options.frameworkComponents, }), apply: (call) => working.apply(call), - renderTableRename: renderRenameTableCall, - tableRenameByHand: renameTableByHandStatements, + ...sqliteTableRenameByHand, }, }); if (!planned.ok) return planned; diff --git a/packages/3-targets/3-targets/sqlite/src/core/migrations/table-rename-calls.ts b/packages/3-targets/3-targets/sqlite/src/core/migrations/table-rename-calls.ts index 6d8e07b3cee5..db4901e14515 100644 --- a/packages/3-targets/3-targets/sqlite/src/core/migrations/table-rename-calls.ts +++ b/packages/3-targets/3-targets/sqlite/src/core/migrations/table-rename-calls.ts @@ -1,5 +1,10 @@ import type { Contract } from '@internal/contract/types'; -import type { ColumnRename, TableRename, TableRenameRequest } from '@internal/family-sql/control'; +import type { + ColumnRename, + TableRename, + TableRenameByHand, + TableRenameRequest, +} from '@internal/family-sql/control'; import type { TargetBoundComponentDescriptor } from '@internal/framework-components/components'; import type { SqlStorage } from '@internal/sql-contract/types'; import type { SqlSchemaIR } from '@internal/sql-schema-ir/types'; @@ -20,6 +25,12 @@ export function renderRenameTableCall(rename: TableRenameRequest): string { return new RenameTableCall(rename.from, rename.to, []).renderTypeScript(); } +/** How a user renames a table by hand on SQLite. */ +export const sqliteTableRenameByHand: TableRenameByHand = { + renderTableRename: renderRenameTableCall, + tableRenameByHand: renameTableByHandStatements, +}; + /** * The call that renames a table, carrying as companions a drop and a create under the new name of * each index whose wire name derives from the old table name, since SQLite cannot rename an index. diff --git a/packages/3-targets/3-targets/sqlite/test/migrations/operation-subjects.test.ts b/packages/3-targets/3-targets/sqlite/test/migrations/operation-subjects.test.ts index 3e326cf0cce2..a5a0c59dd49e 100644 --- a/packages/3-targets/3-targets/sqlite/test/migrations/operation-subjects.test.ts +++ b/packages/3-targets/3-targets/sqlite/test/migrations/operation-subjects.test.ts @@ -6,6 +6,7 @@ import { SqlStorage, StorageTable } from '@internal/sql-contract/types'; import { applicationDomainOf } from '@repo/test-utils'; import { describe, expect, it } from 'vitest'; import { sqliteCallSubjects } from '../../src/core/migrations/operation-subjects'; +import { sqliteTableRenameByHand } from '../../src/core/migrations/table-rename-calls'; import { sqliteCreateNamespace } from '../../src/core/sqlite-unbound-database'; const text = { dataType: 'sqlite/text', codecId: 'sqlite/text@1', nullable: false }; @@ -71,7 +72,7 @@ function subjectsOf(calls: readonly OpFactoryCall[], fieldEvents: readonly Field calls, new Map(fieldEvents.map((fieldEvent) => [fieldEvent.call, fieldEvent])), ), - { fromContract: contract, contract, statements: [] }, + { fromContract: contract, contract, byHand: sqliteTableRenameByHand, statements: [] }, ).dataLoss; } diff --git a/packages/3-targets/6-adapters/postgres/test/migrations/render-typescript.roundtrip.test.ts b/packages/3-targets/6-adapters/postgres/test/migrations/render-typescript.roundtrip.test.ts index ad7ac6a3606b..7df3c3738059 100644 --- a/packages/3-targets/6-adapters/postgres/test/migrations/render-typescript.roundtrip.test.ts +++ b/packages/3-targets/6-adapters/postgres/test/migrations/render-typescript.roundtrip.test.ts @@ -93,6 +93,7 @@ const fixtureConfigSource = [ ' family: sql,', ' target: postgres,', ' adapter: postgresAdapter,', + " migrations: { dir: '.' },", '}) };', '', ].join('\n'); diff --git a/packages/3-targets/6-adapters/sqlite/test/migrations/render-typescript.roundtrip.test.ts b/packages/3-targets/6-adapters/sqlite/test/migrations/render-typescript.roundtrip.test.ts index 4b581bf1233f..57bb43d3d5f9 100644 --- a/packages/3-targets/6-adapters/sqlite/test/migrations/render-typescript.roundtrip.test.ts +++ b/packages/3-targets/6-adapters/sqlite/test/migrations/render-typescript.roundtrip.test.ts @@ -75,6 +75,7 @@ const fixtureConfigSource = [ ' family: sql,', ' target: sqlite,', ' adapter: sqliteAdapter,', + " migrations: { dir: '.' },", '}) };', '', ].join('\n'); diff --git a/projects/migration-statements/deferred.md b/projects/migration-statements/deferred.md index e79746a3f208..cf9f00379269 100644 --- a/projects/migration-statements/deferred.md +++ b/projects/migration-statements/deferred.md @@ -12,6 +12,8 @@ A user keeps `model User` and changes `@@map("users")` to `@@map("app_users")`. **Decided 2026-10-07 by Will:** no statement for this case in this project. Slice 2's refusal keeps slice 1's manual steps for both commands. A statement that can name storage objects rather than models and fields is wanted later; Will floated `--rename table/users:app_users` as an illustration only, and the syntax needs a critical design discussion and a survey of established prior art before anything is chosen. +**What depends on it:** `MigrationDataLoss.keepDataAdvice` (framework components, `control/migration-statements.ts`) exists only because this statement is missing. The SQL family (`keepTableAdvice` in `operation-subjects.ts`) and the MongoDB target (`keepCollectionByHandAdvice`, reached from `collectionDrops` in `mongo-planner.ts`) use it to print manual steps for this one case. When the statement lands, the planner answers this case with it, and both advice texts go, with the field if nothing else uses it. Slice 3a (PR #30695) adds `label` to the same type, `MigrationDataLoss`; whichever of the two PRs merges second combines the two changes. + ## A model move across namespaces (`auth.User:public.User`) **Found:** slice 1, dispatch 3 (2026-10-06). Resolves but is refused with `statementRejected`; needs a `set schema` operation with its working-schema step and companion names. Scheduled for slice 3 with the namespace renames (recorded in `plan.md`). diff --git a/projects/migration-statements/manual-qa-reports/2026-10-09-qa-slice-4a.md b/projects/migration-statements/manual-qa-reports/2026-10-09-qa-slice-4a.md new file mode 100644 index 000000000000..ee574b58f786 --- /dev/null +++ b/projects/migration-statements/manual-qa-reports/2026-10-09-qa-slice-4a.md @@ -0,0 +1,136 @@ +# Manual QA — slice 4a, renames and deletes on MongoDB (TML-3478) + +Branch `tml-3478-mongo-renames` at `f55b1bb9e3`. Built CLI (`packages/1-framework/3-tooling/cli/dist/bin.mjs`), one mongodb-memory-server replica set (MongoDB 8.2), one database per scratch project under `wip/qa-4a/`. Terminal scenarios ran in a real pseudo-terminal (140 columns). Each result was checked in the database. + +**No blocker.** No document was lost or broken without a `--delete`. Every printed piece of advice that I followed left the documents intact. + +## Summary + +| # | Scenario | Verdict | Findings | +| - | -------- | ------- | -------- | +| 1 | `migration plan`, no flags, not interactive | ✅ | The refusal names `Profile` first. After `--rename Profile:User` it names `User.name` and `User.nickname` with both flags. Nothing is written. | +| 2 | Same, in a terminal | ✅ | Wrong answers are explained and asked again. `migration.ts` uses `renameCollection`, `dropIndex`, `renameField`, `unsetField`, `createIndex`, `collMod`. | +| 3 | `db migrate`, then ORM update | ✅ | Documents in `users` with `fullName`, `nickname` gone, unique index `fullName_1`, 3 of 3 ORM updates succeed, `db verify --schema-only` clean, next plan empty. | +| 4 | `db update` twice | ⚠️ | F3 | +| 5 | Rename `Bug.score` in a shared collection | ✅ | `Feature` documents keep `score`; 4 of 4 updates succeed. | +| 6 | Move `Task.note` to `Bug` (delete; rename) | ⚠️ | F6. Data result correct both ways. | +| 7 | `db update --db` with no snapshot | ✅ | `--delete profiles.nickname` works. The longer route in the "why" (store a snapshot, then `--rename`) also works, followed as printed. See F8. | +| 8 | Refusals | ⚠️ | F1, F4, F5. `_id`, discriminator change and `system.x` refusals are clear, and their advice works. | +| 9 | Field removed from an `external` collection | ✅ | Nothing asked, nothing removed. A `--rename` there is refused with a clear reason. | +| 10 | Interrupted field rename | ✅ | `db migrate` finishes; no document holds both names. See F2 for a related case. | +| 11 | Upgrade fragments | ⚠️ | Clear for both audiences. The app fragment says a time-series collection is refused; for fields it is not (F1). | +| 12 | Skill: rename a MongoDB field | ✅ | `references/migrations.md` item 8 gives `--rename` in the right form and the `User._id` rule. Same time-series claim as F1. | +| 13 | Exploratory | ⚠️ | F2, F7, F9 | + +## Findings + +### F1 ⚠️ A field rename or removal on a time-series collection is not refused; `db update` fails with an unexpected error + +TypeScript contract, model `Reading` on time-series collection `readings` (`timeField: 'ts'`, `metaField: 'meta'`), field `value` renamed to `reading`. + +```text +$ prisma db update --no-interactive --rename Reading.value:Reading.reading +✘ [CLI.UNEXPECTED] Unexpected error + why: Unexpected error during db update: Cannot perform an update or delete on a time-series collection when querying on a field that is not the metaField 'meta' +→ Check the error message and try again + +$ prisma db update --no-interactive --delete Reading.value +✘ [CLI.UNEXPECTED] Unexpected error + why: Unexpected error during db update: Cannot perform an update or delete on a time-series collection when querying on a field that is not the metaField 'meta' +``` + +Without a flag the command asks about `Reading.value`. So no answer works, and the user cannot remove a field from a time-series model at all. Before this slice that removal went through silently. No documents changed. + +Expected: a planning refusal with a reason, as for the collection rename (`collection "readings" is a time-series collection`). The app fragment and `skills/prisma-8/references/migrations.md` both say statements refuse a time-series collection. This may overlap the pending decision about TypeScript contracts without a validator, but that decision is about `migrate`; this failure is in `db update`. + +### F2 ⚠️ A rename precheck failure does not say what is wrong, and the earlier index drop stays applied + +A document holds both `name` and `fullName` (written outside Prisma). Unique index on `name`. + +```text +$ prisma db update --no-interactive --rename Profile.name:Profile.fullName +✘ [MIGRATION.RUNNER_FAILED] Operation field.profiles.name.rename(fullName) failed during precheck + why: Migration runner failed +→ Inspect the reported conflict, reconcile schema drift if needed, then re-run `prisma db update` +``` + +The data is safe: no value was overwritten. But the message does not say that a document already holds `fullName`, or which one. The JSON adds only `runnerErrorCode: MIGRATION.PRECHECK_FAILED`. The unique index `name_1` was dropped before the precheck failed and is still gone. + +Expected: the reason ("1 document in `profiles` already holds `fullName`") and what to do. + +### F3 ⚠️ A repeated `db update` reports stale statements one at a time + +After a successful `db update --rename Profile:User --rename User.name:User.fullName --delete User.nickname`, the same command again: + +```text +✘ [MIGRATION.STATEMENT_UNRESOLVED] Cannot resolve statement "--rename Profile:User" + why: The origin contract already has "User" and has no "Profile", so this rename has already happened. ... +→ Leave out --rename Profile:User. +``` + +Without it, the next run names `--rename User.name:User.fullName`; without that, `CLI.CONSENT_UNUSED` names `--delete User.nickname`. Three runs to learn the three flags to drop. Each message is clear. Expected: all stale statements in one error. + +### F4 ⚠️ Dotted field name: the rename refusal's advice is incomplete + +PSL accepts `nick String? @map("a.b")`. The ORM cannot write it (`Document failed validation`), so data only gets there from outside Prisma. With a document holding `"a.b": "bobby"` and the map changed to `"c"`: + +```text +$ prisma db update --no-interactive --rename Profile.a.b:Profile.c +✘ [MIGRATION.PLANNING_FAILED] Migration planning failed + why: Cannot rename field "Profile.a.b" to "Profile.c": the stored field name "a.b" cannot be rewritten +→ MongoDB reads a name containing "." as a path into embedded documents, so a statement cannot move its values. Rename it in the database yourself, or leave out this statement. +``` + +If you leave the statement out, the plan updates the validator (`removed: a.b`) without asking. The values stay in the documents, so those documents then fail every update. "Rename it in the database yourself" gives no steps, and `$rename` cannot target a literal dotted name. Low impact because Prisma cannot write such a field. + +### F5 ⚠️ Following the time-series rename advice drops MongoDB's automatic index + +```text +→ MongoDB cannot rename a time-series collection. Leave out this statement. To keep the documents, first create collection "samples" yourself with the same time-series options and copy them into it; the plan then drops "readings" once --delete names the model. +``` + +Followed as printed: created `samples` with the same options, copied 2 documents, ran `db update --delete Reading`. Documents kept. The plan also ran `Drop index on samples (meta:1, ts:1)`. MongoDB creates that index automatically for a time-series collection with a `metaField`. Not data loss, but slower queries after following the advice. + +### F6 ⚠️ The question for a base field moved to a variant shows a raw MongoDB filter + +```text +◆ Remove field note from collection tasks where {"kind":{"$nin":["bug"]}} would lose the values of field "Task.note". (rename Task.note:Task. or delete) +``` + +The same text is in the plan list and in scenario 5 (`where {"kind":"bug"}`). The result is correct: `Feature` documents lose `note` (or get `remark`), `Bug` documents keep it. Expected plain words, for example "in documents that are not `Bug`". + +### F7 ⚠️ Changing only a model's `@@map` offers only `--delete` + +`model Profile { … @@map("profiles") }` changed to `@@map("people")`: + +```text +✘ [CLI.CONSENT_REQUIRED] "Profile" needs a statement, and the session is not interactive. + why: Drop collection profiles would lose the data of model "Profile". +→ Run the command again with --delete Profile +``` + +Followed as printed, this drops the documents (with the user's consent). The way to keep them is in the skill (item 8: hand-edit `renameCollection(...)` into `migration.ts`). I did that: plan with `--delete Profile`, replace the operations with `renameCollection('profiles', 'people')`, self-emit, `db migrate`. Documents kept. Expected: the CLI says how to keep the documents, as the skill does. + +### F8 ⚠️ The no-snapshot "why" packs three numbered steps into one line + +```text + why: Remove field nickname from collection profiles would lose the data in "profiles.nickname", named by its storage name because the origin contract is unknown; --delete loses that data. If it was renamed, keep its data instead: store the snapshot of the contract the database is at, then run this command again and answer with --rename. 1. Put the contract source back ... 2. Run `prisma db update --advance-ref --dry-run` ... 3. Put the new contract source back ... +``` + +The steps are correct (I followed them and `--rename Profile.nickname:Profile.alias` moved the values). They are hard to read as one paragraph of about 900 characters. + +### F9 ⚠️ A plain `system.x` collection gets an unexpected error + +The rename onto `system.x` is refused clearly. But `db update --delete Profile` for the same change fails with: + +```text +✘ [CLI.UNEXPECTED] Unexpected error + why: Unexpected error during db update: Invalid system namespace: s8h.system.x +``` + +No documents lost: the create fails before the drop. Probably outside this slice. + +## Notes outside the slice + +- `db migrate` did not advance the `db` ref (`advancedRef: null`), so the next `migration plan` failed with `MIGRATION.PLAN_ORIGIN_UNKNOWN` until `migration ref set db `. `db update` advances it. +- Without a `package.json` that lists `@prisma/orm-mongo`, `migration.ts` imports `@internal/target-mongo/migration` and cannot self-emit. With one it imports `@prisma/orm-mongo/target/migration`. This came from my scratch setup, not from the slice. diff --git a/projects/migration-statements/slices/mongo-renames/manual-qa.md b/projects/migration-statements/slices/mongo-renames/manual-qa.md new file mode 100644 index 000000000000..f357bb1e9094 --- /dev/null +++ b/projects/migration-statements/slices/mongo-renames/manual-qa.md @@ -0,0 +1,47 @@ +# Manual QA — slice 4a, renames and deletes on MongoDB (TML-3478) + +> **Be the user.** Run the built CLI against a real MongoDB (mongodb-memory-server replica set) in scratch projects. Read every message as someone who has never seen this feature. Judge what the journeys cannot: whether a refusal tells you what to type next, whether the questions make sense, and whether anything loses or breaks documents without you saying so. +> +> **Out of scope.** Do not re-run `pnpm test`, the journeys or CI lints. Decide from what the commands print and what the database holds, not from the source. +> +> **Spec:** `projects/migration-statements/slices/mongo-renames/spec.md` + +## What this script is testing + +**The problem.** MongoDB refused every `--rename` and printed steps to run by hand in `mongosh`. Removing a field asked nothing, and every document that still held the field then failed each update against Prisma's closed validator. + +**What changed.** +- `--rename Profile:User` renames the collection; `--rename User.name:User.fullName` renames the field in every document (only the variant's documents for a variant's field). +- Removing a field is data loss: it is asked about, and `--delete User.nickname` removes it from every document. +- Both `migration plan` and `db update` do this. +- Some renames are refused with a reason and steps: `_id`, dotted names, time-series collections, a discriminator value change, a collection name MongoDB rejects. + +**Audiences.** End users in a terminal and in scripts; agents using the `prisma-8` skill; extension authors reading the upgrade fragment. + +## Scenarios + +| # | Scenario | What it proves | +| - | -------- | -------------- | +| 1 | PSL project with documents. Rename model `Profile` to `User` (collection `profiles` to `users`), rename `name` to `fullName` (unique index), remove `nickname`. `migration plan` with no flags, non-interactive | The refusal names `nickname` and the flags; nothing is written | +| 2 | Same, in a real terminal, answering by typing `rename …` and `delete` | The questions read clearly; a wrong answer is explained and re-asked; the written `migration.ts` reads like hand-written code (`renameCollection`, `renameField`, `unsetField`) | +| 3 | Apply scenario 2's plan with `db migrate`; then update a document through the ORM | Documents under the new names, `nickname` gone, unique index on `fullName`, updates succeed, `db verify --schema-only` clean, a further plan empty | +| 4 | The same change through `db update`, then run it again | No prompt with the flags; the second run fails and says why | +| 5 | Polymorphic collection: base `Task`, variants `Bug` and `Feature` both store `score`. Rename `Bug.score` to `Bug.severity` | `Feature` documents keep `score`; every document accepts an update | +| 6 | Move a base field `Task.note` to `Bug`. Answer `delete`; in a second project answer `rename Task.note:Task.remark` | `Feature` documents lose `note` (or get `remark`); `Bug` documents keep `note` | +| 7 | `db update --db ` with no snapshot, removing a field | The question names `profiles.nickname` and says why; the printed `--delete` works | +| 8 | Each refusal: rename `_id`; a field mapped to `a.b`; a time-series collection; `Bug` renamed with its discriminator value changed; a collection mapped to `system.x` | Each names the reason; each piece of advice, followed as printed, ends in a correct state with no lost documents | +| 9 | A field removed from an `external` collection (TypeScript contract with `defaultControlPolicy: 'external'`) | Nothing asked, nothing removed | +| 10 | Interrupt a field rename half way (kill the process after some documents moved, or move one by hand), then run `db migrate` again | It finishes; no document holds both names | +| 11 | Read the app and extension upgrade fragments as their audiences **(judgement)** | A user knows what changed; an extension author knows every rename and removal | +| 12 | Ask the skill how to rename a MongoDB field **(judgement)** | `skills/prisma-8/references/migrations.md` leads to `--rename` in the right form | +| 13 | Exploratory: 20 minutes | Anything the scenarios missed | + +## Pre-flight + +1. In the worktree, `git status --short` shows no source changes. +2. `mise exec -- pnpm build`. +3. Scratch projects under `wip/qa-4a/`, one per scenario, with a mongodb-memory-server replica set (`examples/mongo-demo` shows the config). Stop every server by PID at the end. + +## Report + +Write `projects/migration-statements/manual-qa-reports/-qa-slice-4a.md`: a summary table with a verdict per scenario (✅ / ⚠️ / 🛑 Blocker), then each finding numbered `F` with the exact command and output. A blocker is anything that loses or breaks documents without the user saying so, or advice that does so when followed. diff --git a/skills/journey-tests/02c-data-transform-placeholder.md b/skills/journey-tests/02c-data-transform-placeholder.md index 07021b221a1d..ea9ba4961bbb 100644 --- a/skills/journey-tests/02c-data-transform-placeholder.md +++ b/skills/journey-tests/02c-data-transform-placeholder.md @@ -14,7 +14,7 @@ - [ ] Emits, plans, observes a `placeholder(...)` in `migration.ts`. - [ ] Replaces the placeholder with `UPDATE user SET displayName = email WHERE displayName IS NULL`. - [ ] Adds a follow-up step to ALTER COLUMN to NOT NULL. -- [ ] Self-emits the migration (`node migrations//migration.ts`). +- [ ] Self-emits the migration (`node /migration.ts`). - [ ] Applies. ## Success criteria diff --git a/skills/journey-tests/02e-hash-mismatch.md b/skills/journey-tests/02e-hash-mismatch.md index 97509dfafcec..f15e7d231679 100644 --- a/skills/journey-tests/02e-hash-mismatch.md +++ b/skills/journey-tests/02e-hash-mismatch.md @@ -23,7 +23,7 @@ Plan a migration. Edit its `migration.ts` (add a no-op comment). Do NOT self-emi - [ ] Reads the envelope, recognises `MIGRATION.HASH_MISMATCH`. - [ ] Names the cause: `migration.ts` edited after the initial emit. -- [ ] Runs `node migrations//migration.ts` (self-emit). +- [ ] Runs `node /migration.ts` (self-emit). - [ ] Re-runs `db migrate`. ## Success criteria diff --git a/skills/journey-tests/02g-diamond-convergence.md b/skills/journey-tests/02g-diamond-convergence.md index 7b0c2b92fd3d..11cd17c195b7 100644 --- a/skills/journey-tests/02g-diamond-convergence.md +++ b/skills/journey-tests/02g-diamond-convergence.md @@ -20,7 +20,7 @@ The 5-step diamond-convergence procedure: - [ ] **2.** Identify the topic-branch migration directory under `migrations/` (e.g. `migrations/app/20241112-add-tags/`) and treat it as the stale plan. Leave it on disk: it is still an edge any database that already applied it needs. No `rm -rf`. - [ ] **3.** Run `contract emit` then `migration plan --from --name ` to plan a fresh edge from the post-merge origin. The graph now has two edges off the old parent; that is a legal shape. If a database (e.g. the dev database) already applied the topic-branch migration, its marker sits at that tip and has no path to the new head: plan a second edge for it with `migration plan --from --name `, or reset that database. - [ ] **4.** Open the stale topic-branch `migration.ts`; port any custom data-transform logic into the new one. -- [ ] **5.** Self-emit (`node migrations/app//migration.ts`). +- [ ] **5.** Self-emit (`node /migration.ts`). ## Success criteria diff --git a/skills/prisma-8/references/debug.md b/skills/prisma-8/references/debug.md index bad5d318bbdd..241c97e13621 100644 --- a/skills/prisma-8/references/debug.md +++ b/skills/prisma-8/references/debug.md @@ -86,11 +86,11 @@ The single source of truth: read the envelope, find the row by `code`, follow th | `CONFIG.MISSING_EXTENSION_PACKS` | `contract emit` (e.g. contract uses `pgvector.Vector(...)` but config does not list the pgvector descriptor) | Add the descriptors named in `meta.missingExtensionPacks` to `extensions` in `prisma.config.ts`. See `references/contract.md`. | | `MIGRATION.PLANNING_FAILED` | `db init`, `db update`, `migration plan` | Inspect `meta.conflicts`. Recovery is per-conflict — chain to `references/migrations.md`. When `migration plan` raises it because the planner produced no operations, the conflict's `why` gives the next step (`migration new --from `, or `db sign` for a database managed with `db update`). | | `CLI.INIT_MISSING_FLAGS` / `CLI.INIT_INVALID_FLAG_VALUE` / `CLI.INIT_REINIT_NEEDS_FORCE` / `CLI.INIT_*` | `prisma orm init` | Re-run with the flags listed in `meta.missingFlags` or `meta.allowed`; a re-init needs `--confirm `. `CLI.INIT_INSTALL_FAILED` (exit 4) and `CLI.INIT_EMIT_FAILED` (exit 5) are findings on a completed scaffold — the files are on disk; fix and re-run the named step. | -| `MIGRATION.UNFILLED_PLACEHOLDER` | `node migrations/app//migration.ts` (self-emit) or `db migrate` | Edit `migration.ts`, replace the `placeholder("")` named by `meta.slot` with a real query closure, self-emit. See `references/migrations.md`. | +| `MIGRATION.UNFILLED_PLACEHOLDER` | `node /migration.ts` (self-emit) or `db migrate` | Edit `migration.ts`, replace the `placeholder("")` named by `meta.slot` with a real query closure, self-emit. See `references/migrations.md`. | | `MIGRATION.FILE_MISSING` | Reading a migration package | Restore from version control or scaffold a fresh package with `migration plan` / `migration new`. | | `MIGRATION.INVALID_DEFAULT_EXPORT` | Loading `migration.ts` | Use `export default class extends Migration { ... }` (or a factory returning `{ operations, targetId, destination }`). See `references/migrations.md`. | | `MIGRATION.DATA_TRANSFORM_CONTRACT_MISMATCH` | Building a data-transform query plan | Pass the same `endContract` reference to both `this.dataTransform(endContract, …)` and the query-builder context. | -| `MIGRATION.HASH_MISMATCH` | Any read of a migration package (`plan`, `list`, `db migrate`) | `ops.json` / `migration.json` were edited without self-emitting. Run `node migrations/app//migration.ts` to re-emit. | +| `MIGRATION.HASH_MISMATCH` | Any read of a migration package (`plan`, `list`, `db migrate`) | `ops.json` / `migration.json` were edited without self-emitting. Run `node /migration.ts` to re-emit. | | `CONTRACT.MARKER_MISSING` | `db verify` (`error` diagnostic, exit 4), runtime startup (warning) | DB has no marker yet. Run `prisma db init --db ` (baseline empty DB), `db update --db ` (apply contract directly), or `db sign --db ` if the schema already matches the contract. | | `CONTRACT.MARKER_MISMATCH` | `db verify` (exit 4), runtime startup (warning) | Marker disagrees with contract hash (`meta.expected` / `meta.actual`). Either migrate forward (`db migrate` / `db update`), or — if the DB is correct after a manual fix-up — `db sign`. See `references/migrations.md`. | | `CONTRACT.TARGET_MISMATCH` | `db verify`, runtime startup | Contract target ≠ config target; align them (see `meta.expected` / `meta.actual`). | diff --git a/skills/prisma-8/references/migrations.md b/skills/prisma-8/references/migrations.md index 758a7d7336ad..d80701ab111a 100644 --- a/skills/prisma-8/references/migrations.md +++ b/skills/prisma-8/references/migrations.md @@ -38,7 +38,7 @@ Once the contract changes, you choose how the change reaches the database. This - `ops.json` — canonical operation list. Content-addressed; `migrationHash` is computed over this. - `migration.ts` — TypeScript authoring source, **framework-rendered** by `migration plan` (or `migration new`). You edit specific holes in it (see *Fill a placeholder* below) and re-emit `ops.json` / `migration.json` by running it. - **Contract snapshots.** `migration.ts` imports its bookend contracts from the shared, content-addressed store at `migrations/snapshots//contract.json` + `contract.d.ts` (`` is the contract's 64-hex storage hash) — not from files inside the migration package. -- **Self-emit.** Running `node migrations/app//migration.ts` regenerates `ops.json` and `migration.json` from the (possibly edited) TS source. This is the only supported way to update an existing migration package after edits. +- **Self-emit.** Running `node /migration.ts` regenerates `ops.json` and `migration.json` from the (possibly edited) TS source. This is the only supported way to update an existing migration package after edits. `` is the directory `migration plan` or `migration new` printed; it sits under the config's `migrations.dir` (by default `migrations/app/`), so the path depends on that setting. The command works from any directory. Without `--config`, the file uses the config whose migrations directory holds it: the nearest `prisma.config.ts` above it, else the one in the working directory; if neither does, it fails with `CLI.MIGRATION_CONFIG_NOT_FOUND`. If you gave the CLI `--config`, pass the same `--config ` to the file. - **`migration.ts` shape.** Framework-rendered. A class `M extends Migration` (from `@internal/postgres/migration` on Postgres, `@prisma/orm-mongo/target/migration` on Mongo — see the framing block below) that assigns the two snapshot imports to `startContractJson` / `endContractJson` (`Start` is `never` and there is no `startContractJson` on a baseline) and has an `operations` getter returning an array of operation values. **On Postgres the operation factories are methods on the base class** (`this.addColumn({...})`, `this.setNotNull({...})`, `this.dataTransform(...)`) taking one options object; free helpers like `col(...)` build the column descriptors they take. **On Mongo they are free factories** (`createIndex(...)`, `dataTransform(...)`) imported beside `Migration`. The file ends with `MigrationCLI.run(import.meta.url, M)` so executing it self-emits. - **`placeholder(slot)`.** A sentinel the planner emits into the rendered `migration.ts` (from the same `.../migration` import as `Migration`) wherever a data transform is needed. Calling `placeholder(...)` at emit time throws `MIGRATION.UNFILLED_PLACEHOLDER` with `meta.slot` naming the hole. The user replaces the `() => placeholder(...)` arrow with a real query-plan closure (Postgres) or fills `dataTransform({ check, run })` sources (Mongo — see *Fill a placeholder*), then self-emits. - **`this.dataTransform(endContract, name, { check, run })`.** The data-transform factory. `check` is a rowset query whose presence-of-any-row signals "work remains"; `run` is one or more mutation queries that perform the backfill. Both are lazy closures returning query-plans built against `endContract`. The runner wraps `check` as `EXISTS(...)` for precheck and `NOT EXISTS(...)` for postcheck, so the same closure asserts both "there is work" and "the work is done". @@ -70,7 +70,7 @@ Treat the rendered import lines as framework-managed on both targets: | `MIGRATION.FILE_MISSING` | Reading a migration package | `migration.ts`, `migration.json`, or `ops.json` is absent. Recover from version control, re-emit via `migration.ts`, or run `prisma migration new` for a fresh one. | | `MIGRATION.INVALID_DEFAULT_EXPORT` | Loading `migration.ts` | The file's default export is not a `Migration` subclass or factory function. Restore the planner-emitted scaffold from version control or re-run `migration plan` for a clean package. | | `MIGRATION.DATA_TRANSFORM_CONTRACT_MISMATCH` | Building a data-transform query plan | The query builder was instantiated with a contract reference different from the `endContract` passed to `this.dataTransform(...)`. Use the `endContract` imported at module scope for both. | -| `MIGRATION.HASH_MISMATCH` *Migration package is corrupt* | `db migrate` (or any read of the package) | `ops.json` / `migration.json` were edited without self-emitting. Run `node migrations/app//migration.ts` to re-emit, then re-run `db migrate`. | +| `MIGRATION.HASH_MISMATCH` *Migration package is corrupt* | `db migrate` (or any read of the package) | `ops.json` / `migration.json` were edited without self-emitting. Run `node /migration.ts` to re-emit, then re-run `db migrate`. | | `CLI.CONSENT_REQUIRED` | `migration plan` / `db update` run with nobody to answer (`--no-interactive`, CI, `--yes`) | The plan would lose data or (on `db update`) widen access. `meta.unanswered` and `nextActions` list each question with the flags that answer it: `--delete `, `--rename :`, `--allow `. Re-run with the ones you mean. `--confirm` and `--yes` answer none of them. | | `CLI.CONSENT_UNUSED` | `migration plan` / `db update` | Every question was answered, and a `--delete` or `--allow` was left over: a typo, a subject the plan does not touch, or a `--confirm` (which consents to nothing). Fix or drop the flag. A typo next to a question nobody answered is reported as `CLI.CONSENT_REQUIRED` instead, which names the flag that matched nothing. On `db update --dry-run` the same mistake is `MIGRATION.STATEMENT_ANSWERS_NO_QUESTION`. | | `MIGRATION.STATEMENT_DID_NOT_RESOLVE_LOSS` | A rename typed at the data-loss prompt | The plan was made again with the rename and still loses the subject's data. Answer with `delete`, or give a rename whose old name is the subject. | @@ -349,7 +349,7 @@ export default M; MigrationCLI.run(import.meta.url, M); ``` -Self-emit the same way: `node migrations/app//migration.ts`. +Self-emit the same way: `node /migration.ts`. ## Workflow — Author a migration by hand @@ -375,12 +375,13 @@ On Postgres the operations are **methods on the `Migration` base class**, each t **Mongo** factories (from `@prisma/orm-mongo/target/migration`): -- Collections: `createCollection`, `dropCollection`, `validatedCollection`, `setValidation`. +- Collections: `createCollection`, `dropCollection`, `renameCollection(from, to)` (keeps the documents, indexes and validator), `validatedCollection`, `setValidation`. - Indexes: `createIndex`, `dropIndex`. +- Document rewrites: `renameField(collection, from, to, { filter? })` and `unsetField(collection, field, { filter? })`, each one `updateMany` that skips validation; `filter` limits them to one variant's documents, such as `{ filter: { type: 'bug' } }`. - Collection options: `collMod`. - Data transforms: `dataTransform(name, { check, run })` (free factory; `check`/`run` use Mongo query-plan shapes). -Self-emit (`node migrations/app//migration.ts`) after each edit. +Self-emit (`node /migration.ts`) after each edit. ## Workflow — Inspect the live schema @@ -455,7 +456,7 @@ pnpm prisma db schema --db $DATABASE_URL --json Fix and re-run `db migrate`: ```bash -node migrations/app//migration.ts +node /migration.ts pnpm prisma db migrate --db $DATABASE_URL ``` @@ -466,7 +467,7 @@ If the failure was an out-of-band side-effect that left external systems half-ch The concept: `migrationHash` is content-addressed. A mismatch means `migration.json`'s stored hash disagrees with the hash recomputed from `ops.json` (and metadata). The cause is almost always: someone edited `migration.ts` and forgot to self-emit. The remediation is to self-emit the offending package. ```bash -node migrations/app//migration.ts +node /migration.ts pnpm prisma db migrate --db $DATABASE_URL ``` @@ -476,7 +477,7 @@ If self-emit itself fails (e.g. the contract has moved on and the operations no The concept: the planner cannot tell whether a dropped model or field was meant to go, or was renamed. So before `migration plan` writes a package, or `db update` applies, the command asks one question per model, field or storage name an operation would lose data from. `db update` also asks one per model whose rows an operation would open up, such as dropping a row-level-security policy. Each question is answered by a statement: -- `--delete ` lets the data go: `--delete Legacy`, `--delete User.nickname`. A subject no model stores is named by its storage name (`public.audit_log` on Postgres, `audit_log` on SQLite) and can only be deleted. +- `--delete ` lets the data go: `--delete Legacy`, `--delete User.nickname`. A subject no model stores is named by its storage name (`public.audit_log` on Postgres, `audit_log` on SQLite, `profiles.nickname` for a MongoDB field when `db update` has no snapshot of the origin contract) and can only be deleted. On MongoDB, removing a field from a model is asked about, and `--delete` removes it from every document that holds it. - `--rename :` keeps it under a new name: `--rename Profile:User`, or for a field `--rename User.nickname:User.handle` (the new name keeps its model). A field of a renamed model is named through the new name (`--delete User.nickname` after `--rename Profile:User`). - `--allow ` (`db update` only) lets an operation that changes who can read or write the model's rows run: one flag per operation, so a policy drop and disabling row-level security on one model take `--allow User --allow User`. @@ -499,7 +500,7 @@ In a terminal the command asks each question in turn; type `delete`, `allow`, or 6. **Aggregate `check` closure in Postgres `this.dataTransform`.** Returning `count(*)` or `bool_and(...)` breaks the precheck/postcheck contract — both sides resolve to constants. Use a rowset shape: `select('id').where().limit(1)`. 7. **Two contract references in one migration.** Building a query plan against a different contract than the one passed to `this.dataTransform(endContract, ...)` raises `MIGRATION.DATA_TRANSFORM_CONTRACT_MISMATCH`. Always import `endContract` once at module scope and use the same reference. 11. **Calling Postgres operations as free functions.** `addColumn('public', 'user', {...})` does not exist as an import; the operations are `this.addColumn({ schema, table, column })` and friends on the `Migration` base class, with `col(...)` building the column. Only `col`, `rawSql`, `placeholder`, `Migration`, and `MigrationCLI` are imported. -8. **Renaming a model or field without a statement (Postgres, SQLite).** The planner cannot tell a rename from a drop and a create, so without a statement it plans a destructive drop+add. State the rename on the command line: `migration plan --name --rename User:Person --rename Person.email:Person.emailAddress`, or the same `--rename` flags on `db update`. A field is named through its model as the new contract names it. The planner then renames the table or column and the constraints and indexes named after it. For a rename statements do not cover, such as a model that keeps its name but changes its `@@map`, hand-edit `migration.ts` with `...this.renameTable(...)` or `...this.renameColumn(...)`, then self-emit. +8. **Renaming a model or field without a statement.** The planner cannot tell a rename from a drop and a create, so without a statement it plans a destructive drop+add. State the rename on the command line: `migration plan --name --rename User:Person --rename Person.email:Person.emailAddress`, or the same `--rename` flags on `db update`. A field is named through its model as the new contract names it. The planner then renames the table or column and the constraints and indexes named after it; on MongoDB it renames the collection, and the field in every document that holds it (only in that variant's documents for a variant's field). On MongoDB a field is named by its stored name: `id @map("_id")` is `User._id`. For a rename statements do not cover, such as a model that keeps its name but changes its `@@map`, hand-edit `migration.ts` with `...this.renameTable(...)` or `...this.renameColumn(...)` (Postgres, SQLite) or `renameCollection(...)` (MongoDB), then self-emit. 9. **Planning with no `db` ref and no `--from` in a project that already has migrations.** The origin falls through to the empty database, which would make the plan a full-create migration; `migration plan` refuses with `MIGRATION.PLAN_ORIGIN_UNKNOWN` rather than writing it. Pick the exit that matches your intent — the error lists them, and `references/migration-model.md` § *The trap* explains which to choose. 10. **Hand-authoring `migration.ts` from a blank file, or rewriting the rendered import line.** Migration files are framework-rendered — let `prisma migration plan` (or `migration new`) render the package, then edit only the holes the framework leaves for you. On Postgres leave the rendered `@internal/postgres/migration` (or `@internal/sqlite/migration`) import path alone; on Mongo leave `@prisma/orm-mongo/target/migration` as rendered. Add symbols to the existing import line rather than introducing new import paths. @@ -508,7 +509,7 @@ In a terminal the command asks each question in turn; type `delete`, `allow`, or - **Runtime-apply migrations.** Prisma 8 doesn't apply pending migrations from your app's startup code (the "Drizzle pattern" for serverless / edge). Workaround: run `prisma db migrate` from your deploy pipeline before the app starts. If you need runtime-apply built-in, file a feature request via the `references/feedback.md` skill. - **Seeds-as-first-class.** Prisma 8 doesn't ship a `prisma db seed` equivalent. Workaround: write a TypeScript script that imports your `db` instance and runs your setup queries; invoke it from `package.json`'s scripts. If you need first-class seeding, file a feature request via the `references/feedback.md` skill. - **Migration squashing.** Prisma 8 doesn't squash older migrations into a baseline. They accumulate; for very large histories, manual baseline-and-truncate is the path. If you need built-in squashing, file a feature request via the `references/feedback.md` skill. -- **Renames beyond models and fields, and renames on MongoDB.** `--rename` covers models and fields on Postgres and SQLite. It does not rename enum values, namespaces or value objects, and does not move a model to another namespace. On MongoDB, `--rename` fails with `MIGRATION.PLANNING_FAILED` and plans nothing. A data-loss question about a dropped collection offers only `--delete` and says how to keep the documents by hand; a migration written by `migration plan` still drops the collection, so remove that operation or do not apply it where the collection was renamed. Workaround: follow the error's advice and make the rename by hand in `mongosh` on each database, before a plan made without `--rename` is applied there: `renameCollection` for a model, and for a field turn off the collection's validator, drop each unique index that includes the field, then `updateMany({}, { $rename: ... })`. Applying the plan then creates the new indexes and turns the validator back on. There is no Mongo rename operation to write in `migration.ts`. If you need one of these, file a feature request via the `references/feedback.md` skill. +- **Renames beyond models and fields.** `--rename` covers models and fields on Postgres, SQLite and MongoDB. It does not rename enum values, namespaces or value objects (on MongoDB, a field inside a value object), and does not move a model to another namespace. On MongoDB it also refuses `_id`, a field whose stored name contains `.` or starts with `$`, and a time-series collection; each refusal (`MIGRATION.PLANNING_FAILED`) says what to do instead. If you need one of these, file a feature request via the `references/feedback.md` skill. ## Graph and history commands @@ -535,7 +536,7 @@ The CLI collects anonymous usage data by default. To opt out, set `PRISMA_DISABL - [ ] For `migration plan`: ran `migration show ` to review before `db migrate`. - [ ] Filled every `placeholder(...)` in `migration.ts` (if any), built against `endContract`. - [ ] `check` closures are rowset queries, not scalar aggregates. -- [ ] Self-emitted (`node migrations/app//migration.ts`) after editing the TS. +- [ ] Self-emitted (`node /migration.ts`) after editing the TS. - [ ] Ran `db migrate` (or `db update`) and saw it complete. - [ ] Used `db verify` only when diagnosing drift — not as a routine post-apply step. - [ ] Did NOT use `db update` against a shared or production database. diff --git a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md index a37d87200f82..cf39ebd8b754 100644 --- a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md +++ b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.14-to-8.0.0-rc.15/instructions.md @@ -820,7 +820,7 @@ The `migration.ts` that `migration plan` writes for a new SQLite table now gives A migration operation that changes an existing default on PostgreSQL checked afterwards only that the column has a default. The old default passes that check, and the runner skips an operation whose check already passes, so the default stayed as it was and verification then failed with `MIGRATION.SCHEMA_VERIFY_FAILED`. Such an operation now has no check afterwards and always runs; setting a default twice changes nothing. -A migration package an earlier version planned keeps the old check in `ops.json`. If one changes a default and you have not applied it, write its `ops.json` again before you apply it with `prisma db migrate`. Either delete the package and run `prisma migration plan` again, or rewrite its `setDefault` call in `migration.ts` to the form [`migration-ts-column-defaults`](#migration-ts-column-defaults) shows and then run the file (`node migration.ts`). Run unchanged, the file stops with `MIGRATION.OPERATION_OPTION_REMOVED`, because its `setDefault` still passes `defaultSql`. `db update` plans again each time, so it needs nothing. +A migration package an earlier version planned keeps the old check in `ops.json`. If one changes a default and you have not applied it, write its `ops.json` again before you apply it with `prisma db migrate`. Either delete the package and run `prisma migration plan` again, or rewrite its `setDefault` call in `migration.ts` to the form [`migration-ts-column-defaults`](#migration-ts-column-defaults) shows and then run the file with node from the directory that holds `prisma.config.ts` (`node /migration.ts`). Run unchanged, the file stops with `MIGRATION.OPERATION_OPTION_REMOVED`, because its `setDefault` still passes `defaultSql`. `db update` plans again each time, so it needs nothing. ## `cli-error-from-caught` diff --git a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md index dce208bdbc40..7dd66236c1aa 100644 --- a/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md +++ b/skills/prisma-8/upgrading/app/upgrades/8.0.0-rc.16-to-8.0.0-rc.17/instructions.md @@ -585,7 +585,7 @@ The planner and the migration factories now class an operation as `destructive` A migration that is already applied needs nothing: `prisma db migrate` applies its `ops.json` as written, and the database ledger records the migration by its hash. -For each migration package that no database has applied yet and whose `migration.ts` calls `setNotNull` (Postgres), or `dropIndex`, `setValidation` or `collMod` without an `operationClass` (MongoDB), run its `migration.ts` again (`node migration.ts`) so that `ops.json` records the new class. This rewrites `ops.json` and the `migrationHash` in `migration.json`. Do not run it again for a package a database has applied: its new hash would no longer match the hash the database's ledger recorded for it. +For each migration package that no database has applied yet and whose `migration.ts` calls `setNotNull` (Postgres), or `dropIndex`, `setValidation` or `collMod` without an `operationClass` (MongoDB), run its `migration.ts` again with node from the directory that holds `prisma.config.ts` (`node /migration.ts`) so that `ops.json` records the new class. This rewrites `ops.json` and the `migrationHash` in `migration.json`. Do not run it again for a package a database has applied: its new hash would no longer match the hash the database's ledger recorded for it. ## `migration-plan-refuses-data-loss` diff --git a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts index 202cc2113ef4..09fce4a4c07e 100644 --- a/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts +++ b/test/integration/test/cli-journeys/delete-statements-migration.mongo.e2e.test.ts @@ -3,12 +3,12 @@ * * Dropping a collection loses its documents, so `migration plan` and `db update` ask what the drop means; `--delete ` answers. The MongoDB planner refuses rename statements in this release. * - * Journey S1 removes `Event` through `migration plan`: refused without `--delete Event`, planned with it, and applied by `migrate`. Journey S2 does the same through `db update`, where `--confirm ` no longer consents. Journey S3 adds a required field with no refusal. Journey S4 gives a rename, which the planner refuses with `statementRefused`. + * Journey S1 removes `Event` through `migration plan`: refused without `--delete Event`, planned with it, and applied by `migrate`. Journey S2 does the same through `db update`, where `--confirm ` no longer consents. Journey S3 adds a required field with no refusal. Journey S4 gives a rename, which renames the collection and keeps its documents. Journey S5 follows, as printed, the refusal of a field rename whose model keeps its name while its collection changes. Journey S6 follows, through `db update`, the question about a collection whose model keeps its name, and keeps its documents. */ import { copyFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; import { rm } from 'node:fs/promises'; -import { join } from 'node:path'; +import { dirname, join } from 'node:path'; import { timeouts } from '@repo/test-utils'; import { MongoClient } from 'mongodb'; import { MongoMemoryReplSet } from 'mongodb-memory-server'; @@ -16,6 +16,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it } from 'vitest'; import { fixtureAppDir } from '../utils/cli-test-helpers'; import { engineError, + followRenameCollectionAdvice, type JourneyContext, latestMigrationDirName, parseJsonOutput, @@ -23,6 +24,7 @@ import { runDbUpdate, runMigrate, runMigrationPlan, + runPrintedSelfEmit, } from '../utils/journey-test-helpers'; const FIXTURES_DIR = join(fixtureAppDir, 'fixtures/mongo-cli-journeys'); @@ -278,13 +280,14 @@ describe('Journeys (MongoDB): statements that consent to data loss', { ]); }); - it('S4: the planner refuses a rename statement', async () => { + it('S4: a rename statement renames the collection and keeps its documents', async () => { const dbName = 'mongo_delete_rename'; const ctx = setupProject(withDatabase(replSet.getUri(), dbName)); created.add(ctx.testDir); await emit(ctx, 'S4.01'); const create = await runDbUpdate(ctx, ['--no-interactive', '--json']); expect(create.exitCode, `S4.02: db update creates the collections: ${create.stderr}`).toBe(0); + await client.db(dbName).collection('events').insertOne({ name: 'launch' }); useFixture(ctx, 'contract-events-renamed.ts'); await emit(ctx, 'S4.03'); @@ -294,12 +297,144 @@ describe('Journeys (MongoDB): statements that consent to data loss', { 'Event:Occasion', '--json', ]); - expect(renamed.exitCode, 'S4.04: the rename is refused').not.toBe(0); - const refusal = engineError(renamed); - expect(refusal?.code, 'S4.04: planning failed').toBe('MIGRATION.PLANNING_FAILED'); - expect(JSON.stringify(refusal?.meta), 'S4.04: the conflict is the refused statement').toContain( - 'statementRefused', + expect(renamed.exitCode, `S4.04: the rename runs: ${renamed.stderr}`).toBe(0); + expect(await collections(dbName), 'S4.04: the collection is renamed').toEqual([ + 'occasions', + 'users', + ]); + const documents = await client + .db(dbName) + .collection('occasions') + .find({}, { projection: { _id: 0 } }) + .toArray(); + expect(documents, 'S4.04: its documents are kept').toEqual([{ name: 'launch' }]); + }); + + it('S5: the advice for a field rename whose collection changes keeps the documents when followed as printed', async () => { + const dbName = 'mongo_remap_then_rename'; + const ctx = setupProject(withDatabase(replSet.getUri(), dbName)); + created.add(ctx.testDir); + await emit(ctx, 'S5.01'); + const initial = await runMigrationPlan(ctx, ['--name', 'initial']); + expect(initial.exitCode, `S5.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(ctx)).exitCode, 'S5.03: migrate initial').toBe(0); + await client.db(dbName).collection('events').insertOne({ name: 'launch' }); + const origin = latestMigrationDirName(ctx); + + useFixture(ctx, 'contract-events-remapped-title.ts'); + await emit(ctx, 'S5.04'); + const refused = await runMigrationPlan(ctx, [ + '--from', + origin, + '--rename', + 'Event.name:Event.title', + '--json', + ]); + expect(refused.exitCode, 'S5.05: the field rename is refused').not.toBe(0); + expect(JSON.stringify(engineError(refused)?.meta), 'S5.05: the advice').toContain( + 'run node on that migration.ts so it writes ops.json again', + ); + + useFixture(ctx, 'contract-events-remapped.ts'); + await emit(ctx, 'S5.06'); + const asked = await runMigrationPlan(ctx, ['--from', origin, '--json']); + expect(asked.exitCode, 'S5.06: the drop of events needs a statement').toBe(2); + expect( + JSON.stringify(engineError(asked)), + 'S5.06: the question says how to keep the documents', + ).toContain( + 'make renameCollection(\\"events\\", \\"occasions\\") the first operation, imported with the other operations, delete the drop of \\"events\\", and delete the create of \\"occasions\\" if there is one', + ); + const remap = await runMigrationPlan(ctx, [ + '--name', + 'remap', + '--from', + origin, + '--delete', + 'Event', + ]); + expect( + remap.exitCode, + `S5.07: plan the collection change with --delete Event: ${remap.stderr}`, + ).toBe(0); + const remapDir = latestMigrationDirName(ctx); + const migrationTs = join(ctx.testDir, 'migrations/app', remapDir, 'migration.ts'); + const planned = readFileSync(migrationTs, 'utf-8'); + const edited = followRenameCollectionAdvice(planned, 'events', 'occasions'); + expect(edited, `S5.08: migration.ts is edited as the advice says:\n${edited}`).toMatch( + /return \[renameCollection\("events", "occasions"\)/, + ); + expect(edited, `S5.08: the drop and the create are gone:\n${edited}`).not.toMatch( + /dropCollection\(['"]events|createCollection\(['"]occasions/, ); - expect(await collections(dbName), 'S4.04: nothing changed').toEqual(['events', 'users']); + writeFileSync(migrationTs, edited, 'utf-8'); + const emitted = runPrintedSelfEmit(String(engineError(asked)?.why), dirname(migrationTs)); + expect(emitted.exitCode, `S5.09: node on migration.ts, as printed: ${emitted.stderr}`).toBe(0); + const applyRemap = await runMigrate(ctx); + expect(applyRemap.exitCode, `S5.10: migrate: ${applyRemap.stderr}`).toBe(0); + + useFixture(ctx, 'contract-events-remapped-title.ts'); + await emit(ctx, 'S5.11'); + const rename = await runMigrationPlan(ctx, [ + '--name', + 'title', + '--from', + remapDir, + '--rename', + 'Event.name:Event.title', + ]); + expect(rename.exitCode, `S5.12: plan the field rename: ${rename.stderr}`).toBe(0); + const applyRename = await runMigrate(ctx); + expect(applyRename.exitCode, `S5.13: migrate: ${applyRename.stderr}`).toBe(0); + + expect(await collections(dbName), 'S5.14: the collection is renamed').toEqual([ + 'occasions', + 'users', + ]); + const documents = await client + .db(dbName) + .collection('occasions') + .find({}, { projection: { _id: 0 } }) + .toArray(); + expect(documents, 'S5.14: its document is kept under the new field').toEqual([ + { title: 'launch' }, + ]); + }); + + it('S6: db update keeps the documents when the question about a collection whose model keeps its name is followed', async () => { + const dbName = 'mongo_remap_update'; + const ctx = setupProject(withDatabase(replSet.getUri(), dbName)); + created.add(ctx.testDir); + await emit(ctx, 'S6.01'); + const create = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(create.exitCode, `S6.02: db update creates the collections: ${create.stderr}`).toBe(0); + await client.db(dbName).collection('events').insertOne({ name: 'launch' }); + useFixture(ctx, 'contract-events-remapped.ts'); + await emit(ctx, 'S6.03'); + + const refused = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(refused.exitCode, 'S6.04: the drop of events needs a statement').toBe(2); + expect(engineError(refused)?.code, 'S6.04').toBe('CLI.CONSENT_REQUIRED'); + expect( + JSON.stringify(engineError(refused)), + 'S6.04: the question says how to keep the documents', + ).toContain( + 'db.getCollection(\\"events\\").renameCollection(\\"occasions\\") in mongosh, then run db update', + ); + expect(await collections(dbName), 'S6.04: nothing dropped').toContain('events'); + + await client.db(dbName).collection('events').rename('occasions'); + const updated = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(updated.exitCode, `S6.05: db update after the rename: ${updated.stderr}`).toBe(0); + expect(await collections(dbName), 'S6.05: the collection is renamed').toEqual([ + 'occasions', + 'users', + ]); + const documents = await client + .db(dbName) + .collection('occasions') + .find({}, { projection: { _id: 0 } }) + .toArray(); + expect(documents, 'S6.05: its document is kept').toEqual([{ name: 'launch' }]); }); }); diff --git a/test/integration/test/cli-journeys/keep-documents-advice.mongo.e2e.test.ts b/test/integration/test/cli-journeys/keep-documents-advice.mongo.e2e.test.ts new file mode 100644 index 000000000000..68de094734c1 --- /dev/null +++ b/test/integration/test/cli-journeys/keep-documents-advice.mongo.e2e.test.ts @@ -0,0 +1,157 @@ +/** + * The advice to keep the documents of a collection whose model keeps its name, followed as printed when the plan creates the new collection only with its index (MongoDB). + * + * `Event` moves from `events` to `occasions` and has a unique index on `name`, so the plan has no `createCollection("occasions")`: `createIndex` creates it. The journey plans with `--delete Event`, edits `migration.ts` as the question says, runs node on it, and applies it with `db migrate`. + */ + +import { copyFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { rm } from 'node:fs/promises'; +import { dirname, join } from 'node:path'; +import { timeouts } from '@repo/test-utils'; +import { MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { fixtureAppDir } from '../utils/cli-test-helpers'; +import { + engineError, + followRenameCollectionAdvice, + type JourneyContext, + latestMigrationDirName, + parseJsonOutput, + runContractEmit, + runDbVerify, + runMigrate, + runMigrationPlan, + runPrintedSelfEmit, +} from '../utils/journey-test-helpers'; + +const FIXTURES_DIR = join(fixtureAppDir, 'fixtures/mongo-cli-journeys'); +const DB_NAME = 'mongo_keep_indexed_documents'; + +function setupProject(connectionString: string): JourneyContext { + const testDir = join( + fixtureAppDir, + `test-mongo-keep-documents-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + const outputDir = join(testDir, 'output'); + mkdirSync(outputDir, { recursive: true }); + mkdirSync(join(testDir, 'migrations'), { recursive: true }); + writeFileSync( + join(testDir, 'package.json'), + JSON.stringify({ + name: 'mongo-keep-documents-journey', + private: true, + type: 'module', + dependencies: { '@prisma/orm-mongo': 'workspace:0.16.0' }, + }), + 'utf-8', + ); + const config = readFileSync(join(FIXTURES_DIR, 'prisma.config.with-db.ts'), 'utf-8').replace( + /\{\{DB_URL\}\}/g, + () => connectionString, + ); + const configPath = join(testDir, 'prisma.config.ts'); + writeFileSync(configPath, config, 'utf-8'); + return { testDir, configPath, outputDir }; +} + +async function emitFixture(ctx: JourneyContext, name: string, label: string): Promise { + copyFileSync(join(FIXTURES_DIR, name), join(ctx.testDir, 'contract.ts')); + const emitted = await runContractEmit(ctx); + expect(emitted.exitCode, `${label}: emit: ${emitted.stderr}`).toBe(0); +} + +describe('Journey (MongoDB): keeping the documents of a collection created only by its index', { + timeout: timeouts.spinUpMongoMemoryServer, +}, () => { + let replSet: MongoMemoryReplSet; + let client: MongoClient; + let ctx: JourneyContext | undefined; + + beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + instanceOpts: [ + { launchTimeout: timeouts.spinUpMongoMemoryServer, storageEngine: 'wiredTiger' }, + ], + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + }, timeouts.spinUpMongoMemoryServer); + + afterAll(async () => { + if (ctx !== undefined) await rm(ctx.testDir, { recursive: true, force: true }).catch(() => {}); + await client?.close().catch(() => {}); + await replSet?.stop().catch(() => {}); + }, timeouts.spinUpMongoMemoryServer); + + it('renames the collection with its documents and index when migration.ts is edited as printed', async () => { + const url = new URL(replSet.getUri()); + url.pathname = `/${DB_NAME}`; + const project = setupProject(url.toString()); + ctx = project; + await emitFixture(project, 'contract-with-indexed-events.ts', 'K.01'); + const initial = await runMigrationPlan(project, ['--name', 'initial']); + expect(initial.exitCode, `K.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(project)).exitCode, 'K.03: migrate initial').toBe(0); + const db = client.db(DB_NAME); + await db.collection('events').insertMany([{ name: 'launch' }, { name: 'party' }]); + const origin = latestMigrationDirName(project); + + await emitFixture(project, 'contract-indexed-events-remapped.ts', 'K.04'); + const asked = await runMigrationPlan(project, ['--from', origin, '--json']); + expect(asked.exitCode, 'K.05: the drop of events needs a statement').toBe(2); + const question = String(engineError(asked)?.why); + expect(question, 'K.05: the question says where the rename goes').toContain( + 'make renameCollection("events", "occasions") the first operation, imported with the other operations, delete the drop of "events", and delete the create of "occasions" if there is one', + ); + + const remap = await runMigrationPlan(project, [ + '--name', + 'remap', + '--from', + origin, + '--delete', + 'Event', + ]); + expect(remap.exitCode, `K.06: plan with --delete Event: ${remap.stderr}`).toBe(0); + const remapDir = latestMigrationDirName(project); + const migrationTs = join(project.testDir, 'migrations/app', remapDir, 'migration.ts'); + const planned = readFileSync(migrationTs, 'utf-8'); + expect(planned, 'K.07: the index creates occasions, no createCollection does').not.toMatch( + /createCollection\(/, + ); + writeFileSync(migrationTs, followRenameCollectionAdvice(planned, 'events', 'occasions')); + const emitted = runPrintedSelfEmit(question, dirname(migrationTs)); + expect(emitted.exitCode, `K.08: node on migration.ts, as printed: ${emitted.stderr}`).toBe(0); + + const applied = await runMigrate(project); + expect(applied.exitCode, `K.09: db migrate: ${applied.stderr}`).toBe(0); + const collections = (await db.listCollections().toArray()) + .map(({ name }) => name) + .filter((name) => !name.startsWith('_prisma')) + .sort(); + const documents = await db + .collection('occasions') + .find({}, { projection: { _id: 0 } }) + .sort({ name: 1 }) + .toArray(); + const indexes = (await db.collection('occasions').indexes()).map(({ key, unique }) => ({ + key, + unique: unique === true, + })); + expect({ collections, documents, indexes }, 'K.10: the documents and the index moved').toEqual({ + collections: ['occasions', 'users'], + documents: [{ name: 'launch' }, { name: 'party' }], + indexes: [ + { key: { _id: 1 }, unique: false }, + { key: { name: 1 }, unique: true }, + ], + }); + const verify = await runDbVerify(project, ['--schema-only']); + expect(verify.exitCode, `K.11: db verify --schema-only: ${verify.stderr}`).toBe(0); + const further = await runMigrationPlan(project, ['--from', remapDir, '--json']); + expect(further.exitCode, `K.12: a further plan: ${further.stderr}`).toBe(0); + expect(parseJsonOutput<{ noOp: boolean }>(further).noOp, 'K.12: it is empty').toBe(true); + }); +}); diff --git a/test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts b/test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts new file mode 100644 index 000000000000..39d4b3ad2e90 --- /dev/null +++ b/test/integration/test/cli-journeys/migration-file-config-lookup.sqlite.e2e.test.ts @@ -0,0 +1,125 @@ +/** + * Without `--config`, `node migration.ts` uses the config that owns the migration: the one whose migrations directory holds the file. It tries the nearest `prisma.config.ts` above the migration, then the one in the working directory, and refuses when neither owns the file. + */ + +import { spawnSync } from 'node:child_process'; +import { mkdirSync, readdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join, relative } from 'pathe'; +import { describe, expect, it } from 'vitest'; +import { withTempDir, writeProjectManifest } from '../utils/cli-test-helpers'; +import { + type JourneyContext, + runContractEmit, + runMigrationPlan, + sqlitePslConfigFixture, + timeouts, +} from '../utils/journey-test-helpers'; + +const PSL = `// use prisma-8 + +model Profile { + id Int @id + name String +} +`; + +const THROWING_CONFIG = "throw new Error('the outer prisma.config.ts was loaded');\n"; + +/** Writes a SQLite project into `dir` whose config, named `configName`, keeps migrations in `migrationsDir`. */ +function writeProject(dir: string, migrationsDir: string, configName = 'prisma.config.ts') { + mkdirSync(dir, { recursive: true }); + const configPath = join(dir, configName); + const config = readFileSync(sqlitePslConfigFixture, 'utf-8') + .replace('{{DB_PATH}}', () => join(dir, 'journey.db')) + .replace("migrations: { dir: 'migrations' }", () => `migrations: { dir: '${migrationsDir}' }`); + writeFileSync(configPath, config, 'utf-8'); + writeFileSync(join(dir, 'contract.prisma'), PSL, 'utf-8'); + writeProjectManifest(dir); + return { testDir: dir, configPath, outputDir: dir } satisfies JourneyContext; +} + +/** Emits the contract, plans the first migration, and returns the migration's directory. */ +async function planFirstMigration(ctx: JourneyContext, appDir: string): Promise { + const emitted = await runContractEmit(ctx); + expect(emitted.exitCode, `emit: ${emitted.stderr}`).toBe(0); + const planned = await runMigrationPlan(ctx, ['--name', 'initial']); + expect(planned.exitCode, `plan: ${planned.stderr}`).toBe(0); + const [dir] = readdirSync(appDir).filter((name) => name !== 'refs'); + return join(appDir, String(dir)); +} + +function runNode(args: readonly string[], cwd: string) { + const run = spawnSync(process.execPath, args, { cwd, encoding: 'utf-8' }); + return { exitCode: run.status, stdout: run.stdout, stderr: run.stderr }; +} + +const WROTE = { exitCode: 0, stdout: expect.stringContaining('Wrote ops.json + migration.json') }; + +withTempDir(({ createTempDir }) => { + describe('node migration.ts', () => { + it( + 'finds the config from the migration directory, the project directory and its parent', + async () => { + const ctx = writeProject(createTempDir(), './db/migrations'); + const migrationDir = await planFirstMigration(ctx, join(ctx.testDir, 'db/migrations/app')); + const file = join(migrationDir, 'migration.ts'); + + expect( + runNode(['migration.ts'], migrationDir), + 'from the migration directory', + ).toMatchObject(WROTE); + expect( + runNode([relative(ctx.testDir, file)], ctx.testDir), + 'from the project directory', + ).toMatchObject(WROTE); + const parent = dirname(ctx.testDir); + expect(runNode([relative(parent, file)], parent), 'from the parent').toMatchObject(WROTE); + }, + timeouts.spinUpPpgDev, + ); + + it( + 'uses the working directory config when the migrations directory is outside the project', + async () => { + const root = createTempDir(); + writeFileSync(join(root, 'prisma.config.ts'), THROWING_CONFIG, 'utf-8'); + const ctx = writeProject(join(root, 'app'), '../shared-db/migrations'); + const migrationDir = await planFirstMigration(ctx, join(root, 'shared-db/migrations/app')); + + expect( + runNode([relative(ctx.testDir, join(migrationDir, 'migration.ts'))], ctx.testDir), + 'from the project directory', + ).toMatchObject(WROTE); + }, + timeouts.spinUpPpgDev, + ); + + it( + 'refuses when no config it finds owns the migration, and runs with --config', + async () => { + const root = createTempDir(); + writeProject(root, './migrations'); + const ctx = writeProject(join(root, 'inner'), './migrations', 'prisma.sqlite.config.ts'); + const migrationDir = await planFirstMigration(ctx, join(ctx.testDir, 'migrations/app')); + + const refused = runNode(['migration.ts'], migrationDir); + expect(refused, 'without --config').toMatchObject({ + exitCode: 1, + stdout: '', + stderr: expect.stringContaining('CLI.MIGRATION_CONFIG_NOT_FOUND'), + }); + expect(refused.stderr, 'names the config it checked').toContain( + join(root, 'prisma.config.ts'), + ); + expect( + runNode( + ['migration.ts', '--config', relative(migrationDir, ctx.configPath)], + migrationDir, + ), + 'with --config', + ).toMatchObject(WROTE); + }, + timeouts.spinUpPpgDev, + ); + }); +}); diff --git a/test/integration/test/cli-journeys/mongo-db-update-consent.e2e.test.ts b/test/integration/test/cli-journeys/mongo-db-update-consent.e2e.test.ts index 3fd52b2730c4..5fada18d872c 100644 --- a/test/integration/test/cli-journeys/mongo-db-update-consent.e2e.test.ts +++ b/test/integration/test/cli-journeys/mongo-db-update-consent.e2e.test.ts @@ -163,7 +163,7 @@ describe('Journey: Mongo db update asks for consent only to lose data', { expect(unconfirmed.exitCode).toBe(2); expect(parseJsonOutput(unconfirmed)).toMatchObject({ code: 'CLI.CONSENT_REQUIRED', - meta: { unanswered: [{ subject: 'Event', verbs: ['delete'] }] }, + meta: { unanswered: [{ subject: 'Event', verbs: ['rename', 'delete'] }] }, }); const kept = await client.db(dbName).listCollections({ name: 'events' }).toArray(); expect(kept.map(({ name }) => name)).toEqual(['events']); diff --git a/test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts b/test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts new file mode 100644 index 000000000000..10b741d23e08 --- /dev/null +++ b/test/integration/test/cli-journeys/rename-delete-statements.mongo.e2e.test.ts @@ -0,0 +1,409 @@ +/** + * Journeys (MongoDB) for `--rename` and `--delete`, on PSL contracts with documents. + * + * R1 plans and migrates a model rename, a field rename with a unique index, a variant's field + * rename and a field removal. R2 does the same through `db update`, then runs it again. R3 removes + * a field through `db update` with no snapshot of the origin contract. R4 follows the question + * about a field whose model the destination names differently. Every refusal is followed as + * printed. + */ + +import { mkdirSync, rmSync, writeFileSync } from 'node:fs'; +import { rm } from 'node:fs/promises'; +import { join } from 'node:path'; +import { timeouts } from '@repo/test-utils'; +import { type Db, MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, afterEach, beforeAll, describe, expect, it } from 'vitest'; +import { fixtureAppDir } from '../utils/cli-test-helpers'; +import { + type EngineCommandResult, + engineError, + type JourneyContext, + latestMigrationDirName, + parseJsonOutput, + runContractEmit, + runDbUpdate, + runDbVerify, + runMigrate, + runMigrationPlan, +} from '../utils/journey-test-helpers'; + +const BEFORE = `// use prisma-8 + +model Profile { + id ObjectId @id @map("_id") + name String + nickname String + + @@unique([name]) + @@map("profiles") +} + +model Task { + id ObjectId @id @map("_id") + title String + type String + + @@discriminator(type) + @@map("tasks") +} + +model Bug { + id ObjectId @id @map("_id") + score String + + @@base(Task, "bug") +} + +model Feature { + id ObjectId @id @map("_id") + score String + + @@base(Task, "feature") +} +`; + +const AFTER = `// use prisma-8 + +model User { + id ObjectId @id @map("_id") + fullName String + + @@unique([fullName]) + @@map("users") +} + +model Task { + id ObjectId @id @map("_id") + title String + type String + + @@discriminator(type) + @@map("tasks") +} + +model Bug { + id ObjectId @id @map("_id") + severity String + + @@base(Task, "bug") +} + +model Feature { + id ObjectId @id @map("_id") + score String + + @@base(Task, "feature") +} +`; + +const RENAMES = [ + '--rename', + 'Profile:User', + '--rename', + 'User.name:User.fullName', + '--rename', + 'Bug.score:Bug.severity', +]; + +function withDatabase(baseUri: string, dbName: string): string { + const url = new URL(baseUri); + url.pathname = `/${dbName}`; + return url.toString(); +} + +function setupProject(connectionString: string): JourneyContext { + const testDir = join( + fixtureAppDir, + `test-mongo-renames-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + const outputDir = join(testDir, 'output'); + mkdirSync(outputDir, { recursive: true }); + mkdirSync(join(testDir, 'migrations'), { recursive: true }); + writeFileSync( + join(testDir, 'package.json'), + JSON.stringify({ + name: 'mongo-renames-journey', + private: true, + type: 'module', + dependencies: { '@prisma/orm-mongo': 'workspace:0.16.0' }, + }), + ); + const configPath = join(testDir, 'prisma.config.ts'); + writeFileSync( + configPath, + [ + "import { defineConfig as ormConfig } from '@prisma/orm-mongo/config';", + "import { definePrismaConfig } from '@prisma/cli-engine';", + '', + 'export default definePrismaConfig({', + ' orm: ormConfig({', + " contract: './contract.prisma',", + " output: 'output',", + ` db: { connection: ${JSON.stringify(connectionString)} },`, + ' }),', + '});', + '', + ].join('\n'), + ); + return { testDir, configPath, outputDir }; +} + +async function useSchema(ctx: JourneyContext, schema: string, label: string): Promise { + writeFileSync(join(ctx.testDir, 'contract.prisma'), schema); + const emitted = await runContractEmit(ctx); + expect(emitted.exitCode, `${label}: emit: ${emitted.stderr}`).toBe(0); +} + +/** The flags a refusal's next actions print for `verb`, as they would be typed. */ +function printedFlags(result: EngineCommandResult, verb: string): readonly string[] { + const actions = JSON.stringify(engineError(result)?.nextActions ?? []); + const pattern = new RegExp(`--${verb} (?:'([^']+)'|([^\\s"\\\\]+))`, 'g'); + return [...actions.matchAll(pattern)].flatMap((match) => [ + `--${verb}`, + match[1] ?? match[2] ?? '', + ]); +} + +async function seed(db: Db): Promise { + await db.collection('profiles').insertMany([ + { name: 'Ada', nickname: 'a' }, + { name: 'Bo', nickname: 'b' }, + ]); + await db.collection('tasks').insertMany([ + { title: 'crash', type: 'bug', score: 'high' }, + { title: 'dark mode', type: 'feature', score: 'p1' }, + ]); +} + +async function documents(db: Db, collection: string) { + return db + .collection(collection) + .find({}, { projection: { _id: 0 }, sort: { _id: 1 } }) + .toArray(); +} + +async function expectEndState(ctx: JourneyContext, db: Db, label: string): Promise { + const collections = (await db.listCollections().toArray()) + .map(({ name }) => name) + .filter((name) => !name.startsWith('_prisma')) + .sort(); + expect( + { + collections, + users: await documents(db, 'users'), + tasks: await documents(db, 'tasks'), + }, + `${label}: documents under the new names`, + ).toEqual({ + collections: ['tasks', 'users'], + users: [{ fullName: 'Ada' }, { fullName: 'Bo' }], + tasks: [ + { title: 'crash', type: 'bug', severity: 'high' }, + { title: 'dark mode', type: 'feature', score: 'p1' }, + ], + }); + await expect( + db.collection('users').updateOne({ fullName: 'Ada' }, { $set: { fullName: 'Ada L.' } }), + `${label}: a user accepts an update`, + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + db.collection('tasks').updateOne({ type: 'bug' }, { $set: { severity: 'low' } }), + `${label}: a bug accepts an update`, + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + db.collection('tasks').updateOne({ type: 'feature' }, { $set: { score: 'p2' } }), + `${label}: a feature accepts an update`, + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + db.collection('users').insertOne({ fullName: 'Bo' }), + `${label}: the unique index is on the new field`, + ).rejects.toThrow(/duplicate key/); + + const verify = await runDbVerify(ctx, ['--schema-only']); + expect(verify.exitCode, `${label}: db verify --schema-only: ${verify.stderr}`).toBe(0); + const further = await runDbUpdate(ctx, ['--dry-run', '--no-interactive', '--json']); + expect(further.exitCode, `${label}: further plan: ${further.stderr}`).toBe(0); + expect( + parseJsonOutput<{ plan: { operations: readonly unknown[] } }>(further).plan.operations, + `${label}: a further plan is empty`, + ).toEqual([]); +} + +describe('Journeys (MongoDB): --rename and --delete', { + timeout: timeouts.spinUpMongoMemoryServer, +}, () => { + let replSet: MongoMemoryReplSet; + let client: MongoClient; + const created = new Set(); + + beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + instanceOpts: [ + { launchTimeout: timeouts.spinUpMongoMemoryServer, storageEngine: 'wiredTiger' }, + ], + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + }, timeouts.spinUpMongoMemoryServer); + + afterEach(async () => { + for (const dir of created) { + await rm(dir, { recursive: true, force: true }).catch(() => {}); + } + created.clear(); + }); + + afterAll(async () => { + await client?.close().catch(() => {}); + await replSet?.stop().catch(() => {}); + }, timeouts.spinUpMongoMemoryServer); + + function project(dbName: string) { + const ctx = setupProject(withDatabase(replSet.getUri(), dbName)); + created.add(ctx.testDir); + return { ctx, db: client.db(dbName) }; + } + + it('R1: migration plan refuses until the printed --delete is given, then migrate renames and removes', async () => { + const { ctx, db } = project('mongo_renames_plan'); + await useSchema(ctx, BEFORE, 'R1.01'); + const initial = await runMigrationPlan(ctx, ['--name', 'initial']); + expect(initial.exitCode, `R1.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(ctx)).exitCode, 'R1.03: migrate initial').toBe(0); + await seed(db); + const origin = latestMigrationDirName(ctx); + await useSchema(ctx, AFTER, 'R1.04'); + + const refused = await runMigrationPlan(ctx, ['--from', origin, ...RENAMES, '--json']); + expect(refused.exitCode, 'R1.05: the removed field needs a statement').toBe(2); + expect(engineError(refused)?.code, 'R1.05').toBe('CLI.CONSENT_REQUIRED'); + const printed = printedFlags(refused, 'delete'); + expect(printed, 'R1.05: the printed flag').toEqual(['--delete', 'User.nickname']); + + const plan = await runMigrationPlan(ctx, [ + '--name', + 'tidy', + '--from', + origin, + ...RENAMES, + ...printed, + '--json', + ]); + expect(plan.exitCode, `R1.06: plan with the printed flag: ${plan.stderr}`).toBe(0); + const applied = await runMigrate(ctx); + expect(applied.exitCode, `R1.07: migrate: ${applied.stderr}`).toBe(0); + + await expectEndState(ctx, db, 'R1.08'); + }); + + it('R2: db update refuses until the printed --delete is given, then a second run fails on its first statement', async () => { + const { ctx, db } = project('mongo_renames_update'); + await useSchema(ctx, BEFORE, 'R2.01'); + const create = await runDbUpdate(ctx, ['--no-interactive']); + expect(create.exitCode, `R2.02: db update creates: ${create.stderr}`).toBe(0); + await seed(db); + await useSchema(ctx, AFTER, 'R2.03'); + + const refused = await runDbUpdate(ctx, ['--no-interactive', ...RENAMES, '--json']); + expect(refused.exitCode, 'R2.04: the removed field needs a statement').toBe(2); + expect(engineError(refused)?.code, 'R2.04').toBe('CLI.CONSENT_REQUIRED'); + expect(await documents(db, 'profiles'), 'R2.04: nothing changed').toHaveLength(2); + const printed = printedFlags(refused, 'delete'); + expect(printed, 'R2.04: the printed flag').toEqual(['--delete', 'User.nickname']); + + const updated = await runDbUpdate(ctx, ['--no-interactive', ...RENAMES, ...printed]); + expect(updated.exitCode, `R2.05: db update with the printed flag: ${updated.stderr}`).toBe(0); + + await expectEndState(ctx, db, 'R2.06'); + + const again = await runDbUpdate(ctx, ['--no-interactive', ...RENAMES, ...printed, '--json']); + expect(again.exitCode, 'R2.07: the same statements again').not.toBe(0); + expect( + { code: engineError(again)?.code, summary: engineError(again)?.summary }, + 'R2.07: it fails on the first statement', + ).toMatchObject({ + code: 'MIGRATION.STATEMENT_UNRESOLVED', + summary: expect.stringContaining('Profile:User'), + }); + }); + + it('R3: db update with no snapshot names a removed field by its collection, and --delete . removes it', async () => { + const { ctx, db } = project('mongo_renames_no_snapshot'); + await useSchema(ctx, BEFORE, 'R3.01'); + const create = await runDbUpdate(ctx, ['--no-interactive']); + expect(create.exitCode, `R3.02: db update creates: ${create.stderr}`).toBe(0); + await seed(db); + rmSync(join(ctx.testDir, 'migrations'), { recursive: true, force: true }); + await useSchema(ctx, BEFORE.replace(' nickname String\n', ''), 'R3.03'); + + const refused = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(refused.exitCode, 'R3.04: the removed field needs a statement').toBe(2); + const printed = printedFlags(refused, 'delete'); + expect(printed, 'R3.04: named by its collection').toEqual(['--delete', 'profiles.nickname']); + expect( + String(engineError(refused)?.why) + .split('\n') + .map((line) => line.slice(0, 3)), + 'R3.04: the question, then each way to keep the data on a line of its own', + ).toEqual(['Rem', 'If ', '1. ', '2. ', '3. ']); + + const updated = await runDbUpdate(ctx, ['--no-interactive', ...printed]); + expect(updated.exitCode, `R3.05: db update with the printed flag: ${updated.stderr}`).toBe(0); + expect(await documents(db, 'profiles'), 'R3.05: the field is gone').toEqual([ + { name: 'Ada' }, + { name: 'Bo' }, + ]); + }); + + it('R4: a field whose model the destination names differently is renamed once the printed model rename is given', async () => { + const { ctx, db } = project('mongo_renames_model_first'); + await useSchema(ctx, BEFORE, 'R4.01'); + const create = await runDbUpdate(ctx, ['--no-interactive']); + expect(create.exitCode, `R4.02: db update creates: ${create.stderr}`).toBe(0); + await seed(db); + await useSchema( + ctx, + BEFORE.replace('model Profile {', 'model Person {').replace( + ' nickname String', + ' alias String', + ), + 'R4.03', + ); + + const refused = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(refused.exitCode, 'R4.04: the removed field needs a statement').toBe(2); + const question = JSON.stringify(engineError(refused)); + expect(question, 'R4.04: says to rename the model first').toContain( + 'also pass --rename Profile:', + ); + + const modelFirst = await runDbUpdate(ctx, [ + '--no-interactive', + '--rename', + 'Profile:Person', + '--json', + ]); + expect(modelFirst.exitCode, 'R4.05: the field question now offers a rename').toBe(2); + const renameForm = printedFlags(modelFirst, 'rename'); + expect(renameForm, 'R4.05: the printed rename form').toEqual([ + '--rename', + 'Person.nickname:Person.', + ]); + + const renamed = await runDbUpdate(ctx, [ + '--no-interactive', + '--rename', + 'Profile:Person', + '--rename', + 'Person.nickname:Person.alias', + ]); + expect(renamed.exitCode, `R4.06: db update with both renames: ${renamed.stderr}`).toBe(0); + expect(await documents(db, 'profiles'), 'R4.06: the values are kept').toEqual([ + { name: 'Ada', alias: 'a' }, + { name: 'Bo', alias: 'b' }, + ]); + }); +}); diff --git a/test/integration/test/cli-journeys/same-hash-field-changes.mongo.e2e.test.ts b/test/integration/test/cli-journeys/same-hash-field-changes.mongo.e2e.test.ts new file mode 100644 index 000000000000..b6347e50b9cc --- /dev/null +++ b/test/integration/test/cli-journeys/same-hash-field-changes.mongo.e2e.test.ts @@ -0,0 +1,163 @@ +/** + * Field changes on a MongoDB contract written in TypeScript, which gives its collections no validator. + * + * Field names are then not in the contract's storage hash, so a migration that only renames or removes a field would start and end at the same hash, which `db migrate` refuses. `migration plan` refuses the change before it writes anything; `db update` carries it out. Journey H1 renames `Event.name` and follows the refusal's advice through `db update`. Journey H2 removes the field: no statement names a removal, so `migration plan` finds no change and writes nothing, and `db update` carries it out. + */ + +import { copyFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { rm } from 'node:fs/promises'; +import { join } from 'node:path'; +import { timeouts } from '@repo/test-utils'; +import { MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, afterEach, beforeAll, describe, expect, it } from 'vitest'; +import { fixtureAppDir } from '../utils/cli-test-helpers'; +import { + engineError, + getMigrationDirs, + type JourneyContext, + latestMigrationDirName, + parseJsonOutput, + runContractEmit, + runDbUpdate, + runMigrate, + runMigrationPlan, +} from '../utils/journey-test-helpers'; + +const FIXTURES_DIR = join(fixtureAppDir, 'fixtures/mongo-cli-journeys'); + +function withDatabase(baseUri: string, dbName: string): string { + const url = new URL(baseUri); + url.pathname = `/${dbName}`; + return url.toString(); +} + +function setupProject(connectionString: string): JourneyContext { + const testDir = join( + fixtureAppDir, + `test-mongo-same-hash-${Date.now()}-${Math.random().toString(36).slice(2)}`, + ); + const outputDir = join(testDir, 'output'); + mkdirSync(outputDir, { recursive: true }); + mkdirSync(join(testDir, 'migrations'), { recursive: true }); + writeFileSync( + join(testDir, 'package.json'), + JSON.stringify({ + name: 'mongo-same-hash-journey', + private: true, + type: 'module', + dependencies: { '@prisma/orm-mongo': 'workspace:0.16.0' }, + }), + 'utf-8', + ); + copyFileSync(join(FIXTURES_DIR, 'contract-with-events.ts'), join(testDir, 'contract.ts')); + const config = readFileSync(join(FIXTURES_DIR, 'prisma.config.with-db.ts'), 'utf-8').replace( + /\{\{DB_URL\}\}/g, + () => connectionString, + ); + const configPath = join(testDir, 'prisma.config.ts'); + writeFileSync(configPath, config, 'utf-8'); + return { testDir, configPath, outputDir }; +} + +async function useFixture(ctx: JourneyContext, name: string, label: string): Promise { + copyFileSync(join(FIXTURES_DIR, name), join(ctx.testDir, 'contract.ts')); + const emitted = await runContractEmit(ctx); + expect(emitted.exitCode, `${label}: emit: ${emitted.stderr}`).toBe(0); +} + +describe('Journeys (MongoDB): field changes that leave the storage hash unchanged', { + timeout: timeouts.spinUpMongoMemoryServer, +}, () => { + let replSet: MongoMemoryReplSet; + let client: MongoClient; + const created = new Set(); + + beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + instanceOpts: [ + { launchTimeout: timeouts.spinUpMongoMemoryServer, storageEngine: 'wiredTiger' }, + ], + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + }, timeouts.spinUpMongoMemoryServer); + + afterEach(async () => { + for (const dir of created) { + await rm(dir, { recursive: true, force: true }).catch(() => {}); + } + created.clear(); + }); + + afterAll(async () => { + await client?.close().catch(() => {}); + await replSet?.stop().catch(() => {}); + }, timeouts.spinUpMongoMemoryServer); + + async function migratedProject(dbName: string, label: string) { + const ctx = setupProject(withDatabase(replSet.getUri(), dbName)); + created.add(ctx.testDir); + await useFixture(ctx, 'contract-with-events.ts', `${label}.01`); + const initial = await runMigrationPlan(ctx, ['--name', 'initial']); + expect(initial.exitCode, `${label}.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(ctx)).exitCode, `${label}.03: migrate initial`).toBe(0); + await client.db(dbName).collection('events').insertOne({ name: 'launch' }); + return { ctx, origin: latestMigrationDirName(ctx) }; + } + + async function events(dbName: string) { + return client + .db(dbName) + .collection('events') + .find({}, { projection: { _id: 0 } }) + .toArray(); + } + + it('H1: migration plan refuses a field rename, and db update carries it out as advised', async () => { + const dbName = 'mongo_same_hash_rename'; + const { ctx, origin } = await migratedProject(dbName, 'H1'); + await useFixture(ctx, 'contract-events-name-renamed.ts', 'H1.04'); + const rename = ['--rename', 'Event.name:Event.title']; + + const refused = await runMigrationPlan(ctx, ['--from', origin, ...rename, '--json']); + expect(refused.exitCode, 'H1.05: the rename is refused').not.toBe(0); + const error = JSON.stringify(engineError(refused)); + expect(error, 'H1.05: names the field and the reason').toContain( + 'Cannot rename field \\"Event.name\\" to \\"Event.title\\": the migration would start and end at the same contract hash', + ); + expect(error, 'H1.05: says what works').toContain('Use prisma db update for this change'); + expect(getMigrationDirs(ctx), 'H1.05: nothing written').toHaveLength(1); + + const updated = await runDbUpdate(ctx, ['--no-interactive', ...rename, '--json']); + expect(updated.exitCode, `H1.06: db update: ${updated.stderr}`).toBe(0); + expect(await events(dbName), 'H1.06: the value is under the new name').toEqual([ + { title: 'launch' }, + ]); + }); + + it('H2: migration plan finds no change in a field removal, and db update carries it out', async () => { + const dbName = 'mongo_same_hash_remove'; + const { ctx, origin } = await migratedProject(dbName, 'H2'); + await useFixture(ctx, 'contract-events-name-removed.ts', 'H2.04'); + + const unstated = await runMigrationPlan(ctx, ['--from', origin, '--json']); + expect(unstated.exitCode, `H2.05: without a statement: ${unstated.stderr}`).toBe(0); + expect( + parseJsonOutput<{ noOp: boolean }>(unstated).noOp, + 'H2.05: the hash is unchanged, so the plan sees no change', + ).toBe(true); + + expect(getMigrationDirs(ctx), 'H2.05: nothing written').toHaveLength(1); + + const updated = await runDbUpdate(ctx, [ + '--no-interactive', + '--delete', + 'Event.name', + '--json', + ]); + expect(updated.exitCode, `H2.06: db update: ${updated.stderr}`).toBe(0); + expect(await events(dbName), 'H2.06: the field is gone').toEqual([{}]); + }); +}); diff --git a/test/integration/test/cli-journeys/table-map-change.e2e.test.ts b/test/integration/test/cli-journeys/table-map-change.e2e.test.ts new file mode 100644 index 000000000000..b1116400736a --- /dev/null +++ b/test/integration/test/cli-journeys/table-map-change.e2e.test.ts @@ -0,0 +1,171 @@ +/** + * A model that keeps its name while `@@map` gives it a new table (Postgres). + * + * The Postgres twin of `table-map-change.sqlite.e2e.test.ts`. + * + * No statement can rename that table, so the question about the dropped table says how to keep its rows. Journey M1 follows the `migration plan` route as printed: a migration of its own written with `migration new`, the printed `renameTable` call and the printed `node` command, then `db migrate`. Journey M2 follows the `db update` route: the printed SQL, then `db update`. + */ + +import { readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'pathe'; +import { describe, expect, it } from 'vitest'; +import { withTempDir } from '../utils/cli-test-helpers'; +import { + type EngineCommandResult, + engineError, + type JourneyContext, + latestMigrationDirName, + latestMigrationToHash, + parseJsonOutput, + planMigrationAndSelfEmit, + runContractEmit, + runDbUpdate, + runDbVerify, + runMigrate, + runMigrationNew, + runMigrationPlan, + runPrintedSelfEmit, + setupJourney, + sql, + timeouts, + useDevDatabase, +} from '../utils/journey-test-helpers'; + +const FROM_PSL = `// use prisma-8 + +model Profile { + id Int @id + name String @unique + + @@map("profiles") +} +`; + +const TO_PSL = FROM_PSL.replace('@@map("profiles")', '@@map("people")'); + +async function emitContract(ctx: JourneyContext, psl: string, label: string): Promise { + writeFileSync(join(ctx.testDir, 'contract.prisma'), psl, 'utf-8'); + const emit = await runContractEmit(ctx); + expect(emit.exitCode, `${label}: emit: ${emit.stderr}`).toBe(0); +} + +function questionOf(result: EngineCommandResult): string { + return String(engineError(result)?.why); +} + +async function expectRowsKept(connectionString: string, label: string): Promise { + const rows = await sql(connectionString, 'SELECT id, name FROM "public"."people" ORDER BY id'); + const tables = await sql( + connectionString, + "SELECT tablename FROM pg_tables WHERE schemaname = 'public' AND tablename NOT LIKE '\\_prisma%' ORDER BY tablename", + ); + expect( + { rows: rows.rows, tables: tables.rows.map((row) => row['tablename']) }, + `${label}: the rows are in the renamed table`, + ).toEqual({ + rows: [ + { id: 1, name: 'alice' }, + { id: 2, name: 'bob' }, + ], + tables: ['people'], + }); +} + +async function seedRows(connectionString: string): Promise { + await sql( + connectionString, + `INSERT INTO "public"."profiles" (id, name) VALUES (1, 'alice'), (2, 'bob')`, + ); +} + +withTempDir(({ createTempDir }) => { + describe('Journey M1 (Postgres): the migration plan route keeps the rows', () => { + const db = useDevDatabase(); + + it( + 'writes the rename as a migration of its own, as the question prints it', + async () => { + const ctx = setupJourney({ + connectionString: db.connectionString, + createTempDir, + contractMode: 'psl', + }); + await emitContract(ctx, FROM_PSL, 'M1.01'); + const initial = await planMigrationAndSelfEmit(ctx, ['--name', 'initial']); + expect(initial.exitCode, `M1.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(ctx)).exitCode, 'M1.03: migrate initial').toBe(0); + await seedRows(db.connectionString); + const origin = latestMigrationDirName(ctx); + const originHash = latestMigrationToHash(ctx); + await emitContract(ctx, TO_PSL, 'M1.04'); + + const refused = await runMigrationPlan(ctx, ['--from', origin, '--json']); + expect(refused.exitCode, 'M1.05: the drop of profiles needs a statement').toBe(2); + const question = questionOf(refused); + expect(question, 'M1.05: the question says how to keep the rows').toContain( + 'To keep its rows, rename table "profiles" to "people" instead', + ); + + const scaffold = await runMigrationNew(ctx, ['--name', 'people', '--from', originHash]); + expect(scaffold.exitCode, `M1.06: migration new: ${scaffold.stderr}`).toBe(0); + const dir = latestMigrationDirName(ctx); + const call = /add (\.\.\.this\.renameTable\(.*?\)) to the operations/.exec(question)?.[1]; + expect(call, `M1.07: the printed renameTable call in: ${question}`).toBeDefined(); + const migrationTs = join(ctx.testDir, 'migrations/app', dir, 'migration.ts'); + const scaffolded = readFileSync(migrationTs, 'utf-8'); + const edited = scaffolded.replace(/return \[[\s\S]*?\];/, () => `return [${call}];`); + expect(edited, 'M1.07: the call is in the operations').not.toBe(scaffolded); + writeFileSync(migrationTs, edited, 'utf-8'); + const emitted = runPrintedSelfEmit(question, dirname(migrationTs)); + expect(emitted.exitCode, `M1.08: node on migration.ts, as printed: ${emitted.stderr}`).toBe( + 0, + ); + + const applied = await runMigrate(ctx); + expect(applied.exitCode, `M1.09: db migrate: ${applied.stderr}`).toBe(0); + await expectRowsKept(db.connectionString, 'M1.10'); + const verify = await runDbVerify(ctx, ['--schema-only']); + expect(verify.exitCode, `M1.10: db verify --schema-only: ${verify.stderr}`).toBe(0); + const further = await runMigrationPlan(ctx, ['--from', dir, '--json']); + expect(further.exitCode, `M1.11: a further plan: ${further.stderr}`).toBe(0); + expect(parseJsonOutput<{ noOp: boolean }>(further).noOp, 'M1.11: it is empty').toBe(true); + }, + timeouts.spinUpPpgDev, + ); + }); + + describe('Journey M2 (Postgres): the db update route keeps the rows', () => { + const db = useDevDatabase(); + + it( + 'renames the table with the printed SQL, then db update finds nothing that loses data', + async () => { + const ctx = setupJourney({ + connectionString: db.connectionString, + createTempDir, + contractMode: 'psl', + }); + await emitContract(ctx, FROM_PSL, 'M2.01'); + const create = await runDbUpdate(ctx, ['--json']); + expect(create.exitCode, `M2.02: db update creates the table: ${create.stderr}`).toBe(0); + await seedRows(db.connectionString); + await emitContract(ctx, TO_PSL, 'M2.03'); + + const refused = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(refused.exitCode, 'M2.04: the drop of profiles needs a statement').toBe(2); + const printed = /yourself with (.*?), then run prisma db update/.exec( + questionOf(refused), + )?.[1]; + expect(printed, 'M2.04: the printed SQL').toBeDefined(); + await sql(db.connectionString, String(printed)); + + const updated = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(updated.exitCode, `M2.05: db update: ${updated.stderr}`).toBe(0); + await expectRowsKept(db.connectionString, 'M2.06'); + const verify = await runDbVerify(ctx, ['--schema-only']); + expect(verify.exitCode, `M2.06: db verify --schema-only: ${verify.stderr}`).toBe(0); + }, + timeouts.spinUpPpgDev, + ); + }); +}); diff --git a/test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts b/test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts new file mode 100644 index 000000000000..d236edd704a1 --- /dev/null +++ b/test/integration/test/cli-journeys/table-map-change.sqlite.e2e.test.ts @@ -0,0 +1,182 @@ +/** + * A model that keeps its name while `@@map` gives it a new table (SQLite). + * + * No statement can rename that table, so the question about the dropped table says how to keep its rows. Journey M1 follows the `migration plan` route as printed, in a project whose `migrations.dir` is `./db/migrations`: a migration of its own written with `migration new`, the printed `renameTable` call, node run on its `migration.ts`, then `db migrate`. Journey M2 follows the `db update` route: the printed SQL, then `db update`. + */ + +import { readFileSync, writeFileSync } from 'node:fs'; +import { DatabaseSync } from 'node:sqlite'; +import { dirname, join } from 'pathe'; +import { describe, expect, it } from 'vitest'; +import { withTempDir, writeProjectManifest } from '../utils/cli-test-helpers'; +import { + type EngineCommandResult, + engineError, + type JourneyContext, + latestMigrationDirName, + latestMigrationToHash, + parseJsonOutput, + planMigrationAndSelfEmit, + runContractEmit, + runDbUpdate, + runDbVerify, + runMigrate, + runMigrationNew, + runMigrationPlan, + runPrintedSelfEmit, + sqlitePslConfigFixture, + timeouts, +} from '../utils/journey-test-helpers'; + +const FROM_PSL = `// use prisma-8 + +model Profile { + id Int @id + name String @unique + + @@map("profiles") +} +`; + +const TO_PSL = FROM_PSL.replace('@@map("profiles")', '@@map("people")'); + +type SqliteJourney = JourneyContext & { readonly dbPath: string }; + +function setupSqliteJourney( + createTempDir: () => string, + migrationsDir = 'migrations', +): SqliteJourney { + const testDir = createTempDir(); + const dbPath = join(testDir, 'journey.db'); + const configPath = join(testDir, 'prisma.config.ts'); + const config = readFileSync(sqlitePslConfigFixture, 'utf-8') + .replace('{{DB_PATH}}', () => dbPath) + .replace( + "migrations: { dir: 'migrations' }", + () => `migrations: { dir: './${migrationsDir}' }`, + ); + writeFileSync(configPath, config, 'utf-8'); + writeProjectManifest(testDir); + return { testDir, configPath, outputDir: testDir, dbPath, migrationsDir }; +} + +function withDatabase(dbPath: string, run: (db: DatabaseSync) => T): T { + const db = new DatabaseSync(dbPath); + try { + return run(db); + } finally { + db.close(); + } +} + +async function emitContract(ctx: JourneyContext, psl: string, label: string): Promise { + writeFileSync(join(ctx.testDir, 'contract.prisma'), psl, 'utf-8'); + const emit = await runContractEmit(ctx); + expect(emit.exitCode, `${label}: emit: ${emit.stderr}`).toBe(0); +} + +function questionOf(result: EngineCommandResult): string { + return String(engineError(result)?.why); +} + +function expectRowsKept(ctx: SqliteJourney, label: string): void { + const state = withDatabase(ctx.dbPath, (db) => ({ + rows: db.prepare(`SELECT id, name FROM "people" ORDER BY id`).all(), + tables: db + .prepare(`SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name`) + .all() + .map((row) => row['name']) + .filter((name) => !/^(_prisma|sqlite_)/.test(String(name))), + })); + expect(state, `${label}: the rows are in the renamed table`).toEqual({ + rows: [ + { id: 1, name: 'alice' }, + { id: 2, name: 'bob' }, + ], + tables: ['people'], + }); +} + +function seedRows(ctx: SqliteJourney): void { + withDatabase(ctx.dbPath, (db) => + db.exec(`INSERT INTO "profiles" (id, name) VALUES (1, 'alice'), (2, 'bob')`), + ); +} + +withTempDir(({ createTempDir }) => { + describe('Journey M1 (SQLite): the migration plan route keeps the rows', () => { + it( + 'writes the rename as a migration of its own, as the question prints it', + async () => { + const ctx = setupSqliteJourney(createTempDir, 'db/migrations'); + await emitContract(ctx, FROM_PSL, 'M1.01'); + const initial = await planMigrationAndSelfEmit(ctx, ['--name', 'initial']); + expect(initial.exitCode, `M1.02: plan initial: ${initial.stderr}`).toBe(0); + expect((await runMigrate(ctx)).exitCode, 'M1.03: migrate initial').toBe(0); + seedRows(ctx); + const origin = latestMigrationDirName(ctx); + const originHash = latestMigrationToHash(ctx); + await emitContract(ctx, TO_PSL, 'M1.04'); + + const refused = await runMigrationPlan(ctx, ['--from', origin, '--json']); + expect(refused.exitCode, 'M1.05: the drop of profiles needs a statement').toBe(2); + const question = questionOf(refused); + expect(question, 'M1.05: the question says how to keep the rows').toContain( + 'To keep its rows, rename table "profiles" to "people" instead', + ); + + const scaffold = await runMigrationNew(ctx, ['--name', 'people', '--from', originHash]); + expect(scaffold.exitCode, `M1.06: migration new: ${scaffold.stderr}`).toBe(0); + const dir = latestMigrationDirName(ctx); + const call = /add (\.\.\.this\.renameTable\(.*?\)) to the operations/.exec(question)?.[1]; + expect(call, `M1.07: the printed renameTable call in: ${question}`).toBeDefined(); + const migrationTs = join(ctx.testDir, 'db/migrations/app', dir, 'migration.ts'); + const scaffolded = readFileSync(migrationTs, 'utf-8'); + const edited = scaffolded.replace(/return \[[\s\S]*?\];/, () => `return [${call}];`); + expect(edited, 'M1.07: the call is in the operations').not.toBe(scaffolded); + writeFileSync(migrationTs, edited, 'utf-8'); + const emitted = runPrintedSelfEmit(question, dirname(migrationTs)); + expect(emitted.exitCode, `M1.08: node on migration.ts, as printed: ${emitted.stderr}`).toBe( + 0, + ); + + const applied = await runMigrate(ctx); + expect(applied.exitCode, `M1.09: db migrate: ${applied.stderr}`).toBe(0); + expectRowsKept(ctx, 'M1.10'); + const verify = await runDbVerify(ctx, ['--schema-only']); + expect(verify.exitCode, `M1.10: db verify --schema-only: ${verify.stderr}`).toBe(0); + const further = await runMigrationPlan(ctx, ['--from', dir, '--json']); + expect(further.exitCode, `M1.11: a further plan: ${further.stderr}`).toBe(0); + expect(parseJsonOutput<{ noOp: boolean }>(further).noOp, 'M1.11: it is empty').toBe(true); + }, + timeouts.spinUpPpgDev, + ); + }); + + describe('Journey M2 (SQLite): the db update route keeps the rows', () => { + it( + 'renames the table with the printed SQL, then db update finds nothing that loses data', + async () => { + const ctx = setupSqliteJourney(createTempDir); + await emitContract(ctx, FROM_PSL, 'M2.01'); + const create = await runDbUpdate(ctx, ['--json']); + expect(create.exitCode, `M2.02: db update creates the table: ${create.stderr}`).toBe(0); + seedRows(ctx); + await emitContract(ctx, TO_PSL, 'M2.03'); + + const refused = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(refused.exitCode, 'M2.04: the drop of profiles needs a statement').toBe(2); + const sql = /yourself with (.*?), then run prisma db update/.exec(questionOf(refused))?.[1]; + expect(sql, 'M2.04: the printed SQL').toBeDefined(); + withDatabase(ctx.dbPath, (db) => db.exec(String(sql))); + + const updated = await runDbUpdate(ctx, ['--no-interactive', '--json']); + expect(updated.exitCode, `M2.05: db update: ${updated.stderr}`).toBe(0); + expectRowsKept(ctx, 'M2.06'); + const verify = await runDbVerify(ctx, ['--schema-only']); + expect(verify.exitCode, `M2.06: db verify --schema-only: ${verify.stderr}`).toBe(0); + }, + timeouts.spinUpPpgDev, + ); + }); +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-removed.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-removed.ts new file mode 100644 index 000000000000..2cd00ec063df --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-removed.ts @@ -0,0 +1,22 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'events', + fields: { + _id: field.objectId(), + }, +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-renamed.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-renamed.ts new file mode 100644 index 000000000000..b710a78b811d --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-name-renamed.ts @@ -0,0 +1,23 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'events', + fields: { + _id: field.objectId(), + title: field.string(), + }, +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped-title.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped-title.ts new file mode 100644 index 000000000000..fbb0781c9cca --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped-title.ts @@ -0,0 +1,24 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'occasions', + fields: { + _id: field.objectId(), + title: field.string(), + }, + indexes: [index({ title: 1 })], +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped.ts new file mode 100644 index 000000000000..a5dfff876c3a --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-events-remapped.ts @@ -0,0 +1,23 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'occasions', + fields: { + _id: field.objectId(), + name: field.string(), + }, +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-indexed-events-remapped.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-indexed-events-remapped.ts new file mode 100644 index 000000000000..a9efe8e66c4c --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-indexed-events-remapped.ts @@ -0,0 +1,24 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'occasions', + fields: { + _id: field.objectId(), + name: field.string(), + }, + indexes: [index({ name: 1 }, { unique: true })], +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-with-indexed-events.ts b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-with-indexed-events.ts new file mode 100644 index 000000000000..6ccf183f7c74 --- /dev/null +++ b/test/integration/test/fixtures/cli/cli-e2e-test-app/fixtures/mongo-cli-journeys/contract-with-indexed-events.ts @@ -0,0 +1,24 @@ +import { defineContract, field, index, model } from '@internal/mongo/contract-builder'; + +const User = model('User', { + collection: 'users', + fields: { + _id: field.objectId(), + email: field.string(), + name: field.string(), + }, + indexes: [index({ email: 1 }, { unique: true }), index({ name: 1 })], +}); + +const Event = model('Event', { + collection: 'events', + fields: { + _id: field.objectId(), + name: field.string(), + }, + indexes: [index({ name: 1 }, { unique: true })], +}); + +export const contract = defineContract({ + models: { User, Event }, +}); diff --git a/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts b/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts new file mode 100644 index 000000000000..8679842e0bed --- /dev/null +++ b/test/integration/test/mongo/target-runner/runner-field-rewrites.test.ts @@ -0,0 +1,372 @@ +import { MongoControlAdapterImpl } from '@internal/adapter-mongo/control'; +import { MongoControlDriver } from '@internal/driver-mongo/control'; +import type { + MigrationOperationPolicy, + MigrationPlan, +} from '@internal/framework-components/control'; +import { buildFabricatedMigrationEdge } from '@internal/migration-tools/aggregate'; +import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; +import type { AnyMongoMigrationOperation } from '@internal/mongo-query-ast/control'; +import { RawUpdateManyCommand } from '@internal/mongo-query-ast/execution'; +import { MongoMigrationRunner, serializeMongoOps } from '@internal/target-mongo/control'; +import { + collMod, + createIndex, + dataTransform, + dropIndex, + renameField, + unsetField, +} from '@internal/target-mongo/migration'; +import { timeouts } from '@repo/test-utils'; +import { type Db, MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest'; + +const controlAdapter = new MongoControlAdapterImpl(); +const dbUpdatePolicy: MigrationOperationPolicy = { + allowedOperationClasses: ['additive', 'widening', 'destructive'], +}; + +let replSet: MongoMemoryReplSet; +let client: MongoClient; +let db: Db; +const dbName = 'runner_field_rewrites_test'; + +beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + db = client.db(dbName); +}, timeouts.spinUpMongoMemoryServer); + +afterAll(async () => { + await client?.close(); + await replSet?.stop(); +}, timeouts.spinUpMongoMemoryServer); + +beforeEach(async () => { + const collections = await db.listCollections().toArray(); + for (const col of collections) { + await db.dropCollection(col['name'] as string); + } + await controlAdapter.initMarker(new MongoControlDriver(db, client), 'app', { + storageHash: 'origin', + profileHash: 'p1', + }); +}); + +function closedSchema(field: string) { + return { + bsonType: 'object', + required: ['_id', field], + properties: { _id: { bsonType: 'objectId' }, [field]: { bsonType: 'string' } }, + additionalProperties: false, + }; +} + +function contractWith(collections: Record): MongoContract { + return { + target: 'mongo', + targetFamily: 'mongo', + profileHash: 'p1', + capabilities: {}, + extensions: {}, + meta: {}, + roots: {}, + models: {}, + storage: { + storageHash: 'dest', + namespaces: { + __unbound__: { + id: '__unbound__', + kind: 'mongo-namespace', + entries: { collection: collections }, + }, + }, + }, + } as unknown as MongoContract; +} + +const usersWithFullName = contractWith({ + users: new MongoCollection({ + indexes: [{ keys: [{ field: 'fullName', direction: 1 }], unique: true }], + validator: { + jsonSchema: closedSchema('fullName'), + validationLevel: 'strict', + validationAction: 'error', + }, + }), +}); + +const renameNamePlan: readonly AnyMongoMigrationOperation[] = [ + dropIndex('users', [{ field: 'name', direction: 1 }]), + renameField('users', 'name', 'fullName'), + createIndex('users', [{ field: 'fullName', direction: 1 }], { unique: true }), + collMod( + 'users', + { + validator: { $jsonSchema: closedSchema('fullName') }, + validationLevel: 'strict', + validationAction: 'error', + }, + { id: 'validator.users.update', label: 'Update validator on users' }, + ), +]; + +async function run( + operations: readonly AnyMongoMigrationOperation[], + destinationContract: MongoContract, +) { + const plan: MigrationPlan = { + targetId: 'mongo', + operations: JSON.parse(serializeMongoOps(operations)), + origin: { storageHash: 'origin' }, + destination: { storageHash: 'dest' }, + }; + const runner = new MongoMigrationRunner( + controlAdapter.createRunnerDependencies(new MongoControlDriver(db, client)), + ); + return runner.execute({ + plan, + migrationEdges: [ + buildFabricatedMigrationEdge({ + currentMarkerStorageHash: 'origin', + destinationStorageHash: 'dest', + operationCount: operations.length, + }), + ], + destinationContract, + policy: dbUpdatePolicy, + frameworkComponents: [], + }); +} + +async function createClosedUsers(): Promise { + await db.createCollection('users', { + validator: { $jsonSchema: closedSchema('name') }, + validationLevel: 'strict', + validationAction: 'error', + }); + await db.collection('users').createIndex({ name: 1 }, { unique: true, name: 'name_1' }); + await db.collection('users').insertMany([{ name: 'Ada' }, { name: 'Bo' }, { name: 'Cy' }]); +} + +async function documents(collectionName: string) { + return db + .collection(collectionName) + .find({}, { projection: { _id: 0 }, sort: { _id: 1 } }) + .toArray(); +} + +describe('MongoMigrationRunner - field rewrites', () => { + it( + 'renames a field required by a closed validator and indexed uniquely, then rebuilds the index on the new name', + async () => { + await createClosedUsers(); + + const result = await run(renameNamePlan, usersWithFullName); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(4); + const indexes = await db.collection('users').indexes(); + expect({ + documents: await documents('users'), + indexes: indexes.map(({ name, key, unique }) => ({ name, key, unique })), + }).toEqual({ + documents: [{ fullName: 'Ada' }, { fullName: 'Bo' }, { fullName: 'Cy' }], + indexes: [ + { name: '_id_', key: { _id: 1 }, unique: undefined }, + { name: 'fullName_1', key: { fullName: 1 }, unique: true }, + ], + }); + await expect( + db.collection('users').updateOne({ fullName: 'Ada' }, { $set: { fullName: 'Ada L.' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'skips every operation when run again after success', + async () => { + await createClosedUsers(); + const first = await run(renameNamePlan, usersWithFullName); + expect(first.ok).toBe(true); + await controlAdapter.updateMarker(new MongoControlDriver(db, client), 'app', 'dest', { + storageHash: 'origin', + profileHash: 'p1', + }); + + const second = await run(renameNamePlan, usersWithFullName); + + expect(second.ok ? second.value.operationsExecuted : second.failure).toBe(0); + expect(await documents('users')).toEqual([ + { fullName: 'Ada' }, + { fullName: 'Bo' }, + { fullName: 'Cy' }, + ]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'fails at the before-check and changes nothing when a document already holds the new name', + async () => { + await db.collection('users').insertMany([{ name: 'Ada' }, { name: 'Bo', fullName: 'Bo B.' }]); + + const result = await run([renameField('users', 'name', 'fullName')], usersWithFullName); + + expect(result.ok ? result.value : result.failure.code).toBe('MIGRATION.PRECHECK_FAILED'); + expect(await documents('users')).toEqual([ + { name: 'Ada' }, + { name: 'Bo', fullName: 'Bo B.' }, + ]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'finishes the plan once the documents its before-check names are fixed as its message says', + async () => { + await createClosedUsers(); + await db + .collection('users') + .insertOne({ name: 'Dee', fullName: 'Dee D.' }, { bypassDocumentValidation: true }); + + const stopped = await run(renameNamePlan, usersWithFullName); + if (stopped.ok) throw new Error('Expected the rename to stop at its before-check'); + const printedFind = /db\.getCollection\("users"\)\.find\((.*)\) in mongosh/.exec( + stopped.failure.why ?? '', + ); + const found = await db + .collection('users') + .find(JSON.parse(printedFind?.[1] ?? 'null'), { projection: { _id: 0 } }) + .toArray(); + expect({ + summary: stopped.failure.summary, + found, + indexes: (await db.collection('users').indexes()).map(({ name }) => name), + }).toEqual({ + summary: + 'Operation field.users.name.rename(fullName) stopped: some documents in collection users already hold fullName as well as name', + found: [{ name: 'Dee', fullName: 'Dee D.' }], + indexes: ['_id_'], + }); + + await db.collection('users').updateMany( + JSON.parse(printedFind?.[1] ?? 'null'), + { $unset: { fullName: '' } }, + { + bypassDocumentValidation: true, + }, + ); + const finished = await run(renameNamePlan, usersWithFullName); + + expect(finished.ok ? finished.value.operationsExecuted : finished.failure).toBe(3); + expect({ + documents: await documents('users'), + indexes: (await db.collection('users').indexes()).map(({ name }) => name), + }).toEqual({ + documents: [ + { fullName: 'Ada' }, + { fullName: 'Bo' }, + { fullName: 'Cy' }, + { fullName: 'Dee' }, + ], + indexes: ['_id_', 'fullName_1'], + }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'finishes a rename that stopped part way, where some documents already hold only the new name', + async () => { + await db.collection('users').insertMany([{ fullName: 'a' }, { name: 'b' }]); + + const result = await run( + [renameField('users', 'name', 'fullName')], + contractWith({ users: new MongoCollection({}) }), + ); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect(await documents('users')).toEqual([{ fullName: 'a' }, { fullName: 'b' }]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'removes a field from every document that holds it, past a closed validator', + async () => { + await db.createCollection('users', { + validator: { $jsonSchema: closedSchema('name') }, + validationLevel: 'strict', + validationAction: 'error', + }); + await db.collection('users').insertMany([{ name: 'Ada' }, { name: 'Bo' }]); + + const result = await run( + [unsetField('users', 'name')], + contractWith({ + users: new MongoCollection({ + validator: { + jsonSchema: closedSchema('name'), + validationLevel: 'strict', + validationAction: 'error', + }, + }), + }), + ); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect(await documents('users')).toEqual([{}, {}]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'limits both rewrites to documents that match the filter', + async () => { + await db.collection('posts').insertMany([ + { type: 'bug', title: 'Crash', score: 1 }, + { type: 'feature', title: 'Dark mode', score: 2 }, + ]); + + const result = await run( + [ + renameField('posts', 'title', 'headline', { filter: { type: 'bug' } }), + unsetField('posts', 'score', { filter: { type: 'bug' } }), + ], + contractWith({ posts: new MongoCollection({}) }), + ); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); + expect(await documents('posts')).toEqual([ + { type: 'bug', headline: 'Crash' }, + { type: 'feature', title: 'Dark mode', score: 2 }, + ]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'refuses a data transform under the same policy', + async () => { + await db.collection('posts').insertOne({ title: 'Crash' }); + const backfill = dataTransform('backfill-status', { + run: () => ({ + collection: 'posts', + command: new RawUpdateManyCommand('posts', {}, { $set: { status: 'open' } }), + meta: { target: 'mongo', storageHash: 'dest', lane: 'mongo-raw' }, + }), + }); + + const result = await run([backfill], contractWith({ posts: new MongoCollection({}) })); + + expect(result.ok ? result.value : result.failure.code).toBe('MIGRATION.POLICY_VIOLATION'); + expect(await documents('posts')).toEqual([{ title: 'Crash' }]); + }, + timeouts.spinUpMongoMemoryServer, + ); +}); diff --git a/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts b/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts new file mode 100644 index 000000000000..a07d1e821f2a --- /dev/null +++ b/test/integration/test/mongo/target-runner/runner-rename-collection.test.ts @@ -0,0 +1,218 @@ +import { MongoControlAdapterImpl } from '@internal/adapter-mongo/control'; +import { MongoControlDriver } from '@internal/driver-mongo/control'; +import type { MigrationPlan } from '@internal/framework-components/control'; +import { buildFabricatedMigrationEdge } from '@internal/migration-tools/aggregate'; +import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; +import { MongoMigrationRunner, serializeMongoOps } from '@internal/target-mongo/control'; +import { renameCollection } from '@internal/target-mongo/migration'; +import { timeouts } from '@repo/test-utils'; +import { type Db, MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest'; + +const controlAdapter = new MongoControlAdapterImpl(); + +let replSet: MongoMemoryReplSet; +let client: MongoClient; +let db: Db; +const dbName = 'runner_rename_collection_test'; + +beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + db = client.db(dbName); +}, timeouts.spinUpMongoMemoryServer); + +afterAll(async () => { + await client?.close(); + await replSet?.stop(); +}, timeouts.spinUpMongoMemoryServer); + +beforeEach(async () => { + const collections = await db.listCollections().toArray(); + for (const col of collections) { + await db.dropCollection(col['name'] as string); + } +}); + +const CLOSED_PROFILE_SCHEMA = { + bsonType: 'object', + required: ['_id', 'email'], + properties: { + _id: { bsonType: 'objectId' }, + email: { bsonType: 'string' }, + }, + additionalProperties: false, +} as const; + +function usersContract(): MongoContract { + const users = new MongoCollection({ + indexes: [{ keys: [{ field: 'email', direction: 1 }], unique: true }], + validator: { + jsonSchema: { ...CLOSED_PROFILE_SCHEMA }, + validationLevel: 'strict', + validationAction: 'error', + }, + }); + return { + target: 'mongo', + targetFamily: 'mongo', + profileHash: 'p1', + capabilities: {}, + extensions: {}, + meta: {}, + roots: {}, + models: {}, + storage: { + storageHash: 'dest', + namespaces: { + __unbound__: { + id: '__unbound__', + kind: 'mongo-namespace', + entries: { collection: { users } }, + }, + }, + }, + } as unknown as MongoContract; +} + +const renamePlan: MigrationPlan = { + targetId: 'mongo', + operations: JSON.parse(serializeMongoOps([renameCollection('profiles', 'users')])), + origin: { storageHash: 'origin' }, + destination: { storageHash: 'dest' }, +}; + +async function createProfiles(): Promise { + await db.createCollection('profiles', { + validator: { $jsonSchema: CLOSED_PROFILE_SCHEMA }, + validationLevel: 'strict', + validationAction: 'error', + }); + await db.collection('profiles').createIndex({ email: 1 }, { unique: true, name: 'email_1' }); + await db + .collection('profiles') + .insertMany([{ email: 'ada@example.com' }, { email: 'bo@example.com' }]); + await controlAdapter.initMarker(new MongoControlDriver(db, client), 'app', { + storageHash: 'origin', + profileHash: 'p1', + }); +} + +async function runRename() { + const runner = new MongoMigrationRunner( + controlAdapter.createRunnerDependencies(new MongoControlDriver(db, client)), + ); + return runner.execute({ + plan: renamePlan, + migrationEdges: [ + buildFabricatedMigrationEdge({ + currentMarkerStorageHash: 'origin', + destinationStorageHash: 'dest', + operationCount: 1, + }), + ], + destinationContract: usersContract(), + policy: { allowedOperationClasses: ['additive', 'widening', 'destructive'] }, + frameworkComponents: [], + }); +} + +async function collectionNames(): Promise { + const collections = await db.listCollections({ name: { $in: ['profiles', 'users'] } }).toArray(); + return collections.map((c) => c.name).sort(); +} + +async function emails(collectionName: string): Promise { + const docs = await db.collection(collectionName).find().toArray(); + return docs.map((d) => String(d['email'])).sort(); +} + +describe('MongoMigrationRunner - renameCollection', () => { + it( + 'renames a collection and keeps its documents, unique index and closed validator', + async () => { + await createProfiles(); + + const result = await runRename(); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + const [users] = await db.listCollections({ name: 'users' }, { nameOnly: false }).toArray(); + const indexes = await db.collection('users').indexes(); + expect({ + names: await collectionNames(), + emails: await emails('users'), + validator: users && 'options' in users ? users.options?.['validator'] : undefined, + emailIndex: indexes.find((i) => i.name === 'email_1'), + }).toEqual({ + names: ['users'], + emails: ['ada@example.com', 'bo@example.com'], + validator: { $jsonSchema: CLOSED_PROFILE_SCHEMA }, + emailIndex: expect.objectContaining({ key: { email: 1 }, unique: true }), + }); + await expect(db.collection('users').insertOne({ email: 'ada@example.com' })).rejects.toThrow( + /duplicate key/, + ); + await expect( + db.collection('users').insertOne({ email: 'cy@example.com', undeclared: true }), + ).rejects.toThrow(/Document failed validation/); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'skips the rename through its after-check when run again', + async () => { + await createProfiles(); + const first = await runRename(); + expect(first.ok).toBe(true); + const reset = await controlAdapter.updateMarker( + new MongoControlDriver(db, client), + 'app', + 'dest', + { storageHash: 'origin', profileHash: 'p1' }, + ); + expect(reset).toBe(true); + + const second = await runRename(); + + expect(second.ok ? second.value.operationsExecuted : second.failure).toBe(0); + expect({ names: await collectionNames(), emails: await emails('users') }).toEqual({ + names: ['users'], + emails: ['ada@example.com', 'bo@example.com'], + }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'fails at the before-check and changes nothing when the target collection exists', + async () => { + await createProfiles(); + await db.collection('users').insertOne({ email: 'already@example.com' }); + + const result = await runRename(); + + expect( + result.ok ? result.value : { code: result.failure.code, summary: result.failure.summary }, + ).toEqual({ + code: 'MIGRATION.PRECHECK_FAILED', + summary: + 'Operation collection.profiles.rename(users) stopped: collection users already exists, so collection profiles cannot be renamed to it', + }); + expect({ + names: await collectionNames(), + profiles: await emails('profiles'), + users: await emails('users'), + }).toEqual({ + names: ['profiles', 'users'], + profiles: ['ada@example.com', 'bo@example.com'], + users: ['already@example.com'], + }); + }, + timeouts.spinUpMongoMemoryServer, + ); +}); diff --git a/test/integration/test/mongo/target-runner/runner-statements-fixtures.ts b/test/integration/test/mongo/target-runner/runner-statements-fixtures.ts new file mode 100644 index 000000000000..290aa45ccf3c --- /dev/null +++ b/test/integration/test/mongo/target-runner/runner-statements-fixtures.ts @@ -0,0 +1,244 @@ +import { introspectSchema, MongoControlAdapterImpl } from '@internal/adapter-mongo/control'; +import { asNamespaceId } from '@internal/contract/types'; +import { MongoControlDriver } from '@internal/driver-mongo/control'; +import type { + MigrationOperationPolicy, + MigrationPlan, + ResolvedMigrationStatement, +} from '@internal/framework-components/control'; +import { buildFabricatedMigrationEdge } from '@internal/migration-tools/aggregate'; +import { MongoCollection, type MongoContract } from '@internal/mongo-contract'; +import type { AnyMongoMigrationOperation } from '@internal/mongo-query-ast/control'; +import { + MongoMigrationPlanner, + MongoMigrationRunner, + serializeMongoOps, +} from '@internal/target-mongo/control'; +import { timeouts } from '@repo/test-utils'; +import { type Db, MongoClient } from 'mongodb'; +import { MongoMemoryReplSet } from 'mongodb-memory-server'; +import { afterAll, beforeAll, beforeEach } from 'vitest'; + +const controlAdapter = new MongoControlAdapterImpl(); +export const NS = asNamespaceId('__unbound__'); +const policy: MigrationOperationPolicy = { + allowedOperationClasses: ['additive', 'widening', 'destructive'], +}; + +export function closedSchema(fields: readonly string[]) { + return { + bsonType: 'object', + required: ['_id', ...fields], + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ...fields.map((field) => [field, { bsonType: 'string' }]), + ]), + additionalProperties: false, + }; +} + +export interface ModelSpec { + readonly collection: string; + readonly fields: readonly string[]; + readonly base?: string; + readonly variants?: Readonly>; +} + +export function contract( + storageHash: string, + models: Readonly>, + collections: Readonly>, +): MongoContract { + return { + target: 'mongo', + targetFamily: 'mongo', + profileHash: 'p1', + capabilities: {}, + extensions: {}, + meta: {}, + roots: {}, + domain: { + namespaces: { + __unbound__: { + models: Object.fromEntries( + Object.entries(models).map(([name, spec]) => [ + name, + { + fields: Object.fromEntries(spec.fields.map((field) => [field, {}])), + relations: {}, + storage: { collection: spec.collection }, + ...(spec.base === undefined + ? {} + : { base: { namespace: '__unbound__', model: spec.base } }), + ...(spec.variants === undefined + ? {} + : { + discriminator: { field: 'type' }, + variants: Object.fromEntries( + Object.entries(spec.variants).map(([variant, value]) => [ + variant, + { value }, + ]), + ), + }), + }, + ]), + ), + }, + }, + }, + storage: { + storageHash, + namespaces: { + __unbound__: { + id: '__unbound__', + kind: 'mongo-namespace', + entries: { collection: collections }, + }, + }, + }, + } as unknown as MongoContract; +} + +export function closedCollection(fields: readonly string[], unique: readonly string[] = []) { + return new MongoCollection({ + indexes: unique.map((field) => ({ keys: [{ field, direction: 1 as const }], unique: true })), + validator: { + jsonSchema: closedSchema(fields), + validationLevel: 'strict', + validationAction: 'error', + }, + }); +} + +/** + * Starts a MongoDB replica set for the file, empties the database before each test, and returns + * the helpers that plan and run against it. + */ +export function useStatementsDatabase(dbName: string) { + let replSet: MongoMemoryReplSet | undefined; + let client: MongoClient | undefined; + let database: Db | undefined; + const live = (): Db => { + if (database === undefined) throw new Error('The database is not started'); + return database; + }; + const connected = (): MongoClient => { + if (client === undefined) throw new Error('The database is not started'); + return client; + }; + + beforeAll(async () => { + replSet = await MongoMemoryReplSet.create({ + replSet: { count: 1, storageEngine: 'wiredTiger' }, + }); + client = new MongoClient(replSet.getUri()); + await client.connect(); + database = client.db(dbName); + }, timeouts.spinUpMongoMemoryServer); + + afterAll(async () => { + await client?.close(); + await replSet?.stop(); + }, timeouts.spinUpMongoMemoryServer); + + beforeEach(async () => { + const collections = await live().listCollections().toArray(); + for (const col of collections) { + const name = col['name'] as string; + if (!name.startsWith('system.')) await live().dropCollection(name); + } + }); + + async function createClosed( + name: string, + fields: readonly string[], + unique: readonly string[], + documents: readonly Record[], + ) { + await live().createCollection(name, { + validator: { $jsonSchema: closedSchema(fields) }, + validationLevel: 'strict', + validationAction: 'error', + }); + for (const field of unique) { + await live() + .collection(name) + .createIndex({ [field]: 1 }, { unique: true, name: `${field}_1` }); + } + await live() + .collection(name) + .insertMany(documents.map((document) => ({ ...document }))); + } + + async function planAndRun( + from: MongoContract, + to: MongoContract, + statements: readonly ResolvedMigrationStatement[] = [], + ) { + await controlAdapter.initMarker(new MongoControlDriver(live(), connected()), 'app', { + storageHash: from.storage.storageHash, + profileHash: 'p1', + }); + const planned = new MongoMigrationPlanner().plan({ + contract: to, + schema: await introspectSchema(live()), + policy, + fromContract: from, + origin: { storageHash: from.storage.storageHash }, + statements, + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + if (planned.kind !== 'success') throw new Error(JSON.stringify(planned.conflicts, null, 2)); + const operations = planned.plan.operations as readonly AnyMongoMigrationOperation[]; + const plan: MigrationPlan = { + targetId: 'mongo', + operations: JSON.parse(serializeMongoOps(operations)), + origin: { storageHash: from.storage.storageHash }, + destination: { storageHash: to.storage.storageHash }, + }; + const result = await new MongoMigrationRunner( + controlAdapter.createRunnerDependencies(new MongoControlDriver(live(), connected())), + ).execute({ + plan, + migrationEdges: [ + buildFabricatedMigrationEdge({ + currentMarkerStorageHash: from.storage.storageHash, + destinationStorageHash: to.storage.storageHash, + operationCount: operations.length, + }), + ], + destinationContract: to, + policy, + frameworkComponents: [], + }); + return { planned, result }; + } + + async function furtherPlanLabels(to: MongoContract): Promise { + const planned = new MongoMigrationPlanner().plan({ + contract: to, + schema: await introspectSchema(live()), + policy, + fromContract: to, + origin: { storageHash: to.storage.storageHash }, + statements: [], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + if (planned.kind !== 'success') throw new Error(JSON.stringify(planned.conflicts)); + return (planned.plan.operations as readonly AnyMongoMigrationOperation[]).map( + (operation) => operation.label, + ); + } + + async function documents(name: string) { + return live() + .collection(name) + .find({}, { projection: { _id: 0 }, sort: { _id: 1 } }) + .toArray(); + } + + return { db: live, createClosed, planAndRun, furtherPlanLabels, documents }; +} diff --git a/test/integration/test/mongo/target-runner/runner-statements.test.ts b/test/integration/test/mongo/target-runner/runner-statements.test.ts new file mode 100644 index 000000000000..c8298b2f5964 --- /dev/null +++ b/test/integration/test/mongo/target-runner/runner-statements.test.ts @@ -0,0 +1,269 @@ +import { introspectSchema } from '@internal/adapter-mongo/control'; +import type { MigrationOperationPolicy } from '@internal/framework-components/control'; +import { MongoCollection } from '@internal/mongo-contract'; +import { MongoMigrationPlanner } from '@internal/target-mongo/control'; +import { timeouts } from '@repo/test-utils'; +import { describe, expect, it } from 'vitest'; +import { + closedCollection, + contract, + NS, + useStatementsDatabase, +} from './runner-statements-fixtures'; + +const live = useStatementsDatabase('runner_statements_test'); + +const policy: MigrationOperationPolicy = { + allowedOperationClasses: ['additive', 'widening', 'destructive'], +}; + +describe('MongoMigrationPlanner and runner - statements', () => { + it( + 'renames a model and a uniquely indexed field behind a closed validator, keeping every document', + async () => { + await live.createClosed('profiles', ['name'], ['name'], [{ name: 'Ada' }, { name: 'Bo' }]); + const from = contract( + 'from', + { Profile: { collection: 'profiles', fields: ['_id', 'name'] } }, + { profiles: closedCollection(['name'], ['name']) }, + ); + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'fullName'] } }, + { users: closedCollection(['fullName'], ['fullName']) }, + ); + + const { planned, result } = await live.planAndRun(from, to, [ + { + kind: 'rename', + entity: 'model', + from: { namespaceId: NS, model: 'Profile' }, + to: { namespaceId: NS, model: 'User' }, + }, + { + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: 'Profile', field: 'name' }, + to: { namespaceId: NS, model: 'User', field: 'fullName' }, + }, + ]); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(5); + expect(planned.kind === 'success' ? planned.dataLoss : undefined).toEqual([]); + expect({ + collections: ( + await live + .db() + .listCollections({ name: { $in: ['profiles', 'users'] } }) + .toArray() + ).map((c) => c.name), + users: await live.documents('users'), + further: await live.furtherPlanLabels(to), + }).toEqual({ + collections: ['users'], + users: [{ fullName: 'Ada' }, { fullName: 'Bo' }], + further: [], + }); + await expect( + live + .db() + .collection('users') + .updateOne({ fullName: 'Ada' }, { $set: { fullName: 'Ada L.' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect(live.db().collection('users').insertOne({ fullName: 'Bo' })).rejects.toThrow( + /duplicate key/, + ); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'removes a field the destination no longer has from every document', + async () => { + await live.createClosed( + 'users', + ['email', 'nickname'], + [], + [ + { email: 'a@x', nickname: 'a' }, + { email: 'b@x', nickname: 'b' }, + ], + ); + const from = contract( + 'from', + { User: { collection: 'users', fields: ['_id', 'email', 'nickname'] } }, + { users: closedCollection(['email', 'nickname']) }, + ); + const to = contract( + 'to', + { User: { collection: 'users', fields: ['_id', 'email'] } }, + { users: closedCollection(['email']) }, + ); + + const { planned, result } = await live.planAndRun(from, to); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); + expect(planned.kind === 'success' ? planned.dataLoss : undefined).toEqual([ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'User', field: 'nickname' }, + }, + ]); + expect({ + users: await live.documents('users'), + further: await live.furtherPlanLabels(to), + }).toEqual({ + users: [{ email: 'a@x' }, { email: 'b@x' }], + further: [], + }); + await expect( + live + .db() + .collection('users') + .updateOne({ email: 'a@x' }, { $set: { email: 'a@y' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + function readings(storageHash: string, model: string, collection: string) { + return contract( + storageHash, + { [model]: { collection, fields: ['_id', 'at'] } }, + { [collection]: new MongoCollection({ options: { timeseries: { timeField: 'at' } } }) }, + ); + } + + it( + 'plans nothing for a time-series collection that has not changed', + async () => { + const to = readings('to', 'Reading', 'readings'); + + const { result } = await live.planAndRun(contract('from', {}, {}), to); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect(await live.furtherPlanLabels(to)).toEqual([]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'keeps the documents of a time-series collection when its refusal is followed as printed', + async () => { + const from = readings('from', 'Reading', 'readings'); + const to = readings('to', 'Measurement', 'measurements'); + await live.db().createCollection('readings', { timeseries: { timeField: 'at' } }); + await live + .db() + .collection('readings') + .insertOne({ at: new Date('2026-01-01T00:00:00Z') }); + const refused = new MongoMigrationPlanner().plan({ + contract: to, + schema: await introspectSchema(live.db()), + policy, + fromContract: from, + origin: { storageHash: 'from' }, + statements: [ + { + kind: 'rename', + entity: 'model', + from: { namespaceId: NS, model: 'Reading' }, + to: { namespaceId: NS, model: 'Measurement' }, + }, + ], + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + expect(refused.kind === 'failure' ? refused.conflicts[0]?.why : refused.kind).toBe( + 'MongoDB cannot rename a time-series collection. Leave out this statement. To keep the documents, first create collection "measurements" yourself with the same time-series options and copy them into it; the plan then drops "readings" once --delete names the model.', + ); + + await live.db().createCollection('measurements', { timeseries: { timeField: 'at' } }); + const copied = await live + .db() + .collection('readings') + .find({}, { projection: { _id: 0 } }) + .toArray(); + await live.db().collection('measurements').insertMany(copied); + const { planned, result } = await live.planAndRun(from, to); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect( + planned.kind === 'success' ? planned.dataLoss.map(({ subject }) => subject) : [], + ).toEqual([{ kind: 'model', namespaceId: NS, model: 'Reading' }]); + expect({ + measurements: await live + .db() + .collection('measurements') + .find({}, { projection: { _id: 0 } }) + .toArray(), + readings: await live.db().listCollections({ name: 'readings' }).toArray(), + further: await live.furtherPlanLabels(to), + }).toEqual({ + measurements: [{ at: new Date('2026-01-01T00:00:00Z') }], + readings: [], + further: [], + }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'refuses at plan time to rename or remove a field of a live time-series collection, and plans nothing once it is kept', + async () => { + const withFields = (storageHash: string, fields: readonly string[]) => + contract( + storageHash, + { Reading: { collection: 'readings', fields: ['_id', 'at', ...fields] } }, + { + readings: new MongoCollection({ options: { timeseries: { timeField: 'at' } } }), + }, + ); + await live.db().createCollection('readings', { timeseries: { timeField: 'at' } }); + await live + .db() + .collection('readings') + .insertOne({ at: new Date('2026-01-01'), value: 1 }); + const planFor = async ( + to: ReturnType, + statements: Parameters[0]['statements'] = [], + ) => + new MongoMigrationPlanner().plan({ + contract: to, + schema: await introspectSchema(live.db()), + policy, + fromContract: withFields('from', ['value']), + origin: { storageHash: 'from' }, + statements, + frameworkComponents: [], + snapshotsImportPath: '../../snapshots', + }); + + const renamed = await planFor(withFields('to', ['reading']), [ + { + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: 'Reading', field: 'value' }, + to: { namespaceId: NS, model: 'Reading', field: 'reading' }, + }, + ]); + const removed = await planFor(withFields('to', [])); + const kept = await planFor(withFields('to', ['value'])); + + expect({ + renamed: renamed.kind === 'failure' ? renamed.conflicts.map(({ summary }) => summary) : [], + removed: removed.kind === 'failure' ? removed.conflicts.map(({ summary }) => summary) : [], + kept: kept.kind === 'success' ? kept.plan.operations.length : kept.conflicts, + }).toEqual({ + renamed: [ + 'Cannot rename field "Reading.value" to "Reading.reading": collection "readings" is a time-series collection', + ], + removed: [ + 'Cannot remove field "value" from collection "readings": it is a time-series collection', + ], + kept: 0, + }); + }, + timeouts.spinUpMongoMemoryServer, + ); +}); diff --git a/test/integration/test/mongo/target-runner/runner-statements.variants.test.ts b/test/integration/test/mongo/target-runner/runner-statements.variants.test.ts new file mode 100644 index 000000000000..c9634ddc83e1 --- /dev/null +++ b/test/integration/test/mongo/target-runner/runner-statements.variants.test.ts @@ -0,0 +1,195 @@ +import { MongoCollection } from '@internal/mongo-contract'; +import { timeouts } from '@repo/test-utils'; +import { describe, expect, it } from 'vitest'; +import { contract, NS, useStatementsDatabase } from './runner-statements-fixtures'; + +const live = useStatementsDatabase('runner_statements_variants_test'); + +function branchSchema(value: string, fields: readonly string[]) { + return { + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ['type', { enum: [value] }], + ...fields.map((field) => [field, { bsonType: 'string' }]), + ]), + required: ['type'], + additionalProperties: false, + }; +} + +function tasksValidator(base: readonly string[], bug: readonly string[]) { + return { + bsonType: 'object', + required: ['_id', 'type'], + properties: Object.fromEntries([ + ['_id', { bsonType: 'objectId' }], + ['type', { bsonType: 'string' }], + ...base.map((field) => [field, { bsonType: 'string' }]), + ]), + oneOf: [branchSchema('bug', [...base, ...bug]), branchSchema('feature', [...base, 'priority'])], + }; +} + +function tasks(storageHash: string, base: readonly string[], bug: readonly string[]) { + return contract( + storageHash, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type', ...base], + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: bug, base: 'Task' }, + Feature: { collection: 'tasks', fields: ['priority'], base: 'Task' }, + }, + { + tasks: new MongoCollection({ + validator: { + jsonSchema: tasksValidator(base, bug), + validationLevel: 'strict', + validationAction: 'error', + }, + }), + }, + ); +} + +async function createTasksWithNote(): Promise { + await live.db().createCollection('tasks', { + validator: { $jsonSchema: tasksValidator(['note'], []) }, + validationLevel: 'strict', + validationAction: 'error', + }); + await live + .db() + .collection('tasks') + .insertMany([ + { type: 'bug', note: 'b' }, + { type: 'feature', note: 'f' }, + ]); +} + +describe('MongoMigrationPlanner and runner - statements on variants', () => { + it( + "renames a variant's field only in its own documents, so another variant's field of that name keeps its values", + async () => { + await live + .db() + .collection('tasks') + .insertMany([ + { type: 'bug', score: 3 }, + { type: 'feature', score: 5 }, + ]); + const tasks = (storageHash: string, bugField: string) => + contract( + storageHash, + { + Task: { + collection: 'tasks', + fields: ['_id', 'type'], + variants: { Bug: 'bug', Feature: 'feature' }, + }, + Bug: { collection: 'tasks', fields: [bugField], base: 'Task' }, + Feature: { collection: 'tasks', fields: ['score'], base: 'Task' }, + }, + { tasks: new MongoCollection({}) }, + ); + + const { result } = await live.planAndRun(tasks('from', 'score'), tasks('to', 'severity'), [ + { + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: 'Bug', field: 'score' }, + to: { namespaceId: NS, model: 'Bug', field: 'severity' }, + }, + ]); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(1); + expect(await live.documents('tasks')).toEqual([ + { type: 'bug', severity: 3 }, + { type: 'feature', score: 5 }, + ]); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'removes a base field moved to one variant from the documents of every other variant', + async () => { + await createTasksWithNote(); + const to = tasks('to', [], ['note']); + + const { planned, result } = await live.planAndRun(tasks('from', ['note'], []), to); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); + expect(planned.kind === 'success' ? planned.dataLoss : undefined).toEqual([ + { + operationIndex: 0, + subject: { kind: 'field', namespaceId: NS, model: 'Task', field: 'note' }, + }, + ]); + expect({ + tasks: await live.documents('tasks'), + further: await live.furtherPlanLabels(to), + }).toEqual({ + tasks: [{ type: 'bug', note: 'b' }, { type: 'feature' }], + further: [], + }); + await expect( + live + .db() + .collection('tasks') + .updateOne({ type: 'feature' }, { $set: { priority: 'p1' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + live + .db() + .collection('tasks') + .updateOne({ type: 'bug' }, { $set: { note: 'b2' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + }, + timeouts.spinUpMongoMemoryServer, + ); + + it( + 'renames a base field moved to one variant only in the documents of the other variants', + async () => { + await createTasksWithNote(); + const to = tasks('to', ['remark'], ['note']); + + const { result } = await live.planAndRun(tasks('from', ['note'], []), to, [ + { + kind: 'rename', + entity: 'field', + from: { namespaceId: NS, model: 'Task', field: 'note' }, + to: { namespaceId: NS, model: 'Task', field: 'remark' }, + }, + ]); + + expect(result.ok ? result.value.operationsExecuted : result.failure).toBe(2); + expect({ + tasks: await live.documents('tasks'), + further: await live.furtherPlanLabels(to), + }).toEqual({ + tasks: [ + { type: 'bug', note: 'b' }, + { type: 'feature', remark: 'f' }, + ], + further: [], + }); + await expect( + live + .db() + .collection('tasks') + .updateOne({ type: 'feature' }, { $set: { priority: 'p1' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + await expect( + live + .db() + .collection('tasks') + .updateOne({ type: 'bug' }, { $set: { note: 'b2' } }), + ).resolves.toMatchObject({ modifiedCount: 1 }); + }, + timeouts.spinUpMongoMemoryServer, + ); +}); diff --git a/test/integration/test/utils/cli-test-helpers.ts b/test/integration/test/utils/cli-test-helpers.ts index 5541749d33fc..705496b16f14 100644 --- a/test/integration/test/utils/cli-test-helpers.ts +++ b/test/integration/test/utils/cli-test-helpers.ts @@ -494,9 +494,9 @@ class CapturingWritable extends Writable { * own tests use. * * Two process globals are saved and restored around the run, because the - * migration-file CLI is written for a process it owns: config discovery walks - * up from `process.cwd()` (so the helper chdirs to `cwd`, exactly where the - * old spawn pointed the child), and a failing run sets `process.exitCode` + * migration-file CLI is written for a process it owns: config discovery falls + * back to the config in `process.cwd()` (so the helper chdirs to `cwd`, + * exactly where the old spawn pointed the child), and a failing run sets `process.exitCode` * (which must not leak into the vitest worker's exit status when a test * asserts on a migration failure). Tests within a worker run sequentially * under the forks pool, so the temporary chdir cannot interleave with another diff --git a/test/integration/test/utils/journey-test-helpers.ts b/test/integration/test/utils/journey-test-helpers.ts index 82abd945eb55..40e357a7d796 100644 --- a/test/integration/test/utils/journey-test-helpers.ts +++ b/test/integration/test/utils/journey-test-helpers.ts @@ -6,6 +6,7 @@ * so journey tests stay concise and readable. */ +import { spawnSync } from 'node:child_process'; import { copyFileSync, existsSync, @@ -54,6 +55,8 @@ export interface JourneyContext { testDir: string; configPath: string; outputDir: string; + /** The config's `migrations.dir`, relative to `testDir`, when it is not `migrations`. */ + migrationsDir?: string; } // Re-export timeouts so journey tests can import from a single module. @@ -403,7 +406,7 @@ export async function authorMigration( throw new Error('authorMigration: the scaffold has no operations array to fill'); } writeFileSync(migrationTsPath, authored, 'utf-8'); - const emit = await selfEmitMigration(ctx, ['--dir', `migrations/app/${dirName}`]); + const emit = await selfEmitMigration(ctx, ['--dir', join(appMigrationsDir(ctx), dirName)]); return { dirName, emit }; } @@ -550,6 +553,67 @@ export async function selfEmitMigration( return runMigrationFile(migrationTs, args, ctx.testDir); } +/** + * Follows a message that says to run a migration's `migration.ts` with node: runs `node migration.ts` inside `packageDir`, the migration package directory, as a user would. + */ +export function runPrintedSelfEmit(message: string, packageDir: string): CommandResult { + if (!/\bnode on that migration\.ts\b/.test(message)) { + return { + exitCode: -1, + stdout: '', + stderr: `the message does not say to run node on that migration.ts: ${message}`, + }; + } + const run = spawnSync(process.execPath, ['migration.ts'], { cwd: packageDir, encoding: 'utf-8' }); + return { exitCode: run.status ?? -1, stdout: run.stdout, stderr: run.stderr }; +} + +/** + * Splits the body of a `return [ ... ]` operations list into its top-level entries, keeping each entry's source text. + */ +function operationEntries(body: string): string[] { + const entries: string[] = []; + let depth = 0; + let quote: string | undefined; + let start = 0; + for (let i = 0; i < body.length; i++) { + const char = body[i]!; + if (quote !== undefined) { + if (char === '\\') i++; + else if (char === quote) quote = undefined; + } else if (char === "'" || char === '"' || char === '`') quote = char; + else if ('([{'.includes(char)) depth++; + else if (')]}'.includes(char)) depth--; + else if (char === ',' && depth === 0) { + entries.push(body.slice(start, i)); + start = i + 1; + } + } + entries.push(body.slice(start)); + return entries.map((entry) => entry.trim()).filter((entry) => entry !== ''); +} + +/** + * Edits a MongoDB `migration.ts` as the advice to keep a dropped collection's documents says: `renameCollection(from, to)` becomes the first operation, imported with the others, and the drop of `from` and any create of `to` are deleted. + */ +export function followRenameCollectionAdvice(source: string, from: string, to: string): string { + const operations = /(override get operations\(\) \{\s*return \[)([\s\S]*?)(\];\s*\})/; + const match = operations.exec(source); + if (match === null) throw new Error(`no operations list in:\n${source}`); + const isCall = (entry: string, factory: string, collection: string) => + new RegExp(`^${factory}\\(\\s*['"]${collection}['"]`).test(entry); + const kept = operationEntries(match[2]!).filter( + (entry) => !isCall(entry, 'dropCollection', from) && !isCall(entry, 'createCollection', to), + ); + const entries = [`renameCollection(${JSON.stringify(from)}, ${JSON.stringify(to)})`, ...kept]; + return source + .replace( + operations, + (_all, open: string, _body, close: string) => `${open}${entries.join(', ')}${close}`, + ) + .replace(/import \{/, 'import { renameCollection,'); +} + /** * Runs `migration plan` (which scaffolds a draft migration.ts), then runs * that migration.ts in-process so it writes its own ops.json and @@ -567,7 +631,7 @@ export async function planMigrationAndSelfEmit( if (planResult.exitCode !== 0) return planResult; const latest = getLatestMigrationDir(ctx); if (!latest) return planResult; - const emitResult = await selfEmitMigration(ctx, ['--dir', `migrations/app/${latest}`]); + const emitResult = await selfEmitMigration(ctx, ['--dir', join(appMigrationsDir(ctx), latest)]); if (emitResult.exitCode !== 0) { return { ...planResult, @@ -745,7 +809,7 @@ export function migrationStatusAppSpace( * test root (`migrations/app/`). */ function appMigrationsDir(ctx: JourneyContext): string { - return join(ctx.testDir, 'migrations', 'app'); + return join(ctx.testDir, ctx.migrationsDir ?? 'migrations', 'app'); } /** diff --git a/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md b/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md new file mode 100644 index 000000000000..1e72f1003225 --- /dev/null +++ b/upgrade-instructions/pending/mongo-rename-delete-statements/app/instructions.md @@ -0,0 +1,72 @@ +--- +changes: + - id: mongo-rename-statements-carried-out + summary: | + On MongoDB, `migration plan` and `db update` carry out `--rename`: a model rename renames its collection, keeping the documents, indexes and validator, and a field rename renames the field in every document that holds it. They used to refuse every statement with `MIGRATION.PLANNING_FAILED`. A question about a dropped collection now offers `--rename` as well as `--delete`. + - id: mongo-removed-field-asks + summary: | + On MongoDB, removing a field from a model is now data loss. `migration plan` and `db update` ask about it, and fail with `CLI.CONSENT_REQUIRED` where nobody can answer, until `--delete .` or a `--rename` answers it. `--delete` removes the field from every document that holds it. + - id: mongo-preview-get-collection + summary: | + A MongoDB plan's preview writes each collection as `db.getCollection("users")`, for example `db.getCollection("users").createIndex(...)` where it wrote `db.users.createIndex(...)`. `ops.json` does not change. + detection: + glob: "**/*.{ts,mts,cts,js,mjs,cjs,snap}" + matches: + - 'db\.[A-Za-z_][\w]*\.(?:createIndex|dropIndex|drop)\(' + - id: mongo-time-series-update + summary: | + `db update` on a database with a MongoDB time-series collection no longer fails with "Cannot change immutable collection option 'timeseries'" when nothing changed. + - id: storage-question-wording + summary: | + A data-loss question about data named by its storage name, because the origin contract is unknown, ends with `--delete loses that data` instead of `--delete loses its rows`. + detection: + glob: "**/*.{ts,mts,cts,js,mjs,cjs}" + matches: + - '--delete loses its rows' + - id: migration-ts-finds-its-config + summary: | + Running a `migration.ts` with node and no `--config` uses the config that owns the migration, whose migrations directory holds the file: the nearest `prisma.config.ts` above the migration's directory, else the one in the working directory. If neither owns it, it fails with `CLI.MIGRATION_CONFIG_NOT_FOUND`. It used to load only the working directory's config, whichever project it belonged to. +--- + +# MongoDB carries out `--rename` and `--delete` + +## `mongo-rename-statements-carried-out` + +`--rename` now works on MongoDB, with the same syntax as on SQL. A field is named by its contract key, which on MongoDB is its stored name: `id @map("_id")` is `User._id`. + +```sh +prisma migration plan --name tidy --rename Profile:User --rename User.name:User.fullName +prisma db update --rename Profile:User --rename User.name:User.fullName +``` + +- A model rename whose collection name changes renames the collection (`renameCollection`), keeping its documents, indexes and validator. +- A field rename renames the field in every document that holds it (`renameField`), skipping validation. A variant's field is renamed only in that variant's documents. +- The planner refuses, with the reason and what to do instead: a variant rename whose discriminator value changes, `_id`, a field whose stored name contains `.` or starts with `$`, a time-series collection, a target collection that already exists or that MongoDB rejects, and a collection whose control policy is not `managed`. A field removed from a model whose collection is not `managed` is not asked about and not removed. + +If you renamed a collection or a field by hand in `mongosh` because `--rename` was refused, nothing changes for migrations you already applied. From now on, give the `--rename` statement instead. + +A script or test that expects `--rename` on MongoDB to fail with `MIGRATION.PLANNING_FAILED` now gets a plan. A script that reads `meta.unanswered` of `CLI.CONSENT_REQUIRED` for a dropped MongoDB collection now finds `verbs: ['rename', 'delete']` instead of `['delete']`. + +## `mongo-removed-field-asks` + +Removing a field from a MongoDB model used to update the collection's validator and ask nothing. Documents that still held the field then failed every update with `Document failed validation`. Now the plan removes the field from those documents (`unsetField`, a `destructive` operation) and asks first. Where nobody can answer, the command fails with `CLI.CONSENT_REQUIRED`, whose next actions name `--delete User.nickname` and `--rename 'User.nickname:User.'`. + +In a script or CI job that runs `prisma db update --no-interactive` or `prisma migration plan` after a field was removed, add `--delete .` for each removed field, or `--rename` if the field was renamed. When `db update` has no snapshot of the contract the database is at, the field is named by its collection: `--delete profiles.nickname`. + +## `mongo-preview-get-collection` + +A test or a script that compares the preview of a MongoDB plan (`migration plan`, `db update --dry-run`, `migration show`) expects `db.getCollection("")` in place of `db.`, so a collection name such as `my-coll` is valid `mongosh`. Plans and `ops.json` are unchanged. + +## `mongo-time-series-update` + +A repeated `db update` on a database whose contract declares a time-series collection used to fail with "Cannot change immutable collection option 'timeseries'", because MongoDB adds defaults such as `bucketMaxSpanSeconds` that the contract leaves out. The planner now compares only the options the contract declares, so the command succeeds and plans nothing. A workaround that skipped `db update` for such a database is no longer needed. + +## `storage-question-wording` + +A test that asserts the text of a data-loss question about data named by its storage name expects `--delete loses that data` where it expected `--delete loses its rows`. + +## `migration-ts-finds-its-config` + +Without `--config`, `node migration.ts` now picks the config that owns the migration: a config owns it when the migration file is inside that config's `migrations.dir`. It tries the nearest `prisma.config.ts` above the migration's directory first, then the `prisma.config.ts` in the working directory. So the file runs from inside its own directory, from the project directory, or from any other directory, and with `migrations.dir` outside the project, such as `../shared-db/migrations`, when run from the project directory. If neither config owns the migration, it fails with `CLI.MIGRATION_CONFIG_NOT_FOUND`, naming the configs it checked; run it with `--config `. `--config` still overrides the lookup. + +A command that ran the file from the project directory keeps working. A script that ran it from a directory whose `prisma.config.ts` belongs to another project, and relied on that config, now fails with `CLI.MIGRATION_CONFIG_NOT_FOUND`; pass `--config`. `migration new` and `migration plan` given `--config` now repeat it in the `node` command they print. Messages that tell you to run a `migration.ts` say to run it with node, without a path or a working directory. diff --git a/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md b/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md new file mode 100644 index 000000000000..5101fde03895 --- /dev/null +++ b/upgrade-instructions/pending/mongo-rename-delete-statements/extension/instructions.md @@ -0,0 +1,83 @@ +--- +changes: + - id: rename-statements-capability-removed + summary: | + `TargetMigrationsCapability.renameStatements` (with `refused` and `keepDataByHand`), added in 8.0.0-rc.17, is removed: every target's planner carries out rename statements. The CLI no longer appends a target's text to a data-loss question. + detection: + glob: "**/*.{ts,mts,cts}" + matches: + - '(?` implementation adds `renameCollection(command)`. A Mongo driver's `run(wireCommand)` handles `kind: 'renameCollection'`, as the shipped driver does with `db.renameCollection(collection, to)`, without `dropTarget`. An adapter's DDL lowering maps the command to `new RenameCollectionWireCommand(command.collection, command.to)`. + +## `mongo-dml-operations` + +Rename the types where code names them: + +```diff +- import type { MongoDataTransformCheck, MongoDataTransformOperation } from '@prisma/orm-family-mongo/query-ast/control'; ++ import type { MongoDmlCheck, MongoDmlOperation } from '@prisma/orm-family-mongo/query-ast/control'; +``` + +Where code narrows `AnyMongoMigrationOperation` by `operation.operationClass === 'data'`, use the guard instead: + +```diff +- if (operation.operationClass === 'data') { ++ if (isMongoDmlOperation(operation)) { +``` + +`isMongoDmlOperation` comes from `@prisma/orm-family-mongo/query-ast/control`. An operation without `run` is a DDL operation. The `dataTransform` factory keeps its name: it builds a DML operation with class `data`. A hand-written `dataTransform` still has class `data`, so a policy that leaves out `data` (as `db update`'s does) still refuses it, and runs `renameField` and `unsetField`. + +## `mongo-update-many-bypass-validation` + +A hand-built lowered draft of kind `rawUpdateMany` adds `bypassDocumentValidation: false`. A driver's `updateMany` passes `bypassDocumentValidation: true` to MongoDB when the wire command carries it. + +## `mongo-op-factory-call-to-op` + +Code typed against the result of `toOp()` or `renderOps(...)` as `MongoMigrationPlanOperation` widens it to `AnyMongoMigrationOperation`, and narrows with `isMongoDmlOperation` where it needs `execute` or `run`.