diff --git a/.drive/projects/emulator-commands/deferred.md b/.drive/projects/emulator-commands/deferred.md index f78e3206e..0796f1d18 100644 --- a/.drive/projects/emulator-commands/deferred.md +++ b/.drive/projects/emulator-commands/deferred.md @@ -4,3 +4,5 @@ - **What a previous app's databases and buckets become when another app runs from the same directory.** Compute instances are deleted (S1a). Databases and buckets hold data; decide in S1b. - **`[dev]` notices bypass the CLI engine's output.** The emulators hook writes `[dev] compute emulator ready`, the port-change line and the app-switch line with `console.log` to stdout, so a piped `prisma dev` mixes them into the JSON frames. Fixing it needs a channel for the local target's hooks to report through the engine. - **A deployment that arrives while its instance is being deleted can start a process the emulator no longer tracks.** Same race as #353's app delete. +- **`env.json` and `secrets.json` in `.prisma-composer/dev/` are shared by every app run from a directory, and `--fresh` wipes them.** One app's `--fresh` can drop rows another app in the same directory needs on its next redeploy. Pre-existing. Making them per app, like bucket objects (S1b moved those to `buckets///`), would fix it. +- **A service that never listens fails `prisma dev` under prisma/composer#353** (the cron scheduler; `examples/store`). Raised with Will; the fix belongs in #353. diff --git a/.drive/projects/emulator-commands/slices/s1b-data-instances/spec.md b/.drive/projects/emulator-commands/slices/s1b-data-instances/spec.md new file mode 100644 index 000000000..3090470e8 --- /dev/null +++ b/.drive/projects/emulator-commands/slices/s1b-data-instances/spec.md @@ -0,0 +1,76 @@ +# S1b — The Postgres and bucket emulators know nothing about Composer + +Repo: prisma/composer. Branch `feat/emulator-data-instances`, stacked on S1a (`feat/emulator-instances`, prisma/composer#354). PR base: that branch until it merges. + +## At a glance + +After S1a, only the Compute emulator follows ADR-0052. This slice moves the Postgres and bucket emulators to the same model, makes `prisma dev` stop its Postgres servers when it ends, and lets one directory keep the local data of more than one app. + +```text +~/work/shop/.prisma-composer/dev/emulator-instances.json + { "current": "shop", + "apps": { + "shop": { "nodes": { + "quotes": { "kind": "service", "id": "svc_k3f9…", "port": 3000 }, + "catalog": { "kind": "database", "id": "db_02c4…" }, + "uploads": { "kind": "bucket", "id": "bkt-77e1…" } } }, + "shop-v2": { "nodes": { … } } } } +``` + +Two checkouts of one app no longer share a local database or a bucket. + +## Chosen design + +**Postgres emulator.** +- A database is an instance with an emulator-generated id and labels, hosted on a `@prisma/dev` server named after that id. Every operation takes the id. +- A route lists all instances. +- New: **stop** shuts the instance's server down and keeps its data; **start** brings it back on the same port with the same data. Delete removes the server and its data. A stopped database keeps its port, by the same reasoning as Compute: its connection URL is in its dependents' environments. +- Old app-keyed servers and state are discarded on load. If `@prisma/dev` keeps data for them outside the emulator's own directory, remove it there too, or document exactly what is left behind. + +**Bucket emulator.** +- A bucket is an instance with an emulator-generated id and labels. Its name on the S3 wire is that id, replacing `--`. +- Its data root is the in-project directory Composer registers, `.prisma-composer/dev/buckets///` (the app is in the path so two apps in one directory don't share a bucket's files). The emulator stores that path verbatim and never interprets it. +- Credentials are registered per instance. A route lists all instances. +- A bucket has no process, so stop and start don't apply. + +**The record holds every app run from the directory.** This changes S1a's rule that switching apps deletes the previous app's Compute instances. +- `current` names the app `prisma dev` last ran here. `prisma dev url` (S3) resolves addresses in it. +- Switching apps stops the previous app's instances and keeps them, with their data and ports. Switching back resumes them warm. +- `--fresh` deletes the current app's instances and its entry. +- This follows Will's statement that the mapping is keyed by directory, app and topology. It also avoids deleting a database because a root module was renamed or run with a different `--name`. + +**`prisma dev` ends.** On Ctrl-C, SIGTERM or a failure, `prisma dev` stops the current app's service processes, as today, and also its Postgres servers. The next start brings them back with their data. + +**Everything else reaches databases and buckets through the record.** That includes the providers (`Database`, `Connection`, `Bucket`, `BucketKey`), attach, teardown, and `PgWarm`/`OrmMigration`, which already use the provider's URL. The emulators hook ensures them, as it does for Compute. Every instance gets the three `composer.*` labels. + +**Docs.** +- ADR-0052 is extended in place, since it is unreleased, to cover databases, buckets, stop and the multi-app record. ADR-0041's amendment note is adjusted if needed. +- `local-dev.md`, the running-locally guide and the core-concepts skill are updated, removing every "Postgres and buckets are still keyed by the app" caveat. +- The guide's note that `PRISMA_COMPOSER_EMULATORS_DIR` doesn't isolate Postgres goes, because generated ids make checkouts independent. + +## Scope + +In: the Postgres and bucket emulators and their clients; the record's multi-app shape and the switch rule for all kinds; stopping Postgres servers on exit; the data providers, attach and teardown; tests; docs. + +Out: `dev url`, `dev stop` and `emulator list` (S3); removing instances left by deleted directories. + +## Pre-investigated edge cases + +| Case | Expected | +| --- | --- | +| Warm start after Ctrl-C | Same database, same port, data intact, migrations no-op | +| A recorded database missing from the emulator | Re-created, requesting the remembered port; empty data, so migrations replay as on a fresh start | +| Switch app A → B → A | A's services, databases and buckets resume with their data and ports | +| Two checkouts of one app, each writes a row | Each reads only its own row; each lists only its own bucket objects | + +## Slice-specific done conditions + +- [ ] Two directories running one app with a database and a bucket each see only their own rows and objects (real emulators, temporary emulators directory). +- [ ] After `prisma dev` ends, no Postgres server of that app is running; the next start has the data. + +## Dispatch plan + +1. **Postgres emulator and client:** instances, labels, list, stop and start keeping data, delete. *Hands to 2.* +2. **Bucket emulator and client:** instances, labels, list, per-instance credentials. *Hands to 3.* +3. **Composer side:** the multi-app record, the data providers, attach and stop-on-exit, teardown, the two-directory data test. *Hands to 4.* +4. **Docs.** *Hands to:* the PR. diff --git a/docs/design/10-domains/local-dev.md b/docs/design/10-domains/local-dev.md index dbaaf940a..d62c49a86 100644 --- a/docs/design/10-domains/local-dev.md +++ b/docs/design/10-domains/local-dev.md @@ -60,11 +60,13 @@ Dev re-runs [deploy's pipeline](deploy-cli.md#the-pipeline) with these deltas: `localTarget.emulators` hook inspects the loaded graph and ensures one emulator daemon per node kind the topology uses (Compute always; buckets when bucket resources exist; Postgres when `postgres` or `raw-postgres` - resources exist). Once Compute is up, the hook ensures each service's - Compute instance, in `graph.nodes` order, through the directory's record - (see [Which instance backs a node](#which-instance-backs-a-node)). The - Postgres daemon starts no server here: the database provider asks it for - one `@prisma/dev` server per `Database` at provision. + resources exist). Once they are up, the app becomes the directory's + current app, and the hook ensures each node's instance — service, + database or bucket — in `graph.nodes` order, through the directory's + record (see [Which instance backs a node](#which-instance-backs-a-node)). + A recorded database the last session stopped is started here: on a warm + start the converge may change nothing and never call the Database + provider. 5. **Lower + converge** — a dev-generated stack file (ADR-0007's pattern, at `.prisma-composer/dev/alchemy.run.ts`), driven with the extension's `localTarget.providers()` layer and `devState()` (Alchemy's built-in file store, exported by `@prisma/composer/local-target`), always @@ -72,8 +74,9 @@ Dev re-runs [deploy's pipeline](deploy-cli.md#the-pipeline) with these deltas: service, a database, a bucket) by talking to the emulators; converge terminates as always, and the Compute emulator keeps serving. 6. **Attach** — new, dev-only: through `localTarget.attach`, render the front - door (every service's endpoint) and watch for rebuilds, loop. Ctrl-C stops - the app's service instances through the attachment and exits; emulators and + door (every service's endpoint) and watch for rebuilds, loop. Ctrl-C, + SIGTERM or a failure stops the app's service processes and database + servers through the attachment's `stopApp` and exits; emulators and data persist. Logs are not streamed here — the `log` operation reads the same attachment's merged-log view on demand, so `dev`'s own output stays the front door plus lifecycle and rebuild notices. @@ -136,10 +139,10 @@ service's instance, and the emulator owns the processes: another app run from the same directory) → stop it, remove it and its log, and free its port. - **Ctrl-C on the dev command** → the attachment stops the app's service - instances and detaches. Every emulator — Compute, Postgres instances, the - bucket emulator — stays up with its data; the next start reuses the - same instances (same ports, same data). `--fresh` is what removes - instances and data. (A detached mode where services keep serving with no + instances and its database servers, and detaches. Every emulator — + Compute, Postgres, buckets — stays up, and every instance keeps its data + and its port; the next start reuses the same instances (same ports, same + data). `--fresh` is what removes instances and data. (A detached mode where services keep serving with no session is a designed extension, not v1.) **Known gap (proving-pass finding, fixed in #164):** a service actually only restarts @@ -206,6 +209,14 @@ snapshot diffs on bytes — which restart-amplified dependents on every first-after-cold converge. No sanctioned reader consumes sibling rows, so the scoping changes restart behavior only. +The Postgres and bucket emulators follow the same model: a database is an +instance with a generated id (`db_<16 hex digits>`), served by a +`@prisma/dev` server named after it, and a bucket is an instance whose +generated id (`bkt-<16 hex digits>`, as an S3 bucket name allows no +underscore) is also its name on the S3 wire. Both lists all their instances, +store labels verbatim, and keep a database's port until it is deleted; see +[Postgres](#postgres) and [Buckets](#buckets-a-disk-backed-s3-emulator). + The emulator's failure messages name the instance (`instance "svc_…" exited (code 1) before it accepted connections…`). The local providers and the attachment replace the id with the service's address before the message @@ -214,54 +225,84 @@ reaches the user, so `prisma dev` says `service "gateway" …`. ### Which instance backs a node Composer records, per working directory, which emulator instance backs each -node of the app run there, in `.prisma-composer/dev/emulator-instances.json`: +node of each app run there, in `.prisma-composer/dev/emulator-instances.json`: ```json { - "app": "shop", - "nodes": { - "quotes": { "kind": "compute", "id": "svc_0b6e2f4a91c3d758", "port": 3000 }, - "gateway": { "kind": "compute", "id": "svc_81ad5c07e2f94b31", "port": 3001 } + "current": "shop", + "apps": { + "shop": { + "nodes": { + "quotes": { "kind": "service", "id": "svc_0b6e2f4a91c3d758", "port": 3000 }, + "gateway": { "kind": "service", "id": "svc_81ad5c07e2f94b31", "port": 3001 }, + "catalog": { "kind": "database", "id": "db_02c4e81f6a9b3d70", "port": 51300 }, + "uploads": { "kind": "bucket", "id": "bkt-77e1a4c90b2d5f36" } + } + } } } ``` +`current` names the app `prisma dev` last ran from the directory. A record +written by the previous version (`{ "app", "nodes" }`, one app, whose +services have kind `compute`) reads as that one app with `service` nodes and +is rewritten on the next write. A database's and a bucket's key is its +resource name, from the same helper the lowering uses; a local provider +refuses a database or bucket resource without a name. + The record is the only place Composer's names are mapped to instances. The -emulators hook, the local `App` and `Deployment` providers, the attachment -(endpoints, logs, start, stop) and `--fresh` all go through it; no Composer -code finds an instance by its labels. The labels Composer writes on create — +emulators hook, the local `App`, `Deployment`, `Database`, `Connection`, +`Bucket` and `BucketAccessKey` providers, the attachment (endpoints, logs, +start, stop) and `--fresh` all go through it; no Composer code finds an +instance by its labels. The labels Composer writes on create — `composer.directory` (the absolute directory), `composer.app` and `composer.address` — are for people reading an emulator listing. The record can be read from the directory alone, without running the pipeline. -- **Ensuring an instance.** The emulators hook goes through the services in - `graph.nodes` order (dependencies first, ties in declaration order). A - recorded instance the emulator still has is reused. Otherwise a new one is - created and recorded: on a first start, with no port requested, so ports - follow that order from 3000; after the emulator lost the instance, requesting - the recorded port. A service added by a rebuild during a session gets its - instance from the `App` provider the same way. +- **Ensuring an instance.** The emulators hook goes through the app's nodes + in `graph.nodes` order (dependencies first, ties in declaration order): + services, databases (`postgres` and `raw-postgres` resources) and buckets + (`s3` resources). A recorded instance the emulator still has is reused, and + a stopped database is started. Otherwise a new one is created and recorded: + on a first start, with no port requested, so ports follow that order; after + the emulator lost the instance, requesting the recorded port. A node added + by a rebuild during a session gets its instance from its provider the same + way. - **A remembered port that is taken.** It is a preference: the emulator gives the lowest free port instead, and `prisma dev` prints - `[dev] service "
" now uses port : its previous port is in use`. -- **Redeploying a re-created instance.** A re-created instance runs nothing, + `[dev] service "
" now uses port : its previous port is in use` + (`database "
"` for a database). +- **Redeploying a re-created instance.** A re-created instance starts empty, and a warm start's converge changes nothing, so Alchemy would never call the providers. The `Deployment` provider's diff asks for an update when its - instance does not run the recorded artifact, and the `App` provider's diff - asks for one when its instance's URL changed, so dependents get the new URL. + instance does not run the recorded artifact, the `App` provider's when its + instance's URL changed, and the `Database` and `Bucket` providers' when the + recorded instance is not the one they created, so dependents and access + keys follow. A database created again keeps its URL when it gets its old + port, so the local `OrmMigration` provider's diff reads the database's + migration marker and asks for an update whenever it is not at the target; + the committed migrations then replay as on a fresh start. That read is one + connection per database per converge. - **A removed service.** Deleting its `App` resource deletes its instance and its entry. -- **Another app in the same directory.** A directory has one dev instance. - When the record names a different app, the emulators hook deletes that app's - Compute instances, starts a new record, and prints - `[dev] removed the local services of "", the app run here before`. -- **`--fresh`** deletes the recorded Compute instances, then removes the dev - state directory with the record in it. +- **When `prisma dev` ends** (Ctrl-C, SIGTERM, a failure), the attachment's + `stopApp` stops the app's service processes and database servers. + Their data and ports are kept, and the next start brings them back. +- **Another app in the same directory.** Running a different app from the + directory (another entry, another `--name`) stops the previous app's + services and databases and keeps its entry, with their data and ports, and + prints `[dev] stopped the local services of "", the app run here + before; its data is kept`. Running the previous app again resumes them. +- **`--fresh`** deletes the current app's instances of every kind (a + database's data with it) and its entry, and removes the dev state directory + except the record and other apps' bucket objects. A bucket's objects live + under `.prisma-composer/dev/buckets///` for that reason. Two directories running the same app have separate records, so they get -separate instances on separate ports, and neither restarts the other's -services. State the Compute emulator wrote before instances had generated ids -is discarded when it loads; local dev state is never migrated. +separate services on separate ports, separate databases and separate +buckets, and neither restarts or writes into the other's. State an emulator +wrote before instances had generated ids is discarded when it loads; local +dev state is never migrated. ### Process lifetimes @@ -272,14 +313,11 @@ is discarded when it loads; local dev state is never migrated. multi-tenant, holding every long-lived process: the Compute emulator (service children), the Postgres emulator (one `@prisma/dev` server per `Database`, hosted in-process), and the bucket emulator. They survive dev - sessions; `--fresh` removes a directory's instances. The registry root is + sessions; `--fresh` removes the current app's instances. The registry root is `~/.prisma-composer/emulators` unless `PRISMA_COMPOSER_EMULATORS_DIR` names another absolute directory, so by default one machine shares one - set. Compute instances belong to a directory, so two checkouts of one app - never share a service. Postgres and buckets are still keyed by the app: - Postgres instance names come from the app and database, and bucket names - from the app and bucket, so two checkouts of one app still share a - database server and bucket names, whatever their roots. A daemon whose version + set. Instances belong to a directory, so two checkouts of one app never + share a service, a database or a bucket, whatever their roots. A daemon whose version differs from the caller's is replaced, so two Composer versions sharing a root replace each other's daemons back and forth; a separate root per checkout stops that. @@ -301,17 +339,17 @@ semantics): | Resource | Dev behavior | | --- | --- | | `Project` | a local identity record; no platform | -| `Database` | a database on a local `@prisma/dev` Postgres server, hosted by the Postgres emulator | -| `Connection` | the local connection URL | +| `Database` | the database instance the directory's record names for the resource, on its own `@prisma/dev` server hosted by the Postgres emulator; started if stopped, created when missing; `databaseId` is the instance id | +| `Connection` | the URL of the database instance its `Database` names | | `App` | finds the service's Compute instance through the directory's record, creating it when there is none; the instance holds the service's port; `appEndpointDomain = http://localhost:` — which makes origin (ADR-0039) work unchanged. Deleting it deletes the instance | | `Deployment` | unpacks the artifact once per hash, materializes the env, and puts the deployment at the service's Compute instance, which (re)starts the child; updated when its instance does not run the recorded artifact | | `EnvironmentVariable` | a key→value row in the dev state store | -| `Bucket` | a directory under `.prisma-composer/dev/buckets//`, served by the bucket emulator | -| `BucketKey` | credentials registered with the bucket emulator | +| `Bucket` | the bucket instance the directory's record names for the resource, over a directory under `.prisma-composer/dev/buckets///`, served by the bucket emulator; `bucketId` is the instance id, also its S3 name | +| `BucketKey` | credentials registered with the bucket emulator for its bucket | | `ServiceKey` | **unchanged** — mints locally, persists in state | | `S3Credentials` | **unchanged** — mints locally, persists in state | | `PgWarm` | **unchanged** — real `select 1` against the local URL | -| `OrmMigration` | **unchanged** — real migrations against the local URL | +| `OrmMigration` | real migrations against the local URL, as deployed; locally its diff also asks for an update whenever the database is not at the target by its marker, so a database created again empty is migrated | Because module-backed kinds (storage, streams, email) lower to compute services plus databases, they run their **real service code** locally against local @@ -320,7 +358,7 @@ SQLite test server remains a testing utility, not part of the dev loop. ### Postgres -The emulator is a daemon hosting `@prisma/dev` servers, **one named, persistent server per `Database` resource** — server names are derived from the app and database ids, so instances are isolated and survive across dev sessions for warm starts. Migrations are not special-cased: `OrmMigration` runs exactly as it does in a deploy, against the local URL — replay-only (ADR-0022 as revised), so it applies committed migrations and never synthesizes schema. At reconcile time it reloads the emitted `contract.json` identified by `prisma.config.ts`, attests its `storageHash` against the compact contract identity persisted in deploy state, and only then opens the database. Dev-loop schema iteration therefore happens through the ORM's own `prisma db update`, run directly against the emulator database: `db update` moves the database and its marker to the current contract, and the pipeline's migration step no-ops because the marker matches the target. A dev run against a database that was neither updated nor covered by a planned migration hits the same structured refusal a deploy would, naming both exits (`prisma db update` to iterate, `contract emit` + `migration plan` to author the path). `PgWarm` is near-instant locally and is kept (not stubbed) so the provider set stays uniform. +The emulator is a daemon hosting `@prisma/dev` servers, **one named, persistent server per database instance**. Creating an instance starts its server; the instance's generated id names the server, and `@prisma/dev` keeps the server's data under that name in its own data directory (`~/Library/Application Support/prisma-dev-nodejs/` on macOS, `$XDG_DATA_HOME/prisma-dev-nodejs/` on Linux), outside the emulators directory. Stopping an instance shuts its server down and keeps its data and its port; starting it brings it back on the same port with the same data. Deleting it removes the data. A database keeps its port until it is deleted because its connection string is in its dependents' environments. The Postgres emulator's earlier, app-keyed state is removed when it loads, and the data of the servers it named is deleted on the next request that carries a `@prisma/dev` module, unless another live process still serves one. Migrations are not special-cased: `OrmMigration` runs exactly as it does in a deploy, against the local URL — replay-only (ADR-0022 as revised), so it applies committed migrations and never synthesizes schema. At reconcile time it reloads the emitted `contract.json` identified by `prisma.config.ts`, attests its `storageHash` against the compact contract identity persisted in deploy state, and only then opens the database. Dev-loop schema iteration therefore happens through the ORM's own `prisma db update`, run directly against the emulator database: `db update` moves the database and its marker to the current contract, and the pipeline's migration step no-ops because the marker matches the target. A dev run against a database that was neither updated nor covered by a planned migration hits the same structured refusal a deploy would, naming both exits (`prisma db update` to iterate, `contract emit` + `migration plan` to author the path). `PgWarm` is near-instant locally and is kept (not stubbed) so the provider set stays uniform. ### Buckets: a disk-backed S3 emulator @@ -338,13 +376,15 @@ unchanged. The bucket emulator is then a third store implementation plus one machine-global daemonized server (plain `node:http`, which runs under both node and bun), multi-tenant across apps and administered through a loopback admin endpoint (`/_pcdev/…` — the underscore prefix cannot collide with a -valid bucket name). Physical bucket names are prefixed `--` and -carried on the binding's `bucketName`, so consumers follow the binding and -names cannot collide across apps; each bucket's data root is registered as -an in-project path: +valid bucket name). A bucket is an instance whose generated id is its name +on the S3 wire, carried on the binding's `bucketName`, so consumers follow +the binding and names cannot collide. Credentials are registered per bucket, +and a request is verified only against its target bucket's keys. Each +bucket's data root is registered as an in-project path, which the emulator +stores verbatim: - **Objects are plain files at their key paths** — - `.prisma-composer/dev/buckets//`. Browse them, open them, drop a + `.prisma-composer/dev/buckets///`. Browse them, open them, drop a file in and it exists in the bucket. This is a feature, not an implementation detail: inspectable state is half the value of local dev. - Object metadata (content type, user metadata) lives in a sidecar tree, so @@ -423,7 +463,7 @@ Deploy's rule holds: every failure names its fix. (none outstanding from the design phase — see Known limitations below for gaps found during implementation.) -(Settled since the first draft: Postgres runs one named `@prisma/dev` server per `Database` resource; the front door prints every service URL ordered by address depth then name, shallowest first; on a fresh start, services get their ports in dependency order, ties in declaration order — the emulators hook creates their Compute instances one at a time from `graph.nodes` before Alchemy applies the `App` resources concurrently — and a warm start keeps each service's port, because an instance holds its port until it is deleted (ADR-0052); port allocation and the remaining mechanics are described in this document and [ADR-0041](../90-decisions/ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md). Restart latency is measured — see Known limitations.) +(Settled since the first draft: Postgres runs one named `@prisma/dev` server per `Database` resource; the front door prints every service URL ordered by address depth then name, shallowest first; on a fresh start, services get their ports in dependency order, ties in declaration order — the emulators hook creates their instances one at a time from `graph.nodes` before Alchemy applies the resources concurrently — and a warm start keeps each service's and database's port, because an instance holds its port until it is deleted (ADR-0052); port allocation and the remaining mechanics are described in this document and [ADR-0041](../90-decisions/ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md). Restart latency is measured — see Known limitations.) ## Proven against a real app diff --git a/docs/design/90-decisions/ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md b/docs/design/90-decisions/ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md index 32e5b491a..bb5050b71 100644 --- a/docs/design/90-decisions/ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md +++ b/docs/design/90-decisions/ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md @@ -2,7 +2,7 @@ Amended by [ADR-0050](ADR-0050-composer-runs-as-prisma-deploy-and-prisma-dev.md): the command is `prisma dev `, and reading the running app's logs is the `log` operation on `@prisma/composer/control`, not a command. The `prisma-composer` binary is retired. The local pipeline below is unchanged. -Amended by [ADR-0052](ADR-0052-composer-maps-its-nodes-to-emulator-instances-per-directory.md): the emulators know nothing about Composer. A dev instance is no longer identified on the emulators by the app name; Composer records, per working directory, which emulator instance backs each node, and the Compute emulator keys its instances by ids it generates. The Postgres and bucket emulators still key instances by app name until they move to the same model. +Amended by [ADR-0052](ADR-0052-composer-maps-its-nodes-to-emulator-instances-per-directory.md): the emulators know nothing about Composer. A dev instance is no longer identified on the emulators by the app name: the Compute, Postgres and bucket emulators key their instances by ids they generate, and Composer records, per working directory, which instance backs each node of each app run there. When `prisma dev` ends it stops the app's database servers as well as its services. ## Decision diff --git a/docs/design/90-decisions/ADR-0052-composer-maps-its-nodes-to-emulator-instances-per-directory.md b/docs/design/90-decisions/ADR-0052-composer-maps-its-nodes-to-emulator-instances-per-directory.md index aa9b4c9b3..a0325a869 100644 --- a/docs/design/90-decisions/ADR-0052-composer-maps-its-nodes-to-emulator-instances-per-directory.md +++ b/docs/design/90-decisions/ADR-0052-composer-maps-its-nodes-to-emulator-instances-per-directory.md @@ -2,13 +2,18 @@ ## Decision -The local emulators know nothing about Composer. An emulator holds instances under ids it generates. Each instance carries labels that the emulator stores and returns but never reads. Composer records, per working directory, which emulator instance backs each node of the app run there. That record is the only place Composer's names are mapped to instances. +The local emulators know nothing about Composer. Each emulator holds instances under ids it generates: the Compute emulator holds service instances, the Postgres emulator holds databases, and the bucket emulator holds buckets. Each instance carries labels that the emulator stores and returns but never reads. Composer records, per working directory, which emulator instance backs each node of each app run there. That record is the only place Composer's names are mapped to instances. ```text ~/work/shop/.prisma-composer/dev/emulator-instances.json (written by Composer) - { "app": "shop", - "nodes": { "quotes": { "kind": "compute", "id": "svc_k3f9…", "port": 3000 }, - "gateway": { "kind": "compute", "id": "svc_81ad…", "port": 3001 } } } + { "current": "shop", + "apps": { + "shop": { "nodes": { + "quotes": { "kind": "service", "id": "svc_k3f9…", "port": 3000 }, + "gateway": { "kind": "service", "id": "svc_81ad…", "port": 3001 }, + "catalog": { "kind": "database", "id": "db_02c4…", "port": 51300 }, + "uploads": { "kind": "bucket", "id": "bkt-77e1…" } } }, + "shop-v2": { "nodes": { … } } } } Compute emulator (one per machine, knows nothing about Composer) GET /instances → [ { "id": "svc_k3f9…", "port": 3000, "status": "running", @@ -17,44 +22,48 @@ Compute emulator (one per machine, knows nothing about Composer) "composer.address": "quotes" } }, … ] ``` -A second checkout of the same app in `~/work/shop-2` has its own record and its own instances. The two never share a service process or a port. +A second checkout of the same app in `~/work/shop-2` has its own record and its own instances. The two never share a service process, a port, a database or a bucket. -This amends [ADR-0041](ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md), whose local container gives a dev instance a stable identity from the app name alone. The Compute emulator is keyed this way. The Postgres and bucket emulators still key their instances by app name and are moved to the same model separately. +This amends [ADR-0041](ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md), whose local container gives a dev instance a stable identity from the app name alone. ## Reasoning -`prisma dev` runs an app against local emulators: a Compute emulator that runs service processes, a Postgres emulator and a bucket emulator. Each is a long-lived daemon shared by everything on the machine. ADR-0041 identified an app's instances on those daemons by the app's name, so the emulators' routes and state were shaped around apps: `/apps//services/`. Two directories that run the same app, such as two git worktrees of one repository, therefore resolved to the same instances. Each `prisma dev` redeployed over the other's services. That is a common situation, not an edge case: agents work in many worktrees at once and start and stop environments in each. +`prisma dev` runs an app against local emulators: a Compute emulator that runs service processes, a Postgres emulator that hosts `@prisma/dev` database servers, and a bucket emulator that serves the S3 wire protocol over local directories. Each is a long-lived daemon shared by everything on the machine. ADR-0041 identified an app's instances on those daemons by the app's name, so the emulators' routes and state were shaped around apps: `/apps//services/`, a database server named `pcdev--`, a bucket named `--`. Two directories that run the same app, such as two git worktrees of one repository, therefore resolved to the same instances. Each `prisma dev` redeployed over the other's services and wrote into the other's database. That is a common situation, not an edge case: agents work in many worktrees at once and start and stop environments in each. -The emulators should not have to know what a directory or an app is to avoid this. So an instance belongs to whoever created it. A client creates an instance, the emulator generates its id, and every later operation names that id: deploy, start, stop, delete, logs. The emulator never interprets Composer's data. An app name, a node address or a directory reaches it only as a label value, or inside the deployment it is handed to run: the artifact directory, which lies under the project, and the environment, whose keys are named after addresses. The emulator stores both verbatim and passes them to the process it spawns. No route, state key, file name or field the emulator generates carries any of them, and a test fails if one does. The emulators can then be used by something other than Composer, and moved out of this repository, without carrying Composer's model along. +The emulators should not have to know what a directory or an app is to avoid this. So an instance belongs to whoever created it. A client creates an instance, the emulator generates its id, and every later operation names that id: deploy, start, stop, delete, logs, credentials. A database's `@prisma/dev` server is named after its id, and a bucket's id is also its bucket name on the S3 wire. The emulator never interprets Composer's data. An app name, a node address or a directory reaches it only as a label value, or inside a payload it is handed to use: a deployment's artifact directory, which lies under the project, and its environment, whose keys are named after addresses; the path a database's `@prisma/dev` module is loaded from; a bucket's data directory. The emulator stores these verbatim and uses them as given. No route, state key, file name or field the emulator generates carries any of them, and a test per emulator fails if one does. The emulators can then be used by something other than Composer, and moved out of this repository, without carrying Composer's model along. -Labels make a listing readable. Composer writes `composer.directory`, `composer.app` and `composer.address` on every instance it creates, so a person looking at the emulator's list of instances can tell what started each one. Nothing reads them back as identity. If Composer found its instances by searching labels, the labels would be a second, unenforced key. +Labels make a listing readable. Composer writes `composer.directory`, `composer.app` and `composer.address` on every instance it creates, so a person looking at an emulator's list of instances can tell what started each one. Nothing reads them back as identity. If Composer found its instances by searching labels, the labels would be a second, unenforced key. -Composer finds its instances through `emulator-instances.json` in the directory's dev state (`.prisma-composer/dev/`), which is already per directory. It records the app name and, per node address, the emulator kind, the instance id and the instance's port. The emulators hook writes it before converge. It reuses a recorded instance while the emulator still has it, and otherwise creates one and records it. Everything else reaches instances through the record: the local providers during converge, the dev session's endpoints, logs, start and stop, and `--fresh`. The record is readable from the directory alone, so a separate command can answer "where is this node running" without running the pipeline. +Composer finds its instances through `emulator-instances.json` in the directory's dev state (`.prisma-composer/dev/`), which is already per directory. It records every app run from the directory and, per node address, the node's kind (`service`, `database` or `bucket`), the instance id and, for a service or a database, its port; `current` names the app `prisma dev` last ran there. The emulators hook writes it before converge, in graph order: it reuses a recorded instance while the emulator still has it, starts a stopped database, and otherwise creates an instance and records it. Everything else reaches instances through the record: the local providers during converge, the dev session's endpoints, logs, start and stop, and `--fresh`. The record is readable from the directory alone, so a separate command can answer "where is this node running" without running the pipeline. -An instance keeps its port from creation until it is deleted, whether it is running or stopped. A port is baked into more than the process: the service's environment carries it (`COMPOSER_
_PORT`), and its dependents' environments carry its URL. A warm start does not redeploy, because nothing in the app changed and Alchemy, the deploy engine, sees nothing to update. A service that received a different port on its next start would run with an environment built for the old one. +An instance keeps its port from creation until it is deleted, whether it is running or stopped. A port is baked into more than the process: a service's environment carries its own port (`COMPOSER_
_PORT`), and its dependents' environments carry its URL or a database's connection string. A warm start does not redeploy, because nothing in the app changed and Alchemy, the deploy engine, sees nothing to update. An instance that received a different port on its next start would run, or be reached, through an environment built for the old one. -The record also keeps each node's port, for the one case where the port can change: the instance has to be re-created, for example because the emulator lost its state. Composer then asks for the remembered port. A requested port is a preference: when another instance or another process holds it, the emulator gives the lowest free port from 3000, and `prisma dev` prints one line naming the service, the old port and the new one. Failing instead would break `prisma dev` in one worktree whenever another worktree happened to take the port. Because a re-created instance runs nothing, the local Deployment provider asks Alchemy for an update when its instance does not run the recorded artifact, and the local App provider does so when its instance's URL changed, so dependents get the new URL. +When `prisma dev` ends, through Ctrl-C, SIGTERM or a failure, it stops the app's service processes and its database servers. A stopped database keeps its data and its port, and the next `prisma dev` starts it again before converge. The hook has to: on a warm start Alchemy finds nothing to change and never calls the Database provider, so nothing else would start it. A bucket has no process, so there is nothing to stop. -A directory has one dev instance. When `prisma dev` runs a different app from a directory than the one the record names, the previous app's Compute instances are deleted, the record starts over, and `prisma dev` prints which app's local services were removed. Compute instances hold no data, so nothing is lost that a deploy does not recreate. +The record also keeps each node's port, for the one case where it can change: the instance has to be created again, for example because the emulator lost its state. Composer then asks for the remembered port. A requested port is a preference: when another instance or another process holds it, the emulator gives the lowest free port, and `prisma dev` prints one line naming the node, the old port and the new one. Failing instead would break `prisma dev` in one worktree whenever another worktree happened to take the port. A re-created instance starts empty: a service runs nothing and a database has no tables. So the local providers ask Alchemy for an update when their recorded instance changed (Deployment, Database, Bucket) or its URL did (App), and the local migration step asks for one whenever the database is not at its target by its migration marker, because a database created again empty keeps its URL and nothing in Alchemy's inputs changes. + +A directory keeps the local state of every app run from it. Running a different app from the directory, by another entry or another `--name`, stops the previous app's services and databases and keeps them recorded, with their data and ports, and `prisma dev` prints that it did. Running the previous app again resumes them. Deleting a database because a root module was renamed for a while would lose data no deploy recreates. `--fresh` deletes the current app's instances of every kind, a database's data with it, and the app's entry, and leaves other apps' entries alone. ## Consequences -- Two directories running the same app run separate services on separate ports, and neither restarts the other's. +- Two directories running the same app run separate services on separate ports, with separate databases and buckets, and neither restarts or writes into the other's. - A directory's ports are stable across `prisma dev` sessions. A port changes only when its instance is re-created and the old port is taken, and `prisma dev` says so. -- A stopped instance holds its port. Instances left behind by a deleted worktree keep their ports until something deletes them. No command does that yet. -- Local dev state written before this change is discarded: the Compute emulator drops app-keyed state when it loads, and the first `prisma dev` creates new instances. Local dev state is never migrated across versions. -- Every emulator kind moves to this model: instances keyed by emulator-generated ids, labels stored but never read, and a list of all instances. Until the Postgres and bucket emulators do, the app name still identifies their instances, so two directories running one app still share its databases and buckets. +- After `prisma dev` ends, none of the app's service processes or database servers is running. The next start has the same data on the same ports. +- A stopped instance holds its port, and a database its data. Instances left behind by a deleted worktree keep their ports and data until something deletes them. No command does that yet. +- A bucket's objects live in the project, under `.prisma-composer/dev/buckets//`, so another app's `--fresh` in the same directory leaves them alone. +- Local dev state written before this change is discarded: each emulator drops its app-keyed state when it loads (the Postgres emulator also deletes the data of the servers that state named, once no live process serves them), and the first `prisma dev` creates new instances. Local dev state is never migrated across versions. - Composer code must not find an instance by its labels. The record is the only mapping. ## Alternatives considered -- **App-keyed emulators.** The model this replaces. Simple, but two directories running one app share and overwrite each other's services, and the emulators carry Composer's concepts in their routes and state. +- **App-keyed emulators.** The model this replaces. Simple, but two directories running one app share and overwrite each other's services, databases and buckets, and the emulators carry Composer's concepts in their routes and state. - **The app name plus a hash of the directory as the identity.** Separates directories, but keeps app and directory concepts inside the emulators and makes every emulator route depend on how Composer forms that key. - **Releasing a port when its instance stops.** Lets stopped worktrees hold no ports, but breaks warm starts: the port is in the deployment's environment and in its dependents' URLs, and a warm start does not redeploy, so a service started on a new port would run with an environment built for the old one. - **Finding instances by their labels.** Needs no record file, but makes free-form display metadata into an unenforced key that any client could collide with. +- **One app per directory, deleting the previous app's instances on a switch.** Simpler record, but a renamed root module or another `--name` would delete local databases no deploy recreates. ## Related - [ADR-0041](ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md) — local dev runs the deploy pipeline against local providers; amended here. - [ADR-0051](ADR-0051-a-nodes-address-is-its-logical-id-on-the-platform.md) — a node's address, the key of the record. -- [local-dev.md](../10-domains/local-dev.md) — the Compute emulator, the record and port allocation. +- [local-dev.md](../10-domains/local-dev.md) — the emulators, the record, port allocation and stopping on exit. diff --git a/docs/design/90-decisions/README.md b/docs/design/90-decisions/README.md index e2f8d7bf6..dc0f53033 100644 --- a/docs/design/90-decisions/README.md +++ b/docs/design/90-decisions/README.md @@ -62,7 +62,7 @@ _Earlier drafts (ADR-0001, ADR-0002) were retired as the high-level design settl - [ADR-0038](ADR-0038-containers-are-an-extension-descriptor.md) — Container lifecycle (ensure/locate/remove) is an optional `container` descriptor on `ExtensionDescriptor`, the same pattern as `preflight`/`teardown`; the resolved instance is opaque to core and crosses the CLI parent→alchemy child boundary as one framework-named environment variable per extension, via the extension's own `serialize()`/`deserialize()`. `StateDescriptor` names its owning extension so core can hand it that extension's resolved container. Deletes the `crossDomainExceptions` entry that let the CLI import `@internal/lowering` directly — `0-framework` imports nothing again. - [ADR-0039](ADR-0039-a-compute-services-own-origin-is-a-target-resolved-property.md) — A compute service's own platform-assigned origin is a target-resolved property, read as the service's `origin()` — never a declared param, never operator config, never in `config()`. It rides ADR-0031's reserved provider-param channel as the first *service-derived* entry (`valueForService(provisioned, address)`, written for every compute service, exposing or not), sourced from the provisioned App's `appEndpointDomain` (ADR-0048) — made trustworthy pre-promote by the upstream PRO-200 fix. `envParam(…)` remains correct for operator-known origins (custom domains); narrows ADR-0032's `appOrigin` example accordingly. - [ADR-0040](ADR-0040-the-orm-binding-carries-the-url-and-a-lazy-client.md) — `postgres(contract)`'s dependency binding is `{ url, client }`: the raw connection string plus the typed client, constructed lazily and memoized on first `client` access — `hydrate` builds nothing. The contract remains the compatibility interface (hash check and deploy-time migration unchanged, ADR-0022); the binding becomes a strict superset of plain `rawPostgres()`'s `{ url }`, so an app that owns its database client still gets framework-run migrations. Contract validation cost and failure move from `load()` (where one bad input poisoned every input, unattributed) to the first `client` access. Cross-kind satisfaction (`'postgres'` satisfying `'postgres'`) rejected in its favor. -- [ADR-0041](ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md) — `prisma-composer dev` runs the **same deploy pipeline** (Load → assemble → lower → Alchemy converge) against local implementations of the same Alchemy resource types, declared on an optional `localTarget` field of `ExtensionDescriptor` (a lazy thunk resolving a `LocalTargetDescriptor`; subpaths `@prisma/composer/local-target` and `@prisma/composer-prisma-cloud/local-target` — "dev" names only the user-facing command/prefix/state dir) (providers, container, preflight, emulators, attach, teardown — **no** `nodes`/`provisions`, so the lowering cannot diverge; no `state` either — dev uses Alchemy's own `localState()` through `LowerOptions.state`). The target runs **emulators per node kind**: Compute and buckets are machine-global, multi-tenant daemons (the Compute emulator owns the service child processes — deployment PUTs, crash supervision, logs; buckets serve the S3 wire over plain files on disk), while Postgres runs one detached ORM `prisma dev` instance per `Database` resource under the ORM CLI's own manager. Providers provision instances by communicating with the emulators during converge, and the dev command is a view through `attach`; `ServiceKey`/`S3Credentials`/`PgWarm`/`OrmMigration` are shared verbatim. Credential-free by requirement. Rejects a local Management API (reimplements another team's server-side semantics, drifts silently) and per-kind dev descriptors (an open-set parallel seam). *(Amended by ADR-0050: the command is `prisma dev `; logs are the `log` operation, not a command. Amended by ADR-0052: emulator instances are keyed by emulator-generated ids, not the app name, and Composer maps its nodes to them per directory.)* +- [ADR-0041](ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md) — `prisma-composer dev` runs the **same deploy pipeline** (Load → assemble → lower → Alchemy converge) against local implementations of the same Alchemy resource types, declared on an optional `localTarget` field of `ExtensionDescriptor` (a lazy thunk resolving a `LocalTargetDescriptor`; subpaths `@prisma/composer/local-target` and `@prisma/composer-prisma-cloud/local-target` — "dev" names only the user-facing command/prefix/state dir) (providers, container, preflight, emulators, attach, teardown — **no** `nodes`/`provisions`, so the lowering cannot diverge; no `state` either — dev uses Alchemy's own `localState()` through `LowerOptions.state`). The target runs **emulators per node kind**: Compute and buckets are machine-global, multi-tenant daemons (the Compute emulator owns the service child processes — deployment PUTs, crash supervision, logs; buckets serve the S3 wire over plain files on disk), while Postgres the Postgres emulator hosts one `@prisma/dev` server per database instance. Providers provision instances by communicating with the emulators during converge, and the dev command is a view through `attach`; `ServiceKey`/`S3Credentials`/`PgWarm`/`OrmMigration` are shared verbatim. Credential-free by requirement. Rejects a local Management API (reimplements another team's server-side semantics, drifts silently) and per-kind dev descriptors (an open-set parallel seam). *(Amended by ADR-0050: the command is `prisma dev `; logs are the `log` operation, not a command. Amended by ADR-0052: emulator instances are keyed by emulator-generated ids, not the app name, Composer maps each app's nodes to them per directory, and `prisma dev` stops the app's database servers when it ends.)* - [ADR-0042](ADR-0042-service-input-is-one-standard-schema.md) — A compute service declares its entire incoming configuration — config and secrets together — as one Standard Schema (`input`), read back through one typed accessor; `params`/`secrets` and `config()`/`secrets()` are replaced. The framework never introspects the schema (validate-only, per the spec): the operator's binding is the traversable structure (sourcing: literals, `envParam`, `envSecret`), the schema is the black-box judge of legality (invoked at deploy over the resolved binding with secrets as opaque `SecretString` boxes, and again at boot), and secretness is a leaf *type* enforced by validation in both directions. The wire format is one self-describing JSON document row per service with `$secret` pointers to platform variables; an env-bound key whose variable is unset resolves to key-omitted and the schema arbitrates absence — subsuming optional secrets and conditional config (`stripeId` only when `stripeEnabled`) without a framework DSL. - [ADR-0043](ADR-0043-the-control-subpath-is-the-programmatic-deploy-surface.md) — `@prisma/composer/control` is the programmatic deploy surface: typed `deploy`/`destroy`/`dev`/`log` operations (structured inputs/results, no argv/console/exit) implemented in `@internal/cli`'s `src/operations/` and re-exported per ADR-0035; the CLI is a thin renderer over them. The entry's static graph stays import-light — each operation lazily imports its executor, so importing the subpath executes nothing, and a tree that cannot load the deploy stack surfaces as a structured `pipeline` failure — and `PRISMA_COMPOSER_DEPLOYMENT_RESULT_FILE` carries the deploy result across the process boundary: the alchemy child's report hook writes a serializable `DeploymentSummary` to the named file, the operation reads it back best-effort (absent/malformed = undefined summary, never a failure). Distinct from an extension's ADR-0017 `/control` entry. *(Amended by ADR-0049: each operation takes a required `config: { value, file }`, and the effect-conflict diagnosis is retired.)* *(Amended by ADR-0050: the `prisma` CLI renders two of the four operations, as `prisma deploy` and `prisma dev`; `destroy` and `log` are called from scripts.)* - [ADR-0044](ADR-0044-errors-are-structural-envelopes-with-dotted-namespace-codes.md) — Errors are structural envelopes with dotted `NAMESPACE.SUBCODE` codes (the shared prisma/prisma foundation, duplicated pending extraction): structured at origin with why/fix splits, no catch-all codes, bugs carry no code (exit 1 + report hint), recognition is structural (`CliStructuredError.is()`), operation results ride the shared `Result` `ok` discriminator, expected failures exit 2 — with the alchemy child-status passthrough as the documented exception. *(Amended by ADR-0049: `DEPS.EFFECT_VERSION_CONFLICT` is retired.)* @@ -73,4 +73,4 @@ _Earlier drafts (ADR-0001, ADR-0002) were retired as the high-level design settl - [ADR-0049](ADR-0049-composers-configuration-is-the-composer-section-of-prisma-config.md) — Composer's configuration is the `composer` section of `prisma.config.ts`, loaded by the Prisma CLI engine. The validator checks only the fields that identify extension and state descriptors, passes descriptors through as the file's own objects, rejects unknown keys, and takes the nearest declaring file's section whole so the generated stack imports exactly one file. The old `prisma-composer.config.ts` and the `configPath` pointer are refused with `CONFIG.SECTION_MISSING`, `CONFIG.FIELD_RETIRED` and `CONFIG.FILE_RETIRED`, never migrated. The validator returns the section with its declaring file, and the programmatic operations take the same `ComposerConfigSource` (`{ value, file }`) and run the same checks. The Alchemy child evaluates the whole `prisma.config.ts`. The effect pre-flight and `DEPS.EFFECT_VERSION_CONFLICT` are retired: a broken tree fails evaluation of `prisma.config.ts` with the engine's `CLI.CONFIG_UNREADABLE`. *(Amended: the alternatives section lists the rejected ways to keep a Composer-owned effect check.)* - [ADR-0050](ADR-0050-composer-runs-as-prisma-deploy-and-prisma-dev.md) — Composer has no binary of its own: the `prisma` CLI mounts exactly two Composer commands, `prisma deploy ` and `prisma dev `, mounted at the root as bare verbs on purpose rather than under a noun. `destroy` and `log` are operations on `@prisma/composer/control`, called from a script; a teardown script reads `PRISMA_SERVICE_TOKEN` and `PRISMA_WORKSPACE_ID`. Where teardown and logs belong in the `prisma` command tree is not decided; until it is, agents and users need a script for both. A `destroy` command was rejected: the CLI's destructive verb is `delete`, and no Composer command takes `--production`. Retires the `prisma-composer` binary; a CI lint keeps its name out of user-facing files. - [ADR-0051](ADR-0051-a-nodes-address-is-its-logical-id-on-the-platform.md) — A node's address is its identity on Prisma Cloud: Composer writes it, byte for byte, as the `logicalId` of the node's one platform row (Project, App, Database, Bucket) and as the node's `logicalId` in the application topology, so the platform joins topology nodes to rows, and one node across branches, by string equality. A provision ID defaults to the node's name, so changing a name changes identity. The value is never an Alchemy resource ID, a display name or a platform `id`; App, Database and Bucket rows get it as a prop on the upstream resource that creates them, and the Project from container resolution; no separate resource writes it. Changing an address changes the entity's identity; changing a display name changes nothing. -- [ADR-0052](ADR-0052-composer-maps-its-nodes-to-emulator-instances-per-directory.md) — The local emulators know nothing about Composer: an emulator keys its instances by ids it generates and stores labels it never reads. Composer records per working directory, in `.prisma-composer/dev/emulator-instances.json`, which instance backs each node, and nothing else maps Composer's names to instances, so two directories running one app get separate instances. An instance keeps its port until it is deleted, because the port is in the deployment's environment and its dependents' URLs and a warm start does not redeploy; a remembered port is requested, as a preference, only when an instance is re-created. Running a different app from a directory removes the previous app's Compute instances. Amends ADR-0041's app-name local identity. The Postgres and bucket emulators move to this model separately. +- [ADR-0052](ADR-0052-composer-maps-its-nodes-to-emulator-instances-per-directory.md) — The local emulators know nothing about Composer: the Compute, Postgres and bucket emulators key their instances by ids they generate and store labels they never read. Composer records per working directory, in `.prisma-composer/dev/emulator-instances.json`, which instance backs each node of each app run there, and nothing else maps Composer's names to instances, so two directories running one app get separate services, databases and buckets. An instance keeps its port until it is deleted, because the port is in the deployment's environment and its dependents' URLs and a warm start does not redeploy; a remembered port is requested, as a preference, only when an instance is re-created. When `prisma dev` ends it stops the app's services and database servers, keeping their data. Running a different app from a directory stops the previous app's instances and keeps them; `--fresh` deletes only the current app's. Amends ADR-0041's app-name local identity. diff --git a/docs/guides/running-locally.md b/docs/guides/running-locally.md index 0b0c0c9a7..496c88900 100644 --- a/docs/guides/running-locally.md +++ b/docs/guides/running-locally.md @@ -60,45 +60,53 @@ and watches the file. When the file changes, `dev` prints that it changed and must be restarted, and it stops rebuilding: until you stop `dev` and start it again, a build change only repeats that notice. -`Ctrl-C` stops your app's service processes and exits. The local databases, -buckets, and their data stay up, so the next `prisma dev` is a warm -start — same ports, same data. `--fresh` is what wipes this app's local -instances and data before starting. +`Ctrl-C` stops your app's service processes and its local Postgres servers, +and exits; so does `dev` ending on SIGTERM or a failure. The data stays, and +so do the ports, so the next `prisma dev` is a warm start — same ports, same +data. `--fresh` is what wipes this app's local instances and data before +starting. On a fresh start, each service gets the lowest free port from 3000 up, in dependency order: a service gets its port before the services that call it, and services with no dependency between them go in the order you provisioned them. A warm start keeps the ports services already have. -### Each directory runs its own services +### Each directory runs its own services, databases and buckets -The services `dev` runs belong to the directory you run it in. Two checkouts -of the same app, such as two git worktrees, run separate service processes on -separate ports, and a rebuild in one never restarts the other's services. -`dev` records which local service backs each of your services in +The services, databases and buckets `dev` runs belong to the directory you +run it in. Two checkouts of the same app, such as two git worktrees, run +separate service processes on separate ports, with separate databases and +separate buckets: a rebuild in one never restarts the other's services, and +a row or an object written in one is not seen by the other. `dev` records +which local instance backs each node in `.prisma-composer/dev/emulator-instances.json`. -A service keeps its port in its directory from one `dev` to the next, even -while it is stopped. If the local service is gone, for example because a -Composer upgrade discarded the compute emulator's state, the next `dev` -creates it again and asks for the port it had. When another directory has -taken that port, the service gets the lowest free one and `dev` prints one -line: +A service or a database keeps its port in its directory from one `dev` to the +next, even while it is stopped. If a local instance is gone, for example +because a Composer upgrade discarded an emulator's state, the next `dev` +creates it again, empty, and asks for the port it had; a Prisma ORM +database's (`postgres()`) committed migrations then run again, as on a fresh +start, while a `rawPostgres()` database stays empty. When another directory +has taken that port, the instance gets the lowest free one and `dev` prints +one line, naming a service or a database: ``` [dev] service "gateway" now uses port 3004: its previous port 3001 is in use +[dev] database "catalog" now uses port 51302: its previous port 51300 is in use ``` -A directory runs one app at a time. If you run a different app from the same -directory, `dev` removes the previous app's local services first and says so: +A directory keeps the local state of every app you run from it. If you run a +different app from the same directory (another entry file, or another +`--name`), `dev` stops the previous app's services and databases first, keeps +their data and ports, and says so: ``` -[dev] removed the local services of "shop", the app run here before +[dev] stopped the local services of "shop", the app run here before; its data is kept ``` -A stopped service keeps its port reserved, so the services of a checkout you -delete keep their ports. There is no command to remove them yet. +Running `shop` again resumes them. `--fresh` wipes only the app you start +with it. -Local databases and buckets are not separated this way yet: they are still -keyed by the app's name, so two checkouts of one app share them (see -[What's local vs. what's real](#whats-local-vs-whats-real)). +A stopped service or database keeps its port reserved, and a database its +data, so the instances of a checkout you delete keep theirs. There is no +command to remove them yet. `--fresh` is also the fix when a framework upgrade leaves stale rows in this app's local dev state — the symptom is a plan-time error naming an @@ -173,23 +181,19 @@ token, workspace, or network is involved. Three consequences worth knowing: so your data survives `Ctrl-C` and even a reboot until you `--fresh`. That's what makes restarts warm. - **`PRISMA_COMPOSER_EMULATORS_DIR` gives a checkout or a CI job its own - compute and buckets emulators.** By default every `dev` on the machine - shares one set of emulator daemons, registered under - `~/.prisma-composer/emulators`; each directory still gets its own services - on them. Set the variable to an absolute directory - and `dev` starts and finds its compute and buckets daemons, their registry - and their data there instead. Each picks free ports, so several can run side - by side. A relative path is refused with `DEV.EMULATORS_DIR_INVALID`; `~` is - not expanded. `emulatorRegistryRoot()` from - `@prisma/composer-prisma-cloud/local-target` returns the directory in - effect. - - The variable does not isolate local Postgres. Local Postgres servers are - named after the app and the database, not the directory, so two checkouts - of the same app get the same server names. The second checkout's Postgres - emulator adopts the server the first one is running, and `--fresh` or a - teardown in either checkout stops that server and deletes its database for - both. + emulators.** By default every `dev` on the machine shares one set of + emulator daemons, registered under `~/.prisma-composer/emulators`; each + directory still gets its own services, databases and buckets on them. Set + the variable to an absolute directory and `dev` starts and finds its + compute, Postgres and buckets daemons, their registry and their state there + instead. Each picks free ports, so several can run side by side. A relative + path is refused with `DEV.EMULATORS_DIR_INVALID`; `~` is not expanded. + `emulatorRegistryRoot()` from `@prisma/composer-prisma-cloud/local-target` + returns the directory in effect. A local database's data is kept by + `@prisma/dev` in its own data directory + (`~/Library/Application Support/prisma-dev-nodejs/` on macOS, + `$XDG_DATA_HOME/prisma-dev-nodejs/` on Linux), under a name unique to the + database, whatever the variable says. - **The local Postgres is one shared session.** Every connection to a local database lands in the same Postgres session, which outlives your service processes. If you use Bun's `SQL`, pass `prepare: false` diff --git a/packages/0-framework/1-core/core/src/control/app-config.ts b/packages/0-framework/1-core/core/src/control/app-config.ts index 9a7301f30..6db23a473 100644 --- a/packages/0-framework/1-core/core/src/control/app-config.ts +++ b/packages/0-framework/1-core/core/src/control/app-config.ts @@ -285,8 +285,8 @@ export interface LocalTargetAttachment { signal: AbortSignal, opts?: { readonly tail?: number }, ): AsyncIterable<{ readonly service: string; readonly line: string }>; - /** Stop the app's service instances (emulators and data persist). */ - stopServices(): Promise; + /** Stop everything the target runs for the app, so nothing of it keeps running after the session (emulators and data persist). */ + stopApp(): Promise; } /** `/.prisma-composer/dev` — the dev instance's app-scoped state directory (ADR-0041, ADR-0004's tool-state rule). "dev" names the user-facing feature/dir (naming, operator 2026-07-23) — this constant's name and value are unchanged by the localTarget rename. */ diff --git a/packages/0-framework/3-tooling/cli/src/operations/__tests__/operations.test.ts b/packages/0-framework/3-tooling/cli/src/operations/__tests__/operations.test.ts index c5b82723a..2e10e4ec2 100644 --- a/packages/0-framework/3-tooling/cli/src/operations/__tests__/operations.test.ts +++ b/packages/0-framework/3-tooling/cli/src/operations/__tests__/operations.test.ts @@ -1199,7 +1199,7 @@ function fakeAttachment( ): LocalTargetAttachment { return { startServices: () => Promise.resolve(), - stopServices: () => Promise.resolve(), + stopApp: () => Promise.resolve(), endpoints: () => Promise.resolve(endpoints), logs, }; @@ -1313,7 +1313,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { let stops = 0; const attachment: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => { + stopApp: () => { stops += 1; return Promise.resolve(); }, @@ -1347,7 +1347,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { let stops = 0; const attachment: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => { + stopApp: () => { stops += 1; return Promise.resolve(); }, @@ -1380,7 +1380,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { let stops = 0; const attachment: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => { + stopApp: () => { stops += 1; return Promise.resolve(); }, @@ -1418,7 +1418,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { // the rollback must stop this attachment even though startServices // never returned. startServices: () => Promise.reject(new Error('service two failed to bind its port')), - stopServices: () => { + stopApp: () => { stops += 1; return Promise.resolve(); }, @@ -1458,7 +1458,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { fs.writeFileSync(buildOutput, 'v1'); const attachment: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => Promise.resolve(), + stopApp: () => Promise.resolve(), endpoints: () => Promise.resolve([]), logs: async function* () {}, }; @@ -1519,7 +1519,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { const app = makeAppDir('hello-dev'); const attachment: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => Promise.resolve(), + stopApp: () => Promise.resolve(), endpoints: () => Promise.resolve([]), logs: async function* () {}, }; @@ -1585,7 +1585,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { describe('the prisma.config.ts watch', () => { const quietAttachment: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => Promise.resolve(), + stopApp: () => Promise.resolve(), endpoints: () => Promise.resolve([]), logs: async function* () {}, }; @@ -1700,7 +1700,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { const app = makeAppDir('hello-dev'); const attachment: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => Promise.reject(new Error('service pid 123 will not die')), + stopApp: () => Promise.reject(new Error('service pid 123 will not die')), endpoints: () => Promise.resolve([]), logs: async function* () {}, }; @@ -1733,7 +1733,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { const app = makeAppDir('hello-dev'); const attachment: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => Promise.resolve(), + stopApp: () => Promise.resolve(), endpoints: () => Promise.resolve([{ address: 'app', url: 'http://localhost:3000' }]), logs: async function* () {}, }; @@ -1782,7 +1782,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { const app = makeAppDir('hello-dev'); const attachment: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => Promise.resolve(), + stopApp: () => Promise.resolve(), endpoints: () => Promise.resolve([]), logs: async function* () {}, }; @@ -1816,7 +1816,7 @@ describe.skipIf(process.platform === 'win32')('dev()', () => { fs.writeFileSync(path.join(app.dir, '.prisma-composer'), 'not a directory'); const attachment: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => Promise.resolve(), + stopApp: () => Promise.resolve(), endpoints: () => Promise.resolve([]), logs: async function* () {}, }; @@ -1984,7 +1984,7 @@ describe.skipIf(process.platform === 'win32')('log()', () => { let attempts = 0; const flaky: LocalTargetAttachment = { startServices: () => Promise.resolve(), - stopServices: () => Promise.resolve(), + stopApp: () => Promise.resolve(), endpoints: () => { attempts += 1; if (attempts === 1) { diff --git a/packages/0-framework/3-tooling/cli/src/operations/execute-dev.ts b/packages/0-framework/3-tooling/cli/src/operations/execute-dev.ts index c7d5b6d0a..cb8138faa 100644 --- a/packages/0-framework/3-tooling/cli/src/operations/execute-dev.ts +++ b/packages/0-framework/3-tooling/cli/src/operations/execute-dev.ts @@ -184,7 +184,7 @@ export async function executeDev( for (const [id, dev] of resolved) { try { const attachment = await dev.attach({ container: containers.get(id), devDir }); - await attachment.stopServices(); + await attachment.stopApp(); } catch { // Best effort: the converge failure is the error to report. } @@ -263,7 +263,7 @@ export async function executeDev( for (const attachment of attachments) { // Recorded BEFORE the call: startServices can start some services and // then throw, and a partially-started attachment must be rolled back by - // the catch below too (stopServices failures are already swallowed + // the catch below too (stopApp failures are already swallowed // there, so stopping a never-started attachment is harmless). started.push(attachment); try { @@ -384,7 +384,7 @@ export async function executeDev( // teardown continues, `stopped` still fires, `closed` still settles. for (const attachment of attachments) { try { - await attachment.stopServices(); + await attachment.stopApp(); } catch (error) { emit({ kind: 'stop-error', message: failureMessage(error) }); } @@ -403,7 +403,7 @@ export async function executeDev( // back as values — a non-structured escape is a bug and throws (rule 6). watch?.stop(); configWatch?.stop(); - await Promise.all(started.map((a) => a.stopServices().catch(() => undefined))); + await Promise.all(started.map((a) => a.stopApp().catch(() => undefined))); if (CliStructuredError.is(error)) return notOk(error); throw error; } diff --git a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/buckets.test.ts b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/buckets.test.ts index 36c94a392..4a7b5b14b 100644 --- a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/buckets.test.ts +++ b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/buckets.test.ts @@ -1,13 +1,15 @@ /** - * The bucket emulator (local-dev spec § 2 `buckets-main.ts`): `/_pcdev/` - * admin, SigV4 round-trip via a real `@aws-sdk/client-s3`, multi-app - * isolation (same bucket name, two apps, no collisions), 501 multipart, and - * validation. Every test uses a temp `registryRoot` and stops the daemon it - * started. + * The bucket emulator (local-dev spec § 2 `buckets-main.ts`): bucket + * instances keyed by ids the emulator generates, which are also their S3 + * bucket names; per-instance credentials; a SigV4 round-trip via a real + * `@aws-sdk/client-s3`; isolation between buckets; 501 multipart; and + * discarding an earlier version's state. Every test uses a temp + * `registryRoot` and stops the daemon it started. */ import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; import * as fs from 'node:fs'; import * as path from 'node:path'; +import { fileURLToPath } from 'node:url'; import { GetObjectCommand, ListMultipartUploadsCommand, @@ -15,17 +17,19 @@ import { S3Client, } from '@aws-sdk/client-s3'; import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; -import { bucketsClient } from '../client.ts'; -import { ensureDaemon, stopDaemon } from '../daemon.ts'; -import { ensureFreshDaemon, entryFor, tempDir } from './helpers.ts'; +import { type BucketsClient, bucketsClient } from '../client.ts'; +import { daemonStateDir, ensureDaemon, stopDaemon } from '../daemon.ts'; +import { ensureFreshDaemon, entryFor, tempDir, waitFor } from './helpers.ts'; let registryRoot: string; let daemonUrl: string; +let client: BucketsClient; beforeEach(async () => { registryRoot = tempDir('buckets-registry'); const { url } = await ensureFreshDaemon('buckets', registryRoot); daemonUrl = url; + client = bucketsClient({ registryRoot }); }); afterEach(async () => { @@ -44,222 +48,208 @@ function s3For(accessKeyId: string, secretAccessKey: string): S3Client { }); } -describe('bucket + credential admin', () => { - test('PUT bucket registers the physical name and creates the directory; idempotent', async () => { - const client = bucketsClient({ registryRoot }); - const dir = path.join(tempDir('bucket-data'), 'objects'); +function dataDir(): string { + return path.join(tempDir('bucket-data'), 'objects'); +} - await client.putBucket('myapp', 'uploads', dir); - expect(fs.existsSync(dir)).toBe(true); +describe('bucket instances', () => { + test('create makes the data root; list and get return the labels verbatim', async () => { + const dir = dataDir(); + const labels = { 'composer.app': 'My App', note: 'any: /value' }; - // Second PUT with the same args is idempotent — no error, dir untouched. - await client.putBucket('myapp', 'uploads', dir); + const bucket = await client.createInstance({ labels, dir }); + + expect(bucket.id).toMatch(/^bkt-[0-9a-f]{16}$/); + expect(bucket).toEqual({ id: bucket.id, labels, dir }); expect(fs.existsSync(dir)).toBe(true); + expect(await client.listInstances()).toEqual([bucket]); + expect(await client.getInstance(bucket.id)).toEqual(bucket); + expect(await client.getInstance('bkt-0000000000000000')).toBeUndefined(); }); - test('DELETE app removes registrations and credentials but not the object directory', async () => { - const client = bucketsClient({ registryRoot }); - const dir = path.join(tempDir('bucket-data'), 'objects'); - await client.putBucket('myapp', 'uploads', dir); - await client.putCredentials('myapp', 'AKIADELETETEST', 'secret'); + test('delete forgets the bucket and its credentials but leaves its objects', async () => { + const dir = dataDir(); + const bucket = await client.createInstance({ labels: {}, dir }); + await client.putCredentials(bucket.id, 'AKIADELETETEST', 'secret'); fs.writeFileSync(path.join(dir, 'kept.txt'), 'still here'); - await client.deleteApp('myapp'); + await client.deleteInstance(bucket.id); - // The registration is gone: a PUT through the S3 wire against the - // physical bucket now hits an unknown bucket (put has no graceful - // shape, so it throws — surfaced by the SDK as an error). - const s3 = s3For('AKIADELETETEST', 'secret'); + expect(await client.getInstance(bucket.id)).toBeUndefined(); await expect( - s3.send(new PutObjectCommand({ Bucket: 'myapp--uploads', Key: 'x', Body: 'x' })), + s3For('AKIADELETETEST', 'secret').send( + new PutObjectCommand({ Bucket: bucket.id, Key: 'x', Body: 'x' }), + ), ).rejects.toThrow(); - - // Objects already on disk are untouched — only `teardown`'s `fs.rm` owns them. expect(fs.existsSync(path.join(dir, 'kept.txt'))).toBe(true); + await client.deleteInstance(bucket.id); }); - test('re-registering an accessKeyId already owned by a different app is rejected with 409, naming neither secret', async () => { - const client = bucketsClient({ registryRoot }); - await client.putCredentials('myapp', 'AKIACONFLICT', 'myappsecret'); + test('a malformed instance id is rejected with 400', async () => { + const res = await fetch(`${daemonUrl}/_pcdev/instances/uploads`); + expect(res.status).toBe(400); + expect(await res.text()).toContain('"uploads"'); + }); - const res = await fetch(`${daemonUrl}/_pcdev/apps/otherapp/credentials`, { + test('labels that are not all strings are rejected with 400', async () => { + const res = await fetch(`${daemonUrl}/_pcdev/instances`, { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ labels: { count: 1 }, dir: dataDir() }), + }); + expect(res.status).toBe(400); + }); +}); + +describe('credentials', () => { + test('an accessKeyId registered for another bucket is refused with 409, naming neither secret', async () => { + const mine = await client.createInstance({ labels: {}, dir: dataDir() }); + const other = await client.createInstance({ labels: {}, dir: dataDir() }); + await client.putCredentials(mine.id, 'AKIACONFLICT', 'minesecret'); + + const res = await fetch(`${daemonUrl}/_pcdev/instances/${other.id}/credentials`, { method: 'PUT', headers: { 'content-type': 'application/json' }, - body: JSON.stringify({ accessKeyId: 'AKIACONFLICT', secretAccessKey: 'otherappsecret' }), + body: JSON.stringify({ accessKeyId: 'AKIACONFLICT', secretAccessKey: 'othersecret' }), }); expect(res.status).toBe(409); const body = await res.text(); - expect(body).not.toContain('myappsecret'); - expect(body).not.toContain('otherappsecret'); - - // The SAME app re-registering (rotating the secret) still succeeds. - await client.putCredentials('myapp', 'AKIACONFLICT', 'myappsecret-rotated'); - const dir = path.join(tempDir('bucket-data'), 'objects'); - await client.putBucket('myapp', 'uploads', dir); - const s3 = s3For('AKIACONFLICT', 'myappsecret-rotated'); - await s3.send(new PutObjectCommand({ Bucket: 'myapp--uploads', Key: 'k', Body: 'x' })); - expect(fs.existsSync(path.join(dir, 'k'))).toBe(true); + expect(body).not.toContain('minesecret'); + expect(body).not.toContain('othersecret'); + + // The same bucket rotating its secret still succeeds. + await client.putCredentials(mine.id, 'AKIACONFLICT', 'minesecret-rotated'); + await s3For('AKIACONFLICT', 'minesecret-rotated').send( + new PutObjectCommand({ Bucket: mine.id, Key: 'k', Body: 'x' }), + ); + expect(fs.existsSync(path.join(mine.dir, 'k'))).toBe(true); }); - test('a credentials owning app round-trips across a daemon restart', async () => { - const client = bucketsClient({ registryRoot }); - const dir = path.join(tempDir('bucket-data'), 'objects'); - await client.putBucket('myapp', 'uploads', dir); - await client.putCredentials('myapp', 'AKIARESTART', 'restartsecret'); + test('buckets and their credentials survive an emulator restart', async () => { + const bucket = await client.createInstance({ labels: { note: 'kept' }, dir: dataDir() }); + await client.putCredentials(bucket.id, 'AKIARESTART', 'restartsecret'); await stopDaemon('buckets', { registryRoot }); const restarted = await ensureDaemon('buckets', entryFor('buckets'), { registryRoot }); daemonUrl = restarted.url; - // The secret survived: the original app's own credential still works. - const s3 = s3For('AKIARESTART', 'restartsecret'); - await s3.send( - new PutObjectCommand({ Bucket: 'myapp--uploads', Key: 'k', Body: 'still owned by myapp' }), + expect(await bucketsClient({ registryRoot }).getInstance(bucket.id)).toEqual(bucket); + await s3For('AKIARESTART', 'restartsecret').send( + new PutObjectCommand({ Bucket: bucket.id, Key: 'k', Body: 'still owned' }), ); - expect(fs.existsSync(path.join(dir, 'k'))).toBe(true); - - // The OWNER survived too: a different app re-registering the same - // accessKeyId still 409s, not a silent takeover. - const res = await fetch(`${daemonUrl}/_pcdev/apps/otherapp/credentials`, { - method: 'PUT', - headers: { 'content-type': 'application/json' }, - body: JSON.stringify({ accessKeyId: 'AKIARESTART', secretAccessKey: 'stolen' }), - }); - expect(res.status).toBe(409); + expect(fs.existsSync(path.join(bucket.dir, 'k'))).toBe(true); }); }); describe('SigV4 round-trip', () => { - test('put + get round-trips real bytes through the filesystem', async () => { - const client = bucketsClient({ registryRoot }); - const dir = path.join(tempDir('bucket-data'), 'objects'); - await client.putBucket('myapp', 'uploads', dir); - await client.putCredentials('myapp', 'AKIAROUNDTRIP', 'roundtripsecret'); - + test('put + get round-trips real bytes through the data root', async () => { + const bucket = await client.createInstance({ labels: {}, dir: dataDir() }); + await client.putCredentials(bucket.id, 'AKIAROUNDTRIP', 'roundtripsecret'); const s3 = s3For('AKIAROUNDTRIP', 'roundtripsecret'); + await s3.send( new PutObjectCommand({ - Bucket: 'myapp--uploads', + Bucket: bucket.id, Key: 'hello.txt', Body: 'hello bucket emulator', ContentType: 'text/plain', }), ); - expect(fs.existsSync(path.join(dir, 'hello.txt'))).toBe(true); - - const got = await s3.send(new GetObjectCommand({ Bucket: 'myapp--uploads', Key: 'hello.txt' })); - const body = await got.Body?.transformToString(); - expect(body).toBe('hello bucket emulator'); + expect(fs.existsSync(path.join(bucket.dir, 'hello.txt'))).toBe(true); + const got = await s3.send(new GetObjectCommand({ Bucket: bucket.id, Key: 'hello.txt' })); + expect(await got.Body?.transformToString()).toBe('hello bucket emulator'); }); - test('a wrong credential is rejected with a signature failure', async () => { - const client = bucketsClient({ registryRoot }); - const dir = path.join(tempDir('bucket-data'), 'objects'); - await client.putBucket('myapp', 'uploads', dir); - await client.putCredentials('myapp', 'AKIAREALKEY', 'realsecret'); + test('a wrong secret is rejected with a signature failure', async () => { + const bucket = await client.createInstance({ labels: {}, dir: dataDir() }); + await client.putCredentials(bucket.id, 'AKIAREALKEY', 'realsecret'); - const s3 = s3For('AKIAREALKEY', 'wrong-secret'); await expect( - s3.send(new PutObjectCommand({ Bucket: 'myapp--uploads', Key: 'x', Body: 'x' })), + s3For('AKIAREALKEY', 'wrong-secret').send( + new PutObjectCommand({ Bucket: bucket.id, Key: 'x', Body: 'x' }), + ), ).rejects.toThrow(); }); - test('SigV4 verifies against any of the bucket-owning apps OWN accepted credentials', async () => { - const client = bucketsClient({ registryRoot }); - const dir = path.join(tempDir('bucket-data'), 'objects'); - await client.putBucket('myapp', 'uploads', dir); - await client.putCredentials('myapp', 'AKIAFIRST', 'firstsecret'); - await client.putCredentials('myapp', 'AKIASECOND', 'secondsecret'); + test('any of a bucket’s own credentials verifies', async () => { + const bucket = await client.createInstance({ labels: {}, dir: dataDir() }); + await client.putCredentials(bucket.id, 'AKIAFIRST', 'firstsecret'); + await client.putCredentials(bucket.id, 'AKIASECOND', 'secondsecret'); - const s3First = s3For('AKIAFIRST', 'firstsecret'); - await s3First.send(new PutObjectCommand({ Bucket: 'myapp--uploads', Key: 'a', Body: 'a' })); - - const s3Second = s3For('AKIASECOND', 'secondsecret'); - await s3Second.send(new PutObjectCommand({ Bucket: 'myapp--uploads', Key: 'b', Body: 'b' })); + await s3For('AKIAFIRST', 'firstsecret').send( + new PutObjectCommand({ Bucket: bucket.id, Key: 'a', Body: 'a' }), + ); + await s3For('AKIASECOND', 'secondsecret').send( + new PutObjectCommand({ Bucket: bucket.id, Key: 'b', Body: 'b' }), + ); - expect(fs.existsSync(path.join(dir, 'a'))).toBe(true); - expect(fs.existsSync(path.join(dir, 'b'))).toBe(true); + expect(fs.existsSync(path.join(bucket.dir, 'a'))).toBe(true); + expect(fs.existsSync(path.join(bucket.dir, 'b'))).toBe(true); }); }); -describe('multi-app isolation', () => { - test('the same bucket name under two apps maps to distinct physical dirs, no collisions', async () => { - const client = bucketsClient({ registryRoot }); - const dirOne = path.join(tempDir('bucket-data-one'), 'objects'); - const dirTwo = path.join(tempDir('bucket-data-two'), 'objects'); - await client.putBucket('tenant-one', 'data', dirOne); - await client.putBucket('tenant-two', 'data', dirTwo); - await client.putCredentials('tenant-one', 'AKIAONE', 'onesecret'); - await client.putCredentials('tenant-two', 'AKIATWO', 'twosecret'); +describe('isolation between buckets with the same labels', () => { + const labels = { 'composer.app': 'same-app', 'composer.address': 'uploads' }; + test('each keeps its own objects in its own data root', async () => { + const one = await client.createInstance({ labels, dir: dataDir() }); + const two = await client.createInstance({ labels, dir: dataDir() }); + await client.putCredentials(one.id, 'AKIAONE', 'onesecret'); + await client.putCredentials(two.id, 'AKIATWO', 'twosecret'); const s3One = s3For('AKIAONE', 'onesecret'); const s3Two = s3For('AKIATWO', 'twosecret'); - await s3One.send( - new PutObjectCommand({ Bucket: 'tenant-one--data', Key: 'k', Body: 'from tenant one' }), - ); - await s3Two.send( - new PutObjectCommand({ Bucket: 'tenant-two--data', Key: 'k', Body: 'from tenant two' }), - ); - - const gotOne = await s3One.send(new GetObjectCommand({ Bucket: 'tenant-one--data', Key: 'k' })); - const gotTwo = await s3Two.send(new GetObjectCommand({ Bucket: 'tenant-two--data', Key: 'k' })); - expect(await gotOne.Body?.transformToString()).toBe('from tenant one'); - expect(await gotTwo.Body?.transformToString()).toBe('from tenant two'); + await s3One.send(new PutObjectCommand({ Bucket: one.id, Key: 'k', Body: 'from one' })); + await s3Two.send(new PutObjectCommand({ Bucket: two.id, Key: 'k', Body: 'from two' })); - expect(fs.readFileSync(path.join(dirOne, 'k'), 'utf8')).toBe('from tenant one'); - expect(fs.readFileSync(path.join(dirTwo, 'k'), 'utf8')).toBe('from tenant two'); + const gotOne = await s3One.send(new GetObjectCommand({ Bucket: one.id, Key: 'k' })); + const gotTwo = await s3Two.send(new GetObjectCommand({ Bucket: two.id, Key: 'k' })); + expect(await gotOne.Body?.transformToString()).toBe('from one'); + expect(await gotTwo.Body?.transformToString()).toBe('from two'); + expect(fs.readFileSync(path.join(one.dir, 'k'), 'utf8')).toBe('from one'); + expect(fs.readFileSync(path.join(two.dir, 'k'), 'utf8')).toBe('from two'); }); - test("app A's credential is rejected against app B's bucket exactly like a bad signature", async () => { - const client = bucketsClient({ registryRoot }); - const dirTwo = path.join(tempDir('bucket-data-two'), 'objects'); - await client.putBucket('tenant-two', 'data', dirTwo); - await client.putCredentials('tenant-one', 'AKIACROSSONE', 'onesecret'); - await client.putCredentials('tenant-two', 'AKIACROSSTWO', 'twosecret'); - - // tenant-one's credential, CORRECTLY signed, targeting tenant-two's bucket. - const s3CrossApp = s3For('AKIACROSSONE', 'onesecret'); - const crossAppUrl = await getSignedUrl( - s3CrossApp, - new PutObjectCommand({ Bucket: 'tenant-two--data', Key: 'x' }), + test("one bucket's credential is rejected against another exactly like a bad signature", async () => { + const one = await client.createInstance({ labels, dir: dataDir() }); + const two = await client.createInstance({ labels, dir: dataDir() }); + await client.putCredentials(one.id, 'AKIACROSSONE', 'onesecret'); + await client.putCredentials(two.id, 'AKIACROSSTWO', 'twosecret'); + + // One's credential, CORRECTLY signed, targeting two's bucket. + const crossUrl = await getSignedUrl( + s3For('AKIACROSSONE', 'onesecret'), + new PutObjectCommand({ Bucket: two.id, Key: 'x' }), { expiresIn: 900 }, ); - const crossAppRes = await fetch(crossAppUrl, { method: 'PUT', body: 'x' }); - - // tenant-two's OWN accessKeyId, but a wrong secret — a genuinely bad signature. - const s3BadSig = s3For('AKIACROSSTWO', 'not-the-real-secret'); + const crossRes = await fetch(crossUrl, { method: 'PUT', body: 'x' }); + // Two's OWN accessKeyId with a wrong secret — a genuinely bad signature. const badSigUrl = await getSignedUrl( - s3BadSig, - new PutObjectCommand({ Bucket: 'tenant-two--data', Key: 'x' }), + s3For('AKIACROSSTWO', 'not-the-real-secret'), + new PutObjectCommand({ Bucket: two.id, Key: 'x' }), { expiresIn: 900 }, ); const badSigRes = await fetch(badSigUrl, { method: 'PUT', body: 'x' }); - expect(crossAppRes.status).toBe(403); - expect(crossAppRes.status).toBe(badSigRes.status); - expect(await crossAppRes.text()).toBe(await badSigRes.text()); - - expect(fs.existsSync(path.join(dirTwo, 'x'))).toBe(false); + expect(crossRes.status).toBe(403); + expect(crossRes.status).toBe(badSigRes.status); + expect(await crossRes.text()).toBe(await badSigRes.text()); + expect(fs.existsSync(path.join(two.dir, 'x'))).toBe(false); }); }); describe('multipart uploads', () => { test('an initiate-multipart request is rejected with the pinned 501', async () => { - const client = bucketsClient({ registryRoot }); - const dir = path.join(tempDir('bucket-data'), 'objects'); - await client.putBucket('myapp', 'uploads', dir); - await client.putCredentials('myapp', 'AKIAMULTI', 'multisecret'); + const bucket = await client.createInstance({ labels: {}, dir: dataDir() }); + await client.putCredentials(bucket.id, 'AKIAMULTI', 'multisecret'); - const s3 = s3For('AKIAMULTI', 'multisecret'); await expect( - s3.send(new ListMultipartUploadsCommand({ Bucket: 'myapp--uploads' })), + s3For('AKIAMULTI', 'multisecret').send( + new ListMultipartUploadsCommand({ Bucket: bucket.id }), + ), ).rejects.toThrow(); - - // Confirm the actual wire response is exactly the pinned 501 + body - // (the SDK's own error swallows status/body detail). - const res = await fetch(`${daemonUrl}/myapp--uploads?uploads`); + const res = await fetch(`${daemonUrl}/${bucket.id}?uploads`); expect(res.status).toBe(501); expect(await res.text()).toBe( 'multipart upload is not supported by the local dev bucket emulator yet', @@ -267,39 +257,76 @@ describe('multipart uploads', () => { }); }); -describe('validation', () => { - test('PUT bucket rejects an app segment with an uppercase letter, naming the segment', async () => { - const res = await fetch(`${daemonUrl}/_pcdev/apps/BadApp/buckets/data`, { - method: 'PUT', - headers: { 'content-type': 'application/json' }, - body: JSON.stringify({ dir: tempDir('bucket-invalid') }), - }); - expect(res.status).toBe(400); - expect(await res.text()).toContain('BadApp'); +describe('health', () => { + test('health is available at both /health and /_pcdev/health', async () => { + expect((await fetch(`${daemonUrl}/health`)).status).toBe(200); + expect((await fetch(`${daemonUrl}/_pcdev/health`)).status).toBe(200); }); +}); - test('PUT bucket rejects a physical name over the 63-char cap, naming both parts and the cap', async () => { - // Each of app/name individually satisfies the ≤63 path-segment rule, but - // "--" (with its 2-char separator) exceeds the store's own - // 63-char bucket-name cap — the SEPARATE check this endpoint applies. - const app = 'a'.repeat(35); - const name = 'b'.repeat(35); - const res = await fetch(`${daemonUrl}/_pcdev/apps/${app}/buckets/${name}`, { - method: 'PUT', - headers: { 'content-type': 'application/json' }, - body: JSON.stringify({ dir: tempDir('bucket-invalid') }), - }); - expect(res.status).toBe(400); - const body = await res.text(); - expect(body).toContain(app); - expect(body).toContain(name); - expect(body).toContain('63'); +describe('state written by an earlier version', () => { + test('is discarded on load', async () => { + await stopDaemon('buckets', { registryRoot }); + const stateDir = daemonStateDir(registryRoot, 'buckets'); + fs.rmSync(path.join(stateDir, 'instances.json'), { force: true }); + fs.writeFileSync( + path.join(stateDir, 'state.json'), + JSON.stringify({ + buckets: { 'legacy--data': { app: 'legacy', name: 'data', dir: dataDir() } }, + credentials: { AKIALEGACY: { app: 'legacy', secretAccessKey: 's' } }, + }), + ); + + await ensureDaemon('buckets', entryFor('buckets'), { registryRoot }); + + expect(await bucketsClient({ registryRoot }).listInstances()).toEqual([]); + expect(fs.existsSync(path.join(stateDir, 'state.json'))).toBe(false); + }); +}); + +/** + * The emulator never interprets Composer's data: an app name, a node address + * or a directory reaches it only as label values and as the data root it is + * given, which it stores verbatim. + */ +describe('the emulator never interprets Composer data', () => { + test('labels and the data root are stored verbatim, and nothing else carries their values', async () => { + const directory = path.join(tempDir('marker-project'), 'zzdirectorymarker'); + const dir = path.join(directory, '.prisma-composer', 'dev', 'buckets', 'zzaddressmarker'); + const labels = { + 'composer.directory': directory, + 'composer.app': 'zzappmarker', + 'composer.address': 'zzaddressmarker', + }; + const bucket = await client.createInstance({ labels, dir }); + await client.putCredentials(bucket.id, 'AKIAMARKER', 'secret'); + + const stateDir = daemonStateDir(registryRoot, 'buckets'); + const statePath = path.join(stateDir, 'instances.json'); + await waitFor(() => fs.existsSync(statePath), 5000); + const state: unknown = JSON.parse(fs.readFileSync(statePath, 'utf8')); + const record: unknown = Reflect.get(Reflect.get(state as object, 'instances'), bucket.id); + expect(record).toMatchObject({ labels, dir }); + + const opaque = (key: string, value: unknown) => + key === 'labels' || key === 'dir' ? undefined : value; + const listed = JSON.stringify(await client.listInstances(), opaque); + for (const marker of ['zzdirectorymarker', 'zzappmarker', 'zzaddressmarker']) { + expect(JSON.stringify(state, opaque)).not.toContain(marker); + expect(listed).not.toContain(marker); + for (const file of fs.readdirSync(stateDir, { recursive: true })) { + expect(String(file)).not.toContain(marker); + } + } }); - test('health is available at both /health and /_pcdev/health', async () => { - const plain = await fetch(`${daemonUrl}/health`); - const admin = await fetch(`${daemonUrl}/_pcdev/health`); - expect(plain.status).toBe(200); - expect(admin.status).toBe(200); + test("the emulator's code names no app, address, directory or Composer concept", () => { + const source = fs.readFileSync( + fileURLToPath(new URL('../buckets-main.ts', import.meta.url)), + 'utf8', + ); + const code = source.replace(/\/\*[\s\S]*?\*\//g, '').replace(/\/\/.*$/gm, ''); + + expect(code).not.toMatch(/\b(apps?|appName|address(es)?|directory|composer)\b/i); }); }); diff --git a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/compute-deployment.test.ts b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/compute-deployment.test.ts index 45c559607..6d6f1c231 100644 --- a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/compute-deployment.test.ts +++ b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/compute-deployment.test.ts @@ -6,7 +6,7 @@ * started. */ import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; -import { computeClient, type InstanceInfo } from '../client.ts'; +import { computeClient, type ServiceInstanceInfo } from '../client.ts'; import { isPidAlive, stopDaemon } from '../daemon.ts'; import { CRASHING_BOOTSTRAP, @@ -46,7 +46,7 @@ function baseEnv(extra: Record = {}): Record { return { PATH: path, ...extra }; } -async function deployedService(id: string): Promise { +async function deployedService(id: string): Promise { const instance = await computeClient({ registryRoot }).getInstance(id); if (!instance) throw new Error(`instance "${id}" not found`); return instance; diff --git a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/compute-readiness.test.ts b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/compute-readiness.test.ts index a3b22929b..4d6592e32 100644 --- a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/compute-readiness.test.ts +++ b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/compute-readiness.test.ts @@ -11,7 +11,7 @@ import * as net from 'node:net'; import * as path from 'node:path'; import { fileURLToPath } from 'node:url'; import getPort, { portNumbers } from 'get-port'; -import { type ComputeClient, computeClient, type InstanceInfo } from '../client.ts'; +import { type ComputeClient, computeClient, type ServiceInstanceInfo } from '../client.ts'; import { ensureDaemon, isPidAlive, SERVICE_READY_TIMEOUT_ENV, stopDaemon } from '../daemon.ts'; import { CRASH_ONCE_THEN_SILENT_BOOTSTRAP, @@ -69,14 +69,14 @@ async function startDaemon(entry?: string): Promise { */ const OWN_PORTS = portNumbers(3400, 3599); -async function createInstanceOnOwnPort(client: ComputeClient): Promise { +async function createInstanceOnOwnPort(client: ComputeClient): Promise { const port = await getPort({ port: OWN_PORTS }); const instance = await client.createInstance({ labels: {}, port }); expect(instance.port).toBe(port); return instance; } -async function instanceOf(client: ComputeClient, id: string): Promise { +async function instanceOf(client: ComputeClient, id: string): Promise { const instance = await client.getInstance(id); if (!instance) throw new Error(`instance "${id}" not found`); return instance; diff --git a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/postgres.test.ts b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/postgres.test.ts index d2949275e..d1c132139 100644 --- a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/postgres.test.ts +++ b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/__tests__/postgres.test.ts @@ -1,11 +1,11 @@ /** - * The Postgres emulator (local-dev spec § 2 `postgres-main.ts`): ensure/ - * list/delete lifecycle, port stability with data intact across a daemon - * restart, multi-app isolation, a bogus module path, and fresh-allocation - * port retry. Every test uses a temp `registryRoot` and a real, distinct - * app name so its instances (`pcdev--`) never collide with real - * `@prisma/dev` usage elsewhere on this machine, and every test cleans up - * via `DELETE /apps/` so it never leaves persisted PGlite data behind. + * The Postgres emulator (local-dev spec § 2 `postgres-main.ts`): database + * instances keyed by ids the emulator generates, each served by its own + * `@prisma/dev` server; stop and start keeping data and port; delete + * removing the data; port allocation and its retries; adoption of a server + * another process holds; and discarding an earlier version's state. Every + * test uses a temp `registryRoot` and deletes the instances it created, so + * it never leaves persisted PGlite data behind. */ import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; @@ -14,25 +14,89 @@ import * as fs from 'node:fs'; import * as net from 'node:net'; import * as path from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; -import { deleteServer } from '@prisma/dev/internal/state'; +import { deleteServer, getServerStatus } from '@prisma/dev/internal/state'; import getPort, { portNumbers } from 'get-port'; import { Client as PgClient } from 'pg'; -import { postgresClient } from '../client.ts'; -import { stopDaemon } from '../daemon.ts'; -import { instanceNameFor } from '../instance-name.ts'; +import { type PostgresClient, postgresClient } from '../client.ts'; +import { daemonStateDir, ensureDaemon, stopDaemon } from '../daemon.ts'; import { type RegistryScanHost, registryClaimedPorts } from '../prisma-dev-registry.ts'; -import { ensureFreshDaemon, prismaDevModulePath, tempDir } from './helpers.ts'; +import { ensureFreshDaemon, entryFor, prismaDevModulePath, tempDir, waitFor } from './helpers.ts'; let registryRoot: string; +let created: string[]; beforeEach(() => { registryRoot = tempDir('postgres-registry'); + created = []; }); afterEach(async () => { + const client = (() => { + try { + return postgresClient({ registryRoot }); + } catch { + return undefined; + } + })(); + for (const id of created) await client?.deleteInstance(id).catch(() => undefined); await stopDaemon('postgres', { registryRoot }).catch(() => undefined); }); +async function startedClient(): Promise { + await ensureFreshDaemon('postgres', registryRoot); + return postgresClient({ registryRoot }); +} + +async function createDatabase( + client: PostgresClient, + opts: { readonly labels?: Record; readonly port?: number } = {}, +) { + const instance = await client.createInstance({ + labels: opts.labels ?? {}, + ...(opts.port !== undefined ? { port: opts.port } : {}), + prismaDevModulePath: prismaDevModulePath(), + }); + created.push(instance.id); + return instance; +} + +/** + * `@prisma/dev`'s entry reached through a symlink under `under` — a module + * path that lives inside a directory the test chooses, and that disappears + * when that directory is removed, as an upgraded install's old copy does. + */ +function linkedPrismaDevModulePath(under: string): string { + const entry = fs.realpathSync(prismaDevModulePath()); + let packageDir = path.dirname(entry); + while (!fs.existsSync(path.join(packageDir, 'package.json'))) + packageDir = path.dirname(packageDir); + const linkParent = path.join(under, 'node_modules', '@prisma'); + fs.mkdirSync(linkParent, { recursive: true }); + fs.symlinkSync(packageDir, path.join(linkParent, 'dev')); + return path.join(linkParent, 'dev', path.relative(packageDir, entry)); +} + +async function query(url: string, sql: string, params: unknown[] = []): Promise { + const pg = new PgClient({ connectionString: url }); + await pg.connect(); + try { + return (await pg.query(sql, params)).rows; + } finally { + await pg.end().catch(() => undefined); + } +} + +function acceptsConnection(port: number): Promise { + return new Promise((resolve) => { + const socket = net.connect({ port, host: '127.0.0.1' }); + socket.once('connect', () => { + socket.destroy(); + resolve(true); + }); + socket.once('error', () => resolve(false)); + }); +} + /** Occupies a real port with a plain TCP listener — a stand-in for "another Prisma Dev server already holds this port." */ function occupyPort(port: number): Promise { return new Promise((resolve, reject) => { @@ -46,270 +110,231 @@ function closeServer(server: net.Server): Promise { return new Promise((resolve) => server.close(() => resolve())); } -describe('database lifecycle', () => { - test('ensure, list, and delete a database', async () => { - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); - - const ensured = await client.ensureDatabase('pgtest-lifecycle', 'appdb', prismaDevModulePath()); - expect(ensured.url).toMatch(/^postgres:\/\//); - - const listed = await client.listDatabases('pgtest-lifecycle'); - expect(listed).toHaveLength(1); - const entry = listed[0]; - if (!entry) throw new Error('expected one listed database'); - expect(entry.id).toBe('appdb'); - expect(entry.url).toBe(ensured.url); - expect(entry.instanceName).toBe('pcdev-pgtest-lifecycle-appdb'); - - // A real connection actually works — this is a real Postgres server, not a stub. - const pg = new PgClient({ connectionString: ensured.url }); - await pg.connect(); - const res = await pg.query('select 1 as one'); - expect(res.rows[0].one).toBe(1); - await pg.end(); - - await client.deleteApp('pgtest-lifecycle'); - expect(await client.listDatabases('pgtest-lifecycle')).toHaveLength(0); - // 60s, not 30s: this boots a real Postgres and every step is awaited, so - // there is no race to fix here — it is simply slow, and 30s left no room - // for a loaded CI runner (observed timing out at 30151ms against a ~8s - // normal run). The heavier tests below already sit at 45-120s. +describe('database instances', () => { + test('create, list, get and delete a database; the server is a real Postgres', async () => { + const client = await startedClient(); + const labels = { 'composer.app': 'My App', note: 'any: /value' }; + + const db = await createDatabase(client, { labels }); + + expect(db.id).toMatch(/^db_[0-9a-f]{16}$/); + expect(db.labels).toEqual(labels); + expect(db.status).toBe('running'); + expect(db.url).toMatch(/^postgres:\/\//); + expect(Number(new URL(db.url).port)).toBe(db.port); + expect(await query(db.url, 'select 1 as one')).toEqual([{ one: 1 }]); + expect((await client.listInstances()).map((i) => i.id)).toEqual([db.id]); + expect(await client.getInstance(db.id)).toEqual(db); + + await client.deleteInstance(db.id); + + expect(await client.getInstance(db.id)).toBeUndefined(); + expect(await acceptsConnection(db.port)).toBe(false); + // 60s: this boots a real Postgres, and a loaded CI runner is slow. + }, 60_000); + + test('two instances with the same labels are separate databases', async () => { + const client = await startedClient(); + const labels = { 'composer.app': 'same-app', 'composer.address': 'db' }; + const a = await createDatabase(client, { labels }); + const b = await createDatabase(client, { labels }); + + await query(a.url, 'create table marker (v text)'); + + expect(a.port).not.toBe(b.port); + expect(await query(b.url, "select to_regclass('marker') as reg")).toEqual([{ reg: null }]); }, 60_000); - test('ensure is idempotent — a second call returns the same URL without restarting', async () => { - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); + test('a start naming an unknown instance fails and names it; a stop skips it', async () => { + const client = await startedClient(); + const db = await createDatabase(client); - const first = await client.ensureDatabase('pgtest-idempotent', 'appdb', prismaDevModulePath()); - const second = await client.ensureDatabase('pgtest-idempotent', 'appdb', prismaDevModulePath()); - expect(second.url).toBe(first.url); + await expect( + client.startInstances([db.id, 'db_0000000000000000'], prismaDevModulePath()), + ).rejects.toThrow('"db_0000000000000000"'); + await client.stopInstances([db.id, 'db_0000000000000000']); - await client.deleteApp('pgtest-idempotent'); - // Same daemon-boot cost as the test above, so the same budget. + expect((await client.getInstance(db.id))?.status).toBe('stopped'); }, 60_000); }); -describe('Postgres messages larger than one loopback TCP read', () => { - // A regression test for @prisma/dev 0.20.0, whose pglite-socket 0.0.20 - // handed each raw 64 KiB TCP chunk to PGlite as if it were a whole - // message: a bind parameter over that size killed the engine and every - // later connection was refused. Composer now pins a fixed @prisma/dev; - // this test fails against the old one. - test('a 100 KiB bind parameter round-trips and the server keeps accepting connections', async () => { - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); - const { url } = await client.ensureDatabase('pgtest-large', 'appdb', prismaDevModulePath()); +describe('stop and start', () => { + test('a stopped database holds its port and no server; a start brings it back with its data', async () => { + const client = await startedClient(); + const db = await createDatabase(client); + await query(db.url, 'create table kept (v text)'); + await query(db.url, "insert into kept values ('row')"); - const payload = 'x'.repeat(100 * 1024); - const first = new PgClient({ connectionString: url }); - await first.connect(); - try { - const res = await first.query('select length($1::text) as len', [payload]); - expect(res.rows[0].len).toBe(payload.length); - } finally { - await first.end().catch(() => undefined); - } + await client.stopInstances([db.id]); + + expect(await client.getInstance(db.id)).toMatchObject({ status: 'stopped', port: db.port }); + expect(await acceptsConnection(db.port)).toBe(false); + const other = await createDatabase(client, { port: db.port }); + expect(other.port).not.toBe(db.port); - const second = new PgClient({ connectionString: url }); - await second.connect(); - const res = await second.query('select 1 as one'); - expect(res.rows[0].one).toBe(1); - await second.end(); + await client.startInstances([db.id], prismaDevModulePath()); - await client.deleteApp('pgtest-large'); + const restarted = await client.getInstance(db.id); + expect(restarted).toMatchObject({ status: 'running', port: db.port, url: db.url }); + expect(await query(db.url, 'select v from kept')).toEqual([{ v: 'row' }]); + }, 90_000); + + test('starting a running database changes nothing', async () => { + const client = await startedClient(); + const db = await createDatabase(client); + await query(db.url, 'create table still_here (v text)'); + + await client.startInstances([db.id], prismaDevModulePath()); + + expect(await client.getInstance(db.id)).toMatchObject({ status: 'running', url: db.url }); + expect(await query(db.url, "select to_regclass('still_here') is not null as ok")).toEqual([ + { ok: true }, + ]); }, 60_000); -}); -describe('port stability across a daemon restart', () => { - test('a version-skew restart drops the in-process server; PUT brings it back on the SAME port with data intact', async () => { - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); - - const first = await client.ensureDatabase('pgtest-restart', 'appdb', prismaDevModulePath()); - const firstPort = new URL(first.url.replace('postgres://', 'http://')).port; - - // Write a real row via a real `pg` connection before the restart. - const writer = new PgClient({ connectionString: first.url }); - await writer.connect(); - await writer.query('drop table if exists restart_check'); - await writer.query('create table restart_check (id integer primary key)'); - await writer.query('insert into restart_check (id) values (1)'); - await writer.end(); - - // A daemon restart (this package's own version-skew path drops the - // in-process server): stop it outright and start a fresh one on the - // SAME registryRoot — the persisted `apps.json` survives, the live - // `@prisma/dev` server object does not. `stopDaemon` deletes the - // DAEMON's own registry entry, so the outer daemon admin port is free - // to land somewhere new on the next ensure — only the DATABASE port - // (postgres-main's own persisted state) must stay put. A fresh client - // re-resolves the (possibly new) daemon admin port; reusing the old - // `client` would talk to a port nothing is listening on anymore. + test('after an emulator restart a database is stopped, and a start brings it back on its port with its data', async () => { + const client = await startedClient(); + const db = await createDatabase(client); + await query(db.url, 'create table restart_check (id integer primary key)'); + await query(db.url, 'insert into restart_check (id) values (1)'); + await stopDaemon('postgres', { registryRoot }); - await ensureFreshDaemonSamePort(registryRoot); - const clientAfterRestart = postgresClient({ registryRoot }); + await ensureDaemon('postgres', entryFor('postgres'), { registryRoot }); + const after = postgresClient({ registryRoot }); - const second = await clientAfterRestart.ensureDatabase( - 'pgtest-restart', - 'appdb', - prismaDevModulePath(), - ); - const secondPort = new URL(second.url.replace('postgres://', 'http://')).port; - expect(secondPort).toBe(firstPort); + expect(await after.getInstance(db.id)).toMatchObject({ status: 'stopped', port: db.port }); + await after.startInstances([db.id], prismaDevModulePath()); + expect(await after.getInstance(db.id)).toMatchObject({ status: 'running', url: db.url }); + expect(await query(db.url, 'select id from restart_check')).toEqual([{ id: 1 }]); + }, 90_000); + + test('deleting a stopped database removes its data', async () => { + const client = await startedClient(); + const db = await createDatabase(client); + await client.stopInstances([db.id]); - const reader = new PgClient({ connectionString: second.url }); - await reader.connect(); - const res = await reader.query('select id from restart_check'); - expect(res.rows).toEqual([{ id: 1 }]); - await reader.end(); + await client.deleteInstance(db.id); - await clientAfterRestart.deleteApp('pgtest-restart'); - }, 45_000); + expect(await client.getInstance(db.id)).toBeUndefined(); + expect(await getServerStatus(`pcdev-${db.id}`)).toMatchObject({ status: 'no_such_server' }); + }, 60_000); }); -/** - * The daemon-port-retry test helper (`ensureFreshDaemon`) pre-seeds fake - * registry occupants to steer a FRESH registryRoot away from shared-machine - * contention. On a restart of an ALREADY-established registryRoot, the - * daemon's own port is already persisted (frozen — spec § 2 step 5's - * persisted-port case), so this just re-ensures directly without the - * fresh-allocation dance `ensureFreshDaemon` does for a brand-new root. - */ -async function ensureFreshDaemonSamePort(root: string): Promise { - const { ensureDaemon } = await import('../daemon.ts'); - const { entryFor } = await import('./helpers.ts'); - await ensureDaemon('postgres', entryFor('postgres'), { registryRoot: root }); -} +describe('deleting with the caller’s @prisma/dev', () => { + test('deletes the data even when the module the database was created with is gone', async () => { + const client = await startedClient(); + const oldInstall = tempDir('removed-install'); + const db = await client.createInstance({ + labels: {}, + prismaDevModulePath: linkedPrismaDevModulePath(oldInstall), + }); + created.push(db.id); + fs.rmSync(oldInstall, { recursive: true, force: true }); -describe('multi-app isolation', () => { - test("two apps' databases with the same id are distinct, non-colliding instances", async () => { - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); + await client.deleteInstance(db.id, prismaDevModulePath()); - const a = await client.ensureDatabase('pgtest-tenant-a', 'appdb', prismaDevModulePath()); - const b = await client.ensureDatabase('pgtest-tenant-b', 'appdb', prismaDevModulePath()); - expect(a.url).not.toBe(b.url); + expect(await client.getInstance(db.id)).toBeUndefined(); + expect(await getServerStatus(`pcdev-${db.id}`)).toMatchObject({ status: 'no_such_server' }); + }, 60_000); +}); + +describe('Postgres messages larger than one loopback TCP read', () => { + // A regression test for @prisma/dev 0.20.0, whose pglite-socket 0.0.20 + // handed each raw 64 KiB TCP chunk to PGlite as if it were a whole + // message: a bind parameter over that size killed the engine and every + // later connection was refused. Composer now pins a fixed @prisma/dev; + // this test fails against the old one. + test('a 100 KiB bind parameter round-trips and the server keeps accepting connections', async () => { + const client = await startedClient(); + const { url } = await createDatabase(client); - const aWriter = new PgClient({ connectionString: a.url }); - await aWriter.connect(); - await aWriter.query('create table if not exists tenant_marker (v text)'); - await aWriter.query("insert into tenant_marker (v) values ('tenant-a')"); - await aWriter.end(); - - // Tenant B's own database never sees tenant A's table — genuinely - // separate PGlite instances, not a shared one with namespacing. - const bReader = new PgClient({ connectionString: b.url }); - await bReader.connect(); - const res = await bReader.query("select to_regclass('tenant_marker') as reg"); - expect(res.rows[0].reg).toBeNull(); - await bReader.end(); - - await client.deleteApp('pgtest-tenant-a'); - await client.deleteApp('pgtest-tenant-b'); - }, 45_000); + const payload = 'x'.repeat(100 * 1024); + expect(await query(url, 'select length($1::text) as len', [payload])).toEqual([ + { len: payload.length }, + ]); + expect(await query(url, 'select 1 as one')).toEqual([{ one: 1 }]); + }, 60_000); }); -describe('a bogus prismaDevModulePath', () => { - test('surfaces 500 naming the resolution failure without leaking unrelated paths', async () => { - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); +describe('ports', () => { + test('a requested port that is free is given', async () => { + const client = await startedClient(); + const requested = await getPort({ port: portNumbers(58_000, 59_000) }); - const bogusPath = '/definitely/not/a/real/path/to/prisma-dev/index.js'; - await expect(client.ensureDatabase('pgtest-bogus', 'appdb', bogusPath)).rejects.toThrow(); + const db = await createDatabase(client, { port: requested }); - try { - await client.ensureDatabase('pgtest-bogus', 'appdb', bogusPath); - throw new Error('expected ensureDatabase to reject'); - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - // The client wraps the emulator's response as - // `request to the local dev emulator failed (${status}): ${body}` — - // assert the status explicitly, not just that SOME error was thrown. - expect(message).toContain('(500)'); - expect(message).toContain('local dev needs @prisma/dev'); - expect(message).toContain(bogusPath); - // No OTHER filesystem path leaks — e.g. a dynamic `import()` - // failure's own message routinely names a SECOND path (the - // importer's own internal location), which has no business in a - // response body. Every MULTI-SEGMENT absolute-path-shaped substring - // (2+ `/`-separated segments, so "@prisma/dev" itself doesn't count) - // must be the one path the client itself supplied. - const pathsInMessage = [...message.matchAll(/(?:\/[^\s"')/]+){2,}/g)].map((m) => m[0]); - expect([...new Set(pathsInMessage)]).toEqual([bogusPath]); - } - }, 15_000); -}); + expect(db.port).toBe(requested); + }, 60_000); + + test("a deleted database's port can be requested again at once", async () => { + const client = await startedClient(); + const db = await createDatabase(client); + await client.deleteInstance(db.id); + + const again = await createDatabase(client, { port: db.port }); -describe('fresh-allocation port retry (spec § 2 step 5, applied to databasePort)', () => { - test('a bind failure on a fresh allocation retries the next free port', async () => { - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); + expect(again.port).toBe(db.port); + }, 60_000); + test('a bind failure on a fresh allocation moves to the next free port', async () => { + const client = await startedClient(); // Occupy the FIRST candidate database port a fresh allocation would // pick, forcing the retry path. const squatter = await occupyPort(51_300); try { - const ensured = await client.ensureDatabase('pgtest-retry', 'appdb', prismaDevModulePath()); - const port = Number(new URL(ensured.url.replace('postgres://', 'http://')).port); - expect(port).toBeGreaterThan(51_300); - - const pg = new PgClient({ connectionString: ensured.url }); - await pg.connect(); - await pg.query('select 1'); - await pg.end(); + const db = await createDatabase(client); + expect(db.port).toBeGreaterThan(51_300); + expect(await query(db.url, 'select 1 as one')).toEqual([{ one: 1 }]); } finally { await closeServer(squatter); } - - await client.deleteApp('pgtest-retry'); - }, 30_000); + }, 60_000); }); -describe('instance name derivation is linear, not polynomial', () => { - test('a pathological run of separators slugs correctly and stays fast', () => { - const pathological = `a${'-'.repeat(10_000)}b`; - - const start = performance.now(); - const name = instanceNameFor(pathological, pathological); - const elapsedMs = performance.now() - start; +describe('a bogus prismaDevModulePath', () => { + test('surfaces 500 naming the resolution failure without leaking unrelated paths, and creates nothing', async () => { + const client = await startedClient(); + const bogusPath = '/definitely/not/a/real/path/to/prisma-dev/index.js'; - // A run of separators collapses to one `-`, same as any other run — - // no special case for length. `pcdev-a-b-a-b`.slice(0, 63) is still - // well within the cap, so nothing else about the shape changes. - expect(name).toBe('pcdev-a-b-a-b'); - expect(elapsedMs).toBeLessThan(100); - }); + const message = await client + .createInstance({ labels: {}, prismaDevModulePath: bogusPath }) + .then( + () => 'created', + (err: unknown) => (err instanceof Error ? err.message : String(err)), + ); + + expect(message).toContain('(500)'); + expect(message).toContain('local dev needs @prisma/dev'); + expect(message).toContain(bogusPath); + // No OTHER filesystem path leaks — a dynamic `import()` failure's own + // message routinely names a SECOND path (the importer's own location). + const pathsInMessage = [...message.matchAll(/(?:\/[^\s"')/]+){2,}/g)].map((m) => m[0]); + expect([...new Set(pathsInMessage)]).toEqual([bogusPath]); + expect(await client.listInstances()).toEqual([]); + }, 15_000); }); -describe("a server that exists in @prisma/dev's records but not the daemon's", () => { - test('a live server under the daemon-derived name is adopted, not restarted or deleted', async () => { - // The daemon's own state and `@prisma/dev`'s records can disagree: a - // daemon replaced for version skew, or a teardown that declined to - // delete, leaves a server registered under a name the daemon has no - // record of. Starting into that used to surface "already running" as a - // 500 — and the recovery that deleted the state took the data (and the - // daemon) with it. - // - // The foreign server runs in its OWN process, exactly as a leftover - // from a previous daemon does. That is not incidental: the record names - // its host's pid, and teardown stops an adopted server by that pid — so - // a server started inside THIS test process would make the teardown - // kill the test runner. - const app = 'pgtest-adopt'; - const id = 'db'; - const instanceName = instanceNameFor(app, id); +describe("a server another process holds under an instance's name", () => { + test('a start adopts it instead of starting a second one, and the emulator stays up', async () => { + // A daemon replaced for version skew can leave a server live in another + // process under an instance's name. Starting into that used to surface + // "already running" as a 500. The foreign server runs in its OWN + // process: its record names its host's pid, and stopping an adopted + // server signals that pid. + const client = await startedClient(); + const db = await createDatabase(client); + await client.stopInstances([db.id]); + const serverName = `pcdev-${db.id}`; const scriptPath = path.join(tempDir('adopt-host'), 'host.mjs'); fs.writeFileSync( scriptPath, `import { startPrismaDevServer } from ${JSON.stringify(pathToFileURL(prismaDevModulePath()).href)}; -const server = await startPrismaDevServer({ name: ${JSON.stringify(instanceName)}, persistenceMode: 'stateful' }); +const server = await startPrismaDevServer({ name: ${JSON.stringify(serverName)}, databasePort: ${String(db.port)}, persistenceMode: 'stateful' }); console.log(server.database.connectionString); setInterval(() => {}, 1 << 30); `, ); - const host = spawn(process.execPath, [scriptPath], { stdio: ['ignore', 'pipe', 'inherit'] }); const foreignUrl = await new Promise((resolve, reject) => { let out = ''; @@ -323,26 +348,13 @@ setInterval(() => {}, 1 << 30); }); try { - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); - - const ensured = await client.ensureDatabase(app, id, prismaDevModulePath()); + await client.startInstances([db.id], prismaDevModulePath()); - // Adopted: the same live server, reachable — not a second one on a - // different port, and not an error. - expect(ensured.url).toBe(foreignUrl); - const pg = new PgClient({ connectionString: ensured.url }); - await pg.connect(); - await pg.query('select 1'); - await pg.end(); - - // And the daemon stayed up — the failure this guards against killed it. + expect(await client.getInstance(db.id)).toMatchObject({ status: 'running', url: foreignUrl }); + expect(await query(foreignUrl, 'select 1 as one')).toEqual([{ one: 1 }]); expect((await client.health()).version.length).toBeGreaterThan(0); - - await client.deleteApp(app); } finally { host.kill('SIGKILL'); - await deleteServer(instanceName).catch(() => undefined); } }, 60_000); }); @@ -386,37 +398,24 @@ describe("a stale @prisma/dev record claiming a port in the daemon's database ra await squatter.close(); try { - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); - - const ensured = await client.ensureDatabase('pgtest-stale', 'appdb', prismaDevModulePath()); + const client = await startedClient(); - const port = Number(new URL(ensured.url.replace('postgres://', 'http://')).port); - expect(port).not.toBe(squatterPort); - const pg = new PgClient({ connectionString: ensured.url }); - await pg.connect(); - await pg.query('select 1'); - await pg.end(); + const db = await createDatabase(client); - await client.deleteApp('pgtest-stale'); + expect(db.port).not.toBe(squatterPort); + expect(await query(db.url, 'select 1 as one')).toEqual([{ one: 1 }]); } finally { await deleteServer(squatterName).catch(() => undefined); } }, 120_000); }); -describe('a start refusal that leaked the name lock', () => { - test('the next ensure recovers the name instead of failing "already running" forever', async () => { - // The lock leak (see the previous test) cannot always be prevented — - // a record can appear between the daemon's exclusion scan and the - // start. This test forces one deliberately: a stand-in @prisma/dev - // module whose start requests a port another record claims, which - // makes the REAL start acquire the name's lock and then refuse — the - // exact leak. The next ensure with the real module must then recover - // (the daemon leaked the lock itself and the name has no record, so - // deleting the orphaned state is safe) instead of retrying into - // "already running" until the daemon dies. - const app = 'pgtest-leak'; +describe('a create whose start refused a port after taking the name lock', () => { + test('fails, records nothing, and the emulator keeps creating databases', async () => { + // A start can refuse a port after `@prisma/dev` took the name's lock, + // which leaks the lock. This forces one: a stand-in @prisma/dev module + // whose start requests a port another record claims. A failed create + // records no instance, so its leaked name is never used again. const realIndexUrl = pathToFileURL(prismaDevModulePath()).href; const realStateUrl = pathToFileURL( fileURLToPath(import.meta.resolve('@prisma/dev/internal/state')), @@ -452,54 +451,188 @@ export async function startPrismaDevServer(options) { `, ); - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); + const client = await startedClient(); // A real database first, so a record exists for the fixture to clash with. - await client.ensureDatabase(app, 'anchor', prismaDevModulePath()); + const anchor = await createDatabase(client); - // The leak: the real start (behind the fixture) locks the victim's name, - // then refuses the claimed port. The PUT fails visibly. await expect( - client.ensureDatabase(app, 'victim', path.join(fixtureDir, 'index.mjs')), + client.createInstance({ + labels: {}, + prismaDevModulePath: path.join(fixtureDir, 'index.mjs'), + }), ).rejects.toThrow(/belongs to another Prisma Dev server/); - // Recovery: the same name, the real module — must come up, not report - // "already running" against the daemon's own dead attempt. - const recovered = await client.ensureDatabase(app, 'victim', prismaDevModulePath()); - const pg = new PgClient({ connectionString: recovered.url }); - await pg.connect(); - await pg.query('select 1'); - await pg.end(); - - await client.deleteApp(app); + expect((await client.listInstances()).map((i) => i.id)).toEqual([anchor.id]); + const next = await createDatabase(client); + expect(await query(next.url, 'select 1 as one')).toEqual([{ one: 1 }]); }, 120_000); }); -describe('concurrent ensures for different databases', () => { - test('two databases requested at once both come up — starts are serialized', async () => { +describe('concurrent creates', () => { + test('two databases created at once both come up — starts are serialized', async () => { // An app converges its databases in parallel, so the daemon receives // simultaneous ensures for different names. `@prisma/dev`'s start is not // concurrency-safe within a process — two at once pick ports without // seeing each other and one fails — so the daemon runs starts one at a // time. Before that, this pair raced and produced the port refusals whose // retries left half-started servers holding their own name's lock. - await ensureFreshDaemon('postgres', registryRoot); - const client = postgresClient({ registryRoot }); + const client = await startedClient(); - const [a, b] = await Promise.all([ - client.ensureDatabase('pgtest-concurrent', 'alpha', prismaDevModulePath()), - client.ensureDatabase('pgtest-concurrent', 'beta', prismaDevModulePath()), - ]); + const [a, b] = await Promise.all([createDatabase(client), createDatabase(client)]); expect(a.url).not.toBe(b.url); - for (const url of [a.url, b.url]) { - const pg = new PgClient({ connectionString: url }); - await pg.connect(); - await pg.query('select 1'); - await pg.end(); + for (const { url } of [a, b]) expect(await query(url, 'select 1 as one')).toEqual([{ one: 1 }]); + }, 90_000); +}); + +describe('state written by an earlier version', () => { + function writeEarlierState(serverName: string): string { + const stateDir = daemonStateDir(registryRoot, 'postgres'); + fs.mkdirSync(stateDir, { recursive: true }); + fs.writeFileSync( + path.join(stateDir, 'apps.json'), + JSON.stringify({ + 'legacy-app': { + databases: { + db: { id: 'db', instanceName: serverName, databasePort: 51_300, url: 'postgres://x' }, + }, + }, + }), + ); + return stateDir; + } + + test("is removed on load, and its servers' data is deleted on the next create", async () => { + const legacyName = `pcdev-pgtest-legacy-${String(process.pid)}`; + const prismaDev = (await import(pathToFileURL(prismaDevModulePath()).href)) as { + startPrismaDevServer(options: { + name: string; + persistenceMode: 'stateful'; + }): Promise<{ close(): Promise }>; + }; + const legacy = await prismaDev.startPrismaDevServer({ + name: legacyName, + persistenceMode: 'stateful', + }); + await legacy.close(); + const stateDir = writeEarlierState(legacyName); + + try { + const client = await startedClient(); + expect(fs.existsSync(path.join(stateDir, 'apps.json'))).toBe(false); + expect(await client.listInstances()).toEqual([]); + + await createDatabase(client); + + expect(await getServerStatus(legacyName)).toMatchObject({ status: 'no_such_server' }); + } finally { + await deleteServer(legacyName).catch(() => undefined); } + }, 60_000); - await client.deleteApp('pgtest-concurrent'); + test('a server another live process still serves is left alone until that process ends', async () => { + // Under the old names, another emulators directory's daemon could be + // serving a server this state names. Its data must survive, and its + // process must not be signalled: that would end every server it hosts. + const legacyName = `pcdev-pgtest-legacy-live-${String(process.pid)}`; + const scriptPath = path.join(tempDir('legacy-host'), 'host.mjs'); + fs.writeFileSync( + scriptPath, + `import { startPrismaDevServer } from ${JSON.stringify(pathToFileURL(prismaDevModulePath()).href)}; +const server = await startPrismaDevServer({ name: ${JSON.stringify(legacyName)}, persistenceMode: 'stateful' }); +console.log(server.database.connectionString); +setInterval(() => {}, 1 << 30); +`, + ); + const host = spawn(process.execPath, [scriptPath], { stdio: ['ignore', 'pipe', 'inherit'] }); + const hostUrl = await new Promise((resolve, reject) => { + let out = ''; + host.stdout.on('data', (chunk: Buffer) => { + out += chunk.toString(); + const line = out.split('\n')[0]; + if (line !== undefined && out.includes('\n')) resolve(line.trim()); + }); + host.once('error', reject); + host.once('exit', (code) => reject(new Error(`host exited early (${String(code)})`))); + }); + await query(hostUrl, "create table kept (v text); insert into kept values ('live')"); + const stateDir = writeEarlierState(legacyName); + + try { + const client = await startedClient(); + await createDatabase(client); + + expect(await getServerStatus(legacyName)).toMatchObject({ status: 'running' }); + expect(await query(hostUrl, 'select v from kept')).toEqual([{ v: 'live' }]); + const state: unknown = JSON.parse( + fs.readFileSync(path.join(stateDir, 'instances.json'), 'utf8'), + ); + expect(state).toMatchObject({ discardedServers: [legacyName] }); + + const exited = new Promise((resolve) => host.once('exit', resolve)); + host.kill('SIGKILL'); + await exited; + await createDatabase(client); + + expect(await getServerStatus(legacyName)).toMatchObject({ status: 'no_such_server' }); + } finally { + host.kill('SIGKILL'); + await deleteServer(legacyName).catch(() => undefined); + } }, 90_000); }); + +/** + * The emulator never interprets Composer's data: an app name, a node address + * or a directory reaches it only as label values, and the `@prisma/dev` + * module path (under the project's `node_modules`) is stored verbatim. + */ +describe('the emulator never interprets Composer data', () => { + test('labels and the module path are stored verbatim, and nothing else carries their values', async () => { + // The module path lies under the project in real use, so it carries the + // directory; it reaches the state only through its own field. + const client = await startedClient(); + const directory = path.join(tempDir('marker-project'), 'zzdirectorymarker'); + const modulePath = linkedPrismaDevModulePath(directory); + const labels = { + 'composer.directory': directory, + 'composer.app': 'zzappmarker', + 'composer.address': 'zzaddressmarker', + }; + const db = await client.createInstance({ labels, prismaDevModulePath: modulePath }); + created.push(db.id); + + const stateDir = daemonStateDir(registryRoot, 'postgres'); + const statePath = path.join(stateDir, 'instances.json'); + await waitFor( + () => fs.existsSync(statePath) && fs.readFileSync(statePath, 'utf8').includes(db.id), + 5000, + ); + const state: unknown = JSON.parse(fs.readFileSync(statePath, 'utf8')); + const record: unknown = Reflect.get(Reflect.get(state as object, 'instances'), db.id); + expect(record).toMatchObject({ labels, prismaDevModulePath: modulePath }); + + const opaque = (key: string, value: unknown) => + key === 'labels' || key === 'prismaDevModulePath' ? undefined : value; + const withoutLabels = JSON.stringify(state, opaque); + const listed = JSON.stringify(await client.listInstances(), opaque); + for (const marker of ['zzdirectorymarker', 'zzappmarker', 'zzaddressmarker']) { + expect(withoutLabels).not.toContain(marker); + expect(listed).not.toContain(marker); + for (const file of fs.readdirSync(stateDir, { recursive: true })) { + expect(String(file)).not.toContain(marker); + } + } + }, 60_000); + + test("the emulator's code names no app, address, directory or Composer concept", () => { + const source = fs.readFileSync( + fileURLToPath(new URL('../postgres-main.ts', import.meta.url)), + 'utf8', + ); + const code = source.replace(/\/\*[\s\S]*?\*\//g, '').replace(/\/\/.*$/gm, ''); + + expect(code).not.toMatch(/\b(apps?|appName|address(es)?|directory|composer)\b/i); + }); +}); diff --git a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/buckets-main.ts b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/buckets-main.ts index bcfcfed76..9273e536a 100644 --- a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/buckets-main.ts +++ b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/buckets-main.ts @@ -1,53 +1,69 @@ /** * The bucket emulator daemon (local-dev spec § 2 `buckets-main.ts`): the S3 * wire protocol (`@internal/s3-protocol`'s handler + `fsStore`) over - * registered per-app, per-bucket directories, multi-tenant via physical - * names `--`. Admin lives under `/_pcdev/` (the underscore can't - * collide with a valid bucket name). Plain `node:http`. + * registered bucket instances. Each instance is keyed by an id this daemon + * generates, which is also its bucket name on the S3 wire; its objects live + * in the data root its creator registers, a path stored verbatim and never + * interpreted, as are its labels. Credentials are registered per instance. + * Admin lives under `/_pcdev/` (the underscore can't collide with a valid + * bucket name). Plain `node:http`. * * Runs as its own OS process, started by `daemon.ts`'s `ensureDaemon` via * `process.execPath --port --state-dir `. */ +import { randomBytes } from 'node:crypto'; import * as fs from 'node:fs'; import * as http from 'node:http'; import * as path from 'node:path'; import { type Credentials, createS3Handler, fsStore } from '@internal/s3-protocol'; import { readOwnVersion } from './daemon.ts'; -import { isValidSegment } from './segments.ts'; import { readJsonFile, StateFile } from './state-file.ts'; const STATE_MODE = 0o600; -// Mirrors `@internal/s3-protocol`'s own fs-store.ts bucket-name rule (not -// exported — it's the physical-bucket-name check this endpoint applies to -// the composed `--`, not a wire-level concern of the handler). -const PHYSICAL_BUCKET_NAME_RE = /^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/; const MULTIPART_MESSAGE = 'multipart upload is not supported by the local dev bucket emulator yet'; -interface BucketRegistration { - readonly app: string; - readonly name: string; +/** Also the bucket's name on the S3 wire, so it uses only characters a bucket name allows. */ +const INSTANCE_ID_RE = /^bkt-[0-9a-f]{16}$/; + +function newInstanceId(): string { + return `bkt-${randomBytes(8).toString('hex')}`; +} + +interface BucketRecord { + readonly id: string; + readonly labels: Record; + /** Where the bucket's objects live, as its creator registered it. */ readonly dir: string; } interface CredentialRecord { - readonly app: string; + readonly instanceId: string; readonly secretAccessKey: string; } interface BucketsState { - buckets: Record; + instances: Record; credentials: Record; } -function isBucketRegistration(value: unknown): value is BucketRegistration { +function isStringRecord(value: unknown): value is Record { + return ( + typeof value === 'object' && + value !== null && + !Array.isArray(value) && + Object.values(value).every((v) => typeof v === 'string') + ); +} + +function isBucketRecord(value: unknown): value is BucketRecord { return ( typeof value === 'object' && value !== null && - 'app' in value && - typeof value.app === 'string' && - 'name' in value && - typeof value.name === 'string' && + 'id' in value && + typeof value.id === 'string' && + 'labels' in value && + isStringRecord(value.labels) && 'dir' in value && typeof value.dir === 'string' ); @@ -57,8 +73,8 @@ function isCredentialRecord(value: unknown): value is CredentialRecord { return ( typeof value === 'object' && value !== null && - 'app' in value && - typeof value.app === 'string' && + 'instanceId' in value && + typeof value.instanceId === 'string' && 'secretAccessKey' in value && typeof value.secretAccessKey === 'string' ); @@ -68,10 +84,10 @@ function isBucketsState(value: unknown): value is BucketsState { return ( typeof value === 'object' && value !== null && - 'buckets' in value && - typeof value.buckets === 'object' && - value.buckets !== null && - Object.values(value.buckets).every(isBucketRegistration) && + 'instances' in value && + typeof value.instances === 'object' && + value.instances !== null && + Object.values(value.instances).every(isBucketRecord) && 'credentials' in value && typeof value.credentials === 'object' && value.credentials !== null && @@ -79,13 +95,19 @@ function isBucketsState(value: unknown): value is BucketsState { ); } -interface BucketBody { +interface CreateBody { + readonly labels: Record; readonly dir: string; } -function isBucketBody(value: unknown): value is BucketBody { +function isCreateBody(value: unknown): value is CreateBody { return ( - typeof value === 'object' && value !== null && 'dir' in value && typeof value.dir === 'string' + typeof value === 'object' && + value !== null && + 'labels' in value && + isStringRecord(value.labels) && + 'dir' in value && + typeof value.dir === 'string' ); } @@ -156,12 +178,12 @@ async function sendWebResponse(res: http.ServerResponse, webRes: Response): Prom function main(): void { const { port, stateDir } = parseArgs(process.argv.slice(2)); const ownVersion = readOwnVersion(); - const stateJsonPath = path.join(stateDir, 'state.json'); - const stateFile = new StateFile(stateJsonPath, STATE_MODE); + const statePath = path.join(stateDir, 'instances.json'); + const stateFile = new StateFile(statePath, STATE_MODE); - let state: BucketsState = { buckets: {}, credentials: {} }; + let state: BucketsState = { instances: {}, credentials: {} }; - const store = fsStore((physicalName) => state.buckets[physicalName]?.dir); + const store = fsStore((bucket) => state.instances[bucket]?.dir); function json(res: http.ServerResponse, status: number, body: unknown): void { res.writeHead(status, { 'content-type': 'application/json' }); @@ -173,88 +195,78 @@ function main(): void { res.end(body); } - function badSegment(res: http.ServerResponse, segment: string): void { - text( - res, - 400, - `invalid path segment "${segment}": must match /^[a-z0-9][a-z0-9-]*$/ and be at most 63 characters`, - ); + function noContent(res: http.ServerResponse): void { + res.writeHead(204); + res.end(); } - async function handlePutBucket( - req: http.IncomingMessage, - res: http.ServerResponse, - app: string, - name: string, - ): Promise { - const physicalName = `${app}--${name}`; - if (!PHYSICAL_BUCKET_NAME_RE.test(physicalName)) { - return text( - res, - 400, - `invalid bucket "${physicalName}" (app "${app}", name "${name}"): the physical bucket name must match /^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/ and be at most 63 characters`, - ); - } + async function readJsonBody(req: http.IncomingMessage): Promise { const raw = await readBody(req); - let parsed: unknown; try { - parsed = JSON.parse(raw.toString('utf8')); + return JSON.parse(raw.toString('utf8')); } catch { - return text(res, 400, 'malformed JSON body'); + return undefined; } - if (!isBucketBody(parsed)) - return text(res, 400, 'malformed bucket body: expected { "dir": string }'); + } - await fs.promises.mkdir(parsed.dir, { recursive: true }); - state.buckets[physicalName] = { app, name, dir: parsed.dir }; + function instanceView(bucket: BucketRecord): unknown { + return { id: bucket.id, labels: bucket.labels, dir: bucket.dir }; + } + + async function handleCreate(req: http.IncomingMessage, res: http.ServerResponse): Promise { + const body = await readJsonBody(req); + if (!isCreateBody(body)) { + return text( + res, + 400, + 'malformed body: expected { "labels": Record, "dir": string }', + ); + } + await fs.promises.mkdir(body.dir, { recursive: true }); + const id = newInstanceId(); + const bucket: BucketRecord = { id, labels: { ...body.labels }, dir: body.dir }; + state.instances[id] = bucket; await stateFile.write(state); - res.writeHead(204); - res.end(); + return json(res, 201, instanceView(bucket)); } async function handlePutCredentials( req: http.IncomingMessage, res: http.ServerResponse, - app: string, + bucket: BucketRecord, ): Promise { - const raw = await readBody(req); - let parsed: unknown; - try { - parsed = JSON.parse(raw.toString('utf8')); - } catch { - return text(res, 400, 'malformed JSON body'); - } - if (!isCredentialsBody(parsed)) { + const body = await readJsonBody(req); + if (!isCredentialsBody(body)) { return text( res, 400, 'malformed credentials body: expected { "accessKeyId", "secretAccessKey" }', ); } - const existing = state.credentials[parsed.accessKeyId]; - if (existing && existing.app !== app) { + const existing = state.credentials[body.accessKeyId]; + if (existing && existing.instanceId !== bucket.id) { return text( res, 409, - `accessKeyId "${parsed.accessKeyId}" is already registered by a different app`, + `accessKeyId "${body.accessKeyId}" is already registered for another bucket`, ); } - state.credentials[parsed.accessKeyId] = { app, secretAccessKey: parsed.secretAccessKey }; + state.credentials[body.accessKeyId] = { + instanceId: bucket.id, + secretAccessKey: body.secretAccessKey, + }; await stateFile.write(state); - res.writeHead(204); - res.end(); + noContent(res); } - async function handleDeleteApp(res: http.ServerResponse, app: string): Promise { - for (const [key, reg] of Object.entries(state.buckets)) { - if (reg.app === app) delete state.buckets[key]; - } + /** Removes the bucket's registration and credentials; its objects stay where its creator put them. */ + async function handleDelete(res: http.ServerResponse, id: string): Promise { + delete state.instances[id]; for (const [key, cred] of Object.entries(state.credentials)) { - if (cred.app === app) delete state.credentials[key]; + if (cred.instanceId === id) delete state.credentials[key]; } await stateFile.write(state); - res.writeHead(204); - res.end(); + noContent(res); } /** Path-style addressing: the bucket is the first non-empty path segment. Mirrors the handler's own `parseTarget`, which isn't exported. */ @@ -264,20 +276,15 @@ function main(): void { } /** - * Credentials accepted for THIS request's target bucket only — its - * owning app, derived from the bucket's registration (the authoritative - * source for the `--` split; unregistered apps/names may - * themselves contain `--`, so the registration is trusted over re-parsing - * the string). An unknown bucket or an app with no registered credentials - * yields no candidates, which still resolves through the real 403 path - * below — never a bespoke "unknown bucket" shortcut that would reveal - * anything. + * Credentials accepted for THIS request's target bucket only. An unknown + * bucket or one with no registered credentials yields no candidates, which + * still resolves through the real 403 path below — never a bespoke + * "unknown bucket" shortcut that would reveal anything. */ function credentialsForBucket(bucket: string | undefined): Credentials[] { - const owningApp = bucket !== undefined ? state.buckets[bucket]?.app : undefined; - if (owningApp === undefined) return []; + if (bucket === undefined || state.instances[bucket] === undefined) return []; return Object.entries(state.credentials) - .filter(([, rec]) => rec.app === owningApp) + .filter(([, rec]) => rec.instanceId === bucket) .map(([accessKeyId, rec]) => ({ accessKeyId, secretAccessKey: rec.secretAccessKey })); } @@ -302,12 +309,10 @@ function main(): void { const headers = toWebHeaders(req.headers); const candidates: Credentials[] = credentialsForBucket(bucketFromPath(url.pathname)); - // No candidate for this bucket (unknown bucket, or its app has no - // credentials registered): fall through to a single verification + // No candidate for this bucket: fall through to a single verification // attempt against an unusable pair, so the request still resolves - // through the real 403 path instead of a bespoke shortcut. A valid - // signature from a DIFFERENT app's credential never even reaches this - // list, so it fails exactly the same way — same status, same body. + // through the real 403 path. A valid signature from ANOTHER bucket's + // credential never reaches this list, so it fails exactly the same way. if (candidates.length === 0) candidates.push({ accessKeyId: '', secretAccessKey: '' }); let last: Response | undefined; @@ -325,6 +330,42 @@ function main(): void { if (last) await sendWebResponse(res, last); } + async function handleAdmin( + req: http.IncomingMessage, + res: http.ServerResponse, + segments: readonly string[], + ): Promise { + const method = req.method ?? 'GET'; + // segments[0] === '_pcdev' + if (method === 'GET' && segments.length === 2 && segments[1] === 'health') { + return json(res, 200, { version: ownVersion }); + } + if (segments[1] !== 'instances') { + res.writeHead(404); + res.end(); + return; + } + if (segments.length === 2) { + if (method === 'GET') return json(res, 200, Object.values(state.instances).map(instanceView)); + if (method === 'POST') return handleCreate(req, res); + } + const id = segments[2]; + if (id !== undefined && !INSTANCE_ID_RE.test(id)) { + return text(res, 400, `invalid instance id "${id}"`); + } + if (method === 'DELETE' && segments.length === 3 && id !== undefined) { + return handleDelete(res, id); + } + const bucket = id === undefined ? undefined : state.instances[id]; + if (!bucket) return text(res, 404, `no such instance: "${id ?? ''}"`); + if (method === 'GET' && segments.length === 3) return json(res, 200, instanceView(bucket)); + if (method === 'PUT' && segments.length === 4 && segments[3] === 'credentials') { + return handlePutCredentials(req, res, bucket); + } + res.writeHead(404); + res.end(); + } + async function handleRequest(req: http.IncomingMessage, res: http.ServerResponse): Promise { const url = new URL(req.url ?? '/', `http://127.0.0.1:${String(port)}`); const method = req.method ?? 'GET'; @@ -333,39 +374,12 @@ function main(): void { return json(res, 200, { version: ownVersion }); } - if (url.pathname === '/_pcdev/health' || url.pathname.startsWith('/_pcdev/')) { + if (url.pathname.startsWith('/_pcdev/')) { const segments = url.pathname .split('/') .filter((s) => s.length > 0) .map((s) => decodeURIComponent(s)); - // segments[0] === '_pcdev' - - if (method === 'GET' && segments.length === 2 && segments[1] === 'health') { - return json(res, 200, { version: ownVersion }); - } - - if (segments[1] === 'apps' && segments.length >= 3) { - const app = segments[2]; - if (app === undefined || !isValidSegment(app)) return badSegment(res, app ?? ''); - - if (method === 'PUT' && segments.length === 5 && segments[3] === 'buckets') { - const name = segments[4]; - if (name === undefined || !isValidSegment(name)) return badSegment(res, name ?? ''); - return handlePutBucket(req, res, app, name); - } - - if (method === 'PUT' && segments.length === 4 && segments[3] === 'credentials') { - return handlePutCredentials(req, res, app); - } - - if (method === 'DELETE' && segments.length === 3) { - return handleDeleteApp(res, app); - } - } - - res.writeHead(404); - res.end(); - return; + return handleAdmin(req, res, segments); } return handleS3Wire(req, res, url); @@ -390,7 +404,8 @@ function main(): void { process.on('SIGTERM', () => void shutdown()); process.on('SIGINT', () => void shutdown()); - readJsonFile(stateJsonPath, isBucketsState) + discardStateFromEarlierVersions(stateDir, statePath) + .then(() => readJsonFile(statePath, isBucketsState)) .then((loaded) => { if (loaded) state = loaded; // A single, unambiguous line into the daemon's own stdio log once @@ -408,4 +423,19 @@ function main(): void { }); } +/** + * Removes state files this version does not write. Local dev state is never + * migrated across versions; buckets registered by an earlier version are + * gone, and their objects stay where they were. + */ +async function discardStateFromEarlierVersions(stateDir: string, statePath: string): Promise { + const files = await fs.promises.readdir(stateDir).catch(() => []); + await Promise.all( + files + .map((name) => path.join(stateDir, name)) + .filter((file) => file.endsWith('.json') && file !== statePath) + .map((file) => fs.promises.rm(file, { force: true })), + ); +} + main(); diff --git a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/client.ts b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/client.ts index 74b57470d..4c5257399 100644 --- a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/client.ts +++ b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/client.ts @@ -13,7 +13,6 @@ import { isPidAlive, registryFilePath, } from './daemon.ts'; -import { isValidSegment } from './segments.ts'; function notRunningError(name: DaemonName): Error { return new Error( @@ -59,13 +58,6 @@ function resolveBaseUrl(name: DaemonName, opts: DaemonRootOptions): string { return `http://127.0.0.1:${parsed.port}`; } -function encodeSegment(segment: string): string { - if (!isValidSegment(segment)) { - throw new Error(`invalid path segment "${segment}"`); - } - return encodeURIComponent(segment); -} - const ADMIN_FETCH_ATTEMPTS = 3; const ADMIN_FETCH_RETRY_DELAY_MS = 150; @@ -118,22 +110,22 @@ async function expectOk(res: Response): Promise { return res; } -export type InstanceStatus = 'running' | 'backoff' | 'held' | 'stopped'; +export type ServiceInstanceStatus = 'running' | 'backoff' | 'held' | 'stopped'; -export interface InstanceInfo { +export interface ServiceInstanceInfo { readonly id: string; /** Stored and returned verbatim; the emulator never reads them. */ readonly labels: Readonly>; readonly port: number; readonly url: string; - readonly status: InstanceStatus; + readonly status: ServiceInstanceStatus; readonly pid?: number; readonly lastExitCode?: number; readonly artifactHash?: string; readonly logPath: string; } -export interface CreateInstanceRequest { +export interface CreateServiceRequest { readonly labels: Readonly>; /** Used if no instance holds it and it is bindable; otherwise the emulator gives the lowest free port from 3000. */ readonly port?: number; @@ -157,7 +149,7 @@ function isStringRecord(value: unknown): value is Record { ); } -function isInstanceInfo(value: unknown): value is InstanceInfo { +function isServiceInstanceInfo(value: unknown): value is ServiceInstanceInfo { return ( typeof value === 'object' && value !== null && @@ -177,8 +169,8 @@ function isInstanceInfo(value: unknown): value is InstanceInfo { ); } -function isInstanceInfoArray(value: unknown): value is InstanceInfo[] { - return Array.isArray(value) && value.every(isInstanceInfo); +function isServiceInstanceInfoArray(value: unknown): value is ServiceInstanceInfo[] { + return Array.isArray(value) && value.every(isServiceInstanceInfo); } interface HealthBody { @@ -199,11 +191,11 @@ export interface ComputeClient { readonly baseUrl: string; health(): Promise<{ version: string }>; /** `POST /instances` — creates a stopped instance holding a port until it is deleted. Never retried: a retry could create a second instance. */ - createInstance(request: CreateInstanceRequest): Promise; + createInstance(request: CreateServiceRequest): Promise; /** `GET /instances`. */ - listInstances(): Promise; + listInstances(): Promise; /** `GET /instances/`; undefined when the emulator has no such instance. */ - getInstance(id: string): Promise; + getInstance(id: string): Promise; /** `PUT /instances//deployment` — returns once the instance accepts connections on its port. */ putDeployment(id: string, deployment: DeploymentRequest): Promise; /** `GET /instances//logs?follow=1` — yields decoded text chunks; ends when `signal` aborts or the daemon closes the stream. */ @@ -252,7 +244,7 @@ export function computeClient(opts: DaemonRootOptions = {}): ComputeClient { await adminFetch(`${baseUrl}/instances`, postJson(request), { retry: false }), ); const body: unknown = await res.json(); - if (!isInstanceInfo(body)) { + if (!isServiceInstanceInfo(body)) { throw new Error('malformed instance-create response from the compute emulator'); } return body; @@ -261,7 +253,7 @@ export function computeClient(opts: DaemonRootOptions = {}): ComputeClient { async listInstances() { const res = await expectOk(await adminFetch(`${baseUrl}/instances`)); const body: unknown = await res.json(); - if (!isInstanceInfoArray(body)) { + if (!isServiceInstanceInfoArray(body)) { throw new Error('malformed instances listing from the compute emulator'); } return body; @@ -271,7 +263,7 @@ export function computeClient(opts: DaemonRootOptions = {}): ComputeClient { const res = await adminFetch(instancePath(baseUrl, id)); if (res.status === 404) return undefined; const body: unknown = await (await expectOk(res)).json(); - if (!isInstanceInfo(body)) { + if (!isServiceInstanceInfo(body)) { throw new Error('malformed instance response from the compute emulator'); } return body; @@ -321,19 +313,56 @@ export function computeClient(opts: DaemonRootOptions = {}): ComputeClient { }; } +export interface BucketInstanceInfo { + /** Also the bucket's name on the S3 wire. */ + readonly id: string; + /** Stored and returned verbatim; the emulator never reads them. */ + readonly labels: Readonly>; + /** Where the bucket's objects live, as its creator registered it. */ + readonly dir: string; +} + +export interface CreateBucketRequest { + readonly labels: Readonly>; + readonly dir: string; +} + +function isBucketInstanceInfo(value: unknown): value is BucketInstanceInfo { + return ( + typeof value === 'object' && + value !== null && + 'id' in value && + typeof value.id === 'string' && + 'labels' in value && + isStringRecord(value.labels) && + 'dir' in value && + typeof value.dir === 'string' + ); +} + +function isBucketInstanceInfoArray(value: unknown): value is BucketInstanceInfo[] { + return Array.isArray(value) && value.every(isBucketInstanceInfo); +} + +/** Each bucket is an instance keyed by an id the emulator generates, which is also its name on the S3 wire at `baseUrl`. */ export interface BucketsClient { readonly baseUrl: string; health(): Promise<{ version: string }>; - /** `PUT /_pcdev/apps//buckets/`. */ - putBucket(app: string, name: string, dir: string): Promise; - /** `PUT /_pcdev/apps//credentials`. */ - putCredentials(app: string, accessKeyId: string, secretAccessKey: string): Promise; - /** `DELETE /_pcdev/apps/`. */ - deleteApp(app: string): Promise; + /** `POST /_pcdev/instances` — creates the bucket and its data root. Never retried: a retry could create a second bucket. */ + createInstance(request: CreateBucketRequest): Promise; + /** `GET /_pcdev/instances`. */ + listInstances(): Promise; + /** `GET /_pcdev/instances/`; undefined when the emulator has no such instance. */ + getInstance(id: string): Promise; + /** `PUT /_pcdev/instances//credentials` — accepts this key pair for this bucket. A key registered for another bucket is refused. */ + putCredentials(id: string, accessKeyId: string, secretAccessKey: string): Promise; + /** `DELETE /_pcdev/instances/` — forgets the bucket and its credentials; its objects stay in its data root. Deleting an unknown id succeeds. */ + deleteInstance(id: string): Promise; } export function bucketsClient(opts: DaemonRootOptions = {}): BucketsClient { const baseUrl = resolveBaseUrl('buckets', opts); + const adminUrl = `${baseUrl}/_pcdev/instances`; return { baseUrl, @@ -347,21 +376,37 @@ export function bucketsClient(opts: DaemonRootOptions = {}): BucketsClient { return body; }, - async putBucket(app, name, dir) { - const url = `${baseUrl}/_pcdev/apps/${encodeSegment(app)}/buckets/${encodeSegment(name)}`; - await expectOk( - await adminFetch(url, { - method: 'PUT', - headers: { 'content-type': 'application/json' }, - body: JSON.stringify({ dir }), - }), - ); + async createInstance(request) { + const res = await expectOk(await adminFetch(adminUrl, postJson(request), { retry: false })); + const body: unknown = await res.json(); + if (!isBucketInstanceInfo(body)) { + throw new Error('malformed instance-create response from the buckets emulator'); + } + return body; + }, + + async listInstances() { + const res = await expectOk(await adminFetch(adminUrl)); + const body: unknown = await res.json(); + if (!isBucketInstanceInfoArray(body)) { + throw new Error('malformed instances listing from the buckets emulator'); + } + return body; + }, + + async getInstance(id) { + const res = await adminFetch(instancePath(`${baseUrl}/_pcdev`, id)); + if (res.status === 404) return undefined; + const body: unknown = await (await expectOk(res)).json(); + if (!isBucketInstanceInfo(body)) { + throw new Error('malformed instance response from the buckets emulator'); + } + return body; }, - async putCredentials(app, accessKeyId, secretAccessKey) { - const url = `${baseUrl}/_pcdev/apps/${encodeSegment(app)}/credentials`; + async putCredentials(id, accessKeyId, secretAccessKey) { await expectOk( - await adminFetch(url, { + await adminFetch(`${instancePath(`${baseUrl}/_pcdev`, id)}/credentials`, { method: 'PUT', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ accessKeyId, secretAccessKey }), @@ -369,58 +414,66 @@ export function bucketsClient(opts: DaemonRootOptions = {}): BucketsClient { ); }, - async deleteApp(app) { - const url = `${baseUrl}/_pcdev/apps/${encodeSegment(app)}`; - await expectOk(await adminFetch(url, { method: 'DELETE' })); + async deleteInstance(id) { + await expectOk(await adminFetch(instancePath(`${baseUrl}/_pcdev`, id), { method: 'DELETE' })); }, }; } -export interface DatabaseInfo { +export interface DatabaseInstanceInfo { readonly id: string; + /** Stored and returned verbatim; the emulator never reads them. */ + readonly labels: Readonly>; + readonly port: number; readonly url: string; - readonly instanceName: string; - readonly databasePort: number; + readonly status: 'running' | 'stopped'; +} + +export interface CreateDatabaseRequest { + readonly labels: Readonly>; + /** Used if no instance holds it and it is bindable; otherwise the emulator gives the lowest free port in its database range. */ + readonly port?: number; + /** Where the caller resolved `@prisma/dev`; the emulator loads it from here. */ + readonly prismaDevModulePath: string; } -function isDatabaseInfo(value: unknown): value is DatabaseInfo { +function isDatabaseInstanceInfo(value: unknown): value is DatabaseInstanceInfo { return ( typeof value === 'object' && value !== null && 'id' in value && typeof value.id === 'string' && + 'labels' in value && + isStringRecord(value.labels) && + 'port' in value && + typeof value.port === 'number' && 'url' in value && typeof value.url === 'string' && - 'instanceName' in value && - typeof value.instanceName === 'string' && - 'databasePort' in value && - typeof value.databasePort === 'number' + 'status' in value && + (value.status === 'running' || value.status === 'stopped') ); } -function isDatabaseInfoArray(value: unknown): value is DatabaseInfo[] { - return Array.isArray(value) && value.every(isDatabaseInfo); -} - -interface EnsuredDatabase { - readonly url: string; -} - -function isEnsuredDatabase(value: unknown): value is EnsuredDatabase { - return ( - typeof value === 'object' && value !== null && 'url' in value && typeof value.url === 'string' - ); +function isDatabaseInstanceInfoArray(value: unknown): value is DatabaseInstanceInfo[] { + return Array.isArray(value) && value.every(isDatabaseInstanceInfo); } +/** Each database is an instance keyed by an id the emulator generates, served by its own `@prisma/dev` server. */ export interface PostgresClient { readonly baseUrl: string; health(): Promise<{ version: string }>; - /** `PUT /apps//databases/` — ensure a named, persistent `@prisma/dev` server. Idempotent. */ - ensureDatabase(app: string, id: string, prismaDevModulePath: string): Promise<{ url: string }>; - /** `GET /apps//databases`. */ - listDatabases(app: string): Promise; - /** `DELETE /apps/` — closes the app's servers and deletes their persisted data. */ - deleteApp(app: string): Promise; + /** `POST /instances` — creates a database and starts its server; it holds its port until it is deleted. Never retried: a retry could create a second database. */ + createInstance(request: CreateDatabaseRequest): Promise; + /** `GET /instances`. */ + listInstances(): Promise; + /** `GET /instances/`; undefined when the emulator has no such instance. */ + getInstance(id: string): Promise; + /** `POST /instances/start` — starts each listed database's server on its port, with its data. Fails naming any unknown id. */ + startInstances(ids: readonly string[], prismaDevModulePath: string): Promise; + /** `POST /instances/stop` — shuts the servers down and keeps their data and ports; an unknown id is skipped. */ + stopInstances(ids: readonly string[]): Promise; + /** `DELETE /instances/` — stops the server and deletes its data, loading `@prisma/dev` from `prismaDevModulePath` when given (otherwise from where the database was last started). Deleting an unknown id succeeds. */ + deleteInstance(id: string, prismaDevModulePath?: string): Promise; } export function postgresClient(opts: DaemonRootOptions = {}): PostgresClient { @@ -438,35 +491,58 @@ export function postgresClient(opts: DaemonRootOptions = {}): PostgresClient { return body; }, - async ensureDatabase(app, id, prismaDevModulePath) { - const url = `${baseUrl}/apps/${encodeSegment(app)}/databases/${encodeSegment(id)}`; + async createInstance(request) { const res = await expectOk( - await adminFetch(url, { - method: 'PUT', - headers: { 'content-type': 'application/json' }, - body: JSON.stringify({ prismaDevModulePath }), - }), + await adminFetch(`${baseUrl}/instances`, postJson(request), { retry: false }), ); const body: unknown = await res.json(); - if (!isEnsuredDatabase(body)) { - throw new Error('malformed database-ensure response from the postgres emulator'); + if (!isDatabaseInstanceInfo(body)) { + throw new Error('malformed instance-create response from the postgres emulator'); } return body; }, - async listDatabases(app) { - const url = `${baseUrl}/apps/${encodeSegment(app)}/databases`; - const res = await expectOk(await adminFetch(url)); + async listInstances() { + const res = await expectOk(await adminFetch(`${baseUrl}/instances`)); const body: unknown = await res.json(); - if (!isDatabaseInfoArray(body)) { - throw new Error('malformed databases listing from the postgres emulator'); + if (!isDatabaseInstanceInfoArray(body)) { + throw new Error('malformed instances listing from the postgres emulator'); + } + return body; + }, + + async getInstance(id) { + const res = await adminFetch(instancePath(baseUrl, id)); + if (res.status === 404) return undefined; + const body: unknown = await (await expectOk(res)).json(); + if (!isDatabaseInstanceInfo(body)) { + throw new Error('malformed instance response from the postgres emulator'); } return body; }, - async deleteApp(app) { - const url = `${baseUrl}/apps/${encodeSegment(app)}`; - await expectOk(await adminFetch(url, { method: 'DELETE' })); + async startInstances(ids, prismaDevModulePath) { + await expectOk( + await adminFetch(`${baseUrl}/instances/start`, postJson({ ids, prismaDevModulePath })), + ); + }, + + async stopInstances(ids) { + await expectOk(await adminFetch(`${baseUrl}/instances/stop`, postJson({ ids }))); + }, + + async deleteInstance(id, prismaDevModulePath) { + await expectOk( + await adminFetch(instancePath(baseUrl, id), { + method: 'DELETE', + ...(prismaDevModulePath !== undefined + ? { + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ prismaDevModulePath }), + } + : {}), + }), + ); }, }; } diff --git a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/exports/index.ts b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/exports/index.ts index 46a39ae16..bffef9981 100644 --- a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/exports/index.ts +++ b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/exports/index.ts @@ -6,14 +6,17 @@ */ export type { + BucketInstanceInfo, BucketsClient, ComputeClient, - CreateInstanceRequest, - DatabaseInfo, + CreateBucketRequest, + CreateDatabaseRequest, + CreateServiceRequest, + DatabaseInstanceInfo, DeploymentRequest, - InstanceInfo, - InstanceStatus, PostgresClient, + ServiceInstanceInfo, + ServiceInstanceStatus, } from '../client.ts'; export { bucketsClient, computeClient, postgresClient } from '../client.ts'; export type { DaemonName, DaemonRootOptions, RegistryEntry } from '../daemon.ts'; @@ -29,4 +32,3 @@ export { registryFilePath, stopDaemon, } from '../daemon.ts'; -export { instanceNameFor, slug } from '../instance-name.ts'; diff --git a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/instance-name.ts b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/instance-name.ts deleted file mode 100644 index 615e4ade3..000000000 --- a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/instance-name.ts +++ /dev/null @@ -1,36 +0,0 @@ -/** - * `pcdev--` instance-name derivation (local-dev spec § 2 - * `postgres-main.ts`): each half lowercased, every char outside `[a-z0-9]` - * replaced by `-`, runs collapsed, the combined name trimmed to 63 chars. - * - * A separate, side-effect-free module (not defined inline in - * `postgres-main.ts`) so it can be imported directly by tests without also - * running that file's own `main()` — `postgres-main.ts` is a daemon - * entrypoint script, always invoked as a subprocess, and calls `main()` - * unconditionally at module load. - */ - -/** - * Deliberately linear, no ambiguous quantifiers: a per-character replace - * (no `+`, so no run-length backtracking surface), a bounded-quantifier - * collapse (`{2,}`, not alternation-with-quantifiers), and plain - * index-walking for the leading/trailing trim instead of a regex — a - * combined `+`-plus-alternation trim (`/^-+|-+$/g`) is exactly the shape - * CodeQL's polynomial-ReDoS check flags, whether or not this particular - * instance is provably safe. - */ -export function slug(segment: string): string { - const collapsed = segment - .toLowerCase() - .replace(/[^a-z0-9]/g, '-') - .replace(/-{2,}/g, '-'); - let start = 0; - let end = collapsed.length; - while (start < end && collapsed[start] === '-') start++; - while (end > start && collapsed[end - 1] === '-') end--; - return collapsed.slice(start, end); -} - -export function instanceNameFor(app: string, id: string): string { - return `pcdev-${slug(app)}-${slug(id)}`.slice(0, 63); -} diff --git a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/postgres-main.ts b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/postgres-main.ts index 4d70b8af3..8fd832226 100644 --- a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/postgres-main.ts +++ b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/postgres-main.ts @@ -1,9 +1,13 @@ /** * The Postgres emulator daemon (local-dev spec § 2 `postgres-main.ts`): a * small local counterpart of hosted Postgres, hosting `@prisma/dev`'s - * `startPrismaDevServer()` — one named, persistent server per `Database` - * resource, several servers in this one daemon process. Loopback - * `node:http` JSON admin API; state under its `--state-dir`. + * `startPrismaDevServer()` — one named, persistent server per database + * instance, several servers in this one daemon process. Instances are keyed + * by ids this daemon generates; a server is named after its instance's id, + * and `@prisma/dev` keeps its data under that name in its own data + * directory. Labels are stored and returned verbatim, never interpreted. + * Stopping an instance shuts its server down and keeps its data and port. + * Loopback `node:http` JSON admin API; state under its `--state-dir`. * * `@prisma/dev` is imported dynamically from a CALLER-RESOLVED path (each * admin request that needs it carries `prismaDevModulePath`): the local @@ -14,25 +18,24 @@ * Runs as its own OS process, started by `daemon.ts`'s `ensureDaemon` via * `process.execPath --port --state-dir `. */ +import { randomBytes } from 'node:crypto'; import * as fs from 'node:fs'; import * as http from 'node:http'; import * as path from 'node:path'; import { pathToFileURL } from 'node:url'; -import getPort, { portNumbers } from 'get-port'; +import getPort, { clearLockedPorts, portNumbers } from 'get-port'; import { isPidAlive, readOwnVersion } from './daemon.ts'; -import { instanceNameFor } from './instance-name.ts'; import { isStringKeyedRecord, portOfUrl, type RegistryScanHost, registryClaimedPorts, } from './prisma-dev-registry.ts'; -import { isValidSegment } from './segments.ts'; import { readJsonFile, StateFile } from './state-file.ts'; const MIN_DATABASE_PORT = 51_300; const MAX_DATABASE_PORT = 65_535; -const APPS_STATE_MODE = 0o600; +const STATE_MODE = 0o600; /** Spec § 2 step 5's pattern, applied to a fresh database-port allocation. */ const MAX_FRESH_PORT_CANDIDATES = 5; /** @@ -47,7 +50,7 @@ const ALREADY_RUNNING_POLLS = 24; const ALREADY_RUNNING_POLL_MS = 500; const NOT_INSTALLED_MESSAGE = - 'local dev needs @prisma/dev for its local Postgres emulator — @prisma/composer-prisma-cloud declares it as a dependency, but the given module path did not load; reinstall your dependencies.'; + 'local dev needs @prisma/dev for its local Postgres emulator, and the given module path did not load; reinstall your dependencies.'; // The behavior contract's no-value-logging rule, applied to embedded // diagnostics too (spec's diagnostics rule) — masks a connection URL's @@ -65,16 +68,46 @@ function firstLine(text: string): string { interface DatabaseRecord { readonly id: string; - readonly instanceName: string; - readonly databasePort: number; - readonly url: string; + readonly labels: Record; + /** The `@prisma/dev` server's name, which also names its data directory. */ + readonly serverName: string; + /** Held from creation until the instance is deleted, running or stopped: dependents' environments carry the URL. */ + readonly port: number; + url: string; + /** Where the client resolved `@prisma/dev`; every later start and the delete load it from here. */ + prismaDevModulePath: string; } -interface AppRecord { - databases: Record; +interface InstancesState { + instances: Record; + /** `@prisma/dev` servers an earlier version of this emulator created, whose data is deleted once a request brings a `@prisma/dev` module to delete it with. */ + discardedServers: string[]; } -type AppsState = Record; +const INSTANCE_ID_RE = /^db_[0-9a-f]{16}$/; + +function newInstanceId(): string { + return `db_${randomBytes(8).toString('hex')}`; +} + +function serverNameFor(id: string): string { + return `pcdev-${id}`; +} + +function isStringRecord(value: unknown): value is Record { + return ( + typeof value === 'object' && + value !== null && + !Array.isArray(value) && + Object.values(value).every((v) => typeof v === 'string') + ); +} + +function isPort(value: unknown): value is number { + return ( + typeof value === 'number' && Number.isInteger(value) && value >= 1 && value <= MAX_DATABASE_PORT + ); +} function isDatabaseRecord(value: unknown): value is DatabaseRecord { return ( @@ -82,43 +115,97 @@ function isDatabaseRecord(value: unknown): value is DatabaseRecord { value !== null && 'id' in value && typeof value.id === 'string' && - 'instanceName' in value && - typeof value.instanceName === 'string' && - 'databasePort' in value && - typeof value.databasePort === 'number' && + 'labels' in value && + isStringRecord(value.labels) && + 'serverName' in value && + typeof value.serverName === 'string' && + 'port' in value && + isPort(value.port) && 'url' in value && - typeof value.url === 'string' + typeof value.url === 'string' && + 'prismaDevModulePath' in value && + typeof value.prismaDevModulePath === 'string' ); } -function isAppRecord(value: unknown): value is AppRecord { +function isInstancesState(value: unknown): value is InstancesState { return ( typeof value === 'object' && value !== null && - 'databases' in value && - typeof value.databases === 'object' && - value.databases !== null && - Object.values(value.databases).every(isDatabaseRecord) + 'instances' in value && + typeof value.instances === 'object' && + value.instances !== null && + Object.values(value.instances).every(isDatabaseRecord) && + 'discardedServers' in value && + Array.isArray(value.discardedServers) && + value.discardedServers.every((name) => typeof name === 'string') ); } -function isAppsState(value: unknown): value is AppsState { - return typeof value === 'object' && value !== null && Object.values(value).every(isAppRecord); -} - -interface DatabaseBody { +interface CreateBody { + readonly labels: Record; + readonly port?: number; readonly prismaDevModulePath: string; } -function isDatabaseBody(value: unknown): value is DatabaseBody { +function isCreateBody(value: unknown): value is CreateBody { return ( typeof value === 'object' && value !== null && + 'labels' in value && + isStringRecord(value.labels) && + (!('port' in value) || value.port === undefined || isPort(value.port)) && + 'prismaDevModulePath' in value && + typeof value.prismaDevModulePath === 'string' + ); +} + +interface StartBody { + readonly ids: readonly string[]; + readonly prismaDevModulePath: string; +} + +function isStartBody(value: unknown): value is StartBody { + return ( + isIdsBody(value) && 'prismaDevModulePath' in value && typeof value.prismaDevModulePath === 'string' ); } +interface IdsBody { + readonly ids: readonly string[]; +} + +function isIdsBody(value: unknown): value is IdsBody { + return ( + typeof value === 'object' && + value !== null && + 'ids' in value && + Array.isArray(value.ids) && + value.ids.every((id) => typeof id === 'string') + ); +} + +/** Every `@prisma/dev` server name an earlier version's state file lists; an unreadable file lists none. */ +function serverNamesInEarlierState(raw: string): string[] { + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch { + return []; + } + if (!isStringKeyedRecord(parsed)) return []; + return Object.values(parsed).flatMap((group) => { + const databases = isStringKeyedRecord(group) ? group['databases'] : undefined; + if (!isStringKeyedRecord(databases)) return []; + return Object.values(databases).flatMap((db) => { + const name = isStringKeyedRecord(db) ? db['instanceName'] : undefined; + return typeof name === 'string' ? [name] : []; + }); + }); +} + // --- `@prisma/dev`, resolved from the caller-given path ------------------- interface PrismaDevServer { @@ -395,31 +482,23 @@ function readBody(req: http.IncomingMessage): Promise { function main(): void { const { port, stateDir } = parseArgs(process.argv.slice(2)); const ownVersion = readOwnVersion(); - const appsJsonPath = path.join(stateDir, 'apps.json'); - const stateFile = new StateFile(appsJsonPath, APPS_STATE_MODE); + const statePath = path.join(stateDir, 'instances.json'); + const stateFile = new StateFile(statePath, STATE_MODE); - let state: AppsState = {}; - // Live, in-process `@prisma/dev` server objects, keyed by instance name — + let state: InstancesState = { instances: {}, discardedServers: [] }; + // Live, in-process `@prisma/dev` server objects, keyed by instance id — // dropped whenever this process exits (including a version-skew replace - // by `ensureDaemon`); `state` (persisted) is what survives, and a PUT is - // what actually restarts a server from it. + // by `ensureDaemon`); `state` (persisted) is what survives, and a start is + // what actually brings a server back from it. const runtimes = new Map(); - // Every OTHER port (server, shadow database, streams) a LIVE server in - // this process is currently using — unlike `databasePort`, none of these - // are part of this daemon's contract with its caller (only the - // connection string is), so they need no persisted identity across a - // restart, just uniqueness among servers live RIGHT NOW in this one - // process (several servers share one daemon). + /** Instances whose server another live process holds: running, but with no handle in this one. */ + const adopted = new Set(); /** * Server starts run ONE AT A TIME. `@prisma/dev`'s start is not * concurrency-safe within a process: two simultaneous calls pick their * ports without seeing each other and one fails with a port refusal * (verified directly — two concurrent starts with distinct, pinned * database ports fail; the same two started in sequence both succeed). - * The daemon issues concurrent starts whenever an app converges more than - * one database, which is what produced the port refusals, the retries - * whose half-started servers held their own name's lock, and every - * "already running" failure downstream of that. */ let startQueue: Promise = Promise.resolve(); function serializeStart(run: () => Promise): Promise { @@ -435,34 +514,45 @@ function main(): void { void stateFile.write(state); } - function getOrCreateApp(app: string): AppRecord { - let appRec = state[app]; - if (!appRec) { - appRec = { databases: {} }; - state[app] = appRec; - } - return appRec; + function heldPorts(): Set { + return new Set(Object.values(state.instances).map((db) => db.port)); } - function usedDatabasePorts(): Set { - const ports = new Set(); - for (const appRec of Object.values(state)) { - for (const db of Object.values(appRec.databases)) ports.add(db.databasePort); - } - return ports; + async function smallestUnusedPort(exclude: ReadonlySet): Promise { + return getPort({ + port: portNumbers(MIN_DATABASE_PORT, MAX_DATABASE_PORT), + exclude: [...heldPorts(), ...exclude], + }); } - async function smallestUnusedDatabasePort( - min: number, - alsoExclude: ReadonlySet = new Set(), + /** The requested port when no instance or record holds it and it is bindable; otherwise the smallest such port in the database range. */ + async function allocatePort( + requested: number | undefined, + exclude: ReadonlySet, ): Promise { - return getPort({ - port: portNumbers(min, MAX_DATABASE_PORT), - exclude: [...usedDatabasePorts(), ...alsoExclude], - }); + const held = heldPorts(); + if (requested !== undefined && !held.has(requested) && !exclude.has(requested)) { + const given = await getPort({ port: requested, exclude: [...held, ...exclude] }); + if (given === requested) return given; + } + return smallestUnusedPort(exclude); } - const inflightEnsures = new Map>(); + /** In-flight starts and stops per instance, so a stop, a start or a delete never overlaps another on the same server. */ + const inflight = new Map>(); + function exclusive(id: string, run: () => Promise): Promise { + const prior = inflight.get(id) ?? Promise.resolve(); + const next = prior.then(run, run); + const settled = next.then( + () => undefined, + () => undefined, + ); + inflight.set(id, settled); + void settled.then(() => { + if (inflight.get(id) === settled) inflight.delete(id); + }); + return next; + } /** * Names whose `startPrismaDevServer` call failed in THIS process for any @@ -477,126 +567,81 @@ function main(): void { */ const leakedStartNames = new Set(); - /** Concurrent PUTs for the same database coalesce onto one start — a second `startPrismaDevServer` for a name whose first start is still booting fails as "already running". */ - function ensureDatabase( - app: string, - id: string, - prismaDevModulePath: string, - ): Promise<{ url: string }> { - const instanceName = instanceNameFor(app, id); - const inflight = inflightEnsures.get(instanceName); - if (inflight) return inflight; - const run = ensureDatabaseSerialized(app, id, prismaDevModulePath, instanceName).finally(() => { - inflightEnsures.delete(instanceName); - }); - inflightEnsures.set(instanceName, run); - return run; - } - - async function ensureDatabaseSerialized( - app: string, - id: string, - prismaDevModulePath: string, - instanceName: string, - ): Promise<{ url: string }> { - const appRec = getOrCreateApp(app); - const existingRecord = appRec.databases[id]; - - // Already live in THIS process — idempotent, nothing to do. - const runningServer = runtimes.get(instanceName); - if (existingRecord && runningServer) { - return { url: existingRecord.url }; - } - - // Resolved first: its failure carries the pinned "add prisma to your - // devDependencies" message, which a caller with a bad path must see - // rather than the internal-state resolver's own wording. + /** + * Starts the server for `serverName`. A fresh allocation picks its port + * (preferring `requestedPort`) inside the serialized section and may move + * to another port on a refusal; a start of an existing instance is pinned + * to its port, because dependents' environments carry its URL. + */ + async function startServer(opts: { + readonly serverName: string; + readonly prismaDevModulePath: string; + readonly pinnedPort?: number; + readonly requestedPort?: number; + }): Promise<{ readonly url: string; readonly port: number; readonly server?: PrismaDevServer }> { + const { serverName, prismaDevModulePath } = opts; + // Resolved first: its failure carries the pinned "reinstall" message, + // which a caller with a bad path must see rather than the internal-state + // resolver's own wording. const prismaDev = await importPrismaDev(prismaDevModulePath); - // Reconcile with `@prisma/dev`'s own record BEFORE trying to start. - // This daemon's state and that record can disagree — a daemon replaced - // for version skew leaves its in-process servers' records claiming - // "running" behind a dead pid, and a teardown that declined to delete - // (see `deleteApp`) leaves a live one. Starting into either produces the - // "already running" refusal; reading the record first turns both into - // ordinary cases. + // Reconcile with `@prisma/dev`'s own record BEFORE trying to start. A + // daemon replaced for version skew leaves its in-process servers' + // records claiming "running" behind a dead pid; starting into that + // produces the "already running" refusal. const internalState = await importPrismaDevInternalState(prismaDevModulePath); - const recorded = await serverStatusOf(internalState, instanceName); + await discardEarlierServers(internalState); + const recorded = await serverStatusOf(internalState, serverName); if (recorded.live && recorded.pid !== undefined && isPidAlive(recorded.pid)) { - // A live server under our own name: the name is namespaced to this - // app+database, so this IS the database being asked for. Adopt its - // URL. (No handle to close it — `deleteApp` uses `killServer` for - // exactly this case.) + // A live server under this instance's own name in another process: + // it IS this database. Adopt its URL. if (recorded.url !== undefined) { - appRec.databases[id] = { - id, - instanceName, - databasePort: recorded.databasePort ?? portOfUrl(recorded.url) ?? MIN_DATABASE_PORT, + return { url: recorded.url, + port: recorded.databasePort ?? portOfUrl(recorded.url) ?? opts.pinnedPort ?? 0, }; - schedulePersist(); - return { url: recorded.url }; } } else if (recorded.live) { - // Recorded live, but nothing is running — the owning process died - // (daemon replacement, a crash). Clear the process record; `killServer` - // never touches the persisted data, so the database survives. - await internalState.killServer(instanceName).catch(() => undefined); + // Recorded live, but nothing is running — the owning process died. + // Clear the process record; `killServer` never touches the data. + await internalState.killServer(serverName).catch(() => undefined); } - const isFreshAllocation = existingRecord === undefined; - // Assigned inside the serialized section below; a persisted port is - // fixed here and never re-picked. - let dbPort = existingRecord?.databasePort ?? 0; - + const isFreshAllocation = opts.pinnedPort === undefined; + let dbPort = opts.pinnedPort ?? 0; const conflicted = new Set(); let alreadyRunningRetries = 0; let blindRetries = 0; for (let attempt = 1; ; attempt++) { - // ALL FOUR ports are ours to pick. `@prisma/dev`'s own picker - // consults only its record registry, never the OS — under - // concurrent processes it walks onto a port another process has - // bound but not yet recorded and fails the bind ("Is port X in - // use?", seen on CI with turbo running several packages' tests at - // once). Our pick both OS-probes (getPort) and excludes every - // recorded claim, and a refusal of any of the four carries the - // port, which the retry below turns into fresh candidates. - // - // Picks happen INSIDE the serialized section, together with the - // start they feed: an app converges several databases at once, and - // a pick made outside the queue cannot see the ports a sibling's - // queued start is about to bind — both ensures would pick the same - // free candidates, and the loser's refusal leaks its name's lock - // (observed on CI as "already running" with the retry budget spent - // on clash-recover cycles). + // ALL FOUR ports are ours to pick. `@prisma/dev`'s own picker consults + // only its record registry, never the OS, so under concurrent processes + // it walks onto a port another process has bound but not yet recorded. + // Our pick both OS-probes (getPort) and excludes every recorded claim, + // and a refusal of any of the four carries the port, which the retry + // below turns into fresh candidates. Picks happen inside the + // serialized section, together with the start they feed. const auxPorts: number[] = []; try { const server = await serializeStart(async () => { - // Scanned inside the serialized section, once per attempt, so each - // pick sees the claims a sibling's queued start just recorded. - // Picks avoid every port a foreign `@prisma/dev` record claims, or - // the start would be refused (see `registryClaimedPorts`). A - // persisted DATABASE port is pinned regardless — endpoints in - // deploy state reference it. - const registryPorts = await registryClaimedPorts(internalState, instanceName); + const registryPorts = await registryClaimedPorts(internalState, serverName); if (isFreshAllocation) { - dbPort = await smallestUnusedDatabasePort( - MIN_DATABASE_PORT, + dbPort = await allocatePort( + attempt === 1 ? opts.requestedPort : undefined, new Set([...registryPorts, ...conflicted]), ); } const auxExclude = new Set([...registryPorts, ...conflicted, dbPort]); const pickAux = async (): Promise => { - const port = await smallestUnusedDatabasePort(MIN_DATABASE_PORT, auxExclude); - auxExclude.add(port); - auxPorts.push(port); - return port; + const picked = await smallestUnusedPort(auxExclude); + auxExclude.add(picked); + auxPorts.push(picked); + return picked; }; const httpPort = await pickAux(); const shadowPort = await pickAux(); const streamsPort = await pickAux(); return prismaDev.startPrismaDevServer({ - name: instanceName, + name: serverName, databasePort: dbPort, port: httpPort, shadowDatabasePort: shadowPort, @@ -604,161 +649,231 @@ function main(): void { persistenceMode: 'stateful', }); }); - const url = server.database.connectionString; - leakedStartNames.delete(instanceName); - runtimes.set(instanceName, server); - appRec.databases[id] = { id, instanceName, databasePort: dbPort, url }; - schedulePersist(); - return { url }; + leakedStartNames.delete(serverName); + return { url: server.database.connectionString, port: dbPort, server }; } catch (err) { if (isNameAlreadyTaken(err) && alreadyRunningRetries < MAX_ALREADY_RUNNING_RETRIES) { alreadyRunningRetries += 1; - // A refusal on a name THIS process leaked (see `leakedStartNames`) - // with no record behind it is not a foreign holder to wait for — - // the holder is our own dead attempt. Delete the orphaned state - // (there is no record, so no data) to free the lock, and retry. - if (leakedStartNames.has(instanceName)) { - const now = await serverStatusOf(internalState, instanceName); + // A refusal on a name THIS process leaked with no record behind it + // is not a foreign holder to wait for: delete the orphaned state + // (no record, so no data) to free the lock, and retry. + if (leakedStartNames.has(serverName)) { + const now = await serverStatusOf(internalState, serverName); if (!now.recordExists) { - await internalState.deleteServer(instanceName).catch(() => undefined); + await internalState.deleteServer(serverName).catch(() => undefined); continue; } } - // The refusal means a live process holds this server's lock, but - // the RECORD it holds can say three different things, and the - // difference matters (observed on CI: the daemon had no record for - // this database while its name was locked): - // - // - a live record with a live owner: that server IS this database - // (the name is namespaced to this app+database) — adopt its URL; - // - a live record with a dead owner: a crashed owner's leftover — - // clear the process record (never the data) and start again; - // - NO live record at all: the lock is held by a holder that has - // not published its record yet, or one whose record was just - // removed while its lock lingers. Neither is resolvable by - // starting again immediately — the previous code did exactly - // that and burned every attempt in milliseconds. Wait for the - // holder to publish or release, then retry the start. + // A live record with a live owner is this database: adopt it. A + // live record with a dead owner is a crashed owner's leftover: + // clear it and start again. No live record: wait for the holder to + // publish or release, then retry. for (let poll = 1; poll <= ALREADY_RUNNING_POLLS; poll += 1) { - const now = await serverStatusOf(internalState, instanceName); + const now = await serverStatusOf(internalState, serverName); if (now.live && now.pid !== undefined && isPidAlive(now.pid) && now.url !== undefined) { - appRec.databases[id] = { - id, - instanceName, - databasePort: now.databasePort ?? portOfUrl(now.url) ?? dbPort, - url: now.url, - }; - schedulePersist(); - return { url: now.url }; + return { url: now.url, port: now.databasePort ?? portOfUrl(now.url) ?? dbPort }; } if (now.live && (now.pid === undefined || !isPidAlive(now.pid))) { - await internalState.killServer(instanceName).catch(() => undefined); + await internalState.killServer(serverName).catch(() => undefined); break; } await new Promise((resolve) => setTimeout(resolve, ALREADY_RUNNING_POLL_MS)); } continue; } - if (!isNameAlreadyTaken(err)) { - // The start may have failed AFTER `@prisma/dev` acquired the - // name's lock (a port refusal does), which leaks the lock — see - // `leakedStartNames`. Every retry below and every later PUT for - // this name needs to know that. - leakedStartNames.add(instanceName); - } + if (!isNameAlreadyTaken(err)) leakedStartNames.add(serverName); const conflictPort = portConflictOf(err, [dbPort, ...auxPorts]); - // The aux listener ports are never persisted, so a refusal of one is - // always retryable with fresh candidates. The DATABASE port is frozen - // once a record exists (endpoints in deploy state reference it) — - // only a fresh allocation may move it. + // The aux ports are never persisted, so a refusal of one is always + // retryable. The database port is pinned once the instance exists. const canRetryNextPort = conflictPort !== undefined && attempt < MAX_FRESH_PORT_CANDIDATES && (conflictPort !== dbPort || isFreshAllocation); if (!canRetryNextPort) { - // A failure that names no port can still be a transient port - // race — a bind that lost to a process our probe couldn't see - // yet reports only "Is port X in use?" text, not a `.port`. - // Every attempt picks fresh aux candidates, so trying again is - // meaningful; bounded so a genuinely broken start still fails - // visibly with its own message. + // A failure that names no port can still be a transient bind race + // ("Is port X in use?" text, no `.port`); every attempt picks fresh + // aux candidates, so trying again is meaningful, a bounded number + // of times. if (conflictPort === undefined && !isNameAlreadyTaken(err) && blindRetries < 2) { blindRetries += 1; continue; } const reason = err instanceof Error ? err.message : String(err); throw new Error( - `postgres emulator failed to start database "${instanceName}" on port ${String(dbPort)}: ${maskCredentials(firstLine(reason))}`, + `postgres emulator failed to start database "${serverName}" on port ${String(dbPort)}: ${maskCredentials(firstLine(reason))}`, ); } - // The next attempt re-picks every non-pinned port inside the - // serialized section, skipping everything conflicted so far. conflicted.add(conflictPort); } } } - async function deleteApp( - app: string, - prismaDevModulePathHint: string | undefined, - ): Promise { - const appRec = state[app]; - if (!appRec) return; - const entries = Object.values(appRec.databases); - - // Closed one at a time, never concurrently: the servers run pglite's - // native/WASM runtime in THIS process, and the daemon has died silently - // (no JS error, no guard output — a native abort) during exactly this - // teardown window under CI load. Sequential closes remove the only - // concurrency we control there. - for (const db of entries) { - const server = runtimes.get(db.instanceName); - if (server) { - await server.close().catch(() => undefined); - runtimes.delete(db.instanceName); - } - } + /** + * Deletes the data of servers an earlier version of this emulator created, + * once a request brings a `@prisma/dev` module to do it with. Best effort: + * a server another live process still serves, or one whose delete fails, + * is tried again on the next request. + */ + let discarding: Promise | undefined; + function discardEarlierServers(internalState: PrismaDevInternalStateModule): Promise { + if (state.discardedServers.length === 0) return Promise.resolve(); + discarding ??= deleteEarlierServers(internalState).finally(() => { + discarding = undefined; + }); + return discarding; + } - // Deleting the persisted PGlite data needs SOME resolved `@prisma/dev` - // module — the app that owns the databases being deleted is the same - // app whose `prismaDevModulePath` every PUT for it already carried, so - // any one of those already-observed paths works; DELETE itself carries - // no body, so there is nothing more specific to prefer. - if (entries.length > 0 && prismaDevModulePathHint) { - const internalState = await importPrismaDevInternalState(prismaDevModulePathHint); - for (const db of entries) { - // A server this daemon adopted rather than started has no handle to - // close, so stop it by its record first (`killServer` leaves data). - // Never by OUR pid: `killServer` signals whatever pid the record - // names, and a record naming this daemon would make it kill itself. - if (!runtimes.has(db.instanceName)) { - const owner = await serverStatusOf(internalState, db.instanceName); - if (owner.pid !== process.pid) { - await internalState.killServer(db.instanceName).catch(() => undefined); - } - } - // `deleteServer`'s kill path targets the pid in the server's persisted - // state — for an in-daemon server that pid is THIS DAEMON's own. The - // close above marks the state stopped, but the write is asynchronous: - // deleting while the state still says "running" makes the daemon kill - // itself (observed on CI as the daemon dying silently mid-teardown). - // Wait for the close to settle; if it never does, leave the data on - // disk rather than die — the next --fresh gets another chance. - const settled = await settledAfterClose(internalState, db.instanceName); - if (!settled) { - console.error( - `postgres-main: server "${db.instanceName}" still reports running after close — leaving its persisted data in place instead of risking a self-kill.`, - ); + async function deleteEarlierServers(internalState: PrismaDevInternalStateModule): Promise { + const remaining: string[] = []; + for (const name of state.discardedServers) { + try { + const owner = await serverStatusOf(internalState, name); + if (owner.live && owner.pid !== undefined && isPidAlive(owner.pid)) { + // Another process still serves it — under the old names, another + // emulators directory could. Signalling that pid would end every + // server it hosts, so leave this one for a later request. + remaining.push(name); continue; } - await internalState.deleteServer(db.instanceName); + if (owner.live) await internalState.killServer(name).catch(() => undefined); + await internalState.deleteServer(name); + } catch (err) { + console.error( + `postgres-main: could not delete the data of "${name}", left by an earlier version: ${err instanceof Error ? err.message : String(err)}`, + ); + remaining.push(name); } } - - delete state[app]; + state.discardedServers = remaining; schedulePersist(); } + async function createInstance(body: CreateBody): Promise { + const id = newInstanceId(); + const serverName = serverNameFor(id); + let started: Awaited>; + try { + started = await startServer({ + serverName, + prismaDevModulePath: body.prismaDevModulePath, + ...(body.port !== undefined ? { requestedPort: body.port } : {}), + }); + } catch (err) { + await discardFailedServer(serverName, body.prismaDevModulePath); + throw err; + } + const record: DatabaseRecord = { + id, + labels: { ...body.labels }, + serverName, + port: started.port, + url: started.url, + prismaDevModulePath: body.prismaDevModulePath, + }; + if (started.server) runtimes.set(id, started.server); + else adopted.add(id); + state.instances[id] = record; + // Written before the create is acknowledged: an id the caller records + // must survive the daemon dying right after, or its data would be left + // under a name no state lists. + await stateFile.write(state); + return record; + } + + /** + * A failed create records no instance, so nothing will use its server name + * again. A start that failed after `@prisma/dev` took the name's lock leaves + * the name's directory behind with no record; remove it. + */ + async function discardFailedServer(serverName: string, prismaDevModulePath: string) { + if (!leakedStartNames.has(serverName)) return; + try { + const internalState = await importPrismaDevInternalState(prismaDevModulePath); + if (!(await serverStatusOf(internalState, serverName)).recordExists) { + await internalState.deleteServer(serverName); + } + leakedStartNames.delete(serverName); + } catch { + // Best effort: the directory holds no data. + } + } + + function isRunning(id: string): boolean { + return runtimes.has(id) || adopted.has(id); + } + + function startInstance(db: DatabaseRecord, prismaDevModulePath: string): Promise { + return exclusive(db.id, async () => { + db.prismaDevModulePath = prismaDevModulePath; + if (isRunning(db.id)) return; + const started = await startServer({ + serverName: db.serverName, + prismaDevModulePath, + pinnedPort: db.port, + }); + db.url = started.url; + if (started.server) runtimes.set(db.id, started.server); + else adopted.add(db.id); + schedulePersist(); + }); + } + + /** Shuts the server down and keeps its data. */ + async function closeServer(db: DatabaseRecord): Promise { + const server = runtimes.get(db.id); + if (server) { + await server.close().catch(() => undefined); + runtimes.delete(db.id); + return; + } + if (!adopted.has(db.id)) return; + // A server another process holds has no handle here: stop it by its + // record (`killServer` leaves the data). Never by OUR pid — `killServer` + // signals whatever pid the record names. + const internalState = await importPrismaDevInternalState(db.prismaDevModulePath); + const owner = await serverStatusOf(internalState, db.serverName); + if (owner.pid !== process.pid) { + await internalState.killServer(db.serverName).catch(() => undefined); + } + adopted.delete(db.id); + } + + function stopInstance(db: DatabaseRecord): Promise { + return exclusive(db.id, () => closeServer(db)); + } + + /** `prismaDevModulePath` is the caller's: the one the database was last started with may since have been removed. */ + function deleteInstance( + db: DatabaseRecord, + prismaDevModulePath: string | undefined, + ): Promise { + return exclusive(db.id, async () => { + if (prismaDevModulePath !== undefined) db.prismaDevModulePath = prismaDevModulePath; + await closeServer(db); + const internalState = await importPrismaDevInternalState(db.prismaDevModulePath); + // `deleteServer`'s kill path targets the pid in the server's persisted + // state — for an in-daemon server that pid is THIS DAEMON's own. The + // close marks the state stopped, but the write is asynchronous: deleting + // while it still says "running" makes the daemon kill itself. Wait for + // the close to settle; if it never does, leave the data on disk rather + // than die. + if (await settledAfterClose(internalState, db.serverName)) { + await internalState.deleteServer(db.serverName); + } else { + console.error( + `postgres-main: server "${db.serverName}" still reports running after close — leaving its persisted data in place instead of risking a self-kill.`, + ); + } + delete state.instances[db.id]; + // Written before the delete is acknowledged, so a daemon that dies + // right after never brings the database back. + await stateFile.write(state); + // get-port holds every port it returns for up to 30 seconds; the freed + // port must be free again for a database created right after. + clearLockedPorts(); + }); + } + function json(res: http.ServerResponse, status: number, body: unknown): void { res.writeHead(status, { 'content-type': 'application/json' }); res.end(JSON.stringify(body)); @@ -769,27 +884,32 @@ function main(): void { res.end(body); } - function badSegment(res: http.ServerResponse, segment: string): void { - text( - res, - 400, - `invalid path segment "${segment}": must match /^[a-z0-9][a-z0-9-]*$/ and be at most 63 characters`, - ); + function noContent(res: http.ServerResponse): void { + res.writeHead(204); + res.end(); + } + + async function readJsonBody(req: http.IncomingMessage): Promise { + const raw = await readBody(req); + try { + return JSON.parse(raw.toString('utf8')); + } catch { + return undefined; + } } - function databaseView(db: DatabaseRecord): unknown { + function instanceView(db: DatabaseRecord): unknown { return { id: db.id, + labels: db.labels, + port: db.port, url: db.url, - instanceName: db.instanceName, - databasePort: db.databasePort, + status: isRunning(db.id) ? 'running' : 'stopped', }; } - /** The most recently observed `prismaDevModulePath` for any database of `app` — DELETE has no body of its own to carry one. */ - const recentPrismaDevModulePath = new Map(); - function lastKnownPrismaDevModulePath(app: string): string | undefined { - return recentPrismaDevModulePath.get(app); + function unknownInstances(res: http.ServerResponse, ids: readonly string[]): void { + text(res, 404, `no such instance: ${ids.map((id) => `"${id}"`).join(', ')}`); } async function handleRequest(req: http.IncomingMessage, res: http.ServerResponse): Promise { @@ -804,45 +924,70 @@ function main(): void { return json(res, 200, { version: ownVersion }); } - if (segments[0] === 'apps' && segments.length >= 2) { - const app = segments[1]; - if (app === undefined || !isValidSegment(app)) return badSegment(res, app ?? ''); - - if (method === 'PUT' && segments.length === 4 && segments[2] === 'databases') { - const id = segments[3]; - if (id === undefined || !isValidSegment(id)) return badSegment(res, id ?? ''); - const raw = await readBody(req); - let parsed: unknown; - try { - parsed = JSON.parse(raw.toString('utf8')); - } catch { - return text(res, 400, 'malformed JSON body'); - } - if (!isDatabaseBody(parsed)) { + if (segments[0] !== 'instances') { + res.writeHead(404); + res.end(); + return; + } + + if (segments.length === 1) { + if (method === 'GET') return json(res, 200, Object.values(state.instances).map(instanceView)); + if (method === 'POST') { + const body = await readJsonBody(req); + if (!isCreateBody(body)) { return text( res, 400, - 'malformed database body: expected { "prismaDevModulePath": string }', + 'malformed body: expected { "labels": Record, "port"?: number, "prismaDevModulePath": string }', ); } - recentPrismaDevModulePath.set(app, parsed.prismaDevModulePath); - const result = await ensureDatabase(app, id, parsed.prismaDevModulePath); - return json(res, 200, result); + return json(res, 201, instanceView(await createInstance(body))); } + } - if (method === 'GET' && segments.length === 3 && segments[2] === 'databases') { - const appRec = state[app]; - const list = appRec ? Object.values(appRec.databases).map(databaseView) : []; - return json(res, 200, list); + if (method === 'POST' && segments.length === 2 && segments[1] === 'start') { + const body = await readJsonBody(req); + if (!isStartBody(body)) { + return text( + res, + 400, + 'malformed body: expected { "ids": string[], "prismaDevModulePath": string }', + ); + } + const unknown = body.ids.filter((id) => state.instances[id] === undefined); + if (unknown.length > 0) return unknownInstances(res, unknown); + for (const id of body.ids) { + const db = state.instances[id]; + if (db) await startInstance(db, body.prismaDevModulePath); } + return noContent(res); + } - if (method === 'DELETE' && segments.length === 2) { - await deleteApp(app, lastKnownPrismaDevModulePath(app)); - res.writeHead(204); - res.end(); - return; + if (method === 'POST' && segments.length === 2 && segments[1] === 'stop') { + const body = await readJsonBody(req); + if (!isIdsBody(body)) return text(res, 400, 'malformed body: expected { "ids": string[] }'); + // An unknown instance runs nothing, so stopping it is already done. + for (const id of body.ids) { + const db = state.instances[id]; + if (db) await stopInstance(db); } + return noContent(res); + } + + const id = segments[1]; + if (id !== undefined && !INSTANCE_ID_RE.test(id)) { + return text(res, 400, `invalid instance id "${id}"`); + } + const db = id === undefined ? undefined : state.instances[id]; + + if (method === 'DELETE' && segments.length === 2) { + const body = await readJsonBody(req); + const modulePath = isStringKeyedRecord(body) ? body['prismaDevModulePath'] : undefined; + if (db) await deleteInstance(db, typeof modulePath === 'string' ? modulePath : undefined); + return noContent(res); } + if (!db) return unknownInstances(res, id === undefined ? [] : [id]); + if (method === 'GET' && segments.length === 2) return json(res, 200, instanceView(db)); res.writeHead(404); res.end(); @@ -856,12 +1001,10 @@ function main(): void { }); // Close every live `@prisma/dev` server cleanly before exiting — they own - // real PGlite sockets and on-disk WAL state, unlike compute's spawned - // children (a separate OS process the OS reclaims on its own) or - // buckets' plain filesystem store. + // real PGlite sockets and on-disk WAL state. async function shutdown(): Promise { - for (const server of runtimes.values()) { - await server.close().catch(() => undefined); + for (const live of runtimes.values()) { + await live.close().catch(() => undefined); } await stateFile.flush(); server.close(); @@ -872,11 +1015,9 @@ function main(): void { // This daemon hosts `@prisma/dev`'s runtime in-process, and a failed or // abandoned start attempt can leave background async work behind whose - // eventual rejection would otherwise kill the process (bun and node both - // exit on an unhandled rejection). A machine-shared daemon serving every - // app's databases must not die because one attempt's debris rejected — - // log it (the daemon's stdio is teed to its registry log) and keep - // serving; every REQUEST path still reports its own errors as 500s. + // eventual rejection would otherwise kill the process. A machine-shared + // daemon must not die because one attempt's debris rejected — log it and + // keep serving; every REQUEST path still reports its own errors as 500s. process.on('unhandledRejection', (reason) => { console.error( 'postgres-main: unhandled rejection from background work:', @@ -890,9 +1031,10 @@ function main(): void { ); }); - readJsonFile(appsJsonPath, isAppsState) + loadState(stateDir, statePath) .then((loaded) => { - if (loaded) state = loaded; + state = loaded; + schedulePersist(); server.listen(port, '127.0.0.1', () => { console.log(`[dev-emulators] postgres-main listening on 127.0.0.1:${String(port)}`); }); @@ -903,4 +1045,26 @@ function main(): void { }); } +/** + * This version's state, plus the servers named in any other JSON state file + * in the state directory, which an earlier version wrote; those files are + * removed. Local dev state is never migrated; the servers' data is deleted + * on the first request that can (see `discardEarlierServers`). + */ +async function loadState(stateDir: string, statePath: string): Promise { + const loaded = (await readJsonFile(statePath, isInstancesState)) ?? { + instances: {}, + discardedServers: [], + }; + const files = await fs.promises.readdir(stateDir).catch(() => []); + for (const file of files.map((name) => path.join(stateDir, name))) { + if (!file.endsWith('.json') || file === statePath) continue; + const earlier = await fs.promises.readFile(file, 'utf8').catch(() => ''); + const names = serverNamesInEarlierState(earlier); + loaded.discardedServers = [...new Set([...loaded.discardedServers, ...names])]; + await fs.promises.rm(file, { force: true }); + } + return loaded; +} + main(); diff --git a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/segments.ts b/packages/1-prisma-cloud/0-lowering/dev-emulators/src/segments.ts deleted file mode 100644 index 60cc7da13..000000000 --- a/packages/1-prisma-cloud/0-lowering/dev-emulators/src/segments.ts +++ /dev/null @@ -1,11 +0,0 @@ -/** - * API hygiene shared by both daemons (local-dev spec § 2): every ``, - * ``, and `` path segment must match this shape or the request is - * a 400 naming the segment. - */ -export const SEGMENT_RE = /^[a-z0-9][a-z0-9-]*$/; -export const MAX_SEGMENT_LENGTH = 63; - -export function isValidSegment(segment: string): boolean { - return segment.length <= MAX_SEGMENT_LENGTH && SEGMENT_RE.test(segment); -} diff --git a/packages/1-prisma-cloud/0-lowering/local-target/package.json b/packages/1-prisma-cloud/0-lowering/local-target/package.json index f742bea68..5c422962e 100644 --- a/packages/1-prisma-cloud/0-lowering/local-target/package.json +++ b/packages/1-prisma-cloud/0-lowering/local-target/package.json @@ -24,10 +24,13 @@ "tar": "^7.5.21" }, "devDependencies": { + "@aws-sdk/client-s3": "^3.1085.0", "@internal/foundation": "workspace:0.30.0", "@internal/tsdown-config": "workspace:0.30.0", "@types/bun": "^1.3.13", "@types/node": "^26.0.1", + "@types/pg": "^8.20.0", + "pg": "^8.22.0", "tsdown": "^0.22.7", "typescript": "^6.0.3" }, diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/app-data-instances.test.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/app-data-instances.test.ts new file mode 100644 index 000000000..30b83d24a --- /dev/null +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/app-data-instances.test.ts @@ -0,0 +1,218 @@ +/** + * Databases and buckets through the directory's record, against real + * Compute, Postgres and bucket emulators in a temporary emulators directory: + * isolation between directories, stopping and resuming an app's databases + * with their data, switching apps, re-creating a lost database, and + * `--fresh`. + */ +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as net from 'node:net'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { ListObjectsV2Command, PutObjectCommand, S3Client } from '@aws-sdk/client-s3'; +import { + bucketsClient, + type DaemonName, + EMULATORS_DIR_ENV, + ensureDaemon, + postgresClient, + stopDaemon, +} from '@internal/dev-emulators'; +import { Client as PgClient } from 'pg'; +import { type AppNode, ensureAppInstances } from '../app-instances.ts'; +import { + type AppScope, + deleteAppInstances, + type NodeInstance, + readEmulatorInstances, + selectApp, + stopAppInstances, +} from '../emulator-instances.ts'; + +const APP = 'data-app'; +const NODES: readonly AppNode[] = [ + { address: 'catalog', kind: 'database' }, + { address: 'uploads', kind: 'bucket' }, + { address: 'web', kind: 'service' }, +]; +const DAEMONS: readonly DaemonName[] = ['compute', 'postgres', 'buckets']; +const savedEmulatorsDir = process.env[EMULATORS_DIR_ENV]; + +function acceptsConnection(port: number): Promise { + return new Promise((resolve) => { + const socket = net.connect({ port, host: '127.0.0.1' }); + socket.once('connect', () => { + socket.destroy(); + resolve(true); + }); + socket.once('error', () => resolve(false)); + }); +} + +async function query(url: string, sql: string): Promise { + const pg = new PgClient({ connectionString: url }); + await pg.connect(); + try { + return (await pg.query(sql)).rows; + } finally { + await pg.end().catch(() => undefined); + } +} + +// Local dev is not supported on Windows, where the emulators cannot run. +describe.skipIf(process.platform === 'win32')('databases and buckets through the record', () => { + let root: string; + let registryRoot: string; + let scopes: AppScope[]; + + beforeEach(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'local-target-app-data-')); + registryRoot = path.join(root, 'emulators'); + process.env[EMULATORS_DIR_ENV] = registryRoot; + for (const name of DAEMONS) { + const entry = fileURLToPath(import.meta.resolve(`@internal/dev-emulators/${name}-main`)); + await ensureDaemon(name, entry, { registryRoot }); + } + scopes = []; + }); + + afterEach(async () => { + for (const scope of scopes) await deleteAppInstances(scope).catch(() => undefined); + for (const name of DAEMONS) await stopDaemon(name, { registryRoot }).catch(() => undefined); + fs.rmSync(root, { recursive: true, force: true }); + if (savedEmulatorsDir === undefined) delete process.env[EMULATORS_DIR_ENV]; + else process.env[EMULATORS_DIR_ENV] = savedEmulatorsDir; + }); + + function scopeIn(name: string, app = APP): AppScope & { readonly directory: string } { + const directory = path.join(root, name); + const scope = { directory, devDir: path.join(directory, '.prisma-composer', 'dev'), app }; + scopes.push(scope); + return scope; + } + + async function nodeOf(scope: AppScope & { readonly directory: string }, address: string) { + const node: NodeInstance | undefined = (await readEmulatorInstances(scope.directory))?.apps[ + scope.app + ]?.nodes[address]; + if (!node) throw new Error(`no instance recorded for "${address}"`); + return node; + } + + async function databaseOf(scope: AppScope & { readonly directory: string }) { + const instance = await postgresClient().getInstance((await nodeOf(scope, 'catalog')).id); + if (!instance) throw new Error('the recorded database is gone'); + return instance; + } + + async function s3For(scope: AppScope & { readonly directory: string }, key: string) { + const bucket = (await nodeOf(scope, 'uploads')).id; + await bucketsClient().putCredentials(bucket, key, `${key}-secret`); + const s3 = new S3Client({ + region: 'auto', + endpoint: bucketsClient().baseUrl, + forcePathStyle: true, + credentials: { accessKeyId: key, secretAccessKey: `${key}-secret` }, + requestChecksumCalculation: 'WHEN_REQUIRED', + responseChecksumValidation: 'WHEN_REQUIRED', + maxAttempts: 1, + }); + return { s3, bucket }; + } + + test('two directories running the same app each see only their own rows and objects', async () => { + const a = scopeIn('a'); + const b = scopeIn('b'); + await Promise.all([ensureAppInstances(a, NODES), ensureAppInstances(b, NODES)]); + + const dbA = await databaseOf(a); + const dbB = await databaseOf(b); + expect(dbA.id).not.toBe(dbB.id); + expect(dbA.port).not.toBe(dbB.port); + await query(dbA.url, "create table marker (v text); insert into marker values ('a')"); + await query(dbB.url, "create table marker (v text); insert into marker values ('b')"); + const s3A = await s3For(a, 'AKIADIRA'); + const s3B = await s3For(b, 'AKIADIRB'); + await s3A.s3.send(new PutObjectCommand({ Bucket: s3A.bucket, Key: 'only-a', Body: 'a' })); + await s3B.s3.send(new PutObjectCommand({ Bucket: s3B.bucket, Key: 'only-b', Body: 'b' })); + + expect(await query(dbA.url, 'select v from marker')).toEqual([{ v: 'a' }]); + expect(await query(dbB.url, 'select v from marker')).toEqual([{ v: 'b' }]); + const listedA = await s3A.s3.send(new ListObjectsV2Command({ Bucket: s3A.bucket })); + const listedB = await s3B.s3.send(new ListObjectsV2Command({ Bucket: s3B.bucket })); + expect(listedA.Contents?.map((o) => o.Key)).toEqual(['only-a']); + expect(listedB.Contents?.map((o) => o.Key)).toEqual(['only-b']); + }, 120_000); + + test('when the app stops, its database server stops; the next start has the data on the same port', async () => { + const scope = scopeIn('stop'); + await ensureAppInstances(scope, NODES); + const before = await databaseOf(scope); + await query(before.url, "create table kept (v text); insert into kept values ('row')"); + + await stopAppInstances(scope); + + expect((await databaseOf(scope)).status).toBe('stopped'); + expect(await acceptsConnection(before.port)).toBe(false); + + await ensureAppInstances(scope, NODES); + + const after = await databaseOf(scope); + expect(after).toMatchObject({ id: before.id, status: 'running', url: before.url }); + expect(await query(after.url, 'select v from kept')).toEqual([{ v: 'row' }]); + }, 120_000); + + test("switching A → B → A stops A's databases and resumes them with their data and ports", async () => { + const a = scopeIn('switch'); + const b = { ...a, app: 'other-app' }; + scopes.push(b); + await selectApp(a); + await ensureAppInstances(a, NODES); + const before = await databaseOf(a); + await query(before.url, "create table kept (v text); insert into kept values ('a')"); + const recordedA = (await readEmulatorInstances(a.directory))?.apps[APP]; + + expect(await selectApp(b)).toBe(APP); + expect((await databaseOf(a)).status).toBe('stopped'); + await ensureAppInstances(b, NODES); + + expect(await selectApp(a)).toBe('other-app'); + const changes = await ensureAppInstances(a, NODES); + + expect(changes).toEqual([]); + expect((await readEmulatorInstances(a.directory))?.apps[APP]).toEqual(recordedA); + const after = await databaseOf(a); + expect(after).toMatchObject({ status: 'running', url: before.url }); + expect(await query(after.url, 'select v from kept')).toEqual([{ v: 'a' }]); + }, 180_000); + + test('a recorded database the emulator lost is created again, empty, on its remembered port', async () => { + const scope = scopeIn('lost'); + await ensureAppInstances(scope, NODES); + const before = await databaseOf(scope); + await postgresClient().deleteInstance(before.id); + + const changes = await ensureAppInstances(scope, NODES); + + const after = await databaseOf(scope); + expect(changes).toEqual([]); + expect(after.id).not.toBe(before.id); + expect(after.port).toBe(before.port); + expect(await query(after.url, "select to_regclass('kept') as reg")).toEqual([{ reg: null }]); + }, 120_000); + + test("--fresh deletes the app's database, bucket and service, and its entry", async () => { + const scope = scopeIn('fresh'); + await ensureAppInstances(scope, NODES); + const database = await nodeOf(scope, 'catalog'); + const bucket = await nodeOf(scope, 'uploads'); + + await deleteAppInstances(scope); + + expect(await postgresClient().getInstance(database.id)).toBeUndefined(); + expect(await bucketsClient().getInstance(bucket.id)).toBeUndefined(); + expect((await readEmulatorInstances(scope.directory))?.apps[APP]).toBeUndefined(); + }, 120_000); +}); diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/bucket-providers.test.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/bucket-providers.test.ts new file mode 100644 index 000000000..15aa12b57 --- /dev/null +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/bucket-providers.test.ts @@ -0,0 +1,94 @@ +/** + * The local Bucket and BucketAccessKey providers against a real bucket + * emulator in a temporary emulators directory: a Bucket is a bucket + * instance, and its access key is registered on that instance, whose id is + * the bucket's name on the S3 wire. + */ +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import type { ContainerInstance, LocalTargetProvidersInput } from '@internal/core/config'; +import { + bucketsClient, + EMULATORS_DIR_ENV, + ensureDaemon, + stopDaemon, +} from '@internal/dev-emulators'; +import { Bucket, BucketAccessKey } from 'alchemy/Prisma'; +import * as Effect from 'effect/Effect'; +import { LocalBucketAccessKeyProvider, LocalBucketProvider } from '../bucket.ts'; + +const APP = 'bucket-providers-app'; +const savedEmulatorsDir = process.env[EMULATORS_DIR_ENV]; + +const reconcileInput = (id: string, news: Record) => + ({ + id, + fqn: id, + instanceId: id, + news, + olds: undefined, + output: undefined, + session: undefined as never, + bindings: [], + }) as never; + +// Local dev is not supported on Windows, where the emulators cannot run. +describe.skipIf(process.platform === 'win32')('the local Bucket providers', () => { + let root: string; + let registryRoot: string; + let input: LocalTargetProvidersInput; + + beforeEach(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'local-target-bucket-providers-')); + registryRoot = path.join(root, 'emulators'); + process.env[EMULATORS_DIR_ENV] = registryRoot; + const entry = fileURLToPath(import.meta.resolve('@internal/dev-emulators/buckets-main')); + await ensureDaemon('buckets', entry, { registryRoot }); + const container: ContainerInstance = { + input: { appName: APP, stage: undefined }, + serialize: () => 'x', + }; + input = { container, devDir: path.join(root, 'project', '.prisma-composer', 'dev') }; + }); + + afterEach(async () => { + await stopDaemon('buckets', { registryRoot }).catch(() => undefined); + fs.rmSync(root, { recursive: true, force: true }); + if (savedEmulatorsDir === undefined) delete process.env[EMULATORS_DIR_ENV]; + else process.env[EMULATORS_DIR_ENV] = savedEmulatorsDir; + }); + + async function reconcileBucket(name: string): Promise { + const service = await Effect.runPromise( + Bucket.Provider.pipe(Effect.provide(LocalBucketProvider(input))), + ); + return Effect.runPromise(service.reconcile(reconcileInput(name, { project: 'p', name }))); + } + + test('a Bucket creates a bucket instance once, with its data root in the project', async () => { + const first = await reconcileBucket('uploads'); + const again = await reconcileBucket('uploads'); + + expect(again.bucketId).toBe(first.bucketId); + const instance = await bucketsClient().getInstance(first.bucketId); + expect(instance?.dir).toBe(path.join(input.devDir, 'buckets', APP, 'uploads')); + expect(instance?.labels['composer.address']).toBe('uploads'); + }); + + test("a BucketAccessKey names the bucket's instance as the bucket on the S3 wire", async () => { + const bucket = await reconcileBucket('uploads'); + const service = await Effect.runPromise( + BucketAccessKey.Provider.pipe(Effect.provide(LocalBucketAccessKeyProvider(input))), + ); + + const key: BucketAccessKey['Attributes'] = await Effect.runPromise( + service.reconcile(reconcileInput('key', { bucket, name: 'key' })), + ); + + expect(key.bucketName).toBe(bucket.bucketId); + expect(key.endpoint).toBe(bucketsClient().baseUrl); + }); +}); diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/compute-port-order.test.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/compute-port-order.test.ts index acd214585..9da48adab 100644 --- a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/compute-port-order.test.ts +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/compute-port-order.test.ts @@ -7,8 +7,20 @@ import type { ContainerInstance } from '@internal/core/config'; import { EMULATORS_DIR_ENV, ensureDaemon, stopDaemon } from '@internal/dev-emulators'; import { App } from 'alchemy/Prisma'; import * as Effect from 'effect/Effect'; -import { ensureServiceInstances, LocalAppProvider } from '../compute.ts'; -import { computeNodeInstances, ensureComputeInstance } from '../emulator-instances.ts'; +import { ensureAppInstances } from '../app-instances.ts'; +import { LocalAppProvider } from '../compute.ts'; +import { ensureServiceInstance, nodeInstancesOf } from '../emulator-instances.ts'; + +function ensureServiceInstances( + forContainer: ContainerInstance, + devDir: string, + addresses: readonly string[], +) { + return ensureAppInstances( + { devDir, app: forContainer.input.appName }, + addresses.map((address) => ({ address, kind: 'service' as const })), + ); +} const savedEmulatorsDir = process.env[EMULATORS_DIR_ENV]; let registryRoot: string; @@ -58,7 +70,7 @@ async function reconcileApp(container: ContainerInstance, address: string): Prom } async function reservedPorts(app: string): Promise> { - const nodes = await computeNodeInstances({ devDir, app }); + const nodes = await nodeInstancesOf({ devDir, app }, 'service'); return Object.fromEntries(nodes.map(({ address, instance }) => [address, instance.port])); } @@ -92,8 +104,8 @@ test('a failed reservation names the service', async () => { test('reserving the services of an app that already has ports keeps those ports', async () => { const container = fakeContainer('port-order-warm'); const scope = { devDir, app: 'port-order-warm' }; - const { instance: gateway } = await ensureComputeInstance(scope, 'gateway'); - const { instance: quotes } = await ensureComputeInstance(scope, 'quotes'); + const { instance: gateway } = await ensureServiceInstance(scope, 'gateway'); + const { instance: quotes } = await ensureServiceInstance(scope, 'quotes'); await ensureServiceInstances(container, devDir, ['quotes', 'gateway']); @@ -105,7 +117,7 @@ test('a dotted service address reserves the same emulator instance the App provi const container = fakeContainer('port-order-nested'); await ensureServiceInstances(container, devDir, ['shop.quotes']); - const listed = (await computeNodeInstances({ devDir, app: 'port-order-nested' })).map( + const listed = (await nodeInstancesOf({ devDir, app: 'port-order-nested' }, 'service')).map( (node) => node.instance, ); expect(await reconcileApp(container, 'shop.quotes')).toBe(listed[0]?.port ?? -1); diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/emulator-instances.test.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/emulator-instances.test.ts index 51ee8d033..5b16638e2 100644 --- a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/emulator-instances.test.ts +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/emulator-instances.test.ts @@ -18,13 +18,15 @@ import { CliStructuredError } from '@internal/foundation/errors'; import { packageComputeArtifact } from '@internal/lowering/compute'; import { App, Deployment } from 'alchemy/Prisma'; import * as Effect from 'effect/Effect'; -import { ensureServiceInstances, LocalAppProvider, LocalDeploymentProvider } from '../compute.ts'; +import { ensureAppInstances } from '../app-instances.ts'; +import { LocalAppProvider, LocalDeploymentProvider } from '../compute.ts'; import { - deleteRecordedInstances, + type AppInstances, + deleteAppInstances, EMULATOR_INSTANCES_FILE, - type EmulatorInstances, readEmulatorInstances, - retirePreviousApp, + type ServiceNodeInstance, + selectApp, withServiceAddresses, } from '../emulator-instances.ts'; @@ -116,16 +118,39 @@ async function appProvider(devDir: string) { ); } -function mappingOf(directory: string): Promise { - return readEmulatorInstances(directory); +/** The services' instances, ensured through the record as the emulators hook ensures them. */ +function ensureServiceInstances( + forContainer: ContainerInstance, + devDir: string, + addresses: readonly string[], +) { + return ensureAppInstances( + { devDir, app: forContainer.input.appName }, + addresses.map((address) => ({ address, kind: 'service' as const })), + ); +} + +async function mappingOf(directory: string, app = APP): Promise { + return (await readEmulatorInstances(directory))?.apps[app]; } async function nodeOf(directory: string, address: string) { const node = (await mappingOf(directory))?.nodes[address]; - if (!node) throw new Error(`no instance recorded for "${address}" in ${directory}`); + if (node?.kind !== 'service') { + throw new Error(`no service instance recorded for "${address}" in ${directory}`); + } return node; } +async function serviceNodes(directory: string): Promise> { + const nodes = (await mappingOf(directory))?.nodes ?? {}; + return Object.fromEntries( + Object.entries(nodes).flatMap(([address, node]) => + node.kind === 'service' ? [[address, node]] : [], + ), + ); +} + async function pidOf(id: string): Promise { return (await computeClient().getInstance(id))?.pid; } @@ -198,8 +223,8 @@ describe.skipIf(process.platform === 'win32')('against a real Compute emulator', [a, b].flatMap(({ devDir }) => ADDRESSES.map((address) => deploy(devDir, address, 'v1'))), ); - const nodesA = (await mappingOf(a.directory))?.nodes ?? {}; - const nodesB = (await mappingOf(b.directory))?.nodes ?? {}; + const nodesA = await serviceNodes(a.directory); + const nodesB = await serviceNodes(b.directory); const ids = [...Object.values(nodesA), ...Object.values(nodesB)].map((n) => n.id); const ports = [...Object.values(nodesA), ...Object.values(nodesB)].map((n) => n.port); expect(new Set(ids).size).toBe(4); @@ -251,7 +276,9 @@ describe.skipIf(process.platform === 'win32')('against a real Compute emulator', const after = await nodeOf(directory, 'quotes'); expect(after.port).not.toBe(before.port); - expect(changes).toEqual([{ address: 'quotes', previous: before.port, given: after.port }]); + expect(changes).toEqual([ + { kind: 'service', address: 'quotes', previous: before.port, given: after.port }, + ]); }); test('a warm restart after a stop reuses the same instances and ports', async () => { @@ -305,7 +332,7 @@ describe.skipIf(process.platform === 'win32')('against a real Compute emulator', fs.readFileSync(path.join(devDir, EMULATOR_INSTANCES_FILE), 'utf8'), ); const node = await nodeOf(directory, 'quotes'); - expect(onDisk).toEqual({ app: APP, nodes: { quotes: node } }); + expect(onDisk).toEqual({ current: APP, apps: { [APP]: { nodes: { quotes: node } } } }); expect((await computeClient().getInstance(node.id))?.labels).toEqual({ 'composer.directory': directory, 'composer.app': APP, @@ -314,63 +341,89 @@ describe.skipIf(process.platform === 'win32')('against a real Compute emulator', expect(await readEmulatorInstances(path.join(root, 'never-ran'))).toBeUndefined(); }); + test('a single-app record written by an earlier version reads as that one app, its compute nodes as services', async () => { + const { directory, devDir } = projectDir('legacy'); + fs.mkdirSync(devDir, { recursive: true }); + const quotes = { id: 'svc_0123456789abcdef', port: 3999 }; + fs.writeFileSync( + path.join(devDir, EMULATOR_INSTANCES_FILE), + JSON.stringify({ app: APP, nodes: { quotes: { kind: 'compute', ...quotes } } }), + ); + + expect(await readEmulatorInstances(directory)).toEqual({ + current: APP, + apps: { [APP]: { nodes: { quotes: { kind: 'service', ...quotes } } } }, + }); + }); + + const otherApp: ContainerInstance = { + input: { appName: 'other-app', stage: undefined }, + serialize: () => 'x', + }; + + async function liveIds(ids: readonly string[]): Promise { + const live = await Promise.all(ids.map((id) => computeClient().getInstance(id))); + return live.flatMap((instance) => (instance ? [instance.id] : [])); + } + describe('switching apps in a directory', () => { - test("running another app deletes the previous app's compute instances and starts a new record", async () => { + test("running another app stops the previous app's services and keeps its instances and ports", async () => { const { directory, devDir } = projectDir('switch'); + await ensureServiceInstances(container, devDir, ['quotes']); + await deploy(devDir, 'quotes', 'v1'); + const before = await mappingOf(directory); + + const previous = await selectApp({ devDir, app: 'other-app' }); + + expect(previous).toBe(APP); + expect((await readEmulatorInstances(directory))?.current).toBe('other-app'); + expect(await mappingOf(directory)).toEqual(before); + const quotes = await nodeOf(directory, 'quotes'); + expect(await computeClient().getInstance(quotes.id)).toMatchObject({ + status: 'stopped', + port: quotes.port, + }); + }, 30_000); + + test('switching back reuses the same instances and ports', async () => { + const { directory, devDir } = projectDir('switch-back'); await ensureServiceInstances(container, devDir, ADDRESSES); - const previous = Object.values((await mappingOf(directory))?.nodes ?? {}); + const before = await mappingOf(directory); + await selectApp({ devDir, app: 'other-app' }); + await ensureServiceInstances(otherApp, devDir, ['web']); - const retired = await retirePreviousApp({ devDir, app: 'another-app' }); + expect(await selectApp({ devDir, app: APP })).toBe('other-app'); + const changes = await ensureServiceInstances(container, devDir, ADDRESSES); - expect(retired).toBe(APP); - for (const node of previous) - expect(await computeClient().getInstance(node.id)).toBeUndefined(); - expect(await mappingOf(directory)).toEqual({ app: 'another-app', nodes: {} }); + expect(changes).toEqual([]); + expect(await mappingOf(directory)).toEqual(before); }); - test('running the same app again retires nothing', async () => { - const { directory, devDir } = projectDir('same'); + test('running the same app again changes nothing', async () => { + const { devDir } = projectDir('same'); + await selectApp({ devDir, app: APP }); await ensureServiceInstances(container, devDir, ['quotes']); - const quotes = await nodeOf(directory, 'quotes'); - expect(await retirePreviousApp({ devDir, app: APP })).toBeUndefined(); - expect(await computeClient().getInstance(quotes.id)).toBeDefined(); + expect(await selectApp({ devDir, app: APP })).toBeUndefined(); }); }); describe('--fresh', () => { - const otherApp: ContainerInstance = { - input: { appName: 'other-app', stage: undefined }, - serialize: () => 'x', - }; - - async function liveIds(ids: readonly string[]): Promise { - const live = await Promise.all(ids.map((id) => computeClient().getInstance(id))); - return live.flatMap((instance) => (instance ? [instance.id] : [])); - } - - test('deletes the recorded instances even when the record names the app run here before', async () => { - const { directory, devDir } = projectDir('fresh-previous'); - await ensureServiceInstances(container, devDir, ADDRESSES); - const ids = Object.values((await mappingOf(directory))?.nodes ?? {}).map((n) => n.id); - - await deleteRecordedInstances(devDir); - - expect(await liveIds(ids)).toEqual([]); - expect((await mappingOf(directory))?.nodes).toEqual({}); - }); - - test('after switching apps, leaves no instance of either app', async () => { - const { directory, devDir } = projectDir('fresh-switch'); - await ensureServiceInstances(container, devDir, ADDRESSES); - const first = Object.values((await mappingOf(directory))?.nodes ?? {}).map((n) => n.id); - await retirePreviousApp({ devDir, app: 'other-app' }); + test("deletes the app's instances and entry, and leaves another app's alone", async () => { + const { directory, devDir } = projectDir('fresh'); await ensureServiceInstances(otherApp, devDir, ['web']); - const second = Object.values((await mappingOf(directory))?.nodes ?? {}).map((n) => n.id); + await selectApp({ devDir, app: APP }); + await ensureServiceInstances(container, devDir, ADDRESSES); + const mine = Object.values((await mappingOf(directory))?.nodes ?? {}).map((n) => n.id); + const theirs = await mappingOf(directory, 'other-app'); - await deleteRecordedInstances(devDir); + await deleteAppInstances({ devDir, app: APP }); - expect(await liveIds([...first, ...second])).toEqual([]); + expect(await liveIds(mine)).toEqual([]); + expect(await mappingOf(directory)).toBeUndefined(); + expect(await mappingOf(directory, 'other-app')).toEqual(theirs); + const web = theirs?.nodes['web']; + expect(await liveIds(web ? [web.id] : [])).toEqual(web ? [web.id] : []); }); }); diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/postgres-instance-name-drift.test.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/postgres-instance-name-drift.test.ts deleted file mode 100644 index d90d8f841..000000000 --- a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/postgres-instance-name-drift.test.ts +++ /dev/null @@ -1,194 +0,0 @@ -import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; -import * as fs from 'node:fs'; -import * as path from 'node:path'; -import { fileURLToPath } from 'node:url'; -import type { ContainerInstance, LocalTargetProvidersInput } from '@internal/core/config'; -import { - defaultRegistryRoot, - ensureDaemon, - instanceNameFor, - postgresClient, -} from '@internal/dev-emulators'; -import { Connection, Database } from 'alchemy/Prisma'; -import * as Effect from 'effect/Effect'; -import * as Redacted from 'effect/Redacted'; -import { LocalConnectionProvider, LocalDatabaseProvider } from '../postgres.ts'; - -/** - * Delta review finding A (#160): `LocalDatabaseProvider` used to re-derive - * the instance name with its OWN slug, which dropped the daemon's - * leading/trailing-dash trim — a database id or app name ending in a - * hyphen (the wire protocol's own segment rule already forbids a LEADING - * non-alphanumeric, so a trailing hyphen is the pathological shape that - * actually reaches these providers) produced a DIFFERENT concatenated name - * here than the one the daemon actually created the server under (a - * doubled dash at the boundary), so `LocalConnectionProvider`'s - * listing-based lookup threw `noRecordedInstanceError` even though the - * server existed. Both providers now import `instanceNameFor` directly - * from `@internal/dev-emulators` — one implementation, so this can no - * longer drift. This test proves it end to end, against the real daemon, - * for exactly the pathological shape that used to fail: an app name AND a - * database id each ending in a hyphen. - * - * The providers back upstream alchemy's `Prisma.Database` / - * `Prisma.Connection` classes, so the attribute names asserted here are the - * upstream ones (`databaseId`, `connectionId`, `directConnectionString`). - */ - -const APP = 'pgdrifttestapp-'; -const DATABASE_ID = 'orders-'; - -/** On failure, print the machine-global postgres daemon's log tail — a daemon that died mid-request is otherwise undiagnosable from the test's own connection error. */ -async function withDaemonLogOnFailure(run: () => Promise): Promise { - try { - return await run(); - } catch (error) { - try { - const log = fs.readFileSync(path.join(defaultRegistryRoot(), 'postgres.log'), 'utf8'); - console.error('--- postgres daemon log (tail) ---'); - console.error(log.split('\n').slice(-80).join('\n')); - } catch { - console.error('--- postgres daemon log unavailable ---'); - } - throw error; - } -} - -function fakeContainer(appName: string): ContainerInstance { - return { input: { appName, stage: undefined }, serialize: () => 'x' }; -} - -const reconcileInput = (id: string, news: Record) => - ({ - id, - fqn: id, - instanceId: id, - news, - olds: undefined, - output: undefined, - session: undefined as never, - bindings: [], - }) as never; - -// The default, machine-global daemon (the SAME one `postgresClient()` inside -// the providers under test talks to) — this test's whole point is proving -// the providers agree with the REAL daemon, so it must run against the one -// registry those providers actually resolve, not an isolated test seam. -// `ensureDaemon` adopts an already-healthy daemon, so this is cheap and -// idempotent alongside every other suite that also ensures it. -beforeAll(async () => { - const entry = fileURLToPath(import.meta.resolve('@internal/dev-emulators/postgres-main')); - await ensureDaemon('postgres', entry); -}); - -afterAll(async () => { - await postgresClient() - .deleteApp(APP) - .catch(() => undefined); -}); - -describe('instance-name drift (delta review finding A, #160)', () => { - test( - 'a database id and app name with leading/trailing non-alphanumerics: Database ensures, Connection resolves the SAME server', - () => - withDaemonLogOnFailure(async () => { - const input: LocalTargetProvidersInput = { - container: fakeContainer(APP), - devDir: '/dev/null/unused', - }; - - // 1. Ensure through the real daemon (LocalDatabaseProvider's own reconcile). - const databaseService = await Effect.runPromise( - Database.Provider.pipe(Effect.provide(LocalDatabaseProvider(input))), - ); - const databaseAttributes: Database['Attributes'] = await Effect.runPromise( - databaseService.reconcile( - reconcileInput('db', { project: 'p', name: DATABASE_ID, region: 'us-east-1' }), - ), - ); - - // The provider-derived id is exactly the daemon's own derivation — no - // second implementation to drift from it. - expect(databaseAttributes.databaseId).toBe(instanceNameFor(APP, DATABASE_ID)); - expect(databaseAttributes.databaseId).toBe('pcdev-pgdrifttestapp-orders'); - // Proves the trim/collapse actually happened — the pre-fix drift left a - // doubled dash at the "pgdrifttestapp-" + "-" + "orders-" boundary. - expect(databaseAttributes.databaseId.includes('--')).toBe(false); - - // 2. Connection-resolve through the listing (LocalConnectionProvider's - // own reconcile) — before the fix, this threw noRecordedInstanceError - // for this exact pathological pair. - const connectionService = await Effect.runPromise( - Connection.Provider.pipe(Effect.provide(LocalConnectionProvider(input))), - ); - const connectionAttributes: Connection['Attributes'] = await Effect.runPromise( - connectionService.reconcile( - reconcileInput('conn', { database: databaseAttributes, name: 'conn' }), - ), - ); - - expect(connectionAttributes.databaseId).toBe(databaseAttributes.databaseId); - const direct = connectionAttributes.directConnectionString; - if (direct === undefined) throw new Error('expected a direct connection string'); - expect(Redacted.value(direct)).toMatch(/^postgres:\/\//); - - // 3. The daemon's own listing agrees on the same name too — the third - // independent read of the same value. - const listed = await postgresClient().listDatabases(APP); - const entry = listed.find((d) => d.instanceName === databaseAttributes.databaseId); - expect(entry).toBeDefined(); - }), - 30_000, - ); - - test( - 'a nested module\'s dotted database name ("catalog.database"): Database ensures, Connection resolves the SAME server', - () => - withDaemonLogOnFailure(async () => { - // A nested module's Database resource name is hierarchical and - // dot-separated — but the daemon's `` path segment rule - // (/^[a-z0-9][a-z0-9-]*$/) forbids dots, so the provider must slug the - // daemon-facing id (compute.ts's `slugServiceId` seam, applied to - // postgres). Pre-fix, the client's own `encodeSegment` threw - // `invalid path segment "catalog.database"` before the request ever - // left the process — observed live against examples/store. - const DOTTED_ID = 'catalog.database'; - const input: LocalTargetProvidersInput = { - container: fakeContainer(APP), - devDir: '/dev/null/unused', - }; - - const databaseService = await Effect.runPromise( - Database.Provider.pipe(Effect.provide(LocalDatabaseProvider(input))), - ); - const databaseAttributes: Database['Attributes'] = await Effect.runPromise( - databaseService.reconcile( - reconcileInput('db', { project: 'p', name: DOTTED_ID, region: 'us-east-1' }), - ), - ); - - // slug() is idempotent, so the daemon's instanceNameFor(app, slug(name)) - // equals the provider-recorded instanceNameFor(app, name). - expect(databaseAttributes.databaseId).toBe(instanceNameFor(APP, DOTTED_ID)); - expect(databaseAttributes.databaseId).toBe('pcdev-pgdrifttestapp-catalog-database'); - - const connectionService = await Effect.runPromise( - Connection.Provider.pipe(Effect.provide(LocalConnectionProvider(input))), - ); - const connectionAttributes: Connection['Attributes'] = await Effect.runPromise( - connectionService.reconcile( - reconcileInput('conn', { database: databaseAttributes.databaseId, name: 'conn' }), - ), - ); - - expect(connectionAttributes.databaseId).toBe(databaseAttributes.databaseId); - const direct = connectionAttributes.directConnectionString; - if (direct === undefined) throw new Error('expected a direct connection string'); - expect(Redacted.value(direct)).toMatch(/^postgres:\/\//); - - const listed = await postgresClient().listDatabases(APP); - expect(listed.find((d) => d.instanceName === databaseAttributes.databaseId)).toBeDefined(); - }), - 30_000, - ); -}); diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/postgres-module-resolution.test.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/postgres-module-resolution.test.ts index e1f882cca..4a135859b 100644 --- a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/postgres-module-resolution.test.ts +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/postgres-module-resolution.test.ts @@ -3,7 +3,7 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; import { fileURLToPath } from 'node:url'; -import { resolvePrismaDevModulePath } from '../postgres.ts'; +import { resolvePrismaDevModulePath } from '../prisma-dev-module.ts'; const localTargetDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..', '..'); diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/postgres-providers.test.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/postgres-providers.test.ts new file mode 100644 index 000000000..6db4a7fde --- /dev/null +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/postgres-providers.test.ts @@ -0,0 +1,123 @@ +/** + * The local Database and Connection providers against a real Postgres + * emulator in a temporary emulators directory: a Database is a database + * instance, and its Connection resolves the same instance by its id. + */ +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import type { ContainerInstance, LocalTargetProvidersInput } from '@internal/core/config'; +import { + EMULATORS_DIR_ENV, + ensureDaemon, + postgresClient, + stopDaemon, +} from '@internal/dev-emulators'; +import { Connection, Database } from 'alchemy/Prisma'; +import * as Effect from 'effect/Effect'; +import * as Redacted from 'effect/Redacted'; +import { deleteAppInstances, readEmulatorInstances } from '../emulator-instances.ts'; +import { LocalConnectionProvider, LocalDatabaseProvider } from '../postgres.ts'; + +const APP = 'pg-providers-app'; +const savedEmulatorsDir = process.env[EMULATORS_DIR_ENV]; +let root: string; +let registryRoot: string; +let input: LocalTargetProvidersInput; + +const reconcileInput = (id: string, news: Record) => + ({ + id, + fqn: id, + instanceId: id, + news, + olds: undefined, + output: undefined, + session: undefined as never, + bindings: [], + }) as never; + +async function reconcileDatabase(name: string): Promise { + const service = await Effect.runPromise( + Database.Provider.pipe(Effect.provide(LocalDatabaseProvider(input))), + ); + return Effect.runPromise( + service.reconcile(reconcileInput('db', { project: 'p', name, region: 'us-east-1' })), + ); +} + +// Local dev is not supported on Windows, where the emulators cannot run. +describe.skipIf(process.platform === 'win32')('the local Database and Connection providers', () => { + beforeEach(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'local-target-pg-providers-')); + registryRoot = path.join(root, 'emulators'); + process.env[EMULATORS_DIR_ENV] = registryRoot; + const entry = fileURLToPath(import.meta.resolve('@internal/dev-emulators/postgres-main')); + await ensureDaemon('postgres', entry, { registryRoot }); + const container: ContainerInstance = { + input: { appName: APP, stage: undefined }, + serialize: () => 'x', + }; + input = { container, devDir: path.join(root, 'project', '.prisma-composer', 'dev') }; + }); + + afterEach(async () => { + await deleteAppInstances({ devDir: input.devDir, app: APP }).catch(() => undefined); + await stopDaemon('postgres', { registryRoot }).catch(() => undefined); + fs.rmSync(root, { recursive: true, force: true }); + if (savedEmulatorsDir === undefined) delete process.env[EMULATORS_DIR_ENV]; + else process.env[EMULATORS_DIR_ENV] = savedEmulatorsDir; + }); + + test('a Database creates a database instance, and its Connection resolves the same one', async () => { + const database = await reconcileDatabase('catalog.database'); + + const service = await Effect.runPromise( + Connection.Provider.pipe(Effect.provide(LocalConnectionProvider(input))), + ); + const connection: Connection['Attributes'] = await Effect.runPromise( + service.reconcile(reconcileInput('conn', { database, name: 'conn' })), + ); + + const instance = await postgresClient().getInstance(database.databaseId); + expect(instance?.labels['composer.address']).toBe('catalog.database'); + expect(connection.databaseId).toBe(database.databaseId); + const direct = connection.directConnectionString; + if (direct === undefined) throw new Error('expected a direct connection string'); + expect(Redacted.value(direct)).toBe(instance?.url ?? ''); + }, 60_000); + + test('reconciling a Database again reuses its instance, starting it when stopped', async () => { + const first = await reconcileDatabase('orders'); + await postgresClient().stopInstances([first.databaseId]); + + const second = await reconcileDatabase('orders'); + + expect(second.databaseId).toBe(first.databaseId); + expect((await postgresClient().getInstance(first.databaseId))?.status).toBe('running'); + }, 60_000); + + async function databaseDiff(output: Database['Attributes']): Promise { + const service = await Effect.runPromise( + Database.Provider.pipe(Effect.provide(LocalDatabaseProvider(input))), + ); + const diff = service.diff; + if (!diff) throw new Error('the local Database provider has no diff'); + return Effect.runPromise(diff({ output } as never)); + } + + test('a Database whose recorded instance was re-created asks for an update', async () => { + const database = await reconcileDatabase('orders'); + expect(await databaseDiff(database)).toBeUndefined(); + + await postgresClient().deleteInstance(database.databaseId); + const again = await reconcileDatabase('orders'); + + expect(again.databaseId).not.toBe(database.databaseId); + expect(await databaseDiff(database)).toEqual({ action: 'update' }); + const recorded = (await readEmulatorInstances(path.join(root, 'project')))?.apps[APP]; + expect(recorded?.nodes['orders']).toMatchObject({ kind: 'database', id: again.databaseId }); + }, 90_000); +}); diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/teardown.test.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/teardown.test.ts new file mode 100644 index 000000000..190336aae --- /dev/null +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/__tests__/teardown.test.ts @@ -0,0 +1,44 @@ +/** `--fresh`'s removal of the dev state directory keeps what other apps run from the directory still use. */ +import { expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { EMULATOR_INSTANCES_FILE } from '../emulator-instances.ts'; +import { removeAppDevState } from '../teardown.ts'; + +function filesUnder(dir: string): string[] { + return fs + .readdirSync(dir, { recursive: true, withFileTypes: true }) + .filter((entry) => entry.isFile()) + .map((entry) => path.relative(dir, path.join(entry.parentPath, entry.name))) + .sort(); +} + +test("keeps the record and other apps' bucket objects, and removes the rest", () => { + const devDir = fs.mkdtempSync(path.join(os.tmpdir(), 'local-target-teardown-')); + const files = [ + EMULATOR_INSTANCES_FILE, + 'env.json', + 'secrets.json', + 'alchemy.run.ts', + path.join('artifacts', 'abc123', 'bootstrap.js'), + path.join('buckets', 'shop', 'uploads', 'mine.txt'), + path.join('buckets', 'other', 'uploads', 'theirs.txt'), + ]; + for (const file of files) { + fs.mkdirSync(path.dirname(path.join(devDir, file)), { recursive: true }); + fs.writeFileSync(path.join(devDir, file), 'x'); + } + + removeAppDevState(devDir, 'shop'); + + expect(filesUnder(devDir)).toEqual( + [EMULATOR_INSTANCES_FILE, path.join('buckets', 'other', 'uploads', 'theirs.txt')].sort(), + ); +}); + +test('a dev state directory that does not exist is nothing to remove', () => { + expect(() => + removeAppDevState(path.join(os.tmpdir(), 'never-created-dev-dir'), 'shop'), + ).not.toThrow(); +}); diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/app-instances.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/app-instances.ts new file mode 100644 index 000000000..bfcd7f26c --- /dev/null +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/app-instances.ts @@ -0,0 +1,60 @@ +/** + * The emulators hook's half of the directory's record: before Alchemy runs, + * every node of the app gets its emulator instance, one at a time in the + * order given. + */ +import { bucketDataDir } from './bucket.ts'; +import { + type AppScope, + ensureBucketInstance, + ensureDatabaseInstance, + ensureServiceInstance, + type NodeKind, + type PortChange, +} from './emulator-instances.ts'; +import { resolvePrismaDevModulePath } from './prisma-dev-module.ts'; + +export interface AppNode { + readonly address: string; + readonly kind: NodeKind; +} + +/** + * Ensures each node's instance, in the order given: a recorded instance is + * reused (a stopped database is started), a missing one is created again on + * its remembered port, and a new node gets a new instance. A new instance + * gets the smallest free port, so on a fresh start the ports follow this + * order. Returns the nodes whose re-created instance could not get its + * remembered port back. + */ +export async function ensureAppInstances( + scope: AppScope, + nodes: readonly AppNode[], +): Promise { + const changes: PortChange[] = []; + for (const { address, kind } of nodes) { + try { + const { portChange } = await ensureNodeInstance(scope, address, kind); + if (portChange) changes.push(portChange); + } catch (cause) { + const reason = cause instanceof Error ? cause.message : String(cause); + throw new Error(`ensuring the local instance for ${kind} "${address}" failed: ${reason}`, { + cause, + }); + } + } + return changes; +} + +async function ensureNodeInstance( + scope: AppScope, + address: string, + kind: NodeKind, +): Promise<{ readonly portChange?: PortChange }> { + if (kind === 'service') return ensureServiceInstance(scope, address); + if (kind === 'database') { + return ensureDatabaseInstance(scope, address, resolvePrismaDevModulePath()); + } + await ensureBucketInstance(scope, address, bucketDataDir(scope, address)); + return {}; +} diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/bucket.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/bucket.ts index 9365738bb..a62643eb4 100644 --- a/packages/1-prisma-cloud/0-lowering/local-target/src/bucket.ts +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/bucket.ts @@ -15,12 +15,14 @@ import type * as Layer from 'effect/Layer'; import * as Predicate from 'effect/Predicate'; import * as Redacted from 'effect/Redacted'; import { appNameOf } from './app-name.ts'; +import { + type AppScope, + ensureBucketInstance, + recordedInstanceId, + recordKeyOf, +} from './emulator-instances.ts'; import { DEV_TIMESTAMP, projectIdOfInput } from './upstream-attributes.ts'; -/** Upstream lets `name` default to the resource's logical ID; locally a bucket must have a concrete name to register under. */ -const bucketNameOf = (news: { readonly name?: string | undefined }, id: string): string => - news.name ?? id; - /** Reads a bucket id from upstream's `bucket` input: a plain string or a resolved `Prisma.Bucket` attributes record. */ function bucketIdOfInput(value: unknown): string { if (typeof value === 'string') return value; @@ -30,7 +32,24 @@ function bucketIdOfInput(value: unknown): string { return 'local'; } -/** `Bucket` → registers `--` with the bucket emulator, backed by an in-project data root. */ +export const BUCKETS_DIR = 'buckets'; + +/** Where a bucket's objects live: in the project's dev directory, under its app, so `--fresh` removes them with the app. */ +export function bucketDataDir(scope: AppScope, name: string): string { + return path.join(scope.devDir, BUCKETS_DIR, scope.app, name); +} + +function appScopeOf(input: LocalTargetProvidersInput): AppScope { + return { devDir: input.devDir, app: appNameOf(input.container) }; +} + +/** + * `Bucket` → the bucket instance the directory's record names for the + * resource, created when there is none, over an in-project data root. Its + * `bucketId` is the instance id, which is also its S3 bucket name. A Bucket + * whose recorded instance changed is updated, so its access key is + * registered on the new instance. + */ export function LocalBucketProvider( input: LocalTargetProvidersInput, ): Layer.Layer> { @@ -39,12 +58,14 @@ export function LocalBucketProvider( reconcile: ({ news, id }) => Effect.tryPromise({ try: async () => { - const app = appNameOf(input.container); - const name = bucketNameOf(news, id); - const dir = path.join(input.devDir, 'buckets', name); - await bucketsClient().putBucket(app, name, dir); + const name = recordKeyOf('bucket', id, news.name); + const bucket = await ensureBucketInstance( + appScopeOf(input), + name, + bucketDataDir(appScopeOf(input), name), + ); return { - bucketId: name, + bucketId: bucket.id, name, projectId: projectIdOfInput(news.project), createdAt: DEV_TIMESTAMP, @@ -53,6 +74,15 @@ export function LocalBucketProvider( }, catch: (cause) => cause, }), + diff: ({ output }) => + Effect.tryPromise({ + try: async () => { + if (output === undefined) return undefined; + const recorded = await recordedInstanceId(appScopeOf(input), output.name); + return recorded === output.bucketId ? undefined : { action: 'update' as const }; + }, + catch: (cause) => cause, + }), // Objects belong to the developer; only `--fresh` removes them. delete: () => Effect.void, read: ({ output }) => Effect.succeed(output), @@ -68,14 +98,13 @@ export function LocalBucketProvider( * next converge. */ export function LocalBucketAccessKeyProvider( - input: LocalTargetProvidersInput, + _input: LocalTargetProvidersInput, ): Layer.Layer> { const service: Provider.ProviderService = { list: () => Effect.succeed([]), reconcile: ({ news, output, id }) => Effect.tryPromise({ try: async () => { - const app = appNameOf(input.container); const pair = output !== undefined ? { @@ -84,15 +113,15 @@ export function LocalBucketAccessKeyProvider( } : mintKeyPair(); const client = bucketsClient(); - await client.putCredentials(app, pair.accessKeyId, pair.secretAccessKey); const bucketId = bucketIdOfInput(news.bucket); + await client.putCredentials(bucketId, pair.accessKeyId, pair.secretAccessKey); return { bucketAccessKeyId: news.name ?? id, bucketId, accessKeyId: pair.accessKeyId, secretAccessKey: Redacted.make(pair.secretAccessKey), endpoint: client.baseUrl, - bucketName: `${app}--${bucketId}`, + bucketName: bucketId, }; }, catch: (cause) => cause, diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/compute.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/compute.ts index aa8c0cf8d..af3705d20 100644 --- a/packages/1-prisma-cloud/0-lowering/local-target/src/compute.ts +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/compute.ts @@ -10,7 +10,7 @@ import * as crypto from 'node:crypto'; import * as fs from 'node:fs'; import * as path from 'node:path'; -import type { ContainerInstance, LocalTargetProvidersInput } from '@internal/core/config'; +import type { LocalTargetProvidersInput } from '@internal/core/config'; import { computeClient } from '@internal/dev-emulators'; import { App, Deployment, EnvironmentVariable, Project } from 'alchemy/Prisma'; import * as Provider from 'alchemy/Provider'; @@ -23,10 +23,9 @@ import { extractComputeArtifact } from './artifact-extract.ts'; import { envStore, secretsStore } from './dev-store.ts'; import { type AppScope, - deleteComputeInstance, - ensureComputeInstance, - liveComputeInstance, - type PortChange, + deleteNodeInstance, + ensureServiceInstance, + liveServiceInstance, withServiceAddresses, } from './emulator-instances.ts'; import { DEV_TIMESTAMP, projectIdOfInput } from './upstream-attributes.ts'; @@ -200,7 +199,7 @@ export function LocalAppProvider( Effect.tryPromise({ try: async () => { const name = news.displayName ?? id; - const { instance } = await ensureComputeInstance(appScopeOf(input), name); + const { instance } = await ensureServiceInstance(appScopeOf(input), name); const url = localUrl(instance.port); return { appId: name, @@ -220,7 +219,7 @@ export function LocalAppProvider( Effect.tryPromise({ try: async () => { if (output === undefined) return undefined; - const live = await liveComputeInstance(appScopeOf(input), output.name); + const live = await liveServiceInstance(appScopeOf(input), output.name); if (live?.url === output.appEndpointDomain) return undefined; return { action: 'update' as const }; }, @@ -228,7 +227,7 @@ export function LocalAppProvider( }), delete: ({ output }) => Effect.tryPromise({ - try: () => deleteComputeInstance(appScopeOf(input), output.name), + try: () => deleteNodeInstance(appScopeOf(input), output.name), catch: (cause) => cause, }), read: ({ output }) => Effect.succeed(output), @@ -236,35 +235,6 @@ export function LocalAppProvider( return Provider.effect(App, Effect.succeed(service)); } -/** - * Ensures each service's emulator instance one at a time, in the order given, - * before Alchemy applies the `App` resources concurrently. A new instance gets - * the smallest free port, so on a fresh start the ports follow this order; a - * service that already has an instance keeps its port. `serviceAppNames` are - * the `App` resources' `displayName`s. Returns the services whose instance had - * to be re-created and could not get its remembered port back. - */ -export async function ensureServiceInstances( - container: ContainerInstance | undefined, - devDir: string, - serviceAppNames: readonly string[], -): Promise { - const scope = { devDir, app: appNameOf(container) }; - const changes: PortChange[] = []; - for (const service of serviceAppNames) { - try { - const { portChange } = await ensureComputeInstance(scope, service); - if (portChange) changes.push(portChange); - } catch (cause) { - const reason = cause instanceof Error ? cause.message : String(cause); - throw new Error(`ensuring the local service for service "${service}" failed: ${reason}`, { - cause, - }); - } - } - return changes; -} - /** `Prisma.EnvironmentVariable` → a key/value row in `/env.json`. Upstream's value is `Redacted`; env.json holds the plain string the child process is given. */ export function LocalEnvironmentVariableProvider( input: LocalTargetProvidersInput, @@ -337,7 +307,7 @@ export function LocalDeploymentProvider( } const { address, listens } = readComputeManifest(artifactDir); - const { instance } = await ensureComputeInstance(appScopeOf(input), appId); + const { instance } = await ensureServiceInstance(appScopeOf(input), appId); const env = await materializeEnv(input.devDir, address, instance.port); try { await computeClient().putDeployment(instance.id, { @@ -374,7 +344,7 @@ export function LocalDeploymentProvider( Effect.tryPromise({ try: async () => { if (output === undefined) return undefined; - const live = await liveComputeInstance(appScopeOf(input), output.appId); + const live = await liveServiceInstance(appScopeOf(input), output.appId); if (live?.artifactHash === output.artifactHash) return undefined; return { action: 'update' as const }; }, diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/emulator-instances.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/emulator-instances.ts index f16d8bb66..15dfb7693 100644 --- a/packages/1-prisma-cloud/0-lowering/local-target/src/emulator-instances.ts +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/emulator-instances.ts @@ -1,5 +1,5 @@ /** - * Which emulator instance backs each node of the app run from a directory. + * Which emulator instance backs each node of each app run from a directory. * `prisma dev` records it in `/.prisma-composer/dev/`, and every * Composer operation on an emulator instance goes through it. The emulators * know nothing about Composer: the labels written on create are for people @@ -7,40 +7,106 @@ */ import * as path from 'node:path'; import { DEV_DIR } from '@internal/core/config'; -import { computeClient, type InstanceInfo } from '@internal/dev-emulators'; +import { + type BucketInstanceInfo, + bucketsClient, + computeClient, + type DatabaseInstanceInfo, + postgresClient, + type ServiceInstanceInfo, +} from '@internal/dev-emulators'; import { createStore, type DevStore } from './dev-store.ts'; +import { resolvePrismaDevModulePath } from './prisma-dev-module.ts'; export const EMULATOR_INSTANCES_FILE = 'emulator-instances.json'; -export interface ComputeNodeInstance { - readonly kind: 'compute'; +export interface ServiceNodeInstance { + readonly kind: 'service'; readonly id: string; /** The port the instance was given, requested again if the instance has to be re-created. */ readonly port: number; } -export type NodeInstance = ComputeNodeInstance; +export interface DatabaseNodeInstance { + readonly kind: 'database'; + readonly id: string; + /** The port the database was given, requested again if it has to be re-created. */ + readonly port: number; +} -export interface EmulatorInstances { - readonly app: string; +export interface BucketNodeInstance { + readonly kind: 'bucket'; + readonly id: string; +} + +export type NodeInstance = ServiceNodeInstance | DatabaseNodeInstance | BucketNodeInstance; +export type NodeKind = NodeInstance['kind']; + +export interface AppInstances { /** Keyed by node address. */ readonly nodes: Readonly>; } +export interface EmulatorInstances { + /** The app `prisma dev` last ran from the directory. */ + readonly current: string; + /** Every app run from the directory, keyed by app name. */ + readonly apps: Readonly>; +} + /** The app run from the directory that holds `devDir` (`/.prisma-composer/dev`). */ export interface AppScope { readonly devDir: string; readonly app: string; } -/** A service whose remembered port another instance or process now holds. */ +/** A node whose remembered port another instance or process now holds. */ export interface PortChange { + readonly kind: 'service' | 'database'; readonly address: string; readonly previous: number; readonly given: number; } function isNodeInstance(value: unknown): value is NodeInstance { + if (typeof value !== 'object' || value === null) return false; + if (!('id' in value) || typeof value.id !== 'string' || !('kind' in value)) return false; + if (value.kind === 'bucket') return true; + return ( + (value.kind === 'service' || value.kind === 'database') && + 'port' in value && + typeof value.port === 'number' + ); +} + +function isNodes(value: unknown): value is Record { + return typeof value === 'object' && value !== null && Object.values(value).every(isNodeInstance); +} + +function isAppInstances(value: unknown): value is AppInstances { + return typeof value === 'object' && value !== null && 'nodes' in value && isNodes(value.nodes); +} + +function isEmulatorInstances(value: unknown): value is EmulatorInstances { + return ( + typeof value === 'object' && + value !== null && + 'current' in value && + typeof value.current === 'string' && + 'apps' in value && + typeof value.apps === 'object' && + value.apps !== null && + Object.values(value.apps).every(isAppInstances) + ); +} + +/** The single-app record an earlier version wrote: one app and its services, whose kind it called `compute`. */ +interface SingleAppRecord { + readonly app: string; + readonly nodes: Readonly>; +} + +function isEarlierServiceNode(value: unknown): value is { id: string; port: number } { return ( typeof value === 'object' && value !== null && @@ -53,7 +119,7 @@ function isNodeInstance(value: unknown): value is NodeInstance { ); } -function isEmulatorInstances(value: unknown): value is EmulatorInstances { +function isSingleAppRecord(value: unknown): value is SingleAppRecord { return ( typeof value === 'object' && value !== null && @@ -62,71 +128,178 @@ function isEmulatorInstances(value: unknown): value is EmulatorInstances { 'nodes' in value && typeof value.nodes === 'object' && value.nodes !== null && - Object.values(value.nodes).every(isNodeInstance) + Object.values(value.nodes).every(isEarlierServiceNode) + ); +} + +type StoredRecord = EmulatorInstances | SingleAppRecord; + +function isStoredRecord(value: unknown): value is StoredRecord { + return isEmulatorInstances(value) || isSingleAppRecord(value); +} + +/** A single-app record reads as a record holding that one app; the next write rewrites it. */ +function normalized(stored: StoredRecord | undefined): EmulatorInstances | undefined { + if (stored === undefined || isEmulatorInstances(stored)) return stored; + const nodes = Object.fromEntries( + Object.entries(stored.nodes).map(([address, { id, port }]) => [ + address, + { kind: 'service' as const, id, port }, + ]), ); + return { current: stored.app, apps: { [stored.app]: { nodes } } }; } -function directoryOf(devDir: string): string { - const normalized = path.normalize(devDir); - return normalized.endsWith(DEV_DIR) ? normalized.slice(0, -DEV_DIR.length - 1) : normalized; +/** + * A database's or bucket's key in the record: its resource `name`, which the + * lowering sets from the same helper the emulators hook keys it by. A + * resource without a name cannot be matched to the instance the hook + * recorded, so it is refused rather than recorded under another key. + */ +export function recordKeyOf( + kind: 'database' | 'bucket', + resourceId: string, + name: string | undefined, +): string { + if (name === undefined) { + throw new Error( + `local dev: the ${kind} resource "${resourceId}" has no name, so its emulator instance cannot be found in the directory's record.`, + ); + } + return name; +} + +/** The project directory that holds `devDir` (`/.prisma-composer/dev`). */ +export function directoryOf(devDir: string): string { + const normalizedDir = path.normalize(devDir); + return normalizedDir.endsWith(DEV_DIR) + ? normalizedDir.slice(0, -DEV_DIR.length - 1) + : normalizedDir; +} + +function store(devDir: string): DevStore { + return createStore(path.join(devDir, EMULATOR_INSTANCES_FILE), isStoredRecord, undefined); } -function store(devDir: string): DevStore { - return createStore(path.join(devDir, EMULATOR_INSTANCES_FILE), isEmulatorInstances, undefined); +async function readRecord(devDir: string): Promise { + return normalized(await store(devDir).read()); +} + +async function updateRecord( + devDir: string, + mutate: (record: EmulatorInstances | undefined) => EmulatorInstances, +): Promise { + await store(devDir).update((stored) => mutate(normalized(stored))); } /** The record `prisma dev` left in `directory`; undefined when it has not run there. */ export function readEmulatorInstances(directory: string): Promise { - return store(path.join(directory, DEV_DIR)).read(); + return readRecord(path.join(directory, DEV_DIR)); } -/** This app's nodes. A record left by another app run from the same directory counts as empty. */ async function appNodes(scope: AppScope): Promise>> { - const current = await store(scope.devDir).read(); - return current?.app === scope.app ? current.nodes : {}; + return (await readRecord(scope.devDir))?.apps[scope.app]?.nodes ?? {}; } async function updateNodes( scope: AppScope, mutate: (nodes: Record) => void, ): Promise { - await store(scope.devDir).update((current) => { - const nodes = { ...(current?.app === scope.app ? current.nodes : {}) }; + await updateRecord(scope.devDir, (record) => { + const nodes = { ...(record?.apps[scope.app]?.nodes ?? {}) }; mutate(nodes); - return { app: scope.app, nodes }; + return { + current: record?.current ?? scope.app, + apps: { ...record?.apps, [scope.app]: { nodes } }, + }; }); } +function labelsFor(scope: AppScope, address: string): Record { + return { + 'composer.directory': directoryOf(scope.devDir), + 'composer.app': scope.app, + 'composer.address': address, + }; +} + +function idsOf(nodes: Readonly>, kind: NodeKind): string[] { + return Object.values(nodes).flatMap((node) => (node.kind === kind ? [node.id] : [])); +} + /** - * A directory has one dev instance. When the record names another app, that - * app's compute instances are deleted and the record starts over for this - * one. Returns the previous app's name when it was retired. + * Stops the service processes and database servers of `nodes`, keeping + * their data and ports. A kind whose emulator is not running has nothing + * running to stop. */ -export async function retirePreviousApp(scope: AppScope): Promise { - const current = await store(scope.devDir).read(); - if (current === undefined || current.app === scope.app) return undefined; - const client = computeClient(); - for (const instance of Object.values(current.nodes)) await client.deleteInstance(instance.id); - await store(scope.devDir).update(() => ({ app: scope.app, nodes: {} })); - return current.app; +async function stopNodes(nodes: Readonly>): Promise { + const services = idsOf(nodes, 'service'); + const databases = idsOf(nodes, 'database'); + // Both are attempted, so a failed service stop never leaves the database + // servers running; the first failure is reported after both. + const results = await Promise.allSettled([ + services.length > 0 ? whenRunning(() => computeClient().stopInstances(services)) : undefined, + databases.length > 0 ? whenRunning(() => postgresClient().stopInstances(databases)) : undefined, + ]); + const failure = results.find((result) => result.status === 'rejected'); + if (failure) throw failure.reason; } -/** Each compute node of the app, with the instance that backs it. */ -export async function computeNodeInstances( - scope: AppScope, -): Promise<{ readonly address: string; readonly instance: ComputeNodeInstance }[]> { - return Object.entries(await appNodes(scope)).map(([address, instance]) => ({ - address, - instance, +/** Runs `action` against an emulator, treating "the emulator is not running" as nothing to do. */ +async function whenRunning(action: () => Promise): Promise { + let run: Promise; + try { + run = action(); + } catch { + return; + } + await run; +} + +/** + * Makes `scope.app` the directory's current app. When another app ran here + * last, its service processes and database servers are stopped and its + * instances stay recorded, so running it again resumes them with their data + * and ports. Returns the previous app's name when it changed. + */ +export async function selectApp(scope: AppScope): Promise { + const record = await readRecord(scope.devDir); + const previous = record?.current; + if (previous !== undefined && previous !== scope.app) { + await stopNodes(record?.apps[previous]?.nodes ?? {}); + } + await updateRecord(scope.devDir, (latest) => ({ + current: scope.app, + apps: { ...latest?.apps, [scope.app]: latest?.apps[scope.app] ?? { nodes: {} } }, })); + return previous !== undefined && previous !== scope.app ? previous : undefined; +} + +/** Each node of the app of `kind`, with the instance that backs it. */ +export async function nodeInstancesOf( + scope: AppScope, + kind: K, +): Promise<{ readonly address: string; readonly instance: Extract }[]> { + return Object.entries(await appNodes(scope)).flatMap(([address, instance]) => + isOfKind(instance, kind) ? [{ address, instance }] : [], + ); } -/** The compute instance recorded for `address`; undefined when there is none. */ -export async function mappedComputeInstance( +function isOfKind( + instance: NodeInstance, + kind: K, +): instance is Extract { + return instance.kind === kind; +} + +/** The instance of `kind` recorded for `address`; undefined when there is none. */ +async function recordedInstance( scope: AppScope, address: string, -): Promise { - return (await appNodes(scope))[address]; + kind: K, +): Promise | undefined> { + const recorded = (await appNodes(scope))[address]; + return recorded !== undefined && isOfKind(recorded, kind) ? recorded : undefined; } /** @@ -134,68 +307,152 @@ export async function mappedComputeInstance( * reused while the Compute emulator still has it. Otherwise a new one is * created, requesting the recorded port, and recorded. */ -export async function ensureComputeInstance( +export async function ensureServiceInstance( scope: AppScope, address: string, -): Promise<{ readonly instance: ComputeNodeInstance; readonly portChange?: PortChange }> { +): Promise<{ readonly instance: ServiceNodeInstance; readonly portChange?: PortChange }> { const client = computeClient(); - const recorded = await mappedComputeInstance(scope, address); + const recorded = await recordedInstance(scope, address, 'service'); if (recorded && (await client.getInstance(recorded.id))) return { instance: recorded }; const created = await client.createInstance({ - labels: { - 'composer.directory': directoryOf(scope.devDir), - 'composer.app': scope.app, - 'composer.address': address, - }, + labels: labelsFor(scope, address), ...(recorded ? { port: recorded.port } : {}), }); - const instance: ComputeNodeInstance = { kind: 'compute', id: created.id, port: created.port }; + const instance: ServiceNodeInstance = { kind: 'service', id: created.id, port: created.port }; await updateNodes(scope, (nodes) => { nodes[address] = instance; }); - if (recorded && recorded.port !== created.port) { - return { instance, portChange: { address, previous: recorded.port, given: created.port } }; + return { instance, ...portChangeOf('service', address, recorded?.port, created.port) }; +} + +function portChangeOf( + kind: PortChange['kind'], + address: string, + previous: number | undefined, + given: number, +): { readonly portChange?: PortChange } { + return previous !== undefined && previous !== given + ? { portChange: { kind, address, previous, given } } + : {}; +} + +/** + * The running database for `address`. A recorded database the Postgres + * emulator still has is started if it is stopped. Otherwise a new one is + * created, requesting the recorded port, and recorded; it starts empty. + */ +export async function ensureDatabaseInstance( + scope: AppScope, + address: string, + prismaDevModulePath: string, +): Promise<{ readonly instance: DatabaseInstanceInfo; readonly portChange?: PortChange }> { + const client = postgresClient(); + const recorded = await recordedInstance(scope, address, 'database'); + const existing = recorded ? await client.getInstance(recorded.id) : undefined; + if (existing?.status === 'running') return { instance: existing }; + if (existing) { + await client.startInstances([existing.id], prismaDevModulePath); + return { instance: (await client.getInstance(existing.id)) ?? existing }; } - return { instance }; + + const created = await client.createInstance({ + labels: labelsFor(scope, address), + ...(recorded ? { port: recorded.port } : {}), + prismaDevModulePath, + }); + await updateNodes(scope, (nodes) => { + nodes[address] = { kind: 'database', id: created.id, port: created.port }; + }); + return { instance: created, ...portChangeOf('database', address, recorded?.port, created.port) }; +} + +/** + * The bucket for `address`. The recorded bucket is reused while the bucket + * emulator still has it. Otherwise a new one is created over `dir` and + * recorded; the objects already in `dir` are its objects. + */ +export async function ensureBucketInstance( + scope: AppScope, + address: string, + dir: string, +): Promise { + const client = bucketsClient(); + const recorded = await recordedInstance(scope, address, 'bucket'); + const existing = recorded ? await client.getInstance(recorded.id) : undefined; + if (existing) return existing; + + const created = await client.createInstance({ labels: labelsFor(scope, address), dir }); + await updateNodes(scope, (nodes) => { + nodes[address] = { kind: 'bucket', id: created.id }; + }); + return created; } -/** The emulator's view of the instance recorded for `address`; undefined when either is missing. */ -export async function liveComputeInstance( +/** The Compute emulator's view of the instance recorded for `address`; undefined when either is missing. */ +export async function liveServiceInstance( scope: AppScope, address: string, -): Promise { - const recorded = await mappedComputeInstance(scope, address); +): Promise { + const recorded = await recordedInstance(scope, address, 'service'); return recorded ? computeClient().getInstance(recorded.id) : undefined; } +/** The id recorded for `address`; undefined when there is none. */ +export async function recordedInstanceId( + scope: AppScope, + address: string, +): Promise { + return (await appNodes(scope))[address]?.id; +} + +async function deleteInstance(instance: NodeInstance): Promise { + if (instance.kind === 'service') await computeClient().deleteInstance(instance.id); + else if (instance.kind === 'database') { + // The caller's `@prisma/dev`: the one the database was created with may + // since have been removed by an upgrade. + await postgresClient().deleteInstance(instance.id, resolvePrismaDevModulePath()); + } else await bucketsClient().deleteInstance(instance.id); +} + /** Deletes the instance recorded for `address` and forgets it. */ -export async function deleteComputeInstance(scope: AppScope, address: string): Promise { - const recorded = await mappedComputeInstance(scope, address); +export async function deleteNodeInstance(scope: AppScope, address: string): Promise { + const recorded = (await appNodes(scope))[address]; if (!recorded) return; - await computeClient().deleteInstance(recorded.id); + await deleteInstance(recorded); await updateNodes(scope, (nodes) => { delete nodes[address]; }); } +/** The kinds of instance recorded for the app, so a caller can start their emulators first. */ +export async function recordedKinds(scope: AppScope): Promise> { + return new Set(Object.values(await appNodes(scope)).map((node) => node.kind)); +} + /** - * `--fresh`: deletes every instance the directory's record names, whichever - * app wrote it, then empties the record. Teardown runs before the emulators - * hook, so the record may still name the app run here before. + * `--fresh`: deletes the app's recorded instances of every kind and its + * entry. Other apps' entries are left alone. A bucket's objects live in the + * dev directory, which `--fresh` removes separately. */ -export async function deleteRecordedInstances(devDir: string): Promise { - const current = await store(devDir).read(); - if (current === undefined) return; - const client = computeClient(); - for (const instance of Object.values(current.nodes)) await client.deleteInstance(instance.id); - await store(devDir).update(() => ({ app: current.app, nodes: {} })); +export async function deleteAppInstances(scope: AppScope): Promise { + for (const instance of Object.values(await appNodes(scope))) await deleteInstance(instance); + await updateRecord(scope.devDir, (record) => { + const apps = { ...record?.apps }; + delete apps[scope.app]; + return { current: record?.current ?? scope.app, apps }; + }); +} + +/** When `prisma dev` ends: stops the app's service processes and database servers, keeping their data and ports. */ +export async function stopAppInstances(scope: AppScope): Promise { + await stopNodes(await appNodes(scope)); } /** - * The Compute emulator's failure messages name instances by id; `prisma dev` - * names services by address. Rewrites a plain `Error`'s message and leaves - * any other error, such as a structured one, untouched. + * The emulators' failure messages name instances by id; `prisma dev` names + * services by address. Rewrites a plain `Error`'s message and leaves any + * other error, such as a structured one, untouched. */ export function withServiceAddresses( error: unknown, diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/exports/index.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/exports/index.ts index 5d7f97127..ed99e85c3 100644 --- a/packages/1-prisma-cloud/0-lowering/local-target/src/exports/index.ts +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/exports/index.ts @@ -1,10 +1,13 @@ /** `@internal/local-target`'s public surface (local-dev spec § 4): the local deploy target's provider suite — the dev provider bundle plus the dev-instance store and shared plumbing. Implementation sits beside this barrel at `src/`. */ + +export * from '../app-instances.ts'; export * from '../app-name.ts'; export * from '../bucket.ts'; export * from '../compute.ts'; export * from '../dev-store.ts'; export * from '../emulator-instances.ts'; export * from '../postgres.ts'; +export * from '../prisma-dev-module.ts'; export * from '../providers.ts'; export * from '../resolve-package-entry.ts'; export * from '../teardown.ts'; diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/postgres.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/postgres.ts index 40336ac32..d4f39a39f 100644 --- a/packages/1-prisma-cloud/0-lowering/local-target/src/postgres.ts +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/postgres.ts @@ -1,17 +1,15 @@ /** * Local postgres-cluster providers: upstream alchemy's `Prisma.Database` and * `Prisma.Connection` become clients of the `postgres-main` emulator daemon - * (one named, persistent `@prisma/dev` server per `Database` resource). + * (one `@prisma/dev` server per database instance). * `PgWarm`/`OrmMigration` are not here — the hosted ones run against whatever * URL they are handed. Attributes match upstream's shapes; the daemon's * DIRECT connection string maps to `directConnectionString` and - * `databaseUrl`, everything else is left absent. Instance names come from - * `@internal/dev-emulators`' own `instanceNameFor` — a locally re-derived - * slug drifted from the daemon's and broke `Connection`'s lookup. + * `databaseUrl`, everything else is left absent. A Database's `databaseId` + * is its emulator instance's id. */ -import { createRequire } from 'node:module'; import type { LocalTargetProvidersInput } from '@internal/core/config'; -import { instanceNameFor, postgresClient, slug } from '@internal/dev-emulators'; +import { postgresClient } from '@internal/dev-emulators'; import * as Prisma from 'alchemy/Prisma'; import * as Provider from 'alchemy/Provider'; import * as Effect from 'effect/Effect'; @@ -19,6 +17,13 @@ import type * as Layer from 'effect/Layer'; import * as Predicate from 'effect/Predicate'; import * as Redacted from 'effect/Redacted'; import { appNameOf } from './app-name.ts'; +import { + type AppScope, + ensureDatabaseInstance, + recordedInstanceId, + recordKeyOf, +} from './emulator-instances.ts'; +import { resolvePrismaDevModulePath } from './prisma-dev-module.ts'; import { DEV_TIMESTAMP, projectIdOfInput } from './upstream-attributes.ts'; /** Reads a database id from upstream's `database` input: a plain string or a resolved `Prisma.Database` attributes record. */ @@ -29,32 +34,12 @@ function databaseIdOfInput(value: unknown): string | undefined { return undefined; } -function noPrismaDevError(cause: unknown): Error { - return new Error( - `local dev needs @prisma/dev for its local Postgres emulator — @prisma/composer-prisma-cloud declares it as a dependency, but it did not resolve from Composer's own installation; reinstall your dependencies. (${cause instanceof Error ? cause.message : String(cause)})`, - ); -} - -/** - * Composer owns the local Postgres emulator's version: `@prisma/dev` is a - * dependency of `@prisma/composer-prisma-cloud` and is resolved from - * Composer's own package, never from the app. The app's copy (if any) is - * ignored on purpose — an old `@prisma/dev` pulled in transitively (alchemy - * pins `^0.20.0`, whose pglite-socket crashes on any message over 64 KiB) - * would otherwise silently replace the version Composer tested against. - * The daemon imports the returned path dynamically. - */ -export function resolvePrismaDevModulePath(): string { - try { - return createRequire(import.meta.url).resolve('@prisma/dev'); - } catch (cause) { - throw noPrismaDevError(cause); - } -} - /** - * `Prisma.Database` → an ensured `postgres-main` server, one per resource. - * Stores the daemon's returned url as the `directConnectionString` attribute. + * `Prisma.Database` → the database instance the directory's record names for + * the resource, started if it is stopped and created when there is none. + * Stores its url as the `directConnectionString` attribute. A Database whose + * recorded instance changed (re-created after the emulator lost it) is + * updated, so its Connection and dependents get the new instance's URL. */ export function LocalDatabaseProvider( input: LocalTargetProvidersInput, @@ -64,30 +49,16 @@ export function LocalDatabaseProvider( reconcile: ({ id, news }) => Effect.tryPromise({ try: async () => { - const app = appNameOf(input.container); - // Hosted branch-stage deploys omit the display name (see - // descriptors/postgres.ts); local dev never has a branch, so - // `news.name` is normally present — the resource's logical id is - // only a defensive fallback. - const name = news.name ?? id; - const prismaDevModulePath = resolvePrismaDevModulePath(); - // The daemon's `` path segment must match - // /^[a-z0-9][a-z0-9-]*$/ (spec § 2's API hygiene rule) — but a - // Database resource's name is hierarchical and dot-separated for - // a nested module (e.g. "catalog.database"). Same seam as - // compute.ts's `slugServiceId`: the daemon-facing id is the slug; - // `slug` is idempotent, so the daemon's own - // `instanceNameFor(app, slug(name))` equals - // `instanceNameFor(app, name)` — the very name the attributes - // below record and `Connection` looks up. - const { url } = await postgresClient().ensureDatabase( - app, - slug(name), - prismaDevModulePath, + // Hosted branch-stage deploys omit the name; local dev has no branch. + const name = recordKeyOf('database', id, news.name); + const { instance } = await ensureDatabaseInstance( + appScopeOf(input), + name, + resolvePrismaDevModulePath(), ); - const direct = Redacted.make(url); + const direct = Redacted.make(instance.url); return { - databaseId: instanceNameFor(app, name), + databaseId: instance.id, databaseName: name, projectId: projectIdOfInput(news.project), status: 'ready', @@ -107,11 +78,24 @@ export function LocalDatabaseProvider( }, catch: (cause) => cause, }), + diff: ({ output }) => + Effect.tryPromise({ + try: async () => { + if (output === undefined || output.databaseName === undefined) return undefined; + const recorded = await recordedInstanceId(appScopeOf(input), output.databaseName); + return recorded === output.databaseId ? undefined : { action: 'update' as const }; + }, + catch: (cause) => cause, + }), delete: () => Effect.void, }; return Provider.effect(Prisma.Database, Effect.succeed(service)); } +function appScopeOf(input: LocalTargetProvidersInput): AppScope { + return { devDir: input.devDir, app: appNameOf(input.container) }; +} + function noRecordedInstanceError(databaseId: string): Error { return new Error( `no local Postgres instance recorded for databaseId "${databaseId}" — the Database provider ` + @@ -119,26 +103,24 @@ function noRecordedInstanceError(databaseId: string): Error { ); } -/** `Prisma.Connection` → the daemon's live listing, matched by instance name (the Database attributes' `databaseId` IS the instance name). */ +/** `Prisma.Connection` → the database instance the Database attributes' `databaseId` names. */ export function LocalConnectionProvider( - input: LocalTargetProvidersInput, + _input: LocalTargetProvidersInput, ): Layer.Layer> { const service: Provider.ProviderService = { list: () => Effect.succeed([]), reconcile: ({ id, news }) => Effect.tryPromise({ try: async () => { - const app = appNameOf(input.container); const databaseId = databaseIdOfInput(news.database); if (databaseId === undefined) throw noRecordedInstanceError(String(news.database)); - const databases = await postgresClient().listDatabases(app); - const found = databases.find((entry) => entry.instanceName === databaseId); + const found = await postgresClient().getInstance(databaseId); if (found === undefined) throw noRecordedInstanceError(databaseId); const direct = Redacted.make(found.url); return { - connectionId: found.instanceName, + connectionId: found.id, connectionName: news.name ?? id, - databaseId: found.instanceName, + databaseId: found.id, kind: 'postgres', createdAt: DEV_TIMESTAMP, directConnectionString: direct, diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/prisma-dev-module.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/prisma-dev-module.ts new file mode 100644 index 000000000..10d3f5899 --- /dev/null +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/prisma-dev-module.ts @@ -0,0 +1,25 @@ +/** Where local dev loads `@prisma/dev` from, for the Postgres emulator. */ +import { createRequire } from 'node:module'; + +function noPrismaDevError(cause: unknown): Error { + return new Error( + `local dev needs @prisma/dev for its local Postgres emulator — @prisma/composer-prisma-cloud declares it as a dependency, but it did not resolve from Composer's own installation; reinstall your dependencies. (${cause instanceof Error ? cause.message : String(cause)})`, + ); +} + +/** + * Composer owns the local Postgres emulator's version: `@prisma/dev` is a + * dependency of `@prisma/composer-prisma-cloud` and is resolved from + * Composer's own package, never from the app. The app's copy (if any) is + * ignored on purpose — an old `@prisma/dev` pulled in transitively (alchemy + * pins `^0.20.0`, whose pglite-socket crashes on any message over 64 KiB) + * would otherwise silently replace the version Composer tested against. + * The daemon imports the returned path dynamically. + */ +export function resolvePrismaDevModulePath(): string { + try { + return createRequire(import.meta.url).resolve('@prisma/dev'); + } catch (cause) { + throw noPrismaDevError(cause); + } +} diff --git a/packages/1-prisma-cloud/0-lowering/local-target/src/teardown.ts b/packages/1-prisma-cloud/0-lowering/local-target/src/teardown.ts index 2c8637f4e..da0289755 100644 --- a/packages/1-prisma-cloud/0-lowering/local-target/src/teardown.ts +++ b/packages/1-prisma-cloud/0-lowering/local-target/src/teardown.ts @@ -3,14 +3,34 @@ * ADR-0041 D12). The target extension's `dev/teardown.ts` orchestrates * (which app, which emulator clients); this module owns the actual * filesystem operations, so the extension's own source stays free of - * `node:`/`bun:` imports (its invariant 5). Postgres instance removal is - * NOT here (REVISED — operator review of #162): it is a - * `postgresClient().deleteApp(app)` call, made directly alongside the - * compute/buckets clients — no filesystem or child-process work of its own. + * `node:`/`bun:` imports (its invariant 5). Emulator instances are not + * removed here: the extension deletes them through the directory's record. */ import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { BUCKETS_DIR } from './bucket.ts'; +import { EMULATOR_INSTANCES_FILE } from './emulator-instances.ts'; /** `fs.rm(..., { recursive: true, force: true })` for every given path — tolerates absence. */ export function removeLocalPaths(paths: readonly string[]): void { for (const target of paths) fs.rmSync(target, { recursive: true, force: true }); } + +/** + * Removes the dev state directory's contents for `--fresh`, except what + * other apps run from the directory still use: the record of their emulator + * instances and their buckets' objects. This app's objects go. + */ +export function removeAppDevState(devDir: string, app: string): void { + let entries: string[]; + try { + entries = fs.readdirSync(devDir); + } catch { + return; + } + const kept = new Set([EMULATOR_INSTANCES_FILE, BUCKETS_DIR]); + removeLocalPaths([ + ...entries.filter((entry) => !kept.has(entry)).map((entry) => path.join(devDir, entry)), + path.join(devDir, BUCKETS_DIR, app), + ]); +} diff --git a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-attach.test.ts b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-attach.test.ts new file mode 100644 index 000000000..6b885a1ff --- /dev/null +++ b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-attach.test.ts @@ -0,0 +1,90 @@ +/** + * The dev session's stop, against real Compute and Postgres emulators in a + * temporary emulators directory: ending `prisma dev` stops the app's + * database servers as well as its services, keeping their data and ports. + */ +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as net from 'node:net'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + type DaemonName, + EMULATORS_DIR_ENV, + ensureDaemon, + postgresClient, + stopDaemon, +} from '@internal/dev-emulators'; +import { + deleteAppInstances, + ensureAppInstances, + readEmulatorInstances, +} from '@internal/local-target'; +import { PrismaCloudContainer } from '../container.ts'; +import { devAttach } from '../local-target/attach.ts'; + +const APP = 'attach-app'; +const DAEMONS: readonly DaemonName[] = ['compute', 'postgres']; +const savedEmulatorsDir = process.env[EMULATORS_DIR_ENV]; + +function acceptsConnection(port: number): Promise { + return new Promise((resolve) => { + const socket = net.connect({ port, host: '127.0.0.1' }); + socket.once('connect', () => { + socket.destroy(); + resolve(true); + }); + socket.once('error', () => resolve(false)); + }); +} + +// Local dev is not supported on Windows, where the emulators cannot run. +describe.skipIf(process.platform === 'win32')("the dev session's stop", () => { + let root: string; + let registryRoot: string; + let directory: string; + let devDir: string; + + beforeEach(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'target-attach-')); + registryRoot = path.join(root, 'emulators'); + process.env[EMULATORS_DIR_ENV] = registryRoot; + for (const name of DAEMONS) { + const entry = fileURLToPath(import.meta.resolve(`@internal/dev-emulators/${name}-main`)); + await ensureDaemon(name, entry, { registryRoot }); + } + directory = path.join(root, 'project'); + devDir = path.join(directory, '.prisma-composer', 'dev'); + }); + + afterEach(async () => { + await deleteAppInstances({ devDir, app: APP }).catch(() => undefined); + for (const name of DAEMONS) await stopDaemon(name, { registryRoot }).catch(() => undefined); + fs.rmSync(root, { recursive: true, force: true }); + if (savedEmulatorsDir === undefined) delete process.env[EMULATORS_DIR_ENV]; + else process.env[EMULATORS_DIR_ENV] = savedEmulatorsDir; + }); + + test("stops the app's database servers and keeps their ports", async () => { + await ensureAppInstances({ devDir, app: APP }, [{ address: 'catalog', kind: 'database' }]); + const recorded = (await readEmulatorInstances(directory))?.apps[APP]?.nodes['catalog']; + if (recorded?.kind !== 'database') throw new Error('no database recorded'); + expect(await acceptsConnection(recorded.port)).toBe(true); + const container = new PrismaCloudContainer( + { appName: APP, stage: undefined }, + 'local', + undefined, + undefined, + true, + ); + + await (await devAttach({ container, devDir })).stopApp(); + + expect(await postgresClient().getInstance(recorded.id)).toMatchObject({ + status: 'stopped', + port: recorded.port, + }); + expect(await acceptsConnection(recorded.port)).toBe(false); + }, 60_000); +}); diff --git a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-emulators.test.ts b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-emulators.test.ts index a29124b94..2ad1a2437 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-emulators.test.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-emulators.test.ts @@ -5,31 +5,46 @@ import type { ContainerInstance } from '@internal/core/config'; import * as RealDevEmulators from '@internal/dev-emulators'; import * as RealLocalTarget from '@internal/local-target'; import { compute, rawPostgres } from '../exports/index.ts'; +import { scopedFakes } from './scoped-fakes.ts'; const events: string[] = []; -let portChanges: { address: string; previous: number; given: number }[] = []; -let retiredApp: string | undefined; +let portChanges: { + kind: 'service' | 'database'; + address: string; + previous: number; + given: number; +}[] = []; +let previousApp: string | undefined; +let ensureFails = false; + +const fake = scopedFakes(); mock.module('@internal/dev-emulators', () => ({ ...RealDevEmulators, - ensureDaemon: async (name: string) => { + ensureDaemon: fake(RealDevEmulators.ensureDaemon, async (name: string) => { events.push(`daemon ${name}`); return { url: `http://127.0.0.1/${name}` }; - }, + }), })); mock.module('@internal/local-target', () => ({ ...RealLocalTarget, - resolvePackageEntry: (specifier: string) => specifier, - retirePreviousApp: async () => retiredApp, - ensureServiceInstances: async ( - container: ContainerInstance | undefined, - _devDir: string, - addresses: readonly string[], - ) => { - events.push(`ensure ${container?.input.appName ?? '?'}: ${addresses.join(', ')}`); - return portChanges; - }, + resolvePackageEntry: fake(RealLocalTarget.resolvePackageEntry, (specifier: string) => specifier), + selectApp: fake(RealLocalTarget.selectApp, async ({ app }: { app: string }) => { + events.push(`select ${app}`); + return previousApp; + }), + ensureAppInstances: fake( + RealLocalTarget.ensureAppInstances, + async ({ app }: { app: string }, nodes: readonly { address: string; kind: string }[]) => { + events.push(`ensure ${app}: ${nodes.map((n) => `${n.kind} ${n.address}`).join(', ')}`); + if (ensureFails) throw new Error('the second database could not start'); + return portChanges; + }, + ), + stopAppInstances: fake(RealLocalTarget.stopAppInstances, async ({ app }: { app: string }) => { + events.push(`stop ${app}`); + }), })); const { runDevEmulators } = await import('../local-target/emulators.ts'); @@ -67,7 +82,8 @@ beforeEach(() => { events.length = 0; logged.length = 0; portChanges = []; - retiredApp = undefined; + previousApp = undefined; + ensureFails = false; }); afterEach(() => { @@ -88,10 +104,14 @@ test('ensures the getting-started services in dependency order once the compute await runDevEmulators({ graph, container, devDir: '/dev/null/unused' }); - expect(events).toEqual(['daemon compute', 'ensure my-app: quotes, gateway']); + expect(events).toEqual([ + 'daemon compute', + 'select my-app', + 'ensure my-app: service quotes, service gateway', + ]); }); -test('ensures only services, in declaration order when nothing orders them', async () => { +test('ensures services and databases in declaration order when nothing orders them, after starting their emulators', async () => { const graph = Load( module('my-app', ({ provision }) => { provision(compute({ name: 'web', deps: {}, build })); @@ -102,11 +122,19 @@ test('ensures only services, in declaration order when nothing orders them', asy await runDevEmulators({ graph, container, devDir: '/dev/null/unused' }); - expect(events).toEqual(['daemon compute', 'ensure my-app: web, admin', 'daemon postgres']); + expect(events).toEqual([ + 'daemon compute', + 'daemon postgres', + 'select my-app', + 'ensure my-app: service web, database db, service admin', + ]); }); test('prints one line for each service that could not get its remembered port back', async () => { - portChanges = [{ address: 'gateway', previous: 3001, given: 3004 }]; + portChanges = [ + { kind: 'service', address: 'gateway', previous: 3001, given: 3004 }, + { kind: 'database', address: 'db', previous: 51300, given: 51301 }, + ]; const graph = Load( module('my-app', ({ provision }) => { provision(compute({ name: 'gateway', deps: {}, build })); @@ -117,11 +145,12 @@ test('prints one line for each service that could not get its remembered port ba expect(logged.filter((line) => line.includes('port'))).toEqual([ '[dev] service "gateway" now uses port 3004: its previous port 3001 is in use', + '[dev] database "db" now uses port 51301: its previous port 51300 is in use', ]); }); -test("prints one line when the directory's previous app's local services were removed", async () => { - retiredApp = 'old-app'; +test("prints one line when the directory's previous app's local services were stopped", async () => { + previousApp = 'old-app'; const graph = Load( module('my-app', ({ provision }) => { provision(compute({ name: 'web', deps: {}, build })); @@ -130,7 +159,22 @@ test("prints one line when the directory's previous app's local services were re await runDevEmulators({ graph, container, devDir: '/dev/null/unused' }); - expect(logged.filter((line) => line.includes('removed'))).toEqual([ - '[dev] removed the local services of "old-app", the app run here before', + expect(logged.filter((line) => line.includes('stopped'))).toEqual([ + '[dev] stopped the local services of "old-app", the app run here before; its data is kept', ]); }); + +test('a failure ensuring the instances stops what the run started, then reports the failure', async () => { + ensureFails = true; + const graph = Load( + module('my-app', ({ provision }) => { + provision(rawPostgres({ name: 'one' }), { id: 'one' }); + provision(rawPostgres({ name: 'two' }), { id: 'two' }); + }), + ); + + await expect(runDevEmulators({ graph, container, devDir: '/dev/null/unused' })).rejects.toThrow( + 'the second database could not start', + ); + expect(events.slice(-2)).toEqual(['ensure my-app: database one, database two', 'stop my-app']); +}); diff --git a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-teardown.test.ts b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-teardown.test.ts index 92987a370..0c1c37c15 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-teardown.test.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/local-target-teardown.test.ts @@ -2,36 +2,38 @@ import { beforeEach, expect, mock, test } from 'bun:test'; import * as RealDevEmulators from '@internal/dev-emulators'; import * as RealLocalTarget from '@internal/local-target'; import { PrismaCloudContainer } from '../container.ts'; +import { scopedFakes } from './scoped-fakes.ts'; const events: string[] = []; -let computeDeleteFails = false; +let deleteFails = false; -const unreachable = (name: string) => () => ({ - deleteApp: async () => { - throw new Error(`${name} emulator is not running`); - }, -}); +const fake = scopedFakes(); mock.module('@internal/dev-emulators', () => ({ ...RealDevEmulators, - ensureDaemon: async (name: string) => { + ensureDaemon: fake(RealDevEmulators.ensureDaemon, async (name: string) => { events.push(`daemon ${name}`); return { url: `http://127.0.0.1/${name}` }; - }, - postgresClient: unreachable('postgres'), - bucketsClient: unreachable('buckets'), + }), })); mock.module('@internal/local-target', () => ({ ...RealLocalTarget, - resolvePackageEntry: (specifier: string) => specifier, - deleteRecordedInstances: async (devDir: string) => { - events.push(`delete compute ${devDir.endsWith('/.prisma-composer/dev') ? 'recorded' : devDir}`); - if (computeDeleteFails) throw new Error('compute delete failed'); - }, - removeLocalPaths: () => { + resolvePackageEntry: fake(RealLocalTarget.resolvePackageEntry, (specifier: string) => specifier), + recordedKinds: fake( + RealLocalTarget.recordedKinds, + async () => new Set(['service', 'database'] as const), + ), + deleteAppInstances: fake(RealLocalTarget.deleteAppInstances, async ({ app }: { app: string }) => { + events.push(`delete instances of ${app}`); + if (deleteFails) throw new Error('delete failed'); + }), + removeAppDevState: fake(RealLocalTarget.removeAppDevState, (_devDir: string, app: string) => { + events.push(`remove dev state of ${app}`); + }), + removeLocalPaths: fake(RealLocalTarget.removeLocalPaths, () => { events.push('remove local paths'); - }, + }), })); const { runDevTeardown } = await import('../local-target/teardown.ts'); @@ -46,20 +48,30 @@ const container = new PrismaCloudContainer( beforeEach(() => { events.length = 0; - computeDeleteFails = false; + deleteFails = false; }); -test('starts the compute emulator before deleting the recorded instances, so a stopped emulator still forgets the recorded instances', async () => { +const removalEvents = ['remove dev state of my-app', 'remove local paths']; + +test("starts the emulators holding the app's instances before deleting them, so a stopped emulator still forgets them", async () => { await runDevTeardown({ container, stage: undefined }); - expect(events).toEqual(['daemon compute', 'delete compute recorded', 'remove local paths']); + expect(events).toEqual([ + 'daemon compute', + 'daemon postgres', + 'delete instances of my-app', + ...removalEvents, + ]); }); -test('a failed compute delete still removes the local state, then fails the teardown', async () => { - computeDeleteFails = true; +test('a failed delete still removes the local state, then fails the teardown', async () => { + deleteFails = true; - await expect(runDevTeardown({ container, stage: undefined })).rejects.toThrow( - 'compute delete failed', - ); - expect(events).toEqual(['daemon compute', 'delete compute recorded', 'remove local paths']); + await expect(runDevTeardown({ container, stage: undefined })).rejects.toThrow('delete failed'); + expect(events).toEqual([ + 'daemon compute', + 'daemon postgres', + 'delete instances of my-app', + ...removalEvents, + ]); }); diff --git a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/orm-migration-local-diff.integration.test.ts b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/orm-migration-local-diff.integration.test.ts new file mode 100644 index 000000000..6c746f074 --- /dev/null +++ b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/orm-migration-local-diff.integration.test.ts @@ -0,0 +1,89 @@ +/** + * Local dev's `OrmMigration` diff, against a real Postgres: a database the + * emulator created again comes back empty on the same URL, so this diff — + * which reads the migration marker — is the only thing that migrates it. + */ +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as net from 'node:net'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import * as Effect from 'effect/Effect'; +import { applyOrmMigration, resolveTargetRef } from '../orm-migrate.ts'; +import { LocalOrmMigrationProvider, OrmMigration } from '../orm-migration-resource.ts'; +import widgetContractJson from './fixtures/widget-contract/emitted/contract.json' with { + type: 'json', +}; +import { + createTestDatabase, + startTestPostgres, + type TestDatabase, + type TestPostgres, +} from './postgres-harness.ts'; +import { authorWidgetInit, widgetHash } from './widget-migrations-fixture.ts'; + +const pg: TestPostgres | undefined = startTestPostgres(); + +if (pg === undefined) { + console.warn( + '[app-cloud] skipping the local OrmMigration diff test: no Postgres available. ' + + 'Set STATE_TEST_DATABASE_URL or install initdb/pg_ctl on PATH.', + ); +} + +/** A port nothing listens on: bound once by the OS, then released. */ +function freePort(): Promise { + return new Promise((resolve) => { + const server = net.createServer(); + server.listen(0, '127.0.0.1', () => { + const address = server.address(); + const port = typeof address === 'object' && address !== null ? address.port : 0; + server.close(() => resolve(port)); + }); + }); +} + +async function localDiff(url: string): Promise { + const provider = await Effect.runPromise( + OrmMigration.Provider.pipe(Effect.provide(LocalOrmMigrationProvider())), + ); + const diff = provider.diff; + if (!diff) throw new Error('the local OrmMigration provider has no diff'); + const olds = { url, targetHash: widgetHash, invariants: [] }; + return Effect.runPromise(diff({ olds } as never)); +} + +describe.skipIf(pg === undefined)("local dev's OrmMigration diff", () => { + if (pg === undefined) return; + let migrationsDir: string; + let db: TestDatabase; + + beforeAll(async () => { + migrationsDir = fs.mkdtempSync(path.join(os.tmpdir(), 'prisma-composer-local-diff-')); + await authorWidgetInit(migrationsDir); + db = await createTestDatabase(pg.url); + }); + afterAll(async () => { + await db?.drop().catch(() => {}); + if (migrationsDir !== undefined) fs.rmSync(migrationsDir, { recursive: true, force: true }); + }); + + test('a database with no marker, such as one created again empty, asks for an update', async () => { + expect(await localDiff(db.url)).toEqual({ action: 'update' }); + }); + + test('a database at the target by its marker asks for nothing', async () => { + const ref = await resolveTargetRef(migrationsDir, widgetContractJson); + await applyOrmMigration({ url: db.url, contractJson: widgetContractJson, migrationsDir, ref }); + + expect(await localDiff(db.url)).toBeUndefined(); + }); + + test('a database that cannot be reached asks for an update, which reports the failure', async () => { + const port = await freePort(); + + expect(await localDiff(`postgres://postgres:postgres@127.0.0.1:${String(port)}/x`)).toEqual({ + action: 'update', + }); + }); +}); diff --git a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/scoped-fakes.ts b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/scoped-fakes.ts new file mode 100644 index 000000000..27919f092 --- /dev/null +++ b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/scoped-fakes.ts @@ -0,0 +1,21 @@ +import { afterEach, beforeEach } from 'bun:test'; + +/** + * bun's `mock.module` replaces a module for every test file in the run, not + * just the file that calls it. A module mock built with the returned `fake` + * uses the fake only while the calling file's own tests run, and the real + * export otherwise, so tests in other files that use the real emulators + * still get them. + */ +export function scopedFakes() { + let active = false; + beforeEach(() => { + active = true; + }); + afterEach(() => { + active = false; + }); + return (real: (...args: A) => R, fake: (...args: A) => R) => + (...args: A): R => + active ? fake(...args) : real(...args); +} diff --git a/packages/1-prisma-cloud/1-extensions/target/src/descriptors/bucket.ts b/packages/1-prisma-cloud/1-extensions/target/src/descriptors/bucket.ts index 085a94270..d0b79f26e 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/descriptors/bucket.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/descriptors/bucket.ts @@ -6,6 +6,7 @@ import * as Output from 'alchemy/Output'; import * as Prisma from 'alchemy/Prisma'; import * as Effect from 'effect/Effect'; import * as Redacted from 'effect/Redacted'; +import { resourceName } from '../service-app.ts'; import { cloudApplicationOf, logicalIdProp, @@ -28,7 +29,7 @@ export function bucketDescriptor(_o: () => ResolvedCloudOptions): NodeDescriptor const { branchId } = cloudApplicationOf(application); const bkt = yield* Prisma.Bucket(`${id}-bucket`, { project: projectIdOf(application), - name: id, + name: resourceName(id), ...(branchId !== undefined ? { branchId } : {}), ...logicalIdProp(application, address), }); diff --git a/packages/1-prisma-cloud/1-extensions/target/src/descriptors/shared.ts b/packages/1-prisma-cloud/1-extensions/target/src/descriptors/shared.ts index b6f3e1785..42e8af331 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/descriptors/shared.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/descriptors/shared.ts @@ -7,6 +7,7 @@ import * as Effect from 'effect/Effect'; import * as Redacted from 'effect/Redacted'; import type { PointerUpdatedAt } from '../control/pointer-timestamps.ts'; import type { ProviderParamEntry } from '../serializer.ts'; +import { resourceName } from '../service-app.ts'; /** * The provider-side reserved param for one brand's minted values (ADR-0031: @@ -200,7 +201,7 @@ export const stageDatabase = ({ const db = yield* Prisma.Database(`${id}-db`, { project: projectIdOf(application), region: region ?? 'inherit', - ...(branchId !== undefined ? { branchId } : { name: id }), + ...(branchId !== undefined ? { branchId } : { name: resourceName(id) }), ...logicalIdProp(application, address), }); const conn = yield* Prisma.Connection(`${id}-conn`, { database: db, name: id }); diff --git a/packages/1-prisma-cloud/1-extensions/target/src/local-target/attach.ts b/packages/1-prisma-cloud/1-extensions/target/src/local-target/attach.ts index 5e2e81df6..fd6e43629 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/local-target/attach.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/local-target/attach.ts @@ -1,12 +1,17 @@ /** * The dev session's view of the running app (local-dev spec § 5, ADR-0041 D5) * — a Compute-emulator client scoped to this app. Core renders `endpoints()`/ - * `logs()`/`stopServices()` and never learns the emulator's HTTP API + * `logs()`/`stopApp()` and never learns the emulator's HTTP API * (ADR-0038's opacity pattern). */ import type { LocalTargetAttachInput, LocalTargetAttachment } from '@internal/core/config'; import { computeClient } from '@internal/dev-emulators'; -import { type AppScope, computeNodeInstances, withServiceAddresses } from '@internal/local-target'; +import { + type AppScope, + nodeInstancesOf, + stopAppInstances, + withServiceAddresses, +} from '@internal/local-target'; import { prismaCloudContainerOf } from '../container.ts'; const RELIST_INTERVAL_MS = 2000; @@ -72,7 +77,7 @@ async function* mergedLogs( // Log lines are labelled with the service's address — the identity the // front door prints, so a service's endpoint line and its log lines name // it identically. - for (const { address, instance } of await computeNodeInstances(scope)) { + for (const { address, instance } of await nodeInstancesOf(scope, 'service')) { follow(instance.id, address); } }; @@ -110,7 +115,7 @@ export async function devAttach(input: LocalTargetAttachInput): Promise { - const nodes = await computeNodeInstances(scope); + const nodes = await nodeInstancesOf(scope, 'service'); try { await client.startInstances(nodes.map(({ instance }) => instance.id)); } catch (error) { @@ -118,16 +123,14 @@ export async function devAttach(input: LocalTargetAttachInput): Promise { - const nodes = await computeNodeInstances(scope); + const nodes = await nodeInstancesOf(scope, 'service'); return nodes.map(({ address, instance }) => ({ address, url: `http://localhost:${String(instance.port)}`, })); }, logs: (signal, opts) => mergedLogs(scope, signal, opts?.tail ?? 0), - stopServices: async () => { - const nodes = await computeNodeInstances(scope); - await client.stopInstances(nodes.map(({ instance }) => instance.id)); - }, + // Service processes and database servers; their data and ports are kept. + stopApp: () => stopAppInstances(scope), }; } diff --git a/packages/1-prisma-cloud/1-extensions/target/src/local-target/descriptor.ts b/packages/1-prisma-cloud/1-extensions/target/src/local-target/descriptor.ts index 44b0dfcb1..7edc520c0 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/local-target/descriptor.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/local-target/descriptor.ts @@ -16,7 +16,7 @@ import { localTargetProviders } from '@internal/local-target'; import * as Prisma from '@internal/lowering'; import * as Layer from 'effect/Layer'; import { GeneratedParamProvider } from '../generated-param-resource.ts'; -import { OrmMigrationProvider } from '../orm-migration-resource.ts'; +import { LocalOrmMigrationProvider } from '../orm-migration-resource.ts'; import { PgWarmProvider } from '../pg-warm-resource.ts'; import { S3CredentialsProvider } from '../s3-credentials-resource.ts'; import { devAttach } from './attach.ts'; @@ -46,7 +46,7 @@ export function localTargetDescriptor(): LocalTargetDescriptor { Layer.mergeAll( localTargetProviders(input), PgWarmProvider(), - OrmMigrationProvider(), + LocalOrmMigrationProvider(), S3CredentialsProvider(), GeneratedParamProvider(), Prisma.ServiceKeyProvider(), diff --git a/packages/1-prisma-cloud/1-extensions/target/src/local-target/emulators.ts b/packages/1-prisma-cloud/1-extensions/target/src/local-target/emulators.ts index 21df6d008..8c8430215 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/local-target/emulators.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/local-target/emulators.ts @@ -6,9 +6,14 @@ * `postgres` resource kind (REVISED — Postgres is a first-class daemon * since the programmatic `@prisma/dev` adoption, operator review of #162). * - * Once Compute is up, every service's instance is ensured in graph order, so a - * fresh start numbers ports dependencies first; Alchemy's concurrent `App` - * applies would otherwise pick them in arrival order. + * Once the daemons are up, the app becomes the directory's current app (a + * different app run here before has its services and databases stopped, + * data kept), and every node's instance is ensured through the directory's + * record in graph order: services, databases and buckets. A fresh start thus + * numbers ports dependencies first, which Alchemy's concurrent applies would + * otherwise do in arrival order, and a recorded database stopped by the last + * session is started before converge, which on a warm start may never call + * the Database provider at all. * * Idempotent: `ensureDaemon` itself adopts an already-healthy daemon, so * repeated `prisma dev` sessions are cheap. @@ -24,57 +29,71 @@ */ import type { LocalTargetEmulatorsInput } from '@internal/core/config'; import { ensureDaemon } from '@internal/dev-emulators'; -import { appNameOf, ensureServiceInstances, retirePreviousApp } from '@internal/local-target'; +import { + type AppNode, + appNameOf, + ensureAppInstances, + selectApp, + stopAppInstances, +} from '@internal/local-target'; import { PRISMA_CLOUD_EXTENSION_ID } from '../container.ts'; -import { serviceAppName } from '../service-app.ts'; +import { resourceName, serviceAppName } from '../service-app.ts'; import { daemonEntry } from './daemon-entry.ts'; -function usesBuckets(input: LocalTargetEmulatorsInput): boolean { - return input.graph.nodes.some((n) => n.node.kind === 'resource' && n.node.type === 's3'); -} - -function usesPostgres(input: LocalTargetEmulatorsInput): boolean { - return input.graph.nodes.some( - (n) => - n.node.kind === 'resource' && (n.node.type === 'raw-postgres' || n.node.type === 'postgres'), - ); -} +const DATABASE_TYPES = new Set(['raw-postgres', 'postgres']); +const BUCKET_TYPE = 's3'; -/** The `App` name of each of this extension's services, in `graph.nodes` order: dependencies first, ties in declaration order. */ -function serviceAppNames(input: LocalTargetEmulatorsInput): string[] { - return input.graph.nodes - .filter((n) => n.node.kind === 'service' && n.node.extension === PRISMA_CLOUD_EXTENSION_ID) - .map((n) => serviceAppName(n.id)); +/** + * This extension's nodes that run on an emulator, in `graph.nodes` order — + * dependencies first, ties in declaration order — keyed by the name their + * resource is given: a service by its `App` name, a database and a bucket by + * their resource name — the keys the local providers look them up by. + */ +function appNodes(input: LocalTargetEmulatorsInput): AppNode[] { + return input.graph.nodes.flatMap((n): AppNode[] => { + if (n.node.kind === 'service' && n.node.extension === PRISMA_CLOUD_EXTENSION_ID) { + return [{ address: serviceAppName(n.id), kind: 'service' }]; + } + if (n.node.kind === 'resource' && DATABASE_TYPES.has(n.node.type)) { + return [{ address: resourceName(n.id), kind: 'database' }]; + } + if (n.node.kind === 'resource' && n.node.type === BUCKET_TYPE) { + return [{ address: resourceName(n.id), kind: 'bucket' }]; + } + return []; + }); } export async function runDevEmulators(input: LocalTargetEmulatorsInput): Promise { + const nodes = appNodes(input); const { url: computeUrl } = await ensureDaemon('compute', daemonEntry('compute')); console.log(`[dev] compute emulator ready at ${computeUrl}`); - const retired = await retirePreviousApp({ - devDir: input.devDir, - app: appNameOf(input.container), - }); - if (retired !== undefined) { - console.log(`[dev] removed the local services of "${retired}", the app run here before`); - } - const portChanges = await ensureServiceInstances( - input.container, - input.devDir, - serviceAppNames(input), - ); - for (const { address, previous, given } of portChanges) { - console.log( - `[dev] service "${address}" now uses port ${String(given)}: its previous port ${String(previous)} is in use`, - ); - } - - if (usesBuckets(input)) { + if (nodes.some((node) => node.kind === 'bucket')) { const { url: bucketsUrl } = await ensureDaemon('buckets', daemonEntry('buckets')); console.log(`[dev] buckets emulator ready at ${bucketsUrl}`); } - - if (usesPostgres(input)) { + if (nodes.some((node) => node.kind === 'database')) { const { url: postgresUrl } = await ensureDaemon('postgres', daemonEntry('postgres')); console.log(`[dev] postgres emulator ready at ${postgresUrl}`); } + + const scope = { devDir: input.devDir, app: appNameOf(input.container) }; + const previous = await selectApp(scope); + if (previous !== undefined) { + console.log( + `[dev] stopped the local services of "${previous}", the app run here before; its data is kept`, + ); + } + const portChanges = await ensureAppInstances(scope, nodes).catch(async (error: unknown) => { + // `dev` fails here without a session to end, so stop the app's whole + // recorded set, as Ctrl-C would, then report the failure. One directory + // runs one `dev` session at a time. + await stopAppInstances(scope).catch(() => undefined); + throw error; + }); + for (const { kind, address, previous: before, given } of portChanges) { + console.log( + `[dev] ${kind} "${address}" now uses port ${String(given)}: its previous port ${String(before)} is in use`, + ); + } } diff --git a/packages/1-prisma-cloud/1-extensions/target/src/local-target/teardown.ts b/packages/1-prisma-cloud/1-extensions/target/src/local-target/teardown.ts index bc59ef4af..3d7e2c2a3 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/local-target/teardown.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/local-target/teardown.ts @@ -1,14 +1,13 @@ /** * `--fresh` teardown (local-dev spec § 5, ADR-0041 D12, REVISED — operator * review of #162): wholesale LOCAL deletion — never an `alchemy destroy`. - * Removes this app's records on the three machine-global emulator daemons - * (never stops the daemons — other apps may be using them; postgres's - * `DELETE /apps/` closes its servers and deletes their persisted data), - * the dev state directory, and the dev stage's `localState()` directory. - * Compute is started first if it is down, and its delete must succeed, - * because it persists port allocations; Postgres and buckets are skipped - * when unreachable. The local directories are removed even when the - * Compute step fails, and the failure is then reported. + * Deletes the app's emulator instances of every kind through the + * directory's record (a database's delete removes its data) and the app's + * entry in it, the app's part of the dev state directory, and the dev + * stage's `localState()` directory. Other apps run from the directory keep + * their entries and their buckets' objects. The daemons are never stopped — + * other directories may be using them. The local directories are removed + * even when a delete fails, and the failure is then reported. * * Every actual filesystem operation is delegated to `@internal/local-target` * (this extension's own source stays free of `node:`/`bun:` imports — @@ -18,34 +17,39 @@ import type { TeardownInput } from '@internal/core/config'; import { DEV_DIR } from '@internal/core/config'; -import { bucketsClient, ensureDaemon, postgresClient } from '@internal/dev-emulators'; -import { deleteRecordedInstances, removeLocalPaths } from '@internal/local-target'; +import { type DaemonName, ensureDaemon } from '@internal/dev-emulators'; +import { + deleteAppInstances, + type NodeKind, + recordedKinds, + removeAppDevState, + removeLocalPaths, +} from '@internal/local-target'; import { prismaCloudContainerOf } from '../container.ts'; import { daemonEntry } from './daemon-entry.ts'; -async function tolerateUnreachable(action: () => Promise): Promise { - try { - await action(); - } catch { - // Unreachable or absent daemon — the daemon itself is never stopped by - // `--fresh` (other apps may be using it), and there is nothing left to - // remove on it if it isn't running at all. - } -} +const DAEMON_OF: Record = { + service: 'compute', + database: 'postgres', + bucket: 'buckets', +}; export async function runDevTeardown(input: TeardownInput): Promise { const app = prismaCloudContainerOf(input.container).input.appName; const cwd = process.cwd(); const devDir = `${cwd}/${DEV_DIR}`; + const scope = { devDir, app }; - await tolerateUnreachable(() => postgresClient().deleteApp(app)); - // Compute persists instances and their ports across restarts, so a stopped - // daemon would hand the old ports back; start it so the delete always lands. try { - await ensureDaemon('compute', daemonEntry('compute')); - await deleteRecordedInstances(devDir); + // Every emulator holding one of the app's instances is started so the + // delete lands: a stopped emulator keeps its instances, their ports and + // a database's data. + for (const kind of await recordedKinds(scope)) { + await ensureDaemon(DAEMON_OF[kind], daemonEntry(DAEMON_OF[kind])); + } + await deleteAppInstances(scope); } finally { - await tolerateUnreachable(() => bucketsClient().deleteApp(app)); - removeLocalPaths([devDir, `${cwd}/.alchemy/state/${app}/dev`]); + removeAppDevState(devDir, app); + removeLocalPaths([`${cwd}/.alchemy/state/${app}/dev`]); } } diff --git a/packages/1-prisma-cloud/1-extensions/target/src/orm-migrate.ts b/packages/1-prisma-cloud/1-extensions/target/src/orm-migrate.ts index b6576890f..4eca43af5 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/orm-migrate.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/orm-migrate.ts @@ -196,6 +196,24 @@ export function decideMigrationAction( return 'migrate'; } +/** + * Whether the database at `url` is already at `ref` by its marker, read + * without applying anything. A database with no marker, such as one created + * empty, is not. + */ +export async function isAtTargetRef(url: string, ref: PnTargetRef): Promise { + const client = createPostgresControlClient({ + connection: normalizeSslMode(url), + extensions: [], + }); + await client.connect(); + try { + return decideMigrationAction(await client.readMarker(), ref) === 'noop'; + } finally { + await client.close(); + } +} + /** * Bring the database at `url` to the target ref via PN's authored migrations. * Reads the live marker, decides no-op / migrate diff --git a/packages/1-prisma-cloud/1-extensions/target/src/orm-migration-resource.ts b/packages/1-prisma-cloud/1-extensions/target/src/orm-migration-resource.ts index 77b40156a..7afca12a6 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/orm-migration-resource.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/orm-migration-resource.ts @@ -22,7 +22,12 @@ import { Resource } from 'alchemy'; import * as Provider from 'alchemy/Provider'; import * as Effect from 'effect/Effect'; import { loadContractJson, resolveOrmConfig } from './orm-config.ts'; -import { applyOrmMigration, OrmMigrationError, targetStorageHash } from './orm-migrate.ts'; +import { + applyOrmMigration, + isAtTargetRef, + OrmMigrationError, + targetStorageHash, +} from './orm-migrate.ts'; export interface OrmMigrationProps { /** The live DB connection string (an Alchemy Output at wiring time, resolved at apply). */ @@ -137,3 +142,31 @@ export const ormMigrationProviderService: Provider.ProviderService /** The `OrmMigration` provider layer — merged into the extension descriptor's `providers()`. */ export const OrmMigrationProvider = () => Provider.effect(OrmMigration, Effect.succeed(ormMigrationProviderService)); + +/** + * Local dev's `OrmMigration` provider: the same service, plus a diff that + * asks for an update when the database is not at the target by its marker. + * A local database the emulator had to create again starts empty on the + * same URL, so nothing in the props changes and Alchemy would otherwise skip + * the migration. Reading the marker is one connection per converge. + */ +export const LocalOrmMigrationProvider = () => + Provider.effect( + OrmMigration, + Effect.succeed({ + ...ormMigrationProviderService, + diff: ({ olds }) => + Effect.tryPromise({ + // A marker that cannot be read is left to the update, which + // retries the connection and reports a failure properly. + try: async () => { + const atTarget = await isAtTargetRef(olds.url, { + hash: olds.targetHash, + invariants: olds.invariants, + }).catch(() => false); + return atTarget ? undefined : { action: 'update' as const }; + }, + catch: (cause) => cause, + }), + } satisfies Provider.ProviderService), + ); diff --git a/packages/1-prisma-cloud/1-extensions/target/src/service-app.ts b/packages/1-prisma-cloud/1-extensions/target/src/service-app.ts index f9faf3389..49ebe2c62 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/service-app.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/service-app.ts @@ -2,3 +2,8 @@ export function serviceAppName(address: string): string { return address; } + +/** The `name` of a database's `Prisma.Database` and a bucket's `Prisma.Bucket`. Local dev records each one's emulator instance under this name before converge, and the local providers look it up by it, so both must come from here. */ +export function resourceName(nodeId: string): string { + return nodeId; +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 093dbd8bc..78c4f3306 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -886,6 +886,9 @@ importers: specifier: ^7.5.21 version: 7.5.21 devDependencies: + '@aws-sdk/client-s3': + specifier: ^3.1085.0 + version: 3.1087.0 '@internal/foundation': specifier: workspace:0.30.0 version: link:../../../0-framework/0-foundation/foundation @@ -898,6 +901,12 @@ importers: '@types/node': specifier: ^26.0.1 version: 26.1.1 + '@types/pg': + specifier: ^8.20.0 + version: 8.20.4 + pg: + specifier: ^8.22.0 + version: 8.22.0 tsdown: specifier: ^0.22.7 version: 0.22.12(typescript@6.0.3) diff --git a/skills/prisma-composer-core-concepts/SKILL.md b/skills/prisma-composer-core-concepts/SKILL.md index d70772c08..15f6e2022 100644 --- a/skills/prisma-composer-core-concepts/SKILL.md +++ b/skills/prisma-composer-core-concepts/SKILL.md @@ -442,8 +442,8 @@ that surprise: deploy. It watches built output and restarts a service when its build changes. `ready` prints only once every service accepts connections on its port, except a service declared `compute({ ..., listens: false })` (one that never listens, like the cron scheduler), which counts as running once its process starts. A service that exits before it listens, or does not listen within 30 seconds, fails `dev` (and stops the app's other services) with a message naming the service, its port, and its log file: on a first start it is in the deploy engine's output, before the error that the deploy engine exited (`DEV.CONVERGE_FAILED` from the `dev` operation); on a warm start it is the `DEV.SERVICE_START_FAILED` message. `PRISMA_COMPOSER_SERVICE_READY_TIMEOUT_MS` raises the 30-second limit (milliseconds); the compute emulator reads it when it starts, so a running emulator keeps its old value until its process ends. An emulator that does not accept connections fails `dev` and `log` with `DEV.EMULATOR_UNREACHABLE` after a few seconds of retries; nothing else is retried. -2. Ctrl-C stops the app's processes but leaves local databases, buckets, and their data up: the next `dev` is a warm start. Starting clean, wiping this app's local instances and data first, is an explicit opt-in flag. On a fresh start, each service gets the lowest free port from 3000 up, in dependency order: a service before the services that call it, otherwise provision order. A warm start keeps the ports services already have. - Services belong to the directory `dev` runs in: two checkouts of one app (two git worktrees) run separate processes on separate ports and never restart each other's. `.prisma-composer/dev/emulator-instances.json` records which local service backs each service. A service keeps its port in its directory across runs, even while stopped. If its local service is gone (a Composer upgrade discarded the compute emulator's state), the next `dev` re-creates it and asks for the old port; when another directory has taken it, the service gets the lowest free port and `dev` prints `[dev] service "
" now uses port : its previous port is in use`. A directory runs one app at a time: running a different app there removes the previous app's local services and prints `[dev] removed the local services of "", the app run here before`. A stopped service keeps its port reserved, so a deleted checkout's services keep theirs; there is no command to remove them yet. Local databases and buckets are still keyed by the app's name, so two checkouts of one app share them. +2. Ctrl-C (or `dev` ending on SIGTERM or a failure) stops the app's service processes and its local Postgres servers and keeps their data and ports: the next `dev` is a warm start. Starting clean, wiping this app's local instances and data first, is an explicit opt-in flag. On a fresh start, each service gets the lowest free port from 3000 up, in dependency order: a service before the services that call it, otherwise provision order. A warm start keeps the ports services already have. + Services, databases and buckets belong to the directory `dev` runs in: two checkouts of one app (two git worktrees) run separate processes on separate ports, with separate databases and buckets, and never restart or write into each other's. `.prisma-composer/dev/emulator-instances.json` records which local instance backs each node. A service or database keeps its port in its directory across runs, even while stopped. If a local instance is gone (a Composer upgrade discarded an emulator's state), the next `dev` re-creates it, empty, and asks for the old port, and a Prisma ORM database's (`postgres()`) committed migrations run again (a `rawPostgres()` database stays empty); when another directory has taken the port, the instance gets the lowest free one and `dev` prints `[dev] service "
" now uses port : its previous port is in use` (`database "
"` for a database). A directory keeps the local state of every app run from it: running a different app there (another entry, another `--name`) stops the previous app's services and databases, keeps their data and ports, and prints `[dev] stopped the local services of "", the app run here before; its data is kept`; running it again resumes them. `--fresh` wipes only the app it starts. A stopped instance keeps its port reserved and a database its data, so a deleted checkout's instances keep theirs; there is no command to remove them yet. 3. `dev` does not print service logs. The `log` operation follows the already-running app's merged logs; it never builds, provisions, starts, or stops anything: @@ -459,12 +459,11 @@ that surprise: service it calls. 6. The emulators are machine-wide daemons shared by every `dev`, registered under `~/.prisma-composer/emulators`. Set `PRISMA_COMPOSER_EMULATORS_DIR` - to an absolute directory to give a checkout or CI job its own compute and - buckets daemons, registry and data; a relative path (or an unexpanded `~`) - fails with `DEV.EMULATORS_DIR_INVALID`. It does not isolate local Postgres: - servers are named after the app and database, so two checkouts of one app - share a server, and `--fresh` or teardown in either stops it and deletes - the database for both. `emulatorRegistryRoot()` from + to an absolute directory to give a checkout or CI job its own compute, + Postgres and buckets daemons, registry and state; a relative path (or an + unexpanded `~`) fails with `DEV.EMULATORS_DIR_INVALID`. A local database's + data is kept by `@prisma/dev` in its own data directory under a name unique + to the database, whatever the variable says. `emulatorRegistryRoot()` from `@prisma/composer-prisma-cloud/local-target` returns the directory in effect. 7. Windows isn't supported yet. diff --git a/test/integration/test/local-dev-criteria-4-5.integration.ts b/test/integration/test/local-dev-criteria-4-5.integration.ts index 0937d54cd..54b14bd04 100644 --- a/test/integration/test/local-dev-criteria-4-5.integration.ts +++ b/test/integration/test/local-dev-criteria-4-5.integration.ts @@ -9,7 +9,8 @@ * * Criterion 4 (bucket round-trip): PUT an object through the app's own * `/blobs/:key` route, confirm it lands as a plain file under - * `.prisma-composer/dev/buckets/files/`; drop a file directly into + * `.prisma-composer/dev/buckets//files/`, where `` is the app + * the directory's `emulator-instances.json` records as current; drop a file directly into * that directory and GET it back through the app. * * Criterion 5 (value sourcing): with `LOCALDEV_FIXTURE_API_KEY` unset, the @@ -31,6 +32,21 @@ function assert(condition: unknown, message: string): asserts condition { if (!condition) throw new Error(`assertion failed: ${message}`); } +/** The app `dev` records as current in the directory's `emulator-instances.json`; its buckets' objects live under `buckets//`. */ +function currentApp(): string { + const record: unknown = JSON.parse( + fs.readFileSync(path.join(devDir, 'emulator-instances.json'), 'utf8'), + ); + assert( + typeof record === 'object' && + record !== null && + 'current' in record && + typeof record.current === 'string', + 'emulator-instances.json names the current app', + ); + return record.current; +} + function assertEqual(actual: T, expected: T, message: string): void { const same = JSON.stringify(actual) === JSON.stringify(expected); if (!same) { @@ -281,7 +297,8 @@ async function main(): Promise { ); assertEqual(putRes.status, 201, 'PUT /blobs/proving-key.txt through the app'); - const objectPath = path.join(devDir, 'buckets', 'files', 'proving-key.txt'); + const bucketDir = path.join(devDir, 'buckets', currentApp(), 'files'); + const objectPath = path.join(bucketDir, 'proving-key.txt'); const onDisk = await waitForAsync( async () => (fs.existsSync(objectPath) ? fs.readFileSync(objectPath, 'utf8') : undefined), 5_000, @@ -291,7 +308,7 @@ async function main(): Promise { console.log(`[proving] PASS criterion 4 (write direction): ${objectPath}`); // Drop a file directly into the bucket directory, read it back through the app. - const droppedPath = path.join(devDir, 'buckets', 'files', 'dropped-key.txt'); + const droppedPath = path.join(bucketDir, 'dropped-key.txt'); fs.writeFileSync(droppedPath, 'dropped straight onto disk'); const getRes = await waitForAsync( () => diff --git a/test/integration/test/local-dev-store.integration.ts b/test/integration/test/local-dev-store.integration.ts index d121e5028..024f0242e 100644 --- a/test/integration/test/local-dev-store.integration.ts +++ b/test/integration/test/local-dev-store.integration.ts @@ -71,6 +71,7 @@ import { type ChildProcess, spawn, spawnSync } from 'node:child_process'; import * as fs from 'node:fs'; +import * as net from 'node:net'; import * as path from 'node:path'; import { containerEnv } from '@prisma/composer/config'; import { nodeBuild } from '@prisma/composer/node/control'; @@ -240,16 +241,12 @@ interface ServiceInfo { readonly pid?: number; } -/** The services `prisma dev` recorded in storeDir's `.prisma-composer/dev/emulator-instances.json`, as the Compute emulator reports them (plain fetch, never the typed client). */ +/** The services `prisma dev` recorded for the app, as the Compute emulator reports them (plain fetch, never the typed client). */ async function listComputeServices(app: string): Promise { const entry = readEmulatorEntry('compute'); if (entry === undefined) throw new Error('compute emulator registry entry not found'); - const mapping = readJson( - path.join(storeDir, '.prisma-composer', 'dev', 'emulator-instances.json'), - ) as { app: string; nodes: Record } | undefined; - if (mapping === undefined || mapping.app !== app) return []; const services: ServiceInfo[] = []; - for (const [address, node] of Object.entries(mapping.nodes)) { + for (const [address, node] of Object.entries(recordedNodes(app, 'service'))) { const res = await fetch(`http://127.0.0.1:${entry.port}/instances/${node.id}`); if (res.status === 404) continue; if (!res.ok) throw new Error(`compute emulator getInstance failed: ${res.status}`); @@ -258,18 +255,36 @@ async function listComputeServices(app: string): Promise return services; } +/** The nodes of `kind` that `prisma dev` recorded for `app` in storeDir's `.prisma-composer/dev/emulator-instances.json`. */ +function recordedNodes(app: string, kind: string): Record { + const record = readJson( + path.join(storeDir, '.prisma-composer', 'dev', 'emulator-instances.json'), + ) as + | { apps: Record }> } + | undefined; + const nodes = record?.apps[app]?.nodes ?? {}; + return Object.fromEntries(Object.entries(nodes).filter(([, node]) => node.kind === kind)); +} + interface PostgresDatabaseInfo { - readonly instanceName: string; + readonly address: string; + readonly id: string; readonly url: string; + readonly status: string; } -/** The documented postgres-emulator wire protocol (spec § 4, REVISED — operator review of #162: the daemon owns instance state, there is no dev-store postgres.json) — plain fetch, never the typed client. */ +/** The databases `prisma dev` recorded for the app, as the postgres emulator reports them (plain fetch, never the typed client). */ async function listPostgresDatabases(): Promise { const entry = readEmulatorEntry('postgres'); if (entry === undefined) throw new Error('postgres emulator registry entry not found'); - const res = await fetch(`http://127.0.0.1:${entry.port}/apps/${APP_NAME}/databases`); - if (!res.ok) throw new Error(`postgres emulator listDatabases failed: ${res.status}`); - return (await res.json()) as PostgresDatabaseInfo[]; + const databases: PostgresDatabaseInfo[] = []; + for (const [address, node] of Object.entries(recordedNodes(APP_NAME, 'database'))) { + const res = await fetch(`http://127.0.0.1:${entry.port}/instances/${node.id}`); + if (res.status === 404) continue; + if (!res.ok) throw new Error(`postgres emulator getInstance failed: ${res.status}`); + databases.push({ ...((await res.json()) as PostgresDatabaseInfo), address }); + } + return databases; } /** Every address's pid, keyed by address — undefined for a stopped/absent service. */ @@ -351,6 +366,17 @@ async function startDev(fresh: boolean): Promise { * that it survives composer's own signal wiring and reaches the shell is * provable only here. */ +function acceptsConnection(port: number): Promise { + return new Promise((resolve) => { + const socket = net.connect({ port, host: '127.0.0.1' }); + socket.once('connect', () => { + socket.destroy(); + resolve(true); + }); + socket.once('error', () => resolve(false)); + }); +} + async function stopDev(session: DevSession): Promise { if (session.child.exitCode !== null || session.child.signalCode !== null) return; session.child.kill('SIGINT'); @@ -380,11 +406,8 @@ function readJson(file: string): unknown { async function readCatalogDbUrl(): Promise { const databases = await listPostgresDatabases(); - const entry = databases.find((d) => d.instanceName === 'pcdev-store-catalog-database'); - assert( - entry?.url !== undefined, - 'the postgres-main listing must carry a pcdev-store-catalog-database entry', - ); + const entry = databases.find((d) => d.address === 'catalog.database'); + assert(entry?.url !== undefined, 'the record must name a running catalog.database database'); return entry.url; } @@ -636,6 +659,15 @@ async function main(): Promise { // ——— Ctrl-C ——— await stopDev(session); console.log('[proving] session 1 stopped cleanly on SIGINT'); + // Ending `prisma dev` stops the app's database servers too; their data + // and ports are kept for the warm start below. + const stoppedDb = (await listPostgresDatabases()).find((d) => d.address === 'catalog.database'); + assert(stoppedDb?.status === 'stopped', 'catalog.database must be stopped after Ctrl-C'); + const stoppedPort = new URL(stoppedDb.url).port; + assert( + !(await acceptsConnection(Number(stoppedPort))), + `catalog.database's port ${stoppedPort} must refuse connections after Ctrl-C`, + ); // ——— Criterion 3 (warm read) + criterion 6 (same ports) ——— session = await startDev(false); @@ -689,9 +721,9 @@ async function main(): Promise { session = await startDev(true); console.log(`[proving] session 3 (--fresh) ready: ${JSON.stringify(session.endpoints)}`); const freshDbUrl = await readCatalogDbUrl(); - // --fresh removes the app's postgres-main-hosted servers and their - // persisted data (DELETE /apps/ on the daemon, runDevTeardown) and - // recreates them on next provision — migrations reapply as part of THIS + // --fresh deletes the app's recorded database instances, their + // persisted data with them (runDevTeardown), and the converge creates + // them again — migrations reapply as part of THIS // SAME converge (OrmMigration runs on every converge), so the "product" // table exists again by the time the front door prints; the proof is // that it's freshly migrated and EMPTY — the proving row from before diff --git a/test/integration/test/local-dev.integration.ts b/test/integration/test/local-dev.integration.ts index 2bdd2b929..03910c974 100644 --- a/test/integration/test/local-dev.integration.ts +++ b/test/integration/test/local-dev.integration.ts @@ -274,20 +274,36 @@ function readEmulatorEntry( return isRegistryEntry(parsed) ? parsed : undefined; } +/** The nodes of `kind` that `prisma dev` recorded for `app` in fixtureDir's `.prisma-composer/dev/emulator-instances.json`. */ +function recordedNodes(app: string, kind: string): Record { + const record = readJson( + path.join(fixtureDir, '.prisma-composer', 'dev', 'emulator-instances.json'), + ) as + | { apps: Record }> } + | undefined; + const nodes = record?.apps[app]?.nodes ?? {}; + return Object.fromEntries(Object.entries(nodes).filter(([, node]) => node.kind === kind)); +} + interface PostgresDatabaseInfo { + readonly address: string; readonly id: string; readonly url: string; - readonly instanceName: string; - readonly databasePort: number; + readonly status: string; } -/** The documented postgres-emulator wire protocol (spec § 4, REVISED — operator review of #162) — plain fetch, never the typed client. */ +/** The databases `prisma dev` recorded for the app, as the postgres emulator reports them (plain fetch, never the typed client). */ async function listPostgresDatabases(): Promise { const entry = readEmulatorEntry('postgres'); if (entry === undefined) throw new Error('postgres emulator registry entry not found'); - const res = await fetch(`http://127.0.0.1:${entry.port}/apps/${APP_NAME}/databases`); - if (!res.ok) throw new Error(`postgres emulator listDatabases failed: ${res.status}`); - return (await res.json()) as PostgresDatabaseInfo[]; + const databases: PostgresDatabaseInfo[] = []; + for (const [address, node] of Object.entries(recordedNodes(APP_NAME, 'database'))) { + const res = await fetch(`http://127.0.0.1:${entry.port}/instances/${node.id}`); + if (res.status === 404) continue; + if (!res.ok) throw new Error(`postgres emulator getInstance failed: ${res.status}`); + databases.push({ ...((await res.json()) as PostgresDatabaseInfo), address }); + } + return databases; } interface ServiceInfo { @@ -299,16 +315,12 @@ interface ServiceInfo { readonly pid?: number; } -/** The services `prisma dev` recorded in fixtureDir's `.prisma-composer/dev/emulator-instances.json`, as the Compute emulator reports them (plain fetch, never the typed client). */ +/** The services `prisma dev` recorded for the app, as the Compute emulator reports them (plain fetch, never the typed client). */ async function listComputeServices(): Promise { const entry = readEmulatorEntry('compute'); if (entry === undefined) throw new Error('compute emulator registry entry not found'); - const mapping = readJson( - path.join(fixtureDir, '.prisma-composer', 'dev', 'emulator-instances.json'), - ) as { app: string; nodes: Record } | undefined; - if (mapping === undefined || mapping.app !== APP_NAME) return []; const services: ServiceInfo[] = []; - for (const [address, node] of Object.entries(mapping.nodes)) { + for (const [address, node] of Object.entries(recordedNodes(APP_NAME, 'service'))) { const res = await fetch(`http://127.0.0.1:${entry.port}/instances/${node.id}`); if (res.status === 404) continue; if (!res.ok) throw new Error(`compute emulator getInstance failed: ${res.status}`); @@ -554,17 +566,11 @@ async function main(): Promise { 'secrets.json carries the shell-sourced env-param', ); - // 11. The postgres-main daemon's own listing (REVISED — operator review - // of #162: no more dev-store postgres.json, the daemon owns instance - // state) + a real, running `@prisma/dev` server. Keyed by the instance - // name itself (`pcdev--`), not the bare node id. + // 11. The database the directory's record names for `appdb`, as the + // postgres emulator reports it, + a real, running `@prisma/dev` server. const databases = await listPostgresDatabases(); - const dbEntry = databases.find((d) => d.instanceName === 'pcdev-localdevs4fixture-appdb'); - assertEqual( - dbEntry?.instanceName, - 'pcdev-localdevs4fixture-appdb', - 'postgres-main database listing instance name', - ); + const dbEntry = databases.find((d) => d.address === 'appdb'); + assertEqual(dbEntry?.status, 'running', 'the recorded appdb database is running'); const dbUrl = new URL(dbEntry?.url ?? ''); const dbReachable = await probeTcp(dbUrl.hostname, Number(dbUrl.port), 2000); assert( @@ -644,7 +650,7 @@ async function main(): Promise { console.log('PASS: local dev (S4) integration proof'); } finally { process.chdir(originalCwd); - await attachment?.stopServices().catch(() => undefined); + await attachment?.stopApp().catch(() => undefined); // App-scoped teardown through the extension's own public // localTarget.teardown — never stops the daemons themselves (D4/D12),