diff --git a/CHANGELOG.md b/CHANGELOG.md index b5be91d..b9c691c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,8 +4,47 @@ All notable changes to this repository. Attribution rules: `AGENTS.md` §1.1. ## [Unreleased] +### Fixed + +- **Closing the terminal part way through a deployment no longer leaves the app locked.** + `deploy` and `rollback` handled Ctrl+C by giving the release back: the new container + removed and the server's deploy lock released. A closed terminal (SIGHUP), `kill` and a + cancelled CI job (SIGTERM) ended the process where it stood instead, and the lock stayed + until someone removed it by hand. All of them are handled the way Ctrl+C is now, with + Ctrl+Break on Windows, and from before the release starts, so a stop that arrives while + the lock is being taken is not lost. Verified against a Droplet by closing the terminal of + a deployment that was waiting on its health check: 1.1.2 left the lock and the new + container behind, and this released the lock, removed the container and left the + previous deployment serving. Not yet run by closing a console window on Windows. + (Gowtham) +- **A remote build from Windows no longer opens the server's Docker on a local port.** + Windows' OpenSSH cannot forward to a Unix socket, so the build reached the server's + Docker through `127.0.0.1:` with nothing authenticating it. For the length of a + build any process on the machine could use it, and so could a web page in a browser, by + rebinding a name to 127.0.0.1; Docker on the server is root on the server. The build now + goes through a named pipe with a random name, which a browser cannot open and only your + account, SYSTEM and Administrators can write to, and each connection is carried by its + own `ssh ... docker system dial-stdio` under the pinned host key. The warning the old + forward printed is gone with it. Verified from Windows 11 against a Droplet, the first + time `--build remote` has run from Windows: a first deployment in 127 s and a second in + 73 s, the image on the server and not on the Windows machine, and streaming passing. + (Gowtham) +- **`server add` works from Windows.** It could not start: the `ssh-keyscan` Windows ships + cannot scan a current server, and ends with `choose_kex: unsupported KEX method + sntrup761x25519-sha512@openssh.com` against Ubuntu 24.04. Found by running it from + Windows against a Droplet. When the scan fails, the host key is now read by letting + `ssh` record it, in one connection that offers no way to authenticate, and it goes + through the same printed fingerprint and `--yes` as a scanned key. The fingerprint + printed matched the server's own. (Gowtham) + ### Docs +- **The roadmap says where the CLI goes next.** A new section lists what is planned and + not built: machine-readable output, and more than one instance of an app, named as the + largest piece of work ahead with its design under "v2: correctness at scale". The fourth + sequencing principle now names the user the server target brought: anyone deploying + Next.js to a server they own. (Gowtham) + - **The evidence page compares nextship with Coolify, measured.** The same app deployed with Coolify 4.3.23 and nextship 1.1.2 on identical Droplets: setup, cold and warm deploy times, image size (987 MB against 407 MB), the memory each tool uses beside the diff --git a/README.md b/README.md index f724f95..9e38f8d 100644 --- a/README.md +++ b/README.md @@ -353,7 +353,7 @@ deployed, and a page that did not compile failed once, with no retry. | Option | Default | Meaning | |---|---|---| -| `--build ` | `remote` | `remote` builds on the server's own Docker, reached through an SSH forward of its socket, so nothing but the build context leaves your machine and the image is built natively for the server. `local` builds here for the server's architecture and streams the image with `docker save \| ssh docker load`. Recorded in `nextship.json`. A server under 2 GB of RAM is refused for remote builds | +| `--build ` | `remote` | `remote` builds on the server's own Docker, reached over SSH through a private socket, or a named pipe on Windows, so nothing but the build context leaves your machine, no port is opened, and the image is built natively for the server. `local` builds here for the server's architecture and streams the image with `docker save \| ssh docker load`. Recorded in `nextship.json`. A server under 2 GB of RAM is refused for remote builds | | `--memory ` | an even share of 80% of RAM across the server's apps | The container's memory limit, such as `512m` | ``` @@ -1205,7 +1205,9 @@ Bun's adapter keeps a list of its own the same way. | **v1.1** | Any Linux server over SSH | Released in 1.1.0. Verified end to end on a local test server, a Proxmox VM and DigitalOcean Droplets | | **Next** | The rest of the server target's live checklist | Planned, not tied to a version: a Hetzner VM, an arm64 machine, Debian 12, a certificate for a real domain, and the GitHub Actions workflow | -Beyond v1.0, each with the trigger that would justify it: correctness at scale (a +Planned for the CLI, none of it built: two defects to fix, machine-readable output, and +more than one instance of an app, which is the largest piece of work ahead. Beyond v1.0, +each with the trigger that would justify it: correctness at scale (a shared cache and distributed tags, needed once there is more than one instance), git-driven previews, a hosted control plane, edge performance (which is where serving static assets from a CDN now lives), and other frameworks. The roadmap also records diff --git a/docs/design.md b/docs/design.md index 6b848b3..de6ba0d 100644 --- a/docs/design.md +++ b/docs/design.md @@ -602,12 +602,79 @@ The server runs a `nextship` user in the `docker` group. **Membership of the `do group is root-equivalent**, and the docs say so rather than implying the user is unprivileged. +#### Stopping a release from outside (Implemented) + +A release holds the app's deploy lock from the moment its container starts until traffic +has switched or the release has been given up, and nothing breaks that lock automatically. +A release that is stopped is therefore told, through one abort signal `deploy` and +`rollback` both watch, so it removes its new container and lets the lock go. Ctrl+C was +handled that way from the start. A closed terminal, which arrives as SIGHUP, and `kill` or +a cancelled CI job, which arrive as SIGTERM, were not: each ended the process where it +stood and left the app locked until someone removed the lock by hand. All of them abort +the same signal now, and on Windows so does Ctrl+Break. The listeners are attached before +the release starts, so a stop that arrives while the lock is being taken is acted on as +soon as the release can be given back. After a stop, a failed write to a terminal that is +gone is ignored rather than allowed to end the process before the lock is released. + +Verified on Linux with real signals: a process holding a stand-in lock was sent SIGHUP, +SIGTERM and SIGINT in turn, and each time released it and exited by itself. + +Then against a real server. Gowtham ran these on 2026-10-09, on a DigitalOcean Droplet (`s-2vcpu-4gb`, Ubuntu 24.04, `blr1`), from Ubuntu 24.04 under WSL. An app +made to take three minutes to start was deployed inside a terminal of its own, and that +terminal was closed once the lock was held and the deployment was waiting on the health +check. With 1.1.2 from npm the process died, and 25 s later the lock was still on the +server, naming a process that no longer existed, with the new container still running +beside the old one. With this fix, 25 s later the lock was gone, the new container had +been removed, the previous one was still healthy and answering 200, and no nextship +process was left. A normal deployment afterwards went live with nothing removed by hand. + +**Not yet run:** closing a console window on Windows part way through a deployment. +Windows delivers that as SIGHUP and ends the process a few seconds later whatever it is +doing, and giving a release back takes two or three SSH logins, each a new connection +there. One attempt closed its window during the build, before any lock was taken, and +showed nothing either way. + #### Build and delivery (Implemented) The default build mode is **remote**: the image is built by the server's own Docker daemon, reached through `ssh -L` forwarding a local socket to `/var/run/docker.sock`. That keeps the pinned host key in force, which `DOCKER_HOST=ssh://` would bypass, builds for the server's own architecture natively, and sends only the build context rather than a finished image. +The socket is in a directory only the user running nextship can enter. + +**From Windows the daemon is reached through a named pipe, not a port.** Windows' OpenSSH +cannot forward to a Unix socket, and the first answer was a loopback TCP port, which +nothing authenticates: for as long as a build ran, any process on the machine could drive +the server's Docker, which is root on the server, and so could a web page open in a +browser, by rebinding a name to 127.0.0.1. Now nextship listens on a named pipe with a +random name and gives each connection the Docker client opens its own +`ssh ... docker system dial-stdio`, the command Docker itself uses to carry a connection +over SSH, started with nextship's own options so the pinned key still applies. A browser +cannot open a named pipe, and no port is opened on either machine. A pipe Node creates +takes Windows' default permissions for one, read from such a pipe on Windows 11: full +control for the user who made it, SYSTEM and Administrators, and read only for everyone +else, who therefore cannot send the daemon a request. Verified on Windows 11 with Docker +29.7.2: through the bridge module the Docker client reported the daemon's version and +built an image with a BuildKit secret, over 14 connections with none lost, the dialled +command running against the local daemon. + +Then against a real server, from Windows 11 with its own OpenSSH (9.5p2) and no WSL +involved. Gowtham ran these on 2026-10-09, on a DigitalOcean Droplet (`s-2vcpu-4gb`, Ubuntu 24.04, `blr1`). `server add --yes` took 61 s. The first `deploy --yes` took +127 s and a second, after a one-line change, 73 s. The image existed on the server +(411 MB) and not on the Windows machine, the output carried no warning about a port, +streaming passed three runs of three with a first byte at 113 to 209 ms, and the Edge +route answered. This is the first time `--build remote` has run from Windows. + +That run also found that `server add` could not start from Windows at all. The +`ssh-keyscan` Windows ships cannot scan a current server: it offers a key exchange it was +not built with, and ends with `choose_kex: unsupported KEX method +sntrup761x25519-sha512@openssh.com` against Ubuntu 24.04's OpenSSH 9.6. `ssh` on the same +machine negotiates correctly, so when the scan fails the key is now read by letting `ssh` +record it: one connection that offers no way to authenticate, sends nothing, and ends at +"permission denied" after the key exchange has written the server's key to a file that is +thrown away. The key then goes through the same printed fingerprint and `--yes` as a +scanned one, and the fingerprint it printed matched the server's own +`ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub`. A server under 2 GB of RAM is refused for remote builds with `--build local` as the action. A remote build whose connection to the daemon drops is built once more over a new forward, and only then. After a failed build the builder has two kinds of evidence: its @@ -868,7 +935,7 @@ Not verified yet, and the live checklist in `docs/roadmap.md`: | A hosting provider: a Hetzner VM, and arm64 on any real machine | An amd64 Proxmox VM and an amd64 DigitalOcean Droplet have passed (above). arm64 has run only against the container stand-in, and Hetzner has not been used | | A certificate issued for a real domain, and streaming over HTTPS | Needs public DNS pointing at a reachable server; neither the Proxmox VM nor the Droplet used so far has had a domain pointed at it | | The GitHub Actions workflow in `vm.md` | Not yet run; the `vm` CI job itself now passes on every push to `main` (`ssh localhost` stands in for a server) | -| `--build remote` from Windows over a loopback forward | No Windows machine was used | +| Closing a console window on Windows part way through a deployment | A closed terminal on Linux gives the lock back (Stopping a release, above); Windows has not been run | #### Limitations diff --git a/docs/roadmap.md b/docs/roadmap.md index bee1592..46028e1 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -261,7 +261,7 @@ tried yet: | `server add` on Debian 12 (item 2) | Debian 12 is a supported distribution that has only run on the stand-in | | `domain add` with a real domain: the certificate is issued, and streaming works over HTTPS (item 4) | No server used so far had a domain pointed at it | | A deployment from the GitHub Actions workflow in `vm.md` with `--build local` (item 11) | The documented CI path has never run | -| `--build remote` from Windows over the loopback forward | No Windows machine has built on a server | +| Closing a console window on Windows part way through a deployment | A closed terminal on Linux gives the lock back, on a real server. Windows ends the process a few seconds after the window closes, and that has not been run | Known from the Droplet runs, to fix or document here: `conformance/vm/e2e.sh` needs a server with no other app on it, and after adding swap `server add` prints `done swap already on`, @@ -421,6 +421,17 @@ driver. --- +## Where the CLI goes next (Planned, not tied to a version) + +The CLI's job is deploying Next.js to infrastructure you own, and what is planned is +doing that job more completely. None of this exists yet. Each row is built, verified on a +real server and documented on its own, and none has a date. + +| Order | Item | State and what decides it | +|---|---|---| +| 1 | **Machine-readable output.** `--json` on the commands that report state or act, so CI and scripts read a result rather than scrape text | Planned | +| 2 | **More than one instance of an app.** A shared cache, tag invalidation that reaches every instance, and deployments that roll across them with no failed request | Planned, and the largest piece of work ahead. Its design is "v2: correctness at scale" below | + ## Beyond v1.0 Not scheduled. Recorded so that the decision to skip them stays deliberate. Each @@ -529,5 +540,6 @@ Recorded so the decision stays visible rather than looking like an oversight. 3. **Seams, not stubs.** Where a later version will swap an implementation, the interface appears in the change that first consumes it, never before ([`../AGENTS.md`](../AGENTS.md) §3.1). -4. **Build for the current user.** Right now that user is one person with one app. - Features justified only by imagined future users wait until those users exist. +4. **Build for the current user.** Through 1.0 that user was one person with one app. + Since the server target it is anyone deploying Next.js to a server they own. Features + justified only by imagined future users still wait until those users exist. diff --git a/docs/vm.md b/docs/vm.md index 712c3d8..acad1a6 100644 --- a/docs/vm.md +++ b/docs/vm.md @@ -17,7 +17,8 @@ Author: Ragul D your `ssh`. - Nothing listening on ports 80 or 443. Caddy takes both. - An OpenSSH client on the machine you run nextship from. macOS, Linux and Windows 10 and - later include one. + later include one. On Windows nothing else is needed: `server add` and a remote build + have run from Windows 11's own OpenSSH, with no WSL. ## 2. Setting a server up @@ -71,6 +72,10 @@ does not protect any port you publish from another container yourself. - **Nothing is interpreted by a shell on your machine.** `ssh` is started with an argument list. On the server, every value placed in a command is validated against a strict pattern or single-quoted. +- **A remote build opens no port.** The server's Docker is reached through a socket in a + directory only you can enter, and from Windows through a named pipe only your account + can write to. Docker on the server is root on the server, so nothing else on your + machine, a web page in your browser included, is given a way to it. - **`nextship` is root-equivalent.** Membership of the `docker` group lets a user start a privileged container, so anyone who can log in as `nextship` controls the server. Treat its keys as root's. It is a separate user so nextship's files and containers have one @@ -108,9 +113,11 @@ again. **Recovering.** A container that exits is restarted by Docker. A container that stays up but stops answering its health check is restarted by the watchdog after three failed -minutes, and the journal says so under `nextship-watchdog`. A lock left by a deployment -that was killed is reported with its owner after 30 minutes and never removed -automatically: confirm nothing is running, then remove +minutes, and the journal says so under `nextship-watchdog`. A deployment or rollback +stopped with Ctrl+C, by closing its terminal, or by `kill` gives its lock back before it +exits. One that was ended some other way, a lost power supply or `kill -9`, can still +leave its lock: that is reported with its owner after 30 minutes and never removed +automatically. Confirm nothing is running, then remove `/etc/nextship/apps//lock` as the `nextship` user. **Reboots.** Every app container runs with `--restart unless-stopped`, Docker is enabled at diff --git a/packages/cli/src/build-attempts.test.ts b/packages/cli/src/build-attempts.test.ts index f5b45f9..58b9913 100644 --- a/packages/cli/src/build-attempts.test.ts +++ b/packages/cli/src/build-attempts.test.ts @@ -25,7 +25,6 @@ function builders(answers: Array) { const builder: ImageBuilder & { closed: number } = { dockerHost: `unix:///tmp/forward-${opened.length}.sock`, cacheScope: null, - warning: null, closed: 0, close: async () => { builder.closed++ diff --git a/packages/cli/src/deploy.ts b/packages/cli/src/deploy.ts index c0bb878..be763e7 100644 --- a/packages/cli/src/deploy.ts +++ b/packages/cli/src/deploy.ts @@ -23,6 +23,7 @@ import { CONTAINER_PORT } from './image/dockerfile.js' import { client as apiClient } from './owned-app.js' import { DEFAULT_REGION } from './targets/digitalocean-target.js' import type { EnvRecord, Target } from './targets/target.js' +import { untilStopped } from './interrupt.js' import { detail, ok, step, warn } from './util/log.js' export interface DeployOptions { @@ -167,7 +168,6 @@ export async function deploy(project: ProjectInfo, options: DeployOptions): Prom const { build, image } = await buildWithReconnect( () => client.builder(), async (builder) => { - if (builder.warning) warn(builder.warning) const placement = { platform, dockerHost: builder.dockerHost, cacheScope: builder.cacheScope } const build = await buildProject(project, placement, serverKey) return { build, image: await packageImage(project, build) } @@ -197,58 +197,55 @@ export async function deploy(project: ProjectInfo, options: DeployOptions): Prom const serving = owned ? (await client.deployments(owned.id)).some((deployment) => deployment.live) : false step(owned ? `Updating app "${name}"` : `Creating app "${name}"`) - const released = await client.release(owned?.id ?? null, { - name, - repository: name, - tag: deploymentId, - port: CONTAINER_PORT, - instanceSize: options.instanceSize ?? null, - memory: options.memory ?? null, - healthPath: build.manifest.healthPath, - }) - const appId = released.appId + // Ctrl+C, a closed terminal and `kill` all arrive from outside while a driver may + // be holding something for this release: on a server it is the app's deploy lock, + // which nothing breaks automatically. They are listened for from before the + // release starts, so one that arrives while the lock is being taken is acted on + // as soon as the release can be given back, rather than ending the process there. + const appId = await untilStopped(async (signal) => { + const released = await client.release(owned?.id ?? null, { + name, + repository: name, + tag: deploymentId, + port: CONTAINER_PORT, + instanceSize: options.instanceSize ?? null, + memory: options.memory ?? null, + healthPath: build.manifest.healthPath, + }) - // From here to the end of the release, every path out has to reach either - // awaitRelease or abandonRelease. A driver may be holding something for this - // release: on a server it is the app's deploy lock, which nothing breaks - // automatically, so an early return or a thrown error here used to leave a - // lock a person had to go and remove by hand, with the new container still - // running beside it. - // - // Ctrl+C is the same problem arriving from outside. Handled rather than left - // to kill the process mid-wait, so the release is given back the same way. - const stopping = new AbortController() - const interrupt = (): void => stopping.abort() - process.on('SIGINT', interrupt) + // From here to the end of the release, every path out has to reach either + // awaitRelease or abandonRelease, or an early return or a thrown error leaves a + // lock a person has to go and remove by hand, with the new container still + // running beside it. + let handedOff = false + try { + // Written before waiting, so a timeout still leaves the app recorded as ours + // rather than orphaned and unadoptable on the next run. + const config: ProjectConfig = { + ...settings, + ...(delivered.store ? { registry: delivered.store } : {}), + appId: released.appId, + } + await writeConfig(project.root, config) + detail('recorded in nextship.json') - let handedOff = false - try { - // Written before waiting, so a timeout still leaves the app recorded as ours - // rather than orphaned and unadoptable on the next run. - const config: ProjectConfig = { - ...settings, - ...(delivered.store ? { registry: delivered.store } : {}), - appId, - } - await writeConfig(project.root, config) - detail('recorded in nextship.json') + if (!released.deploymentId) { + warn(`${client.displayName} reported no deployment to follow. Check its control panel.`) + return null + } - if (!released.deploymentId) { - warn(`${client.displayName} reported no deployment to follow. Check its control panel.`) - return + step('Waiting for the deployment to go live') + handedOff = true + await client.awaitRelease(released.appId, released.deploymentId, detail, serving, signal) + return released.appId + } finally { + // awaitRelease ends the release itself, whether it succeeded or threw, so + // this only covers the paths that never reached it. + if (!handedOff) await client.abandonRelease(released.appId, released.deploymentId) } + }) - step('Waiting for the deployment to go live') - handedOff = true - await client.awaitRelease(appId, released.deploymentId, detail, serving, stopping.signal) - } finally { - process.off('SIGINT', interrupt) - // awaitRelease ends the release itself, whether it succeeded or threw, so - // this only covers the paths that never reached it. - if (!handedOff) await client.abandonRelease(appId, released.deploymentId) - } - - await reportUrls(client, appId) + if (appId !== null) await reportUrls(client, appId) } /** diff --git a/packages/cli/src/interrupt.test.ts b/packages/cli/src/interrupt.test.ts new file mode 100644 index 0000000..d7361b2 --- /dev/null +++ b/packages/cli/src/interrupt.test.ts @@ -0,0 +1,124 @@ +/** + * @nextship/cli: a release is told to stop, not killed + * + * The defect was a process that died holding a server's deploy lock because the + * terminal closed. The first tests pin the mechanism in this process. The last one + * runs a real process and sends it real signals, because the whole point is what + * the operating system does to a process that does not listen: in that test the + * child gives its "lock" back only if it outlives the signal. + * + * Author: Gowtham + */ + +import test from 'node:test' +import assert from 'node:assert/strict' +import { spawn } from 'node:child_process' +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' +import { fileURLToPath, pathToFileURL } from 'node:url' + +import { STOP_SIGNALS, untilStopped } from './interrupt.js' + +const listeners = (): number[] => STOP_SIGNALS.map((name) => process.listenerCount(name)) + +for (const name of STOP_SIGNALS) { + test(`${name} aborts the work's signal instead of ending the process`, async () => { + const before = listeners() + const result = await untilStopped(async (signal) => { + assert.equal(signal.aborted, false) + process.emit(name) + return signal.aborted + }) + assert.equal(result, true) + assert.deepEqual(listeners(), before, 'the listeners are removed afterwards') + }) +} + +test('a stop that arrives before the work looks is still there when it does', async () => { + const seen = await untilStopped(async (signal) => { + process.emit('SIGTERM') + await new Promise((resolve) => setTimeout(resolve, 20)) + return signal.aborted + }) + assert.equal(seen, true) +}) + +test('work that throws still has its listeners removed', async () => { + const before = listeners() + await assert.rejects( + untilStopped(async () => { + throw new Error('the release failed') + }), + /the release failed/ + ) + assert.deepEqual(listeners(), before) +}) + +test('after a stop, a failed write to the terminal no longer ends the process', async () => { + const before = process.stdout.listenerCount('error') + await untilStopped(async () => { + process.emit('SIGHUP') + assert.equal(process.stdout.listenerCount('error'), before + 1) + assert.equal(process.stderr.listenerCount('error') > 0, true) + }) + assert.equal(process.stdout.listenerCount('error'), before) +}) + +test('with no stop, the result is returned and nothing is aborted', async () => { + assert.equal(await untilStopped(async (signal) => (signal.aborted ? 'stopped' : 'finished')), 'finished') +}) + +/** + * A process that takes a lock, waits, and gives the lock back when told to stop. + * It reports in a file, since after a hangup there may be no terminal to print to. + */ +const HOLDER = ` +import { writeFileSync } from 'node:fs' +const { untilStopped } = await import(process.argv[2]) +const report = process.argv[3] +await untilStopped(async (signal) => { + writeFileSync(report, 'locked') + // A release always has something open, an ssh process at least. A timer stands in + // for it here: with nothing open, Node would exit before any signal arrived. + const open = setInterval(() => {}, 1000) + await new Promise((resolve) => signal.addEventListener('abort', resolve, { once: true })) + clearInterval(open) + // What a real release does on the way out takes a moment and prints as it goes. + await new Promise((resolve) => setTimeout(resolve, 150)) + console.log('giving the lock back') + writeFileSync(report, 'released') +}) +` + +for (const name of ['SIGHUP', 'SIGTERM', 'SIGINT'] as const) { + // Windows has no signals to send another process: `kill` there ends it outright, + // which is what closing a console does ten seconds after delivering SIGHUP. + test(`a real process sent ${name} gives its lock back before it exits`, { skip: process.platform === 'win32' }, async () => { + const directory = mkdtempSync(path.join(tmpdir(), 'nextship-interrupt-')) + try { + const script = path.join(directory, 'holder.mjs') + const report = path.join(directory, 'report') + writeFileSync(script, HOLDER) + const module = pathToFileURL(path.join(path.dirname(fileURLToPath(import.meta.url)), 'interrupt.js')).href + + const child = spawn(process.execPath, [script, module, report], { stdio: 'ignore' }) + const exited = new Promise((resolve) => child.on('close', (code) => resolve(code))) + for (let waited = 0; waited < 5000; waited += 25) { + try { + if (readFileSync(report, 'utf8') === 'locked') break + } catch { + // Not written yet. + } + await new Promise((resolve) => setTimeout(resolve, 25)) + } + assert.equal(readFileSync(report, 'utf8'), 'locked', 'the process took its lock') + + child.kill(name) + assert.equal(await exited, 0, 'it exited on its own, not by the signal') + assert.equal(readFileSync(report, 'utf8'), 'released') + } finally { + rmSync(directory, { recursive: true, force: true }) + } + }) +} diff --git a/packages/cli/src/interrupt.ts b/packages/cli/src/interrupt.ts new file mode 100644 index 0000000..125d730 --- /dev/null +++ b/packages/cli/src/interrupt.ts @@ -0,0 +1,52 @@ +/** + * @nextship/cli: stopping a release from outside + * + * A release on a server holds the app's deploy lock, and nothing breaks that lock + * automatically. A release that is stopped therefore has to be told, so it can put + * things back and let the lock go, rather than be killed where it stands. Ctrl+C + * was handled that way from the start. Closing the terminal was not: that arrives + * as SIGHUP, which ends a process that does not listen for it, and it left the app + * locked until a person removed the lock by hand. `kill`, and a CI runner + * cancelling a job, arrive as SIGTERM and did the same. + * + * All of them now abort one signal the release watches. Listeners are attached + * before the release starts, not after, so a stop that arrives while the lock is + * being taken is remembered and acted on once the release can be given back. + * + * Author: Gowtham + */ + +/** + * SIGBREAK is Ctrl+Break on Windows, where it is the one of these besides Ctrl+C + * a console can send. Windows delivers a closed console as SIGHUP and ends the + * process about ten seconds later whatever it is doing, which is long enough to + * release a lock and not long enough to finish a deployment. + */ +export const STOP_SIGNALS: NodeJS.Signals[] = + process.platform === 'win32' ? ['SIGINT', 'SIGHUP', 'SIGTERM', 'SIGBREAK'] : ['SIGINT', 'SIGHUP', 'SIGTERM'] + +/** + * Runs `work` with a signal that aborts when the process is asked to stop, and + * leaves the process's signal handling as it found it. + */ +export async function untilStopped(work: (signal: AbortSignal) => Promise): Promise { + const stopping = new AbortController() + const stop = (): void => { + // After a hangup the terminal is gone, and a write to it fails with an error + // event nobody is listening for, which would end the process before the + // release had been given back. The output has no reader left to miss it. + process.stdout.on('error', ignore) + process.stderr.on('error', ignore) + stopping.abort() + } + for (const name of STOP_SIGNALS) process.on(name, stop) + try { + return await work(stopping.signal) + } finally { + for (const name of STOP_SIGNALS) process.off(name, stop) + process.stdout.off('error', ignore) + process.stderr.off('error', ignore) + } +} + +function ignore(): void {} diff --git a/packages/cli/src/rollback.ts b/packages/cli/src/rollback.ts index 6b78068..6dc3007 100644 --- a/packages/cli/src/rollback.ts +++ b/packages/cli/src/rollback.ts @@ -20,6 +20,7 @@ import { NextshipError } from './errors.js' import type { ProjectInfo } from './detect.js' import { ownedApp } from './owned-app.js' import type { DeploymentRecord } from './targets/target.js' +import { untilStopped } from './interrupt.js' import { detail, ok, step } from './util/log.js' export interface RollbackOptions { @@ -73,16 +74,9 @@ export async function rollback(project: ProjectInfo, options: RollbackOptions): step('Rolling back') // A rollback starts a container and waits for it exactly as a deploy does, so - // Ctrl+C here is handled rather than left to kill the process holding the - // server's deploy lock. - const stopping = new AbortController() - const interrupt = (): void => stopping.abort() - process.on('SIGINT', interrupt) - try { - await app.target.rollback(app.appId, target.id, detail, stopping.signal) - } finally { - process.off('SIGINT', interrupt) - } + // Ctrl+C, a closed terminal and `kill` are handled rather than left to end the + // process holding the server's deploy lock. + await untilStopped((signal) => app.target.rollback(app.appId, target.id, detail, signal)) const address = await app.target.address(app.appId) ok( diff --git a/packages/cli/src/targets/digitalocean-target.ts b/packages/cli/src/targets/digitalocean-target.ts index 0875f8f..f2c2f48 100644 --- a/packages/cli/src/targets/digitalocean-target.ts +++ b/packages/cli/src/targets/digitalocean-target.ts @@ -152,7 +152,7 @@ export class DigitalOceanTarget implements Target { } async builder(): Promise { - return { dockerHost: null, cacheScope: null, warning: null, close: async () => {} } + return { dockerHost: null, cacheScope: null, close: async () => {} } } /** diff --git a/packages/cli/src/targets/target.ts b/packages/cli/src/targets/target.ts index 1babd8a..f3bdfc4 100644 --- a/packages/cli/src/targets/target.ts +++ b/packages/cli/src/targets/target.ts @@ -187,8 +187,6 @@ export interface ImageBuilder { * machine deploying the app reuses one cache there. Null keeps the local rule. */ cacheScope: string | null - /** Something the user must know about how this builder is reached, or null. */ - warning: string | null /** * Whether a build that just failed lost its connection to the daemon rather than * failing on its own merits, asked of the daemon itself. Absent where there is no diff --git a/packages/cli/src/targets/vm/docker-bridge.test.ts b/packages/cli/src/targets/vm/docker-bridge.test.ts new file mode 100644 index 0000000..bb7d39e --- /dev/null +++ b/packages/cli/src/targets/vm/docker-bridge.test.ts @@ -0,0 +1,192 @@ +/** + * @nextship/cli: the bridge between a local address and a dialled process + * + * The bridge is tested with stand-ins for `ssh ... docker system dial-stdio`: a + * process that echoes, one that answers after the request is complete, and one that + * exits the way ssh does when it loses the server. The address is a named pipe on + * Windows and a socket file elsewhere, so the same tests run on every platform CI + * has. That the real Docker client builds through it is in `docs/design.md` §9.3. + * + * Author: Gowtham + */ + +import test from 'node:test' +import assert from 'node:assert/strict' +import { spawn } from 'node:child_process' +import { mkdtempSync, rmSync } from 'node:fs' +import { connect, type Socket } from 'node:net' +import { tmpdir } from 'node:os' +import path from 'node:path' + +import { openDockerBridge, windowsPipe, type Dialled } from './docker-bridge.js' + +/** A stand-in for the dialled ssh: Node running one line. */ +const stand = (script: string) => (): Dialled => spawn(process.execPath, ['-e', script], { stdio: ['pipe', 'pipe', 'ignore'] }) + +const ECHO = 'process.stdin.pipe(process.stdout)' +/** Answers only once the request is complete, as an HTTP exchange over a hijacked connection does. */ +const ANSWER_AT_END = "let n = 0; process.stdin.on('data', (c) => (n += c.length)); process.stdin.on('end', () => process.stdout.write('got ' + n))" +const SSH_LOST_SERVER = 'process.exit(255)' +const COMMAND_FAILED = 'process.exit(1)' + +/** An address to listen on, and what removes it afterwards. */ +function address(): { path: string; remove: () => void } { + if (process.platform === 'win32') return { path: windowsPipe().path, remove: () => {} } + const directory = mkdtempSync(path.join(tmpdir(), 'nextship-bridge-')) + return { path: path.join(directory, 'docker.sock'), remove: () => rmSync(directory, { recursive: true, force: true }) } +} + +function client(target: string): Promise { + return new Promise((resolve, reject) => { + const socket = connect({ path: target, allowHalfOpen: true }) + socket.once('connect', () => resolve(socket)) + socket.once('error', reject) + }) +} + +/** Everything the socket receives until the other side is done sending. */ +function received(socket: Socket): Promise { + return new Promise((resolve) => { + let text = '' + socket.on('data', (chunk: Buffer) => (text += chunk.toString())) + for (const event of ['end', 'close', 'error']) socket.on(event, () => resolve(text)) + }) +} + +/** Resolves once the socket has received exactly `expected`, then lets the connection go. */ +function receives(socket: Socket, expected: string): Promise { + return new Promise((resolve) => { + let text = '' + socket.on('data', (chunk: Buffer) => { + text += chunk.toString() + if (text.length >= expected.length) { + socket.destroy() + resolve(text) + } + }) + for (const event of ['end', 'close', 'error']) socket.on(event, () => resolve(text)) + }) +} + +test('the pipe name is unguessable and matches the DOCKER_HOST that reaches it', () => { + const first = windowsPipe() + const second = windowsPipe() + assert.match(first.path, /^\\\\\.\\pipe\\nextship-docker-[0-9a-f]{32}$/) + assert.equal(first.dockerHost, `npipe:////./pipe/${first.path.split('\\').pop()}`) + assert.notEqual(first.path, second.path) +}) + +test('bytes sent are carried to the dialled process and its output comes back', async () => { + const where = address() + const bridge = await openDockerBridge(where.path, stand(ECHO)) + try { + const socket = await client(where.path) + const all = receives(socket, 'GET /_ping HTTP/1.1\r\n\r\n') + socket.write('GET /_ping HTTP/1.1\r\n\r\n') + assert.equal(await all, 'GET /_ping HTTP/1.1\r\n\r\n') + assert.equal(bridge.lost(), false) + } finally { + await bridge.close() + where.remove() + } +}) + +// A Windows named pipe cannot close one direction and keep the other, and Docker's +// client there never asks it to, so this holds only where the address is a socket. +test('closing the sending side completes the request without cutting off the answer', { skip: process.platform === 'win32' }, async () => { + const where = address() + const bridge = await openDockerBridge(where.path, stand(ANSWER_AT_END)) + try { + const socket = await client(where.path) + const all = received(socket) + socket.write('12345') + socket.end('67890') + assert.equal(await all, 'got 10') + } finally { + await bridge.close() + where.remove() + } +}) + +test('every connection gets a process of its own', async () => { + const where = address() + let dialled = 0 + const bridge = await openDockerBridge(where.path, () => { + dialled += 1 + return stand(ECHO)() + }) + try { + const sockets = await Promise.all([client(where.path), client(where.path), client(where.path)]) + const answers = sockets.map((socket, index) => receives(socket, `connection ${index}`)) + sockets.forEach((socket, index) => socket.write(`connection ${index}`)) + assert.deepEqual(await Promise.all(answers), ['connection 0', 'connection 1', 'connection 2']) + assert.equal(dialled, 3) + } finally { + await bridge.close() + where.remove() + } +}) + +test("ssh's own failure code marks the connection as lost, and closes the client", async () => { + const where = address() + const bridge = await openDockerBridge(where.path, stand(SSH_LOST_SERVER)) + try { + const socket = await client(where.path) + assert.equal(await received(socket), '') + // The client is let go when the process's output ends, which can be a moment + // before its exit code is known. A build asks only after it has failed. + for (let waited = 0; waited < 2000 && !bridge.lost(); waited += 10) await new Promise((resolve) => setTimeout(resolve, 10)) + assert.equal(bridge.lost(), true) + } finally { + await bridge.close() + where.remove() + } +}) + +test('a remote command that fails is not a lost connection', async () => { + const where = address() + const bridge = await openDockerBridge(where.path, stand(COMMAND_FAILED)) + try { + const socket = await client(where.path) + await received(socket) + // Long enough for the exit code to have arrived, so this is not passing early. + await new Promise((resolve) => setTimeout(resolve, 300)) + assert.equal(bridge.lost(), false) + } finally { + await bridge.close() + where.remove() + } +}) + +test('close ends connections still open and stops listening', async () => { + const where = address() + const bridge = await openDockerBridge(where.path, stand(ECHO)) + try { + const socket = await client(where.path) + const echoed = new Promise((resolve) => socket.once('data', resolve)) + const all = received(socket) + socket.write('still open') + await echoed + await bridge.close() + await all + assert.equal(bridge.lost(), false, 'a process this side stopped is not a lost connection') + await assert.rejects(client(where.path)) + } finally { + where.remove() + } +}) + +test('an address that cannot be listened on is an error with a next action', async () => { + const where = address() + const first = await openDockerBridge(where.path, stand(ECHO)) + try { + await assert.rejects(openDockerBridge(where.path, stand(ECHO)), (error: unknown) => { + assert.match((error as Error).message, /Could not open a local address for the server's Docker/) + assert.match((error as { action: string }).action, /--build local/) + return true + }) + } finally { + await first.close() + where.remove() + } +}) diff --git a/packages/cli/src/targets/vm/docker-bridge.ts b/packages/cli/src/targets/vm/docker-bridge.ts new file mode 100644 index 0000000..bf98e36 --- /dev/null +++ b/packages/cli/src/targets/vm/docker-bridge.ts @@ -0,0 +1,105 @@ +/** + * @nextship/cli: a private local address for a server's Docker + * + * A remote build needs the Docker client on this machine to reach the daemon on + * the server. Elsewhere that is an SSH forward to a Unix socket in a directory only + * this user can enter. Windows' OpenSSH cannot forward to a Unix socket, and the + * first answer was a loopback TCP port, which nothing authenticates: for as long as + * a build ran, any process on the machine could drive the server's Docker, which is + * root on the server, and so could a web page open in a browser, by rebinding a + * name to 127.0.0.1. + * + * This listens on a local address a browser cannot open, a named pipe on Windows, + * and gives each connection its own `docker system dial-stdio` on the server: the + * command Docker itself uses to carry a connection over SSH, reached here through + * nextship's own ssh so the pinned host key still applies. No port is opened on + * either machine. + * + * A named pipe Node creates takes Windows' default permissions for one: full + * control for the user who made it, SYSTEM and Administrators, and read only for + * everyone else, who therefore cannot send the daemon a request. Read from a pipe + * this module made, on Windows 11. + * + * Author: Gowtham + * Design: ../../../../../docs/design.md §9.3 + */ + +import { randomBytes } from 'node:crypto' +import type { ChildProcessByStdio } from 'node:child_process' +import { createServer, type Socket } from 'node:net' +import type { Readable, Writable } from 'node:stream' +import { NextshipError } from '../../errors.js' + +/** The process that carries one connection: bytes in on stdin, bytes back on stdout. */ +export type Dialled = ChildProcessByStdio + +/** ssh's own exit code when the connection failed, as opposed to the remote command's. */ +const SSH_FAILED = 255 + +export interface DockerBridge { + /** Whether a connection ended because ssh lost the server, rather than because Docker was done with it. */ + lost(): boolean + /** Stops listening and ends every connection still open. */ + close(): Promise +} + +/** A pipe name nobody else is using, and the `DOCKER_HOST` that reaches it. */ +export function windowsPipe(): { path: string; dockerHost: string } { + const name = `nextship-docker-${randomBytes(16).toString('hex')}` + return { path: `\\\\.\\pipe\\${name}`, dockerHost: `npipe:////./pipe/${name}` } +} + +/** + * Listens on `path` and carries each connection over the process `dial` starts, + * bytes in to its stdin and its stdout back out. + */ +export async function openDockerBridge(path: string, dial: () => Dialled): Promise { + const open = new Set<{ socket: Socket; child: Dialled }>() + let lost = false + + // Half-open, so that a client closing its sending side to say a request is + // complete can still read the answer, where the address supports that. A Windows + // named pipe does not, and Docker's client there never asks for it. + const server = createServer({ allowHalfOpen: true }, (socket) => { + const child = dial() + const connection = { socket, child } + open.add(connection) + socket.pipe(child.stdin) + child.stdout.pipe(socket, { end: false }) + // The process's last output is sent before the client is let go. Its exit is + // not waited for: on Windows the process's `close` never comes while the client + // still holds the pipe that feeds its stdin. + child.stdout.on('end', () => socket.end(() => socket.destroy())) + // The client going away ends the process, and with it the command on the server. + socket.on('error', () => child.kill()) + socket.on('close', () => child.kill()) + child.stdin.on('error', () => {}) + child.on('error', () => { + lost = true + open.delete(connection) + socket.destroy() + }) + child.on('exit', (code) => { + if (code === SSH_FAILED) lost = true + open.delete(connection) + }) + }) + + await new Promise((resolve, reject) => { + server.once('error', (error) => + reject(new NextshipError(`Could not open a local address for the server's Docker: ${error.message}`, 'Check the message above, or build here with `--build local`.')) + ) + server.listen(path, resolve) + }) + + return { + lost: () => lost, + close: async () => { + for (const { socket, child } of open) { + child.kill() + socket.destroy() + } + await new Promise((resolve) => server.close(() => resolve())) + }, + } +} diff --git a/packages/cli/src/targets/vm/host-key.test.ts b/packages/cli/src/targets/vm/host-key.test.ts index 177051f..26fa695 100644 --- a/packages/cli/src/targets/vm/host-key.test.ts +++ b/packages/cli/src/targets/vm/host-key.test.ts @@ -11,7 +11,7 @@ import { test } from 'node:test' import assert from 'node:assert/strict' -import { fingerprint, hostKeyAlgorithms, knownHostsLine, parseKeyLine, parseKeyscan, preferredKey } from './host-key.js' +import { fingerprint, hostKeyAlgorithms, knownHostsLine, parseKeyLine, parseKeyscan, preferredKey, recordArguments } from './host-key.js' const ED25519 = 'AAAAC3NzaC1lZDI1NTE5AAAAIBAzLBUdetTLzoVHIVLcV5jAUBaPHnXQVkBwGe8ZAFq1' const RSA = @@ -58,3 +58,26 @@ test('the negotiated algorithm is the pinned key type, with SHA-2 signatures for test('the known_hosts line names the alias, not the host', () => { assert.equal(knownHostsLine('nextship-server', { type: 'ssh-ed25519', key: ED25519 }), `nextship-server ssh-ed25519 ${ED25519}\n`) }) + +test('recording a key through ssh offers no credentials and trusts nothing it keeps', () => { + const args = recordArguments('203.0.113.10', 2222, '/tmp/x/known_hosts', 'ssh-ed25519') + const option = (name: string): string | undefined => args.find((entry) => entry.startsWith(`${name}=`)) + assert.equal(option('PreferredAuthentications'), 'PreferredAuthentications=none') + assert.equal(option('BatchMode'), 'BatchMode=yes') + assert.equal(option('UserKnownHostsFile'), 'UserKnownHostsFile=/tmp/x/known_hosts') + assert.equal(option('GlobalKnownHostsFile'), 'GlobalKnownHostsFile=none') + assert.equal(option('HostKeyAlgorithms'), 'HostKeyAlgorithms=ssh-ed25519') + assert.deepEqual(args.slice(-4), ['-p', '2222', '--', 'nextship-host-key@203.0.113.10']) +}) + +test('an RSA key is asked for through its SHA-2 signature algorithms', () => { + assert.ok(recordArguments('example.com', 22, 'k', 'ssh-rsa').includes('HostKeyAlgorithms=rsa-sha2-512,rsa-sha2-256')) +}) + +test('the line ssh records, with or without a port in the host, is read like a scanned one', () => { + const key = 'AAAAC3NzaC1lZDI1NTE5AAAAIDoUZlBI2DUVz7mhIViV2kTXdcpny/Fl/PixLiXQCabc' + assert.deepEqual(parseKeyscan(`203.0.113.10 ssh-ed25519 ${key} +`), [{ type: 'ssh-ed25519', key }]) + assert.deepEqual(parseKeyscan(`[203.0.113.10]:2222 ssh-ed25519 ${key} +`), [{ type: 'ssh-ed25519', key }]) +}) diff --git a/packages/cli/src/targets/vm/host-key.ts b/packages/cli/src/targets/vm/host-key.ts index 6b01905..dbf102f 100644 --- a/packages/cli/src/targets/vm/host-key.ts +++ b/packages/cli/src/targets/vm/host-key.ts @@ -19,8 +19,12 @@ */ import { createHash } from 'node:crypto' +import { existsSync } from 'node:fs' +import { mkdtemp, readFile, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import path from 'node:path' import { NextshipError } from '../../errors.js' -import { captureStrict } from '../../util/exec.js' +import { capture, captureStrict } from '../../util/exec.js' /** Key types in order of preference. ed25519 first: smallest, fastest, no parameter choices to get wrong. */ const PREFERRED_TYPES = ['ssh-ed25519', 'ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ssh-rsa'] @@ -88,22 +92,35 @@ export function knownHostsLine(alias: string, key: HostKey): string { return `${alias} ${key.type} ${key.key}\n` } -/** Reads the server's host key over the network, for trust on first use. */ +/** + * Reads the server's host key over the network, for trust on first use. + * + * `ssh-keyscan` first. The one Windows ships (OpenSSH for Windows 9.5) cannot scan + * a current server at all: it offers a key exchange it was not built with, and + * ends with `choose_kex: unsupported KEX method sntrup761x25519-sha512@openssh.com` + * against the OpenSSH 9.6 of Ubuntu 24.04. Found by running `server add` from + * Windows against a Droplet. `ssh` on the same machine negotiates correctly, so + * when the scan fails the key is read by letting `ssh` record it instead. + */ export async function scanHostKey(host: string, port: number): Promise { - let output: string + let keys: HostKey[] try { - output = await captureStrict('ssh-keyscan', ['-T', '15', '-t', 'ed25519,ecdsa,rsa', '-p', String(port), host], { + const output = await captureStrict('ssh-keyscan', ['-T', '15', '-t', 'ed25519,ecdsa,rsa', '-p', String(port), host], { cwd: process.cwd(), env: process.env, }) + keys = parseKeyscan(output) } catch (error) { - throw new NextshipError( - `Could not read the host key of ${host}:${port}.`, - `Check that the server is up and that SSH listens on port ${port}. ${error instanceof Error ? error.message : ''}`.trim() - ) + keys = await recordedHostKeys(host, port) + if (keys.length === 0) { + throw new NextshipError( + `Could not read the host key of ${host}:${port}.`, + `Check that the server is up and that SSH listens on port ${port}. ${error instanceof Error ? error.message : ''}`.trim() + ) + } } - const key = preferredKey(parseKeyscan(output)) + const key = preferredKey(keys) if (!key) { throw new NextshipError( `${host}:${port} offered no host key nextship recognises.`, @@ -112,3 +129,44 @@ export async function scanHostKey(host: string, port: number): Promise } return key } + +/** + * The arguments that make `ssh` write a server's host key of one type into + * `knownHosts` and stop. It offers no way to authenticate, so it sends no key and + * no password and ends at "permission denied", after the key exchange has + * recorded what the server presented. Nothing is trusted by this: the file is + * thrown away, and the key goes through the same printed fingerprint and `--yes` + * as a scanned one. + */ +export function recordArguments(host: string, port: number, knownHosts: string, type: string): string[] { + return [ + '-o', 'BatchMode=yes', + '-o', 'PreferredAuthentications=none', + '-o', 'StrictHostKeyChecking=no', + '-o', `UserKnownHostsFile=${knownHosts}`, + '-o', 'GlobalKnownHostsFile=none', + '-o', `HostKeyAlgorithms=${hostKeyAlgorithms({ type, key: '' })}`, + '-o', 'CheckHostIP=no', + '-o', 'ConnectTimeout=15', + '-o', 'LogLevel=ERROR', + '-p', String(port), + '--', `nextship-host-key@${host}`, + ] +} + +/** The server's host keys as `ssh` records them, the first type it presents in order of preference. */ +async function recordedHostKeys(host: string, port: number): Promise { + const directory = await mkdtemp(path.join(tmpdir(), 'nextship-host-key-')) + try { + const knownHosts = path.join(directory, 'known_hosts') + for (const type of PREFERRED_TYPES) { + // Expected to fail: the connection is refused once the key has been recorded. + await capture('ssh', recordArguments(host, port, knownHosts, type), { cwd: process.cwd() }) + const recorded = existsSync(knownHosts) ? parseKeyscan(await readFile(knownHosts, 'utf8')) : [] + if (recorded.length > 0) return recorded + } + return [] + } finally { + await rm(directory, { recursive: true, force: true }) + } +} diff --git a/packages/cli/src/targets/vm/vm-target.ts b/packages/cli/src/targets/vm/vm-target.ts index 887dedd..db3f5ad 100644 --- a/packages/cli/src/targets/vm/vm-target.ts +++ b/packages/cli/src/targets/vm/vm-target.ts @@ -36,12 +36,12 @@ import { hostname } from 'node:os' import path from 'node:path' import { connect as tlsConnect } from 'node:tls' import { lookup, resolve4 } from 'node:dns/promises' -import { connect as netConnect, createServer } from 'node:net' import { NextshipError } from '../../errors.js' import { docsUrl } from '../../links.js' import type { ProjectConfig, ServerRecord } from '../../config.js' import { DEFAULT_KEEP } from '../../images.js' import { pipeline } from '../../util/exec.js' +import { openDockerBridge, windowsPipe } from './docker-bridge.js' import type { AppAddress, AppRef, @@ -622,25 +622,24 @@ export class VmTarget implements Target { * A remote build reaches the server's Docker daemon through an SSH forward of * its socket, with the same pinned host key as every other connection. * `DOCKER_HOST=ssh://` would be simpler and would bypass that pinning, because - * Docker starts its own ssh with the user's ordinary host key checking. + * Docker starts its own ssh with the user's ordinary host key checking. The + * forward lands on a Unix socket in a directory only this user can enter. */ async builder(): Promise { const cacheScope = `${this.name}@${this.server.host}:${this.server.port}` - if (this.build === 'local') return { dockerHost: null, cacheScope: null, warning: null, close: async () => {} } + if (this.build === 'local') return { dockerHost: null, cacheScope: null, close: async () => {} } const ssh = await this.ssh() // The server's clock, not this machine's, so a later look at its journal starts // where this build did however far apart the two clocks are. const since = (await ssh.run('date +%s', 'read the server clock')).trim() - // Windows' OpenSSH cannot forward to a local Unix socket, so it forwards a - // loopback port, which any local process could use while the build runs. - const windows = process.platform === 'win32' - const port = windows ? await freePort() : 0 - const local = windows ? `127.0.0.1:${port}` : path.join(ssh.privateDir(), 'docker.sock') + if (process.platform === 'win32') return this.bridgedBuilder(cacheScope, since) + + const local = path.join(ssh.privateDir(), 'docker.sock') // A forward that was killed leaves its socket file behind, and readiness below is the // file existing, so a second forward on the same path looked ready before it had bound // and the build met a dead socket. Found by cutting a forward mid-build on a Droplet. - if (!windows) rmSync(local, { force: true }) + rmSync(local, { force: true }) const args = [ '-nNT', ...ssh.options({ multiplex: false }), @@ -654,26 +653,19 @@ export class VmTarget implements Target { child.stderr?.on('data', (chunk: Buffer) => (stderr += chunk.toString())) const exited = new Promise((resolve) => child.on('close', (code) => resolve(code ?? 255))) - const ready = async (): Promise => (windows ? canConnect(port) : existsSync(local)) const deadline = Date.now() + 20_000 for (;;) { - if (await ready()) break + if (existsSync(local)) break const done = await Promise.race([exited, sleep(250).then(() => null)]) if (done !== null || Date.now() > deadline) { child.kill() - throw new NextshipError( - `Could not reach Docker on ${this.server.host} through SSH: ${stderr.trim() || 'the forward did not open'}`, - 'Check `nextship server add` reports the server as set up, or build here with `--build local`.' - ) + throw this.dockerUnreachable(stderr.trim() || 'the forward did not open') } } return { - dockerHost: windows ? `tcp://127.0.0.1:${port}` : `unix://${local}`, + dockerHost: `unix://${local}`, cacheScope, - warning: windows - ? `The server's Docker is reachable on 127.0.0.1:${port} by any process on this machine until the build finishes.` - : null, // Two kinds of evidence, either enough. The forward ending on its own comes first: // a build fails the moment its forward dies, before the daemon at the far end has // logged anything. When the forward is still up, the daemon's journal says whether @@ -687,6 +679,39 @@ export class VmTarget implements Target { } } + /** + * The builder on Windows, whose OpenSSH cannot forward to a Unix socket. Each + * connection the Docker client opens is carried by its own ssh running + * `docker system dial-stdio` on the server, behind a named pipe (`docker-bridge.ts`). + * Docker is asked for its version first, so a server it cannot be reached on fails + * here with the reason rather than inside the build as a broken pipe. + */ + private async bridgedBuilder(cacheScope: string, since: string): Promise { + const ssh = await this.ssh() + const reachable = await ssh.exec("docker version --format '{{.Server.Version}}'") + if (reachable.code !== 0) throw this.dockerUnreachable(reachable.stderr.trim() || `exit code ${reachable.code}`) + + const pipe = windowsPipe() + const bridge = await openDockerBridge(pipe.path, () => + spawn('ssh', ['-T', ...ssh.options({ multiplex: false }), ...ssh.destination(), 'docker system dial-stdio'], { + stdio: ['pipe', 'pipe', 'ignore'], + }) + ) + return { + dockerHost: pipe.dockerHost, + cacheScope, + lostConnection: async () => bridge.lost() || (await this.buildSessionLost(since)), + close: () => bridge.close(), + } + } + + private dockerUnreachable(reason: string): NextshipError { + return new NextshipError( + `Could not reach Docker on ${this.server.host} through SSH: ${reason}`, + 'Check `nextship server add` reports the server as set up, or build here with `--build local`.' + ) + } + /** * Whether the server's Docker logged a build session as lost since `since`, in seconds * since the epoch by the server's clock. The nextship user reads the journal through @@ -1688,28 +1713,6 @@ export class VmTarget implements Target { const sleep = (ms: number): Promise => new Promise((resolve) => setTimeout(resolve, ms)) -async function freePort(): Promise { - return new Promise((resolve, reject) => { - const server = createServer() - server.on('error', reject) - server.listen(0, '127.0.0.1', () => { - const address = server.address() - server.close(() => resolve(typeof address === 'object' && address ? address.port : 0)) - }) - }) -} - -async function canConnect(port: number): Promise { - return new Promise((resolve) => { - const socket = netConnect(port, '127.0.0.1') - socket.once('connect', () => { - socket.destroy() - resolve(true) - }) - socket.once('error', () => resolve(false)) - }) -} - /** * What a `domain add` plan should warn about DNS. Only a warning: the record is * often created after attaching, and Caddy retries the certificate once it diff --git a/site/content/docs/vm.mdx b/site/content/docs/vm.mdx index 5e194d8..18e3ee9 100644 --- a/site/content/docs/vm.mdx +++ b/site/content/docs/vm.mdx @@ -46,8 +46,8 @@ steps. Running `server add` again changes nothing. nextship deploy --yes ``` -The image is built on the server's own Docker by default, through an SSH forward that keeps -the pinned host key. `--build local` builds on your machine instead and streams the image +The image is built on the server's own Docker by default, reached over SSH with the pinned +host key, through a private socket or, on Windows, a named pipe. No port is opened. `--build local` builds on your machine instead and streams the image over SSH, which suits a server with little memory. If the build's connection to the server's Docker drops mid-build, which has happened right after `server add` on a fresh server, `deploy` says so and builds once more; any other build failure is reported at once.