Skip to content

The server target's next phase: hardening, open tabs across deploys, instant rollback, previews, HTTPS on the address - #7

Open
Ragul2105 wants to merge 12 commits into
mainfrom
feat/finalized-roadmap
Open

Ragul2105 wants to merge 12 commits into
mainfrom
feat/finalized-roadmap

Conversation

@Ragul2105

Copy link
Copy Markdown
Collaborator

The server target's next phase, in the order docs/roadmap.md records, each shipped and verified on its own.

What changes

  • Hardened app containers. Read-only root filesystem, a 64 MB noexec /tmp, no capabilities, no-new-privileges, 512 processes. The December 2025 break-ins through CVE-2025-55182 rewrote next.config.js and ran miners from /tmp; none of that can persist now. Behaviour change: an app that writes outside .next fails with EROFS (documented).
  • Cache guard. A timer keeps what each app writes at runtime under 10% of the disk (at least 1 GiB), least recently used first, never deleting a file the build produced.
  • Open tabs keep working across a deployment. The replaced deployment keeps running for an hour, and Caddy sends it every request that names its build (x-deployment-id, ?dpl=). This sidesteps Server Action responses bypass deploymentId version skew detection, crashing stale tabs with "module factory is not available" vercel/next.js#99165. A retire timer stops it afterwards.
  • Instant rollback to that deployment: a Caddy switch to the running container, about 210 to 240 ms.
  • Previews. --preview <name> deploys and manages a second app on the same server, found again from the server record so CI needs only the committed nextship.json, with noindex and its own env. A workflow for one preview per pull request is in docs/vm.md §6.
  • HTTPS on the server's address. On a public IPv4 server, the first app also answers https://<address> with Let's Encrypt's six-day IP certificate.
  • Faster switch. The health wait went from 4.1 s to 1.1 s; a warm deploy is now about 13.5 s, most of it next build.
  • doctor prints crontab lines for vercel.json crons, and a next.config snippet for the routes that carry over unchanged.
  • Nightly compatibility suite on the newest canary, published to an evidence branch that the evidence page reads.
  • macOS: the prune tests pass there now, and CI runs macOS.
  • Recorded as not built: promoting one build to another environment (a spike showed Turbopack folds placeholder values), and stripping x-middleware-subrequest (no supported Next.js version is affected).

How it was verified

  • conformance/vm/e2e.sh passes 25 of 25 on a local Ubuntu 24.04 stand-in, with new checks for hardening, the cache guard, skew routing, instant rollback, retire and previews.
  • The cache guard was run on a 12 GB-disk stand-in, including evicting a live page and image.
  • HTTPS on the address was verified against Pebble.
  • Unit tests pass on Linux (347 of 348, 1 skipped) and macOS; the conformance tests and the site build pass.

Not yet verified (needs GitHub or a real server)

  • The nightly publish job: runs only after this reaches main.
  • The preview-per-pull-request workflow in docs/vm.md §6: never run in GitHub Actions.
  • HTTPS on the address against Let's Encrypt itself, on a server with a public IPv4 address.
  • Existing servers need nextship server add again for setup version 3 (cache guard and retire timers). server status says so.

🤖 Generated with Claude Code

Ragul2105 and others added 8 commits September 28, 2026 22:46
An app on a server now runs read-only: only its two .next volumes and a
64 MB noexec /tmp are writable, every capability is dropped,
no-new-privileges is set and it is limited to 512 processes. The
December 2025 break-ins through CVE-2025-55182 on self-hosted Next.js
rewrote next.config.js and lockfiles and ran miners from /tmp; none of
that can persist now. e2e.sh checks it from inside a deployed container
after it has regenerated a page and optimized an image.

server add also installs a cache guard, run every ten minutes, that
keeps each app's runtime writes under 10% of the disk (never under
1 GiB), deleting the least recently used first. Only files Next.js
wrote at runtime are counted, never the build's own. Verified on a
stand-in with a 12 GB disk, including evicting a live page and image:
both were served, rendered again and cached again, also after a
restart. Setup version 3.

The full e2e suite passes on a local Ubuntu 24.04 stand-in, 21 of 21.
The roadmap records the next phase of the server target in order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… site

The compatibility suite now runs every night against the newest Next.js
canary instead of weekly. A scheduled run publishes its score to the
evidence branch through conformance/publish-evidence.mjs: latest.json
and ninety nights of history.json. A run that scored no suites or lacks
its Next.js version is refused rather than published, and a run started
by hand is never published.

The evidence page reads latest.json in the browser. Checked in the
browser pane in both states: with no file yet it says the result could
not be read and links to the runs; with a result it shows the date,
Next.js version and commit, suites and assertions passing, and the
failing suites. The markdown served to agents gives the JSON's address.

Not yet run on GitHub: the first scheduled run after this reaches main
is what verifies the publish job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A spike built a client component with placeholder NEXT_PUBLIC_ values on
Next.js 16.3.4 with Turbopack. A value used as a string or in a template
literal survived and could be replaced, but FLAG === 'on' ? a : b
compiled to the losing branch alone, and URL.length > 30 ? a : b
compiled to the branch the placeholder's own length chose while the
placeholder still appeared in the bundle. The second is undetectable, so
a replacement would look complete and ship an app that decided on the
placeholder. The roadmap moves it to Deliberately not built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k, previews

A tab loaded before a deployment names its build on every request:
x-deployment-id on navigations and Server Actions, ?dpl= on scripts,
measured in a browser against Next.js 16.3.4. The deployment a new build
replaces now keeps running for an hour, and Caddy routes requests naming
it there, with the live container as the fallback once it stops. That
sidesteps Next.js issue #99165, stale tabs failing on Server Actions. The
same build started again by env push is never kept, since both would
answer to one deployment id. A new retire timer stops the kept container
when its hour is over, and images prune keeps its image.

Rollback to the kept deployment switches Caddy to its running container
instead of starting one, 208 ms in the end-to-end test.

--preview <name> scopes deploy, rollback, logs, env, domain, images and
destroy to a second app on the same server, <app>-<name>. Its server
record names the app it previews, so a CI run with only the committed
nextship.json can find it; nextship.json is never written for it, its
site never claims the server's address and sends X-Robots-Tag noindex.
Found in review: destroy would have cleared the main app's id from
nextship.json for a preview; it no longer touches it.

e2e.sh passes 25 of 25 on a local Ubuntu 24.04 stand-in, with new
checks for skew routing, instant rollback, retire and a preview.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Timed stage by stage, a warm deployment of a one-line change took 16.6 s
and 4.1 s of it was waiting for a container that was ready at once to be
seen as healthy: Docker's first check came two seconds after start and
the deploy polled every two seconds. Both are half a second now. Three
warm deployments then took 15.0, 13.5 and 13.3 s with a 1.1 s wait. Most
of what remains is next build, so the roadmap records that under ten
seconds is not reachable by nextship alone.

e2e.sh passes 25 of 25 with the shorter intervals, including zero
downtime across a deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cron job in vercel.json never runs off Vercel, and nothing reports it.
doctor now prints a crontab line for each, calling its route with
CRON_SECRET as a bearer token, as Vercel does, and says Vercel runs
schedules in UTC. Redirects, rewrites and headers that use only fields
next.config accepts are printed as a next.config snippet; any other, or a
redirect that does not say whether it is permanent, is named rather than
guessed.

This is the part of a Vercel migration plan that needs no Vercel account.
Domains and env var names need Vercel's API and stay in the roadmap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five prune tests created their fixtures under the temporary directory as
named. On macOS that is /var, a link to /private/var, and prune compares
the paths it resolves through links with the root it was given, so every
traced file looked outside the build root. The fixtures are now created
at the real path. prune itself was never affected in the image, where the
root is /src. CI now runs the CLI tests on macOS too, since it ran only
Linux and Windows and never saw this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a server added by a public IPv4 address, the default app's Caddy site
also answers https://<address>, asking Let's Encrypt for a certificate
under its shortlived profile, the only one it issues IP certificates
under since they became generally available on 2026-01-15. deploy
reports the https address only once a TLS connection to it verifies,
and the certificate check no longer sends an IP address as the server
name. Hostnames, IPv6 (fixed in Caddy after the 2.10.2 setup installs)
and private or documentation ranges get no certificate of their own.

Verified against Pebble, Let's Encrypt's test server, on a private
network: Caddy 2.10.2 running the site renderSite produced obtained a
certificate through HTTP-01 on port 80 despite the catch-all HTTP site,
naming IP Address:10.77.0.10 and valid for six days, and a client
trusting only Pebble's root fetched the app over HTTPS. Not yet run
against Let's Encrypt on a real server; the docs say so. e2e.sh still
passes 25 of 25.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
nextship 543f4cf Commit Preview URL

Branch Preview URL
Oct 10 2026, 06:51 PM

@gowtham472 gowtham472 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, Ragul. The design is clear, the tests are real (25 of 25 in e2e.sh, CI green on Linux, Windows and macOS), and the "not yet verified" list is honest. Not ready to merge yet: keeping the previous container running adds a timer, a second container and a second file, and most of what follows is where those meet.

This is from reading the whole diff, not from running it on a server, so each item needs confirming, the Caddy one above all.

Must fix before merging

  1. A server set up by 1.1.x keeps the previous container for good. vm-target.ts (the kept filter after recordLive). Deploy now leaves the replaced container running and relies on the retire timer, but a setup version 2 server has no timer, and deploy neither refuses nor warns: only server status mentions the old setup. The container runs with its full memory limit until the next deployment, and the @previous route stays. Deploy should refuse, or keep nothing, until server add has run again.

  2. The retire timer can stop the live container. retire.sh, the docker stop. It stops whatever previous names once the time has passed, and never checks that the container is not the one Caddy serves. writeDeployments writes deployments.json and previous as two separate commands, and rollbackInstantly switches Caddy before either. A rollback near the hour, or a CLI that dies between the switch and the second write, leaves previous naming the container that is now live, and the timer stops it. --restart unless-stopped does not bring back a stopped container.

  3. rollbackInstantly checks before it takes the lock. It reads the history and inspects the container, then locks, and does not check again. A deploy that goes live in between stops that container (it is no longer the kept one), and the rollback then points Caddy at it.

  4. https://<address> may make plain HTTP redirect before a certificate exists. caddy.ts, the https://${options.ipCertificate} block. Caddy's automatic HTTPS adds an HTTP redirect for a host it manages a certificate for, and a host-specific redirect can win over the catch-all http://:80 site. If Let's Encrypt is slow or cannot issue (443 closed upstream, a rate limit), the only address a first deploy printed would redirect to a URL with no valid certificate. Pebble always issues, so the test cannot show this. Please check the rendered site with caddy adapt, and run once with issuance failing.

  5. Destroying an app orphans its previews. destroyApp removes the named app only, and the plan does not mention previews. Afterwards nextship.json has no appId, so --preview <name> can never match previewOf again, and the previews keep running with their volumes and domains.

Should fix, or decide

  1. The read-only root has no way out, and deploy cannot see the failure. An app that writes a SQLite file or uploads under its working directory deploys, passes the health check on a prerendered route, and then fails with EROFS on its first write. Either a declared writable path, or a check that warns at deploy time.

  2. The memory limit does not count the kept container or previews. memoryLimit still divides 80% of RAM by the number of apps. One app on a 2 GB server gets 1600m, and for an hour two containers are each allowed that. Previews are full apps, and containers already running keep the larger share.

  3. The docs describe HTTPS on the address as working. README and site/content/docs/vm.mdx say the first app answers https://<address>, while this pull request lists that as not yet run against Let's Encrypt. AGENTS.md §5: "Never claim a capability in docs or output that has not been observed working on a real target." Either run it on a public server first, or move the claim to the roadmap until then.

Minor

  1. Cache eviction order depends on access times. cache-guard.sh sorts by %A@. Under relatime that changes at most once a day, and under noatime never, so the guard would evict by age, the hottest entries included.

  2. A long app name with a preview name gives the wrong error. previewAppName can pass 63 characters, and the refusal that follows tells the user to rename the app.

Happy to look again once 1 to 5 are in. After the merge, the plan is a live run on a Droplet (the upgrade from a 1.1.2 server included), then 1.2.0, since the read-only filesystem changes behaviour.

Ragul2105 and others added 4 commits October 10, 2026 23:39
Conflicts in CHANGELOG.md, docs/vm.md, deploy.ts and vm-target.ts.
deploy.ts takes main's untilStopped structure, with the preview's
previewOf on the release and no nextship.json write for a preview.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review point 9. The cache guard sorted by access time alone. Under
relatime that changes at most once a day, and under noatime never, where
every file ties and the busiest entries went with the idle. It now sorts
by the later of a file's last access and last write, so a volume that
keeps no access times is evicted oldest written first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…imit

Review points 5 and 10. Destroying an app removed its id from
nextship.json, which is what finds its previews, so they kept their
containers, volumes and domains with nothing able to reach them. The
plan now names each preview and they are destroyed first, images
included.

A preview name too long for the app went on to the app name check, whose
refusal said to rename the app. It is refused before that, with the
length the preview name may be.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… writable folders

Review points 1 to 4 and 6 to 8 of pull request 7.

1. A server set up by 1.1.x has no retire timer. A deployment reads the
   server's setup version and on an older one keeps nothing, stops the
   replaced container as before, and says to run server add again.
2. The retire timer could stop the live container. previous is emptied
   before Caddy is pointed anywhere and written with the history in one
   command, and retire.sh stops nothing the Caddy site names on its new
   "# live" line.
3. An instant rollback read the history and the container before taking
   the lock. It reads both once the lock is held.
4. With only https://<address> named, Caddy redirected plain HTTP on the
   address to HTTPS before any certificate existed: measured, a 308 with
   a CA that could not be reached. The site names both schemes, and with
   issuance failing HTTP answers 200; with it then succeeding, both do.
6. "writable" in nextship.json lists folders the app may write to, each
   a volume per app that outlives a deployment. doctor warns about
   source that looks like it writes files with none declared; destroy
   and server move say what happens to that data.
7. The default memory limit counts two containers per app where the
   replaced deployment is kept, so both together stay within 80% of RAM.
8. The README and the site no longer describe HTTPS on the address,
   which has not run against Let's Encrypt; the roadmap holds it.

Also found while fixing 2: nothing kept was written as a lone newline,
which the timer would have logged as unreadable every minute.

e2e.sh passes 28 of 28 on a local Ubuntu 24.04 stand-in, with new checks
for a writable folder across a deployment, the timer refusing the live
container, a server with the older setup, and an app's preview going
with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Ragul2105

Copy link
Copy Markdown
Collaborator Author

Thanks for the review, Gowtham. All ten points are addressed, and main (1.1.3) is merged in, so the conflicts are gone. e2e.sh passes 28 of 28 on a local Ubuntu 24.04 stand-in; unit tests pass on Linux (384 of 385, one skipped) and macOS.

Must fix

  1. Servers set up by 1.1.x. A deployment reads the server's setup version. Below 3 it keeps nothing, stops the replaced container as 1.1.x did, writes no @previous route, and says to run server add again. Chosen over refusing so CI keeps deploying after a CLI upgrade. e2e marks the server as version 2, deploys, and checks one container runs and the site has no @previous.
  2. The timer stopping the live container. Three layers: previous is emptied before Caddy is pointed anywhere; the history and previous are written in one remote command, each to a temporary file moved into place; and retire.sh stops nothing the Caddy site names on a new # live <container> line (or when that line is missing). e2e writes the live container into previous with its time up, runs the timer, and checks it refused and logged it.
  3. Lock order in the instant rollback. It takes the lock first, then reads the history and inspects the container. A kept container that is no longer healthy falls back to starting its image.
  4. The HTTP redirect. You were right. With a CA that could not be reached, http://<address> answered 308 to an address with no certificate. The site now names both http://<address> and https://<address>, which makes Caddy serve HTTP rather than redirect. Re-verified against Pebble in two phases: with Pebble down, HTTP on the address answered 200 with no redirect; with Pebble then started, Caddy's retry got the certificate within 60 s and both schemes answered 200. Still not run against Let's Encrypt.
  5. Orphaned previews. destroy names each preview of the app in its plan and destroys them first, images included. e2e deploys a preview, destroys the app, and checks no container, app record or volume is left.

Should fix

  1. Read-only root. Both halves: "writable": ["data"] in nextship.json gives a volume per folder, per app, that outlives a deployment (validated: relative, no leading dot, not under node_modules, no nesting), and doctor warns when the source looks like it writes files and none is declared. destroy says it deletes that data and server move says it does not copy it. e2e writes a note from the first build and reads it from the second.
  2. Memory limit. Where the server keeps a replaced deployment, each app counts as two containers in the 80% share: one app on 2 GB gets 800m, not 1600m twice. Previews were already counted as apps. This lowers the default; --memory still overrides, and the plan shows the figure. Say if you would rather shrink only the kept container.
  3. Docs. The README and the site no longer describe HTTPS on the address. design.md keeps the record as "built, not yet run against Let's Encrypt", and the roadmap holds it until a run on a public address.

Minor

  1. Eviction order. Sorted by the later of last access and last write, so noatime evicts oldest written first.
  2. Long names. previewAppName refuses a name over 63 characters with the length the preview name may be.

Found along the way

  • My first fix for 2 wrote a lone newline when nothing was kept, which the timer would have logged as unreadable every minute. Fixed in the same commit.
  • The hardening check in e2e.sh now allows the mount point Docker creates for a declared writable folder (A /src/data and its parent), and nothing else.

Still open, for the Droplet run

  • The upgrade from a real 1.1.2 server.
  • HTTPS on the address against Let's Encrypt itself.
  • The nightly publish job and the preview workflow, which only run on GitHub.

Commits: 60d3c62 (9), f58c2b3 (5, 10), 543f4cf (1 to 4, 6 to 8).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants