From d748a37762cace8084c5782b62d063a9f2fa002a Mon Sep 17 00:00:00 2001 From: Matthew Zhao Date: Sun, 30 Aug 2026 02:41:35 +0000 Subject: [PATCH] feat: support celld 0.4 --- README.md | 95 +++++++++++-------- .../celld-operator_v1alpha1_workerapp.yaml | 2 +- docs/celld-behaviors.md | 56 +++++------ internal/controller/fleet_resources.go | 18 ++-- internal/controller/fleetstate.go | 50 +++++++--- internal/controller/rollout.go | 10 ++ internal/controller/rollout_test.go | 66 ++++++++++--- .../controller/workerapp_controller_test.go | 20 ++-- 8 files changed, 201 insertions(+), 116 deletions(-) diff --git a/README.md b/README.md index f932dd2..a3605b1 100644 --- a/README.md +++ b/README.md @@ -124,7 +124,7 @@ spec: hostnames: ["chat.acme.example.com"] # routed via the shared Gateway appVersion: sha-abc123 # which bucket deployment is live celld: - image: ghcr.io/denoland/celld:v0.3.0 + image: ghcr.io/denoland/celld:v0.4.0 updateStrategy: Rolling # Recreate for non-rolling celld version changes replicas: 3 bucket: @@ -132,7 +132,7 @@ spec: endpoint: https://ACCOUNT.r2.cloudflarestorage.com region: auto credentialsFrom: - iamRole: arn:aws:iam::123456789012:role/celld-chat # IRSA; or secretRef + iamRole: arn:aws:iam::123456789012:role/celld-chat # IRSA; or secretRef; omit for EKS Pod Identity # bucket: # Azure Blob Storage: the name is the container, # name: az://platform-cells/apps/chat # storageAccount: platformcells # the account is separate (AZURE_STORAGE_ACCOUNT_NAME), @@ -172,44 +172,48 @@ chat Ready sha-abc123 3 0 2m ## Deploying updates -celld nodes load their application deployment from the bucket **at startup -only**, so publishing a new version serves nothing until the fleet restarts. -The workflow: +celld v0.4 polls the bucket's `deploy/current.json` every 30 seconds and +adopts a new deployment in place. New requests use the new generation while +requests already running finish on the old one; resident Durable Objects move +at safe points. `celld deploy` therefore begins serving without a pod restart. + +Keep the `WorkerApp`'s expected version in sync as well: ```sh celld deploy . --bucket s3://platform-cells/apps/chat ... # publish kubectl patch workerapp chat -n tenant-acme --type merge \ - -p '{"spec":{"appVersion":"sha-def456"}}' # roll + -p '{"spec":{"appVersion":"sha-def456"}}' ``` -(Or bump `appVersion` in git and let your GitOps tool apply it.) - -Prefer `celld deploy` to be the whole story? Set `appVersion: auto` and the -operator follows the bucket's `deploy/current.json` itself — a new publish -rolls the fleet within one `--deploy-poll-interval` (default 60s), and -`status.rolledOutAppVersion` reports the concrete version being served. In -pinned mode the same read powers a `DeployTrackingReady: VersionMismatch` -warning when the bucket pointer and the CR disagree (nodes always load the -bucket's version) — but only for fleets using `secretRef` credentials; the -operator does not read the bucket for `iamRole` fleets. Tracking reads use the -fleet's `secretRef` credentials, or the operator's ambient AWS identity -otherwise, and support `s3://` buckets only: a `gs://` or `az://` fleet must -pin `appVersion` (`DeployTrackingReady: UnsupportedStore` says so). +(Or bump `appVersion` in git and let your GitOps tool apply it.) The operator +retains a conservative gated restart when this field changes. v0.4 does not +need that restart to adopt code, but it gives the CR an explicit convergence +point and keeps `status.rolledOutAppVersion` tied to the declared deployment. + +Prefer `celld deploy` to be the whole story? Set `appVersion: auto`. The +operator follows the bucket's `deploy/current.json`; a new publish updates the +expected version within one `--deploy-poll-interval` (default 60s), while celld +itself normally adopts within 30s. In pinned mode the same read powers a +`DeployTrackingReady: VersionMismatch` warning when the bucket pointer and the +CR disagree — but only for fleets using `secretRef` credentials; the operator +does not read the bucket for `iamRole` fleets. Tracking reads use the fleet's +`secretRef` credentials, or the operator's ambient AWS identity otherwise, +and support `s3://` buckets only: a `gs://` or `az://` fleet must pin +`appVersion` (`DeployTrackingReady: UnsupportedStore` says so). A deployment can also need a newer celld than the fleet runs: the manifest -names the features it uses (static assets, Wasm, and since celld 0.3 cron -triggers, D1, sqlite-vec), and an older node rejects it at startup. That -surfaces as the first released pod never becoming Ready, which holds the -rollout there. Bump `spec.celld.image` first, then deploy. - -The operator then runs a **gated rolling update**, not a vanilla one. celld's -documented rule is: after restarting a node, wait until *every* node reports -`restoring=0` before restarting the next — and that restore work lands on the -*peers* that absorbed the drained node's cells, which a stock rolling update -cannot see. The operator owns the StatefulSet partition, releases one ordinal -at a time, and steps only when (a) every released pod runs the new revision -and is Ready, and (b) a live sweep of every pod's `/state` shows fleet-wide -`restoring=0`. Progress is visible in `status.rollout.waitingOn`. +names the features it uses. v0.4 adds Workers KV, Queues, Workflows, and R2 +bindings to the existing static-assets, Wasm, cron, D1, and sqlite-vec +surface. Upgrade `spec.celld.image` before publishing a deployment that needs +the new runtime. + +Pod-template changes still run a **gated rolling update**, not a vanilla one. +The operator owns the StatefulSet partition, releases one ordinal at a time, +and steps only when (a) every released pod runs the new revision and is Ready, +and (b) a live sweep of every pod's `/state` shows fleet-wide +`restoring=0`. v0.4 also gates each replacement's first healthy response on +fleet recovery; the operator's sweep is the conservative fleet-wide backstop. +Progress is visible in `status.rollout.waitingOn`. ### celld version upgrades @@ -222,11 +226,17 @@ event by design; the CR has to ask for it. A jump that skips releases is checked against every boundary in between. Note celld ships security fixes for its latest release only — plan to track head. +v0.4 adds a 40-second complete shutdown bound around drain-token waiting, +handoff, and local durability shutdown. The operator pins that bound and gives +the pod 60 seconds of termination grace, so kubelet does not cut the seal with +SIGKILL. + | Change | Rolling? | Why | | --- | --- | --- | | v0.1 ↔ v0.2 | No, either direction | Mixed fleets break: ownership records changed address semantics and block objects changed format | | v0.2.1 → v0.3.0 | **Yes** | A v0.3 node that cannot replicate to a v0.2 peer falls back to bucket proofs until the peer upgrades | | v0.3 → v0.2 | No | A v0.2 binary cannot read writes still waiting in v0.3's replicated log or bundle objects, so the downgrade can lose acknowledged writes. `Recreate` drains every node first, which seals each log (`node-log close: sealed epoch` in the shutdown log); check for that line before trusting the downgrade | +| v0.3 ↔ v0.4 | No, either direction | v0.4's versioned peer tunnel refuses v0.3 peers, and v0.3 cannot read v0.4's epoch-qualified large Workers KV value references. Use `Recreate`: stop every v0.3 node before starting v0.4 | celld 0.3 also changed the default write-acknowledgement proof from `bucket` to `fleet` (two follower nodes hold the write on disk, the bucket upload @@ -246,7 +256,8 @@ celld has no metrics endpoint yet, so the operator polls each pod's internal | `celld_restoring` | Cold activations in flight | | `celld_evicting` | Cells being evicted | | `celld_shedding` | 1 while pressure-shedding — the hard out-of-capacity signal | -| `celld_rss_bytes` / `celld_in_use_bytes` | Process RSS and the memory the cells hold (celld 0.3+); the gap is allocator retention shedding cannot return | +| `celld_rss_bytes` / `celld_in_use_bytes` | Process RSS and RSS minus allocator retention | +| `celld_cgroup_working_set_bytes` / `celld_cgroup_current_bytes` | v0.4 active cgroup memory used by pressure shedding, and the complete cgroup charge used by the absolute cap; absent outside a readable Linux cgroup | | `celld_container_restarts` / `celld_self_fenced` | Kubelet restart count, and 1 if the last exit was a celld self-fence (exit code 3) — a fence loop means the store or the bucket credential is broken | | `celld_state_up` | 1 if `/state` answered the last poll | @@ -351,24 +362,24 @@ the hammer its own `--prefix`. ## Security notes -- **Bucket credentials are fleet-admin authority.** Scope one IAM role - (or Azure identity) per fleet to that fleet's prefix and nothing else; - prefer IRSA / AKS workload identity over static keys - (`credentialsFrom.secretRef` exists for stores without identity auth). +- **Bucket credentials are fleet-admin authority.** Scope one AWS role + (or Azure identity) per fleet to that fleet's prefix and nothing else. + celld v0.4 reads EKS Pod Identity's injected container credentials with no + pod-spec changes; IRSA uses `iamRole`. Prefer either over `secretRef`. The credential needs put and delete under the prefix: celld's startup probe writes and removes one object under `probe/`. - celld terminates no TLS anywhere: public TLS belongs to the Gateway, and the pod network should be encrypted (Istio ambient, or a CNI with WireGuard) because the peer protocol relies on network confidentiality. -- The deployed Worker owns every public path except `/__celld/health`; - application authentication is the application's job. celld ignores +- The deployed Worker owns every public path except + `/.well-known/celld/health`; application authentication is the + application's job. celld ignores `X-Forwarded-Host` / `X-Forwarded-Proto` unless `spec.trustForwardedHeaders` is set — turn it on only when every hop in front of the fleet replaces both headers. - The internal listener's `/cell/` and `/do/` routes run application code - unauthenticated; only the D1 route (`/__d1/`, what `celld d1` uses) - authenticates with the fleet secret. Keep `:8081` behind the operator's - policies. + unauthenticated. Reserved runtime classes use the HMAC-authenticated + `/runtime/` protocol. Keep `:8081` behind the operator's policies. ## Development diff --git a/config/samples/celld-operator_v1alpha1_workerapp.yaml b/config/samples/celld-operator_v1alpha1_workerapp.yaml index c06aa60..a0e39fa 100644 --- a/config/samples/celld-operator_v1alpha1_workerapp.yaml +++ b/config/samples/celld-operator_v1alpha1_workerapp.yaml @@ -9,7 +9,7 @@ spec: hostnames: ["chat.acme.ezghcloud.com"] appVersion: sha-abc123 celld: - image: ghcr.io/denoland/celld:v0.3.0 + image: ghcr.io/denoland/celld:v0.4.0 updateStrategy: Rolling replicas: 3 bucket: diff --git a/docs/celld-behaviors.md b/docs/celld-behaviors.md index a3f0719..b63f9a9 100644 --- a/docs/celld-behaviors.md +++ b/docs/celld-behaviors.md @@ -3,35 +3,35 @@ The reconciler's guardrails and the rollout state machine exist because of specific, verifiable celld behaviors. This file is the index the code comments cite (F1–F14). Source of truth: celld's docs and source -(verified against v0.3.0); if upstream changes a row, find its citations +(verified against v0.4.0); if upstream changes a row, find its citations in this repo and revisit them. | # | celld behavior | What this operator does about it | | --- | --- | --- | -| F1 | Two listeners: public (`--listen`, serves the Worker, reserves only `/__celld/health`) and internal (`--internal-listen`, peer protocol + operator API). The operator API is **unauthenticated** except `/__d1/` (fleet-secret HMAC, see the wire notes). | Separate Services; the internal listener gets NetworkPolicy + (where Istio enforces) AuthorizationPolicy, and is never routed. | +| F1 | Two listeners: public (`--listen`, serves the Worker, reserves only `/.well-known/celld/health`) and internal (`--internal-listen`, versioned peer protocol + alpha operator API). The operator API is unauthenticated except its command-specific checks. | Separate Services; the internal listener gets NetworkPolicy + (where Istio enforces) AuthorizationPolicy, and is never routed. | | F2 | Peers dial each other at the `--advertise` address; peer requests carry an HMAC, body signature, clock limit and replay protection, but celld terminates no TLS; docs require a trusted network or encrypted overlay. | Stable per-pod DNS via the headless Service; encryption is the cluster's job (mesh or CNI). | | F3 | One fleet runs one application deployment. | Fleet-per-app tenancy; one WorkerApp = one fleet = one bucket prefix with its own credentials, so a full runtime compromise reaches only that app. | -| F4 | Nodes load the app deployment from the bucket's `deploy/current.json` **at startup only**. The manifest names the features it needs (`assets-v1`, `wasm-v1`; since v0.3.0 also `cron-v1`, `d1-v1`, `sqlite-vec-v1`) and a node that predates one rejects the deployment up front. | Deploys are bucket-publish **plus rolling restart**; the pod-template app-version annotation is the rollout trigger, and `appVersion: auto` follows the pointer directly. A deployment that needs a feature the running celld lacks fails the first released pod's readiness and holds the rollout there — bump `celld.image` before deploying such an app. | -| F5 | Rolling-update rule: after restarting a node, wait for **every** node to report `restoring=0` before the next restart — the restore work lands on the *peers*, not the replacement pod. `restoring` counts each cold route holding or awaiting an activation permit (v0.3.0 also reports its parts: `activating`, `activation_waiting`, `capacity_waiting`). | The operator owns the StatefulSet partition and steps it one ordinal at a time, gating on a live fleet-wide `/state` sweep. | -| F6 | SIGTERM starts a graceful drain: health flips to 503, new requests get 503+close, cells hand off to peers (at most `CELLD_RELEASES`=128 at once), bounded by `CELLD_SHUTDOWN_DRAIN_MS` (default 25000); an idle node exits at once. After the drain the node seals its node log with one bucket CAS (`node-log close: sealed epoch`), which is what makes a later downgrade safe (F8). | Termination grace (40 s) above the drain bound leaves the seal its headroom; 503 retries at the edge; readiness on the health path; **no HTTP liveness probe on it** (it would kill draining nodes). | +| F4 | v0.4 reads `deploy/current.json` at startup, polls it every 30 seconds (`CELLD_DEPLOY_POLL_S`), and adopts a new deployment in place; `POST /reload` adopts immediately. Existing requests finish on the old generation and resident Durable Objects move at safe points. The manifest names the runtime features it needs. | `appVersion` remains the CR's expected deployment and `auto` follows the pointer. The pod-template annotation retains a conservative gated restart as an explicit convergence fence, although v0.4 normally adopts before it. Upgrade `celld.image` before publishing an app that requires new runtime features. | +| F5 | For runtime rollouts, v0.4 paces handoff inside the stopping node and gates a replacement's first healthy response on fleet capacity, pressure, and restore backlog. `/state.restoring` remains the activation backlog (`activating` + `activation_waiting` + `capacity_waiting`). | The operator still owns the StatefulSet partition and steps one ordinal at a time, conservatively requiring a live fleet-wide `/state` sweep with `restoring=0` in addition to the replacement's readiness gate. | +| F6 | SIGTERM starts a graceful drain: health flips to 503, new requests get 503+close, and cells hand off in batches (`CELLD_RELEASES`, default 8). `CELLD_DRAIN_TOKEN_WAIT_MS` (default 30000) serializes simultaneous donors; `CELLD_SHUTDOWN_DRAIN_MS` (default 25000) is the no-progress interval; `CELLD_SHUTDOWN_TOTAL_MS` (default 40000) bounds the complete stop. After handoff the node seals its log. | Pin the 40 s celld total bound and set Kubernetes termination grace to 60 s; readiness uses the health path; liveness is TCP because an HTTP probe would kill a draining node. Recreate scales to zero only after every old pod exits. | | F7 | The bucket must provide conditional create (`If-None-Match: *`), conditional overwrite (`If-Match`/ETag CAS), and read-after-write consistency, enforced atomically. celld speaks the S3, GCS and (since v0.3.0) Azure Blob dialects (`bucket.rs`). Qualified: Amazon S3, Cloudflare R2, Google Cloud Storage, Tigris, Azure Blob Storage; not: MinIO CE, Backblaze B2, Hetzner, DigitalOcean Spaces, Azurite. An `az://` name is a *container*; the account comes from `AZURE_STORAGE_ACCOUNT_NAME` and exactly one credential family (account key, VM/AKS-node managed identity via IMDS, or AKS workload identity); `--endpoint` is rejected for `gs://` and `az://`. Bucket credentials are fleet-admin authority. | Qualified-store guidance in the README; `celld diagnose` + `hack/cas-hammer` for anything else (F14); per-fleet credential scoping. `spec.bucket.storageAccount` renders `AZURE_STORAGE_ACCOUNT_NAME`, `credentialsFrom.azureClientID` wires AKS workload identity, `secretRef` injects an account key; the CRD refuses `az://` without an account and an endpoint on `gs://`/`az://`. | -| F8 | Some celld upgrades are explicitly **not rolling-safe** (v0.1 ↔ v0.2: mixed fleets break). v0.2.1 → v0.3.0 **is** rolling-safe (a v0.3 node that cannot replicate to a v0.2 peer falls back to bucket proofs until the peer upgrades), but the **downgrade** v0.3 → v0.2 can lose acknowledged writes unless every node's shutdown log shows `node-log close: sealed epoch`. Security fixes land on the latest release only. | The `breakingBoundaries` table is directional and checks every boundary a jump crosses; Rolling across a flagged crossing is refused with the upstream reason unless the CR says `Recreate`, which also surfaces the reason while the fleet stops. Support policy is latest-only. | +| F8 | v0.1 ↔ v0.2 cannot mix. v0.2.1 → v0.3.0 is rolling-safe, but v0.3 → v0.2 can lose acknowledged writes unless each v0.3 node sealed its log. v0.3 ↔ v0.4 cannot mix: the v0.4 peer tunnel refuses v0.3 and v0.3 cannot read v0.4 epoch-qualified large Workers KV references. Security fixes land on the latest release only. | `breakingBoundaries` checks every crossed boundary and direction. Rolling across a known hazard is refused with the upstream reason unless the CR says `Recreate`; v0.3 ↔ v0.4 therefore stops every old node before any new one starts. | | F9 | Telemetry (`CELLD_OTEL=1`) writes Parquet traces/logs to the fleet bucket or OTLP; W3C `traceparent` propagates both ways. celld exports no metrics endpoint. | Telemetry env wired from `spec.telemetry`; the operator polls `/state` and re-exports Prometheus metrics itself. | -| F10 | No rebalancing on node join; placement is traffic-driven. Pressure shedding has two ceilings (`logic/pressure.rs` `PressureConfig::from_limits`): the threshold `CELLD_MAX_RSS_MB` (default 80% of what the process may use, which celld reads from the cgroup limit in `machine.rs` `total_memory_bytes` before `/proc/meminfo`) applies to the memory the **cells hold** (`in_use_bytes` = RSS minus allocator retention, the only memory shedding can return); an absolute cap at 95% of the limit applies to the process **RSS**. Each latch releases at 80% of its own ceiling; `/state` reports the reason (`memory` or `rss-hard`) and both numbers; `0` disables both. | `CELLD_MAX_RSS_MB` is set to 80% of the container limit — under the cap, so shedding keeps its recovery property, and visible in the pod spec. `celld_rss_bytes` and `celld_in_use_bytes` are exported; autoscaling scales up early (new capacity absorbs slowly) and treats shedding as the hard out-of-capacity trigger. | -| F11 | `celld deploy` = esbuild + `wrangler.jsonc`/`.json` with a strict key allowlist (v0.3.0 adds `triggers.crons` and `d1_databases`); unknown keys fail loudly. `--dry-run` bundles without writing. | CI-validated deploys; the operator never touches the build. | +| F10 | No rebalancing on node join; placement is traffic-driven. v0.4's pressure threshold (`CELLD_MAX_RSS_MB`, default 80% of available memory) uses the greater of allocator-adjusted RSS (`in_use_bytes`) and active cgroup memory (`cgroup_working_set_bytes`). The absolute 95% cap uses complete cgroup charge (`cgroup_current_bytes`), falling back to process RSS when cgroup data is unavailable. `0` disables both. | Set `CELLD_MAX_RSS_MB` to 80% of the container limit. Export all four memory measurements; autoscaling scales up early and treats shedding as the hard out-of-capacity trigger. | +| F11 | `celld deploy` = esbuild + `wrangler.jsonc`/`.json` with a strict key allowlist; v0.4 adds Workers KV, Queues, Workflows, R2 bindings, and broader compatibility settings. Unknown keys fail loudly; `--dry-run` bundles without writing. | CI-validated deploys; the operator never touches the build. | | F12 | Self-fencing: a node whose lease lapses, or whose core actor or replication process dies, logs a `SELF-FENCE:` line and exits with code **3**. The state is terminal, and upstream **requires** a supervisor that restarts the process without an attempt limit and waits at least one lease lifetime (`CELLD_TTL_MS`, default 10 s) between attempts. | Kubernetes restarts the container without a limit, but CrashLoopBackOff is sufficient only when the effective kubelet delay before each retry is at least `CELLD_TTL_MS`. The default 10 s initial delay covers the default lease lifetime; a larger `CELLD_TTL_MS` requires a correspondingly longer kubelet backoff or an explicit supervisor delay. The operator exports `celld_container_restarts` and `celld_self_fenced` (last exit code 3) so a fence loop is visible rather than silent. | | F13 | Fleet durability (`CELLD_DURABILITY`, default `fleet` since v0.3.0; `bucket` is the pre-0.3 behavior): a write is acknowledged when **two follower nodes** hold it on local disk (write-all, ack-all) or the bucket upload proves it, whichever first; the bucket remains the tiering target. A follower failure degrades the leader to bucket proofs until it re-recruits; a takeover of a dead leader recovers from at least one reachable sealed follower before it restores. A one-node fleet behaves as `bucket`. | `spec.durability` pins the proof; a soft hostname topology spread keeps a leader and its followers off one host, since until the upload lands an acknowledged write exists only on their emptyDirs; the per-pod `CELLD_WATCH` emptyDir is otherwise unchanged (its loss is the node-failure case upstream designs for). | | F14 | Startup storage probe (`CELLD_STORAGE_PROBE`, default on): every node writes and deletes one small object under `probe/` to test the four conditional-write transitions (create, reject-create, update, reject-stale) and stops if the store fails any — a store that ignores the precondition would otherwise make the node self-fence in a loop. `celld diagnose` runs the same test (`--read-only` skips it). celld reserves `probe/`, `cells/`, `nodes/`, `node-cells/`, `fleet/`, `deploy/`, `deploy-blobs/`, `wake/` and `telemetry/` under the prefix. | Per-fleet credentials must allow put and delete under the prefix (they already must for cells); README qualification guidance is `celld diagnose` for the sequential contract and `hack/cas-hammer` for atomicity under racers (S3 API only). | ## Wire-format notes (learned from source, not docs) -- `/state` (`actor.rs` `state_json`) returns - `{"ownership","occupied","evicting","restoring","activating","activation_waiting","capacity_waiting","phases","shedding","rss_bytes","in_use_bytes","residents","published","publishes","stops"}` - where `shedding` is `null` or a **reason string** (`"memory"` for the - cell-memory threshold, `"rss-hard"` for the absolute RSS cap), not a - boolean, and `restoring` is the activation backlog. The three breakdown - counters and both memory numbers are new in v0.3.0 (absent on older - nodes, so they decode as zero). +- `/state` (`actor.rs` `state_json`) keeps the v0.3 counters and in v0.4 + adds `quiescing`, `releasing`, `adopting`, `handed_off`, + `handoff_failed`, `cgroup_working_set_bytes`, and + `cgroup_current_bytes`. It also adds a `deployment` object containing + the current version and generation, draining generations, swapping count, + and each resident cell's generation. `shedding` remains `null` or a reason + string; the cgroup measurements are `null` when unavailable. - `deploy/current.json` is `{script_name?, version, prefix, rollout}`; `version` is the ID `celld deploy` prints. The manifest it points at carries a `features` list (F4). @@ -41,15 +41,14 @@ in this repo and revisit them. indefinitely by default. - Boolean celld variables accept only `0` or `1`; a supplied invalid value stops startup, so the operator writes `1`, never `true`. -- The internal listener also serves `/cell/` (resolve *or activate*), - `/evict/`, `/do/` (send a direct Durable Object request; refuses a D1 - database), `/__d1/SCOPE` (SQL to a D1 database, authenticated with the - fleet secret from the bucket — what `celld d1` uses), `POST /shutdown`, - `POST /shutdown?handoff=preserve`, and `/__celld/probe` — unused by the - operator today, available for ops tooling. Note `/cell/` and `/do/` mean - an unauthenticated caller who reaches the port can execute application - code and mutate cell state, not merely inspect it; only the D1 route - authenticates. +- The unauthenticated internal operator routes include `/state`, `/cell/`, + `/evict/`, `/do/`, `POST /reload`, `POST /shutdown`, and + `POST /shutdown?handoff=preserve`. `/cell/` and `/do/` can execute + application code and mutate cell state, not merely inspect it. Reserved + runtime classes (D1, Workers KV, Queues, Workflows) use the + HMAC-authenticated `/runtime/` protocol instead. `/peer/probe` and + the versioned peer-tunnel paths are internal protocol surfaces; the + operator does not call them. - Cron triggers (`triggers.crons`) run as one reserved cell per script (`.cron: