From 2225e2a46038a58edc29dcb87adf7855f5d1613d Mon Sep 17 00:00:00 2001 From: Matthew Zhao Date: Fri, 21 Aug 2026 04:02:02 +0000 Subject: [PATCH 1/2] feat: support celld 0.3 Re-verify the behaviors index against celld v0.3.0 and adapt the operator to what changed since v0.2.0. - Azure Blob Storage is a qualified backend: accept az:// buckets, add bucket.storageAccount (AZURE_STORAGE_ACCOUNT_NAME) and credentialsFrom.azureClientID (AKS workload identity); CEL rules refuse an az:// bucket without an account and an endpoint on gs:// or az://. - Default durability moved from bucket to fleet: add spec.durability and a soft hostname topology spread, since an acknowledged write now lives on follower disks until the bucket upload lands. - v0.2.1 -> v0.3.0 is rolling-safe but the downgrade can lose acknowledged writes: breakingBoundaries is directional, reports the upstream reason, and checks every boundary a multi-hop jump crosses. - Memory model: the threshold applies to cell-held memory under a 95% RSS cap; export celld_rss_bytes / celld_in_use_bytes from the new /state fields, plus celld_container_restarts / celld_self_fenced for the documented supervisor contract (exit code 3). - spec.trustForwardedHeaders wires CELLD_TRUST_FORWARDED_HEADERS. - appVersion auto on a non-S3 bucket reports UnsupportedStore. - Docs: F-table now F1-F14 (self-fence, fleet durability, storage probe), README store qualification uses celld diagnose (put_cas_contract was removed upstream), image pins bumped to v0.3.0, chart CRD resynced. Upgrading the operator rolls every fleet once: the topology spread changes the pod template hash. Co-Authored-By: Claude Fable 5 Signed-off-by: Matthew Zhao --- CONTRIBUTING.md | 2 +- README.md | 111 ++++++++++---- SECURITY.md | 10 +- api/v1alpha1/workerapp_types.go | 83 ++++++++-- .../bases/celld-operator.io_workerapps.yaml | 73 +++++++-- .../celld-operator_v1alpha1_workerapp.yaml | 10 +- .../crd/celld-operator.io_workerapps.yaml | 98 +++++++++--- docs/celld-behaviors.md | 67 +++++--- internal/controller/deploytracker.go | 21 +++ internal/controller/fleet_resources.go | 77 ++++++++-- internal/controller/fleetstate.go | 92 +++++++++-- internal/controller/rollout.go | 135 ++++++++++++----- internal/controller/rollout_test.go | 56 +++++-- .../controller/workerapp_controller_test.go | 143 +++++++++++++++++- 14 files changed, 808 insertions(+), 170 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6cb4e70..f1d4cb9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -28,7 +28,7 @@ every push. - Anything that touches the rollout state machine, the fencing-adjacent paths (partition stepping, `/state` gating), or the guardrails in `fleet_resources.go` should cite the celld behavior it depends on, the - way the existing code comments do (the F1–F11 table in + way the existing code comments do (the F1–F14 table in [docs/celld-behaviors.md](docs/celld-behaviors.md) is the index). If upstream celld changed, update the table. - The `breakingBoundaries` table in `internal/controller/rollout.go` must diff --git a/README.md b/README.md index dc071c1..f932dd2 100644 --- a/README.md +++ b/README.md @@ -53,12 +53,14 @@ under your operation), not for anonymous hostile code. - **A qualified object store.** celld's ownership fencing requires atomically enforced conditional writes (`If-None-Match: *`, `If-Match`) and read-after-write consistency. Qualified: Amazon S3, Cloudflare R2, Google - Cloud Storage, Tigris. Not qualified: MinIO community edition, Backblaze B2, - DigitalOcean Spaces, Hetzner. celld speaks only the S3 and GCS dialects, so a - store without an S3-compatible API (Azure Blob among them) cannot back a - fleet at all. For anything else, run - [store qualification](#store-qualification) first — an unenforced condition - silently splits cell ownership. + Cloud Storage, Tigris, Azure Blob Storage (celld 0.3+). Not qualified: MinIO + community edition, Backblaze B2, DigitalOcean Spaces, Hetzner, Azurite. + celld speaks only the S3, GCS and Azure Blob dialects (`s3://`, `gs://`, + `az://`), so a store without one of those APIs cannot back a fleet at all. + For anything else, run [store qualification](#store-qualification) first — + an unenforced condition silently splits cell ownership. Every node also + probes the store's conditional writes at startup and refuses to serve if + they fail. - Optional, each degrading gracefully to a status condition when absent: - **Gateway API** CRDs + an implementation (Istio recommended) for hostname ingress. @@ -122,15 +124,20 @@ spec: hostnames: ["chat.acme.example.com"] # routed via the shared Gateway appVersion: sha-abc123 # which bucket deployment is live celld: - image: ghcr.io/denoland/celld:v0.2.0 - updateStrategy: Rolling # Recreate for non-rolling celld upgrades + image: ghcr.io/denoland/celld:v0.3.0 + updateStrategy: Rolling # Recreate for non-rolling celld version changes replicas: 3 bucket: - name: s3://platform-cells/apps/chat # bucket + per-app prefix + name: s3://platform-cells/apps/chat # bucket + per-app prefix (s3://, gs://, or az://) endpoint: https://ACCOUNT.r2.cloudflarestorage.com region: auto credentialsFrom: iamRole: arn:aws:iam::123456789012:role/celld-chat # IRSA; or secretRef + # bucket: # Azure Blob Storage: the name is the container, + # name: az://platform-cells/apps/chat + # storageAccount: platformcells # the account is separate (AZURE_STORAGE_ACCOUNT_NAME), + # credentialsFrom: + # azureClientID: 11111111-2222-3333-4444-555555555555 # AKS workload identity; or secretRef resources: memoryGi: 8 # ~1000 resident cells per 8 GiB maxResidentCells: 1000 @@ -140,6 +147,8 @@ spec: # type: ClusterIP # hostnames at all — consumers use -celld..svc:8080 # annotations: {} # or LoadBalancer + annotations for a private LB websockets: true # long idle timeouts, sticky-friendly, slow scale-down + # durability: fleet # celld's default since 0.3; "bucket" waits for the upload before acking + # trustForwardedHeaders: true # request.url from X-Forwarded-*; only behind a proxy that replaces both autoscaling: enabled: true minReplicas: 3 @@ -184,8 +193,14 @@ warning when the bucket pointer and the CR disagree (nodes always load the bucket's version) — but only for fleets using `secretRef` credentials; the operator does not read the bucket for `iamRole` fleets. Tracking reads use the fleet's `secretRef` credentials, or the operator's ambient AWS identity -otherwise, and support `s3://` buckets only: a `gs://` fleet must pin -`appVersion`. +otherwise, and support `s3://` buckets only: a `gs://` or `az://` fleet must +pin `appVersion` (`DeployTrackingReady: UnsupportedStore` says so). + +A deployment can also need a newer celld than the fleet runs: the manifest +names the features it uses (static assets, Wasm, and since celld 0.3 cron +triggers, D1, sqlite-vec), and an older node rejects it at startup. That +surfaces as the first released pod never becoming Ready, which holds the +rollout there. Bump `spec.celld.image` first, then deploy. The operator then runs a **gated rolling update**, not a vanilla one. celld's documented rule is: after restarting a node, wait until *every* node reports @@ -199,13 +214,26 @@ and is Ready, and (b) a live sweep of every pod's `/state` shows fleet-wide ### celld version upgrades Bumping `spec.celld.image` rolls the same way — except across boundaries that -upstream flags as **not rolling-safe** (v0.1 ↔ v0.2: mixed fleets break). -Those are refused with `phase: Degraded` unless the CR explicitly sets +upstream flags as **not rolling-safe**. Those are refused with +`phase: Degraded` and the upstream reason unless the CR explicitly sets `celld.updateStrategy: Recreate`, which scales the fleet to zero, waits for every node to drain, then starts the new version. That is an availability -event by design; the CR has to ask for it. Note celld ships security fixes +event by design; the CR has to ask for it. A jump that skips releases is +checked against every boundary in between. Note celld ships security fixes for its latest release only — plan to track head. +| Change | Rolling? | Why | +| --- | --- | --- | +| v0.1 ↔ v0.2 | No, either direction | Mixed fleets break: ownership records changed address semantics and block objects changed format | +| v0.2.1 → v0.3.0 | **Yes** | A v0.3 node that cannot replicate to a v0.2 peer falls back to bucket proofs until the peer upgrades | +| v0.3 → v0.2 | No | A v0.2 binary cannot read writes still waiting in v0.3's replicated log or bundle objects, so the downgrade can lose acknowledged writes. `Recreate` drains every node first, which seals each log (`node-log close: sealed epoch` in the shutdown log); check for that line before trusting the downgrade | + +celld 0.3 also changed the default write-acknowledgement proof from `bucket` +to `fleet` (two follower nodes hold the write on disk, the bucket upload +trails): a rolling upgrade switches each node as it restarts, and +`spec.durability: bucket` pins the old behavior if you want write latency +and durability to keep depending on the bucket alone. + ## Autoscaling celld has no metrics endpoint yet, so the operator polls each pod's internal @@ -218,6 +246,8 @@ celld has no metrics endpoint yet, so the operator polls each pod's internal | `celld_restoring` | Cold activations in flight | | `celld_evicting` | Cells being evicted | | `celld_shedding` | 1 while pressure-shedding — the hard out-of-capacity signal | +| `celld_rss_bytes` / `celld_in_use_bytes` | Process RSS and the memory the cells hold (celld 0.3+); the gap is allocator retention shedding cannot return | +| `celld_container_restarts` / `celld_self_fenced` | Kubelet restart count, and 1 if the last exit was a celld self-fence (exit code 3) — a fence loop means the store or the bucket credential is broken | | `celld_state_up` | 1 if `/state` answered the last poll | With `spec.autoscaling.enabled`, the operator materializes a KEDA @@ -236,10 +266,23 @@ them wrong via templates because there are no templates: - **Liveness is TCP-only.** celld's health path answers 503 during a graceful drain; an HTTP liveness probe there would kill nodes mid-handoff. - **Termination grace (40s) exceeds the drain bound** (`CELLD_SHUTDOWN_DRAIN_MS`, - 25s), so the kubelet never SIGKILLs a draining node. + 25s), so the kubelet never SIGKILLs a draining node, and the node-log seal + that follows the drain gets its headroom. - **`CELLD_MAX_RSS_MB` is set explicitly** to ~80% of the container memory limit, so the ceiling is visible in the pod spec rather than inferred (celld - derives the same 80% from the cgroup limit on its own). + derives the same 80% from the cgroup limit on its own). celld applies it to + the memory the cells hold and keeps its own absolute cap at 95% of the limit + on the process RSS; 80% stays under that cap, so shedding can still recover + the node. +- **Fleet pods spread across hosts** (soft hostname topology spread). With + celld 0.3's fleet durability an acknowledged write lives on two follower + disks until the bucket upload lands; co-located followers would make one + host failure lose it. +- **Self-fenced nodes come back.** celld exits with code 3 after a + `SELF-FENCE:` and requires a supervisor that restarts it without limit, + spacing attempts by at least one lease lifetime (10s); the kubelet's + restart policy and CrashLoopBackOff do exactly that, and + `celld_self_fenced` makes a fence loop visible. - **The internal listener stays internal.** `:8081` (peer protocol plus an *unauthenticated* operator API) is reachable only from fleet pods and the operator's namespace via NetworkPolicy, reinforced by an Istio @@ -262,6 +305,7 @@ them wrong via templates because there are no templates: | --- | --- | | `Available` / `Progressing` / `Degraded` | Standard phase reflection; `Degraded` message names the refusal (e.g. a breaking upgrade without Recreate) | | `BucketCredentialsReady` | `iamRole: auto` provisioning is not implemented yet — annotate the fleet ServiceAccount yourself | +| `DeployTrackingReady` | `appVersion: auto` cannot follow the bucket: unreachable (holding the last known version), or an `UnsupportedStore` (`gs://`/`az://` — pin `appVersion`); in pinned mode, `VersionMismatch` when the bucket pointer disagrees with the CR | | `IngressReady` | Gateway API CRDs missing, or route error — hostnames are not routed | | `MeshPolicyReady` | Istio absent — NetworkPolicy alone guards `:8081` | | `AutoscalingReady` | KEDA absent — `spec.autoscaling` has no effect | @@ -289,29 +333,42 @@ which fails *silently*, as two nodes owning one cell. Before trusting a store that is not on the qualified list (and after every store upgrade): ```sh -# 1. celld's own sequential contract test (from the celld repo): -CELLD_CAS_LIVE=1 CELLD_CAS_BUCKET= CELLD_CAS_ENDPOINT= \ - cargo test -p celld put_cas_contract -- --nocapture - -# 2. This repo's concurrency hammer — N racers, exactly one winner per round: +# 1. celld's own sequential contract test: create, reject-create, update, +# reject-stale, against the real bucket (any of s3://, gs://, az://). +# Every node repeats it at startup (CELLD_STORAGE_PROBE) and refuses to +# serve if it fails. +celld diagnose --bucket [--endpoint ] [--region ] + +# 2. This repo's concurrency hammer — N racers, exactly one winner per round +# (S3 API only): go run ./hack/cas-hammer --bucket --endpoint \ --writers 8 --rounds 32 ``` Exit 1 from the hammer means the store cannot fence celld cells. Do not run a -fleet on it. +fleet on it. Both tests write: the probe uses the reserved `probe/` prefix, +the hammer its own `--prefix`. ## Security notes -- **Bucket credentials are fleet-admin authority.** Scope one IAM role per - fleet to that fleet's prefix and nothing else; prefer IRSA/workload - identity over static keys (`credentialsFrom.secretRef` exists for stores - without role auth). +- **Bucket credentials are fleet-admin authority.** Scope one IAM role + (or Azure identity) per fleet to that fleet's prefix and nothing else; + prefer IRSA / AKS workload identity over static keys + (`credentialsFrom.secretRef` exists for stores without identity auth). + The credential needs put and delete under the prefix: celld's startup + probe writes and removes one object under `probe/`. - celld terminates no TLS anywhere: public TLS belongs to the Gateway, and the pod network should be encrypted (Istio ambient, or a CNI with WireGuard) because the peer protocol relies on network confidentiality. - The deployed Worker owns every public path except `/__celld/health`; - application authentication is the application's job. + application authentication is the application's job. celld ignores + `X-Forwarded-Host` / `X-Forwarded-Proto` unless + `spec.trustForwardedHeaders` is set — turn it on only when every hop in + front of the fleet replaces both headers. +- The internal listener's `/cell/` and `/do/` routes run application code + unauthenticated; only the D1 route (`/__d1/`, what `celld d1` uses) + authenticates with the fleet secret. Keep `:8081` behind the operator's + policies. ## Development diff --git a/SECURITY.md b/SECURITY.md index 0379f6a..fe7ad78 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -25,10 +25,12 @@ exposure: prefix, and rotate on suspicion of disclosure. The operator reads these credentials (for config-hash rotation tracking and deploy-pointer tracking) but never logs or re-exports them. -- **celld's internal listener (:8081) is unauthenticated upstream.** The - operator fences it with NetworkPolicy and (where Istio is present) - AuthorizationPolicy; weakening either is equivalent to granting cell - eviction and shutdown rights to whatever can reach the pod network. +- **celld's internal listener (:8081) is unauthenticated upstream** (only + the D1 SQL route checks the fleet secret; `/cell/` and `/do/` run + application code for any caller). The operator fences it with + NetworkPolicy and (where Istio is present) AuthorizationPolicy; weakening + either is equivalent to granting cell activation, eviction and shutdown + rights to whatever can reach the pod network. - celld itself is not safe for hostile multi-tenant use; the isolation model is fleet-per-application. See [docs/celld-behaviors.md](docs/celld-behaviors.md) (F3) before hosting diff --git a/api/v1alpha1/workerapp_types.go b/api/v1alpha1/workerapp_types.go index 4e224f9..5a5a620 100644 --- a/api/v1alpha1/workerapp_types.go +++ b/api/v1alpha1/workerapp_types.go @@ -29,7 +29,8 @@ import ( // UpdateStrategy selects how a celld version change rolls through the fleet. // Rolling is the partition-stepped, restoring-gated path (docs/celld-behaviors.md); // Recreate scales to zero first, for upstream releases that forbid mixed -// fleets. A Rolling request across a known-breaking celld boundary is refused. +// fleets or flag a downgrade as lossy. A Rolling request across a +// known-breaking celld boundary is refused (F8). // +kubebuilder:validation:Enum=Rolling;Recreate type UpdateStrategy string @@ -61,28 +62,52 @@ type BucketCredentials struct { // +optional IAMRole string `json:"iamRole,omitempty"` - // secretRef names a Secret with static credentials - // (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY), for stores where - // role-based auth is unavailable. + // secretRef names a Secret whose keys are injected as environment: + // AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY for an s3:// bucket, or + // AZURE_STORAGE_ACCOUNT_KEY for an az:// container. For stores where + // identity-based auth is unavailable. // +optional SecretRef string `json:"secretRef,omitempty"` + + // azureClientID is the client ID of a Microsoft Entra workload identity + // for an az:// container (AKS workload identity). The fleet + // ServiceAccount is annotated with it and fleet pods carry the + // azure.workload.identity/use label, so the AKS webhook injects the + // federated token celld reads (AZURE_CLIENT_ID, AZURE_TENANT_ID, + // AZURE_FEDERATED_TOKEN_FILE, AZURE_AUTHORITY_HOST). The identity + // needs Storage Blob Data Contributor on the container. + // +optional + AzureClientID string `json:"azureClientID,omitempty"` } // BucketSpec locates the fleet's slice of the object store. +// +kubebuilder:validation:XValidation:rule="!self.name.startsWith('az://') || (has(self.storageAccount) && self.storageAccount.size() > 0)",message="an az:// bucket requires storageAccount (the Azure storage account; the bucket name is the container)" +// +kubebuilder:validation:XValidation:rule="!has(self.endpoint) || self.endpoint.size() == 0 || self.name.startsWith('s3://')",message="endpoint applies to s3:// buckets only; celld rejects an endpoint for gs:// and az://" type BucketSpec struct { - // name is the fleet bucket and prefix, e.g. "s3://platform-cells/apps/chat" - // or "gs://platform-cells/apps/chat". The store must satisfy celld's - // fencing contract (conditional create/overwrite, read-after-write); - // see docs/celld-behaviors.md for the qualified list. + // name is the fleet bucket and prefix, e.g. "s3://platform-cells/apps/chat", + // "gs://platform-cells/apps/chat", or "az://platform-cells/apps/chat" + // (for az:// the bucket is a Blob Storage container in the account + // named by storageAccount). The store must satisfy celld's fencing + // contract (conditional create/overwrite, read-after-write); see + // docs/celld-behaviors.md for the qualified list. // +required - // +kubebuilder:validation:Pattern=`^(s3|gs)://.+` + // +kubebuilder:validation:Pattern=`^(s3|gs|az)://.+` + // +kubebuilder:validation:MaxLength=1024 Name string `json:"name"` // endpoint is the S3-compatible endpoint URL, when not AWS S3. - // Rejected by celld for gs:// buckets. + // Rejected by celld for gs:// and az:// buckets. // +optional + // +kubebuilder:validation:MaxLength=2048 Endpoint string `json:"endpoint,omitempty"` + // storageAccount is the Azure storage account that holds an az:// + // container (AZURE_STORAGE_ACCOUNT_NAME). Required for az://, ignored + // otherwise. + // +optional + // +kubebuilder:validation:MaxLength=64 + StorageAccount string `json:"storageAccount,omitempty"` + // region is the storage region, when it cannot be inferred. // +optional Region string `json:"region,omitempty"` @@ -95,7 +120,9 @@ type BucketSpec struct { // ResourcesSpec sizes one fleet pod. The operator derives the container // limit, CELLD_MAX_RSS_MB (~80% of the limit, set explicitly so the ceiling // is visible in the pod spec; celld would derive the same value from the -// cgroup limit itself), and admission caps from these two numbers. +// cgroup limit itself), and admission caps from these two numbers. celld +// applies that threshold to the memory its cells hold and keeps its own +// absolute cap at 95% of the limit on the process RSS (docs/celld-behaviors.md F10). type ResourcesSpec struct { // memoryGi is the container memory limit per pod, in GiB. // +optional @@ -231,6 +258,22 @@ func (t *TelemetrySpec) ResolvedSink() TelemetrySink { return TelemetrySinkBucket } +// Durability selects celld's write-acknowledgement proof (CELLD_DURABILITY, +// docs/celld-behaviors.md F13). +// +kubebuilder:validation:Enum=fleet;bucket +type Durability string + +const ( + // DurabilityFleet (celld's default since v0.3.0) acknowledges a write + // once two follower nodes hold it on disk, and tiers it to the bucket + // behind. A one-node fleet behaves as bucket. + DurabilityFleet Durability = "fleet" + // DurabilityBucket acknowledges a write only after it is in the bucket + // (celld's pre-0.3 behavior): higher write latency, no reliance on + // follower disks. + DurabilityBucket Durability = "bucket" +) + // WorkerAppSpec defines the desired state of WorkerApp. type WorkerAppSpec struct { // hostnames route to this app. One route object is reconciled in the @@ -286,6 +329,24 @@ type WorkerAppSpec struct { // +optional Telemetry TelemetrySpec `json:"telemetry,omitzero"` + + // durability selects how celld proves a write before acknowledging it: + // fleet (celld's default: two follower nodes fsync it, the bucket + // upload follows) or bucket (the write is in the bucket first). Unset + // leaves celld's default. Changing it restarts the fleet through the + // ordinary gated rollout. + // +optional + Durability Durability `json:"durability,omitempty"` + + // trustForwardedHeaders lets X-Forwarded-Host and X-Forwarded-Proto + // set the scheme and host of request.url (CELLD_TRUST_FORWARDED_HEADERS). + // celld ignores both headers by default, so behind a TLS-terminating + // ingress a Worker sees its pod address as its URL. Enable it only when + // every hop in front of the fleet replaces both headers (ingress-nginx + // does; Envoy-based gateways set X-Forwarded-Proto but not always + // X-Forwarded-Host), since celld takes the last value of each. + // +optional + TrustForwardedHeaders bool `json:"trustForwardedHeaders,omitempty"` } // WorkerAppPhase summarizes the fleet at a glance. diff --git a/config/crd/bases/celld-operator.io_workerapps.yaml b/config/crd/bases/celld-operator.io_workerapps.yaml index 4c867b9..1dfab1b 100644 --- a/config/crd/bases/celld-operator.io_workerapps.yaml +++ b/config/crd/bases/celld-operator.io_workerapps.yaml @@ -118,6 +118,16 @@ spec: credentialsFrom: description: credentialsFrom selects the fleet's bucket credential. properties: + azureClientID: + description: |- + azureClientID is the client ID of a Microsoft Entra workload identity + for an az:// container (AKS workload identity). The fleet + ServiceAccount is annotated with it and fleet pods carry the + azure.workload.identity/use label, so the AKS webhook injects the + federated token celld reads (AZURE_CLIENT_ID, AZURE_TENANT_ID, + AZURE_FEDERATED_TOKEN_FILE, AZURE_AUTHORITY_HOST). The identity + needs Storage Blob Data Contributor on the container. + type: string iamRole: description: |- iamRole is an IAM role ARN assumed via the pod's service account @@ -126,30 +136,50 @@ spec: type: string secretRef: description: |- - secretRef names a Secret with static credentials - (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY), for stores where - role-based auth is unavailable. + secretRef names a Secret whose keys are injected as environment: + AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY for an s3:// bucket, or + AZURE_STORAGE_ACCOUNT_KEY for an az:// container. For stores where + identity-based auth is unavailable. type: string type: object endpoint: description: |- endpoint is the S3-compatible endpoint URL, when not AWS S3. - Rejected by celld for gs:// buckets. + Rejected by celld for gs:// and az:// buckets. + maxLength: 2048 type: string name: description: |- - name is the fleet bucket and prefix, e.g. "s3://platform-cells/apps/chat" - or "gs://platform-cells/apps/chat". The store must satisfy celld's - fencing contract (conditional create/overwrite, read-after-write); - see docs/celld-behaviors.md for the qualified list. - pattern: ^(s3|gs)://.+ + name is the fleet bucket and prefix, e.g. "s3://platform-cells/apps/chat", + "gs://platform-cells/apps/chat", or "az://platform-cells/apps/chat" + (for az:// the bucket is a Blob Storage container in the account + named by storageAccount). The store must satisfy celld's fencing + contract (conditional create/overwrite, read-after-write); see + docs/celld-behaviors.md for the qualified list. + maxLength: 1024 + pattern: ^(s3|gs|az)://.+ type: string region: description: region is the storage region, when it cannot be inferred. type: string + storageAccount: + description: |- + storageAccount is the Azure storage account that holds an az:// + container (AZURE_STORAGE_ACCOUNT_NAME). Required for az://, ignored + otherwise. + maxLength: 64 + type: string required: - name type: object + x-kubernetes-validations: + - message: an az:// bucket requires storageAccount (the Azure storage + account; the bucket name is the container) + rule: '!self.name.startsWith(''az://'') || (has(self.storageAccount) + && self.storageAccount.size() > 0)' + - message: endpoint applies to s3:// buckets only; celld rejects an + endpoint for gs:// and az:// + rule: '!has(self.endpoint) || self.endpoint.size() == 0 || self.name.startsWith(''s3://'')' celld: description: CelldSpec pins the celld runtime for the fleet. properties: @@ -169,6 +199,17 @@ spec: required: - image type: object + durability: + description: |- + durability selects how celld proves a write before acknowledging it: + fleet (celld's default: two follower nodes fsync it, the bucket + upload follows) or bucket (the write is in the bucket first). Unset + leaves celld's default. Changing it restarts the fleet through the + ordinary gated rollout. + enum: + - fleet + - bucket + type: string hostnames: description: |- hostnames route to this app. One route object is reconciled in the @@ -191,7 +232,9 @@ spec: ResourcesSpec sizes one fleet pod. The operator derives the container limit, CELLD_MAX_RSS_MB (~80% of the limit, set explicitly so the ceiling is visible in the pod spec; celld would derive the same value from the - cgroup limit itself), and admission caps from these two numbers. + cgroup limit itself), and admission caps from these two numbers. celld + applies that threshold to the memory its cells hold and keeps its own + absolute cap at 95% of the limit on the process RSS (docs/celld-behaviors.md F10). properties: maxResidentCells: default: 1000 @@ -261,6 +304,16 @@ spec: - otlp type: string type: object + trustForwardedHeaders: + description: |- + trustForwardedHeaders lets X-Forwarded-Host and X-Forwarded-Proto + set the scheme and host of request.url (CELLD_TRUST_FORWARDED_HEADERS). + celld ignores both headers by default, so behind a TLS-terminating + ingress a Worker sees its pod address as its URL. Enable it only when + every hop in front of the fleet replaces both headers (ingress-nginx + does; Envoy-based gateways set X-Forwarded-Proto but not always + X-Forwarded-Host), since celld takes the last value of each. + type: boolean vars: description: VarsSpec supplies Worker variables and secrets. properties: diff --git a/config/samples/celld-operator_v1alpha1_workerapp.yaml b/config/samples/celld-operator_v1alpha1_workerapp.yaml index 048a94d..c06aa60 100644 --- a/config/samples/celld-operator_v1alpha1_workerapp.yaml +++ b/config/samples/celld-operator_v1alpha1_workerapp.yaml @@ -9,19 +9,27 @@ spec: hostnames: ["chat.acme.ezghcloud.com"] appVersion: sha-abc123 celld: - image: ghcr.io/denoland/celld:v0.2.0 + image: ghcr.io/denoland/celld:v0.3.0 updateStrategy: Rolling replicas: 3 bucket: name: s3://platform-cells/apps/chat credentialsFrom: iamRole: auto + # Azure Blob Storage instead: the name is the container, the account + # is separate, and AKS workload identity replaces the IAM role. + # name: az://platform-cells/apps/chat + # storageAccount: platformcells + # credentialsFrom: + # azureClientID: 11111111-2222-3333-4444-555555555555 resources: memoryGi: 8 maxResidentCells: 1000 vars: secretRef: chat-vars websockets: true + # durability: fleet # celld's default; "bucket" waits for the upload before acking + # trustForwardedHeaders: true # only behind a proxy that replaces X-Forwarded-Host and -Proto autoscaling: enabled: true minReplicas: 3 diff --git a/dist/chart/templates/crd/celld-operator.io_workerapps.yaml b/dist/chart/templates/crd/celld-operator.io_workerapps.yaml index ed0703c..23bda1f 100644 --- a/dist/chart/templates/crd/celld-operator.io_workerapps.yaml +++ b/dist/chart/templates/crd/celld-operator.io_workerapps.yaml @@ -90,7 +90,6 @@ spec: minReplicas: default: 2 description: minReplicas is the scale floor; keep >= 2 for HA. - Also the PDB floor. format: int32 minimum: 1 type: integer @@ -100,9 +99,10 @@ spec: properties: p95LatencyMs: description: |- - p95LatencyMs adds a gateway-side latency target so traffic-bound, - stateless-Worker-heavy apps scale even at low cell counts. Unset - disables the latency signal. + p95LatencyMs adds a latency target so traffic-bound, + stateless-Worker-heavy apps scale even at low cell counts. The query + reads Istio destination telemetry, so it needs Istio metrics in the + same Prometheus. Unset disables the latency signal. format: int32 minimum: 1 type: integer @@ -125,6 +125,16 @@ spec: credentialsFrom: description: credentialsFrom selects the fleet's bucket credential. properties: + azureClientID: + description: |- + azureClientID is the client ID of a Microsoft Entra workload identity + for an az:// container (AKS workload identity). The fleet + ServiceAccount is annotated with it and fleet pods carry the + azure.workload.identity/use label, so the AKS webhook injects the + federated token celld reads (AZURE_CLIENT_ID, AZURE_TENANT_ID, + AZURE_FEDERATED_TOKEN_FILE, AZURE_AUTHORITY_HOST). The identity + needs Storage Blob Data Contributor on the container. + type: string iamRole: description: |- iamRole is an IAM role ARN assumed via the pod's service account @@ -133,30 +143,50 @@ spec: type: string secretRef: description: |- - secretRef names a Secret with static credentials - (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY), for stores where - role-based auth is unavailable. + secretRef names a Secret whose keys are injected as environment: + AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY for an s3:// bucket, or + AZURE_STORAGE_ACCOUNT_KEY for an az:// container. For stores where + identity-based auth is unavailable. type: string type: object endpoint: description: |- endpoint is the S3-compatible endpoint URL, when not AWS S3. - Rejected by celld for gs:// buckets. + Rejected by celld for gs:// and az:// buckets. + maxLength: 2048 type: string name: description: |- - name is the fleet bucket and prefix, e.g. "s3://platform-cells/apps/chat" - or "gs://platform-cells/apps/chat". The store must satisfy celld's - fencing contract (conditional create/overwrite, read-after-write); - see docs/celld-behaviors.md for the qualified list. - pattern: ^(s3|gs)://.+ + name is the fleet bucket and prefix, e.g. "s3://platform-cells/apps/chat", + "gs://platform-cells/apps/chat", or "az://platform-cells/apps/chat" + (for az:// the bucket is a Blob Storage container in the account + named by storageAccount). The store must satisfy celld's fencing + contract (conditional create/overwrite, read-after-write); see + docs/celld-behaviors.md for the qualified list. + maxLength: 1024 + pattern: ^(s3|gs|az)://.+ type: string region: description: region is the storage region, when it cannot be inferred. type: string + storageAccount: + description: |- + storageAccount is the Azure storage account that holds an az:// + container (AZURE_STORAGE_ACCOUNT_NAME). Required for az://, ignored + otherwise. + maxLength: 64 + type: string required: - name type: object + x-kubernetes-validations: + - message: an az:// bucket requires storageAccount (the Azure storage + account; the bucket name is the container) + rule: '!self.name.startsWith(''az://'') || (has(self.storageAccount) + && self.storageAccount.size() > 0)' + - message: endpoint applies to s3:// buckets only; celld rejects an + endpoint for gs:// and az:// + rule: '!has(self.endpoint) || self.endpoint.size() == 0 || self.name.startsWith(''s3://'')' celld: description: CelldSpec pins the celld runtime for the fleet. properties: @@ -176,10 +206,22 @@ spec: required: - image type: object + durability: + description: |- + durability selects how celld proves a write before acknowledging it: + fleet (celld's default: two follower nodes fsync it, the bucket + upload follows) or bucket (the write is in the bucket first). Unset + leaves celld's default. Changing it restarts the fleet through the + ordinary gated rollout. + enum: + - fleet + - bucket + type: string hostnames: description: |- - hostnames route to this app on the shared Gateway; one HTTPRoute per - hostname is reconciled in the app's namespace. + hostnames route to this app. One route object is reconciled in the + app's namespace carrying every hostname; what kind depends on the + operator's --ingress-mode. items: type: string type: array @@ -195,8 +237,11 @@ spec: resources: description: |- ResourcesSpec sizes one fleet pod. The operator derives the container - limit, CELLD_MAX_RSS_MB (~80% of the limit, set explicitly — the upstream - default is not cgroup-aware), and admission caps from these two numbers. + limit, CELLD_MAX_RSS_MB (~80% of the limit, set explicitly so the ceiling + is visible in the pod spec; celld would derive the same value from the + cgroup limit itself), and admission caps from these two numbers. celld + applies that threshold to the memory its cells hold and keeps its own + absolute cap at 95% of the limit on the process RSS (docs/celld-behaviors.md F10). properties: maxResidentCells: default: 1000 @@ -266,6 +311,16 @@ spec: - otlp type: string type: object + trustForwardedHeaders: + description: |- + trustForwardedHeaders lets X-Forwarded-Host and X-Forwarded-Proto + set the scheme and host of request.url (CELLD_TRUST_FORWARDED_HEADERS). + celld ignores both headers by default, so behind a TLS-terminating + ingress a Worker sees its pod address as its URL. Enable it only when + every hop in front of the fleet replaces both headers (ingress-nginx + does; Envoy-based gateways set X-Forwarded-Proto but not always + X-Forwarded-Host), since celld takes the last value of each. + type: boolean vars: description: VarsSpec supplies Worker variables and secrets. properties: @@ -280,8 +335,10 @@ spec: type: object websockets: description: |- - websockets selects the WebSocket ingress profile: session affinity, - long idle timeouts, and conservative scale-down (docs/celld-behaviors.md). + websockets selects the WebSocket profile: long edge timeouts where the + ingress mode can express them (httproute and ingress; not + virtualservice) and a conservative scale-down window. No session + affinity is configured. type: boolean required: - appVersion @@ -395,7 +452,8 @@ spec: waitingOn: description: |- waitingOn names the gate the rollout is blocked on, e.g. - "celld-2: Ready" or "fleet: restoring=3". Empty when not waiting. + "chat-celld-2: not ready" or "fleet: restoring=3". Empty when not + waiting. type: string type: object type: object diff --git a/docs/celld-behaviors.md b/docs/celld-behaviors.md index 9f2685a..3695bde 100644 --- a/docs/celld-behaviors.md +++ b/docs/celld-behaviors.md @@ -2,41 +2,58 @@ The reconciler's guardrails and the rollout state machine exist because of specific, verifiable celld behaviors. This file is the index the code -comments cite (F1–F11). Source of truth: celld's docs and source -(verified against v0.2.0); if upstream changes a row, find its citations +comments cite (F1–F14). Source of truth: celld's docs and source +(verified against v0.3.0); if upstream changes a row, find its citations in this repo and revisit them. | # | celld behavior | What this operator does about it | | --- | --- | --- | -| F1 | Two listeners: public (`--listen`, serves the Worker, reserves only `/__celld/health`) and internal (`--internal-listen`, peer protocol + **unauthenticated** operator API). | Separate Services; the internal listener gets NetworkPolicy + (where Istio enforces) AuthorizationPolicy, and is never routed. | -| F2 | Peers dial each other at the `--advertise` address; celld terminates no TLS; docs require a trusted network or encrypted overlay. | Stable per-pod DNS via the headless Service; encryption is the cluster's job (mesh or CNI). | +| F1 | Two listeners: public (`--listen`, serves the Worker, reserves only `/__celld/health`) and internal (`--internal-listen`, peer protocol + operator API). The operator API is **unauthenticated** except `/__d1/` (fleet-secret HMAC, see the wire notes). | Separate Services; the internal listener gets NetworkPolicy + (where Istio enforces) AuthorizationPolicy, and is never routed. | +| F2 | Peers dial each other at the `--advertise` address; peer requests carry an HMAC, body signature, clock limit and replay protection, but celld terminates no TLS; docs require a trusted network or encrypted overlay. | Stable per-pod DNS via the headless Service; encryption is the cluster's job (mesh or CNI). | | F3 | One fleet runs one application deployment. | Fleet-per-app tenancy; one WorkerApp = one fleet = one bucket prefix with its own credentials, so a full runtime compromise reaches only that app. | -| F4 | Nodes load the app deployment from the bucket's `deploy/current.json` **at startup only**. | Deploys are bucket-publish **plus rolling restart**; the pod-template app-version annotation is the rollout trigger, and `appVersion: auto` follows the pointer directly. | -| F5 | Rolling-update rule: after restarting a node, wait for **every** node to report `restoring=0` before the next restart — the restore work lands on the *peers*, not the replacement pod. | The operator owns the StatefulSet partition and steps it one ordinal at a time, gating on a live fleet-wide `/state` sweep. | -| F6 | SIGTERM starts a graceful drain: health flips to 503, new requests get 503+close, cells hand off to peers, bounded by `CELLD_SHUTDOWN_DRAIN_MS` (default 25000). | Termination grace above the drain bound; 503 retries at the edge; readiness on the health path; **no HTTP liveness probe on it** (it would kill draining nodes). | -| F7 | The bucket must provide conditional create (`If-None-Match: *`), conditional overwrite (`If-Match`/ETag CAS), and read-after-write consistency, enforced atomically. celld speaks only the S3 and GCS dialects (`bucket.rs` `StorageBackend`), so stores without an S3-compatible API — Azure Blob among them — cannot back a fleet regardless of their guarantees. Bucket credentials are fleet-admin authority. | Qualified-store guidance in the README; `hack/cas-hammer` races conditional PUTs to verify atomicity; per-fleet credential scoping. | -| F8 | Some celld upgrades are explicitly **not rolling-safe** (v0.1 ↔ v0.2: mixed fleets break). Security fixes land on the latest release only. | The `breakingBoundaries` table refuses Rolling across flagged pairs unless the CR says `Recreate`; support policy is latest-only. | +| F4 | Nodes load the app deployment from the bucket's `deploy/current.json` **at startup only**. The manifest names the features it needs (`assets-v1`, `wasm-v1`; since v0.3.0 also `cron-v1`, `d1-v1`, `sqlite-vec-v1`) and a node that predates one rejects the deployment up front. | Deploys are bucket-publish **plus rolling restart**; the pod-template app-version annotation is the rollout trigger, and `appVersion: auto` follows the pointer directly. A deployment that needs a feature the running celld lacks fails the first released pod's readiness and holds the rollout there — bump `celld.image` before deploying such an app. | +| F5 | Rolling-update rule: after restarting a node, wait for **every** node to report `restoring=0` before the next restart — the restore work lands on the *peers*, not the replacement pod. `restoring` counts each cold route holding or awaiting an activation permit (v0.3.0 also reports its parts: `activating`, `activation_waiting`, `capacity_waiting`). | The operator owns the StatefulSet partition and steps it one ordinal at a time, gating on a live fleet-wide `/state` sweep. | +| F6 | SIGTERM starts a graceful drain: health flips to 503, new requests get 503+close, cells hand off to peers (at most `CELLD_RELEASES`=128 at once), bounded by `CELLD_SHUTDOWN_DRAIN_MS` (default 25000); an idle node exits at once. After the drain the node seals its node log with one bucket CAS (`node-log close: sealed epoch`), which is what makes a later downgrade safe (F8). | Termination grace (40 s) above the drain bound leaves the seal its headroom; 503 retries at the edge; readiness on the health path; **no HTTP liveness probe on it** (it would kill draining nodes). | +| F7 | The bucket must provide conditional create (`If-None-Match: *`), conditional overwrite (`If-Match`/ETag CAS), and read-after-write consistency, enforced atomically. celld speaks the S3, GCS and (since v0.3.0) Azure Blob dialects (`bucket.rs`). Qualified: Amazon S3, Cloudflare R2, Google Cloud Storage, Tigris, Azure Blob Storage; not: MinIO CE, Backblaze B2, Hetzner, DigitalOcean Spaces, Azurite. An `az://` name is a *container*; the account comes from `AZURE_STORAGE_ACCOUNT_NAME` and exactly one credential family (account key, VM/AKS-node managed identity via IMDS, or AKS workload identity); `--endpoint` is rejected for `gs://` and `az://`. Bucket credentials are fleet-admin authority. | Qualified-store guidance in the README; `celld diagnose` + `hack/cas-hammer` for anything else (F14); per-fleet credential scoping. `spec.bucket.storageAccount` renders `AZURE_STORAGE_ACCOUNT_NAME`, `credentialsFrom.azureClientID` wires AKS workload identity, `secretRef` injects an account key; the CRD refuses `az://` without an account and an endpoint on `gs://`/`az://`. | +| F8 | Some celld upgrades are explicitly **not rolling-safe** (v0.1 ↔ v0.2: mixed fleets break). v0.2.1 → v0.3.0 **is** rolling-safe (a v0.3 node that cannot replicate to a v0.2 peer falls back to bucket proofs until the peer upgrades), but the **downgrade** v0.3 → v0.2 can lose acknowledged writes unless every node's shutdown log shows `node-log close: sealed epoch`. Security fixes land on the latest release only. | The `breakingBoundaries` table is directional and checks every boundary a jump crosses; Rolling across a flagged crossing is refused with the upstream reason unless the CR says `Recreate`, which also surfaces the reason while the fleet stops. Support policy is latest-only. | | F9 | Telemetry (`CELLD_OTEL=1`) writes Parquet traces/logs to the fleet bucket or OTLP; W3C `traceparent` propagates both ways. celld exports no metrics endpoint. | Telemetry env wired from `spec.telemetry`; the operator polls `/state` and re-exports Prometheus metrics itself. | -| F10 | No rebalancing on node join; placement is traffic-driven. Pressure shedding defaults to 80% of what the process may use, which celld reads from the cgroup limit (`main/machine.rs` `total_memory_bytes`) before falling back to `/proc/meminfo`. | `CELLD_MAX_RSS_MB` is set explicitly from the container limit anyway, so the ceiling is visible in the pod spec; autoscaling scales up early (new capacity absorbs slowly) and treats shedding as the hard out-of-capacity trigger. | -| F11 | `celld deploy` = esbuild + `wrangler.jsonc`/`.json` with a strict key allowlist; unknown keys fail loudly. `--dry-run` bundles without writing. | CI-validated deploys; the operator never touches the build. | +| F10 | No rebalancing on node join; placement is traffic-driven. Pressure shedding has two ceilings (`logic/pressure.rs` `PressureConfig::from_limits`): the threshold `CELLD_MAX_RSS_MB` (default 80% of what the process may use, which celld reads from the cgroup limit in `machine.rs` `total_memory_bytes` before `/proc/meminfo`) applies to the memory the **cells hold** (`in_use_bytes` = RSS minus allocator retention, the only memory shedding can return); an absolute cap at 95% of the limit applies to the process **RSS**. Each latch releases at 80% of its own ceiling; `/state` reports the reason (`memory` or `rss-hard`) and both numbers; `0` disables both. | `CELLD_MAX_RSS_MB` is set to 80% of the container limit — under the cap, so shedding keeps its recovery property, and visible in the pod spec. `celld_rss_bytes` and `celld_in_use_bytes` are exported; autoscaling scales up early (new capacity absorbs slowly) and treats shedding as the hard out-of-capacity trigger. | +| F11 | `celld deploy` = esbuild + `wrangler.jsonc`/`.json` with a strict key allowlist (v0.3.0 adds `triggers.crons` and `d1_databases`); unknown keys fail loudly. `--dry-run` bundles without writing. | CI-validated deploys; the operator never touches the build. | +| F12 | Self-fencing: a node whose lease lapses, or whose core actor or replication process dies, logs a `SELF-FENCE:` line and exits with code **3**. The state is terminal, and upstream **requires** a supervisor that restarts the process without an attempt limit and waits at least one lease lifetime (`CELLD_TTL_MS`, default 10 s) between attempts. | Kubernetes restarts the container without a limit; the kubelet's CrashLoopBackOff (10 s doubling to 5 min) spaces every restart after the first by at least one lease lifetime. The operator exports `celld_container_restarts` and `celld_self_fenced` (last exit code 3) so a fence loop is visible rather than silent. | +| F13 | Fleet durability (`CELLD_DURABILITY`, default `fleet` since v0.3.0; `bucket` is the pre-0.3 behavior): a write is acknowledged when **two follower nodes** hold it on local disk (write-all, ack-all) or the bucket upload proves it, whichever first; the bucket remains the tiering target. A follower failure degrades the leader to bucket proofs until it re-recruits; a takeover of a dead leader recovers from at least one reachable sealed follower before it restores. A one-node fleet behaves as `bucket`. | `spec.durability` pins the proof; a soft hostname topology spread keeps a leader and its followers off one host, since until the upload lands an acknowledged write exists only on their emptyDirs; the per-pod `CELLD_WATCH` emptyDir is otherwise unchanged (its loss is the node-failure case upstream designs for). | +| F14 | Startup storage probe (`CELLD_STORAGE_PROBE`, default on): every node writes and deletes one small object under `probe/` to test the four conditional-write transitions (create, reject-create, update, reject-stale) and stops if the store fails any — a store that ignores the precondition would otherwise make the node self-fence in a loop. `celld diagnose` runs the same test (`--read-only` skips it). celld reserves `probe/`, `cells/`, `nodes/`, `node-cells/`, `fleet/`, `deploy/`, `deploy-blobs/`, `wake/` and `telemetry/` under the prefix. | Per-fleet credentials must allow put and delete under the prefix (they already must for cells); README qualification guidance is `celld diagnose` for the sequential contract and `hack/cas-hammer` for atomicity under racers (S3 API only). | ## Wire-format notes (learned from source, not docs) -- `/state` returns `{"ownership","occupied","evicting","restoring","shedding",...}` - where `shedding` is `null` or a **reason string** (e.g. `"rss"`), not a - boolean, and `restoring` is the activation backlog. +- `/state` (`actor.rs` `state_json`) returns + `{"ownership","occupied","evicting","restoring","activating","activation_waiting","capacity_waiting","phases","shedding","rss_bytes","in_use_bytes","residents","published","publishes","stops"}` + where `shedding` is `null` or a **reason string** (`"memory"` for the + cell-memory threshold, `"rss-hard"` for the absolute RSS cap), not a + boolean, and `restoring` is the activation backlog. The three breakdown + counters and both memory numbers are new in v0.3.0 (absent on older + nodes, so they decode as zero). - `deploy/current.json` is `{script_name?, version, prefix, rollout}`; - `version` is the ID `celld deploy` prints. + `version` is the ID `celld deploy` prints. The manifest it points at + carries a `features` list (F4). - `CELLD_VARS_FILE` is env-file format (`NAME=value` lines, `#` comments, optional quotes), hence the `vars.env` Secret key convention. - `CELLD_IDLE_EVICT_S` is **disabled unless set**: idle cells stay resident indefinitely by default. +- Boolean celld variables accept only `0` or `1`; a supplied invalid value + stops startup, so the operator writes `1`, never `true`. - The internal listener also serves `/cell/` (resolve *or activate*), - `/evict/`, `/do/` (send a direct Durable Object request), + `/evict/`, `/do/` (send a direct Durable Object request; refuses a D1 + database), `/__d1/SCOPE` (SQL to a D1 database, authenticated with the + fleet secret from the bucket — what `celld d1` uses), `POST /shutdown`, `POST /shutdown?handoff=preserve`, and `/__celld/probe` — unused by the - operator today, available for ops tooling. Note `/cell/` and `/do/` mean an - unauthenticated caller who reaches the port can execute application code - and mutate cell state, not merely inspect it. + operator today, available for ops tooling. Note `/cell/` and `/do/` mean + an unauthenticated caller who reaches the port can execute application + code and mutate cell state, not merely inspect it; only the D1 route + authenticates. +- Cron triggers (`triggers.crons`) run as one reserved cell per script + (`.cron: