Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions packages/scale-zero-pg/deploy/compute-app.template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,16 @@ spec:
initContainers:
- name: wait-timeline
image: docker.io/neondatabase/neon:8464@sha256:7a4f124917bb929964b2d696d710f19584f80bb9bd51b2af4a6e2425434c761f
# Pod Security `restricted`. Both images name their USER (neon / postgres,
# uid 1000:1000 in these digests), which the kubelet cannot verify as
# non-root, so the numeric id is spelled out.
securityContext:
allowPrivilegeEscalation: false
capabilities: { drop: ["ALL"] }
seccompProfile: { type: RuntimeDefault }
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
envFrom:
- configMapRef: { name: compute-config-__APP__ }
command: ["/bin/sh", "-c"]
Expand All @@ -87,6 +97,13 @@ spec:
- name: compute
image: docker.io/neondatabase/compute-node-v17:8464@sha256:13ab146d3e7bbabb25a8532f315ac443e7512351d1ede0bab586def5c70e26c3 # PINNED pair with storage (ops "Upgrades")
imagePullPolicy: IfNotPresent
securityContext:
allowPrivilegeEscalation: false
capabilities: { drop: ["ALL"] }
seccompProfile: { type: RuntimeDefault }
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
command: ["/bin/sh", "/compute-files/entrypoint.sh"]
envFrom:
- configMapRef: { name: compute-config-__APP__ }
Expand Down
49 changes: 49 additions & 0 deletions packages/scale-zero-pg/docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -2381,6 +2381,55 @@ that `compute:55433` is unreachable from a non-gateway pod and **fails** (not wa
if it is not — so a regression in the policies, or a CNI that only partially enforces,
is caught instead of passing silently.

## Pod Security: per-app compute pods meet `restricted`

Every container of the per-app compute pods the operator renders (the writer and
the read-replica pool, init containers included) carries a container
`securityContext` that satisfies the Kubernetes Pod Security `restricted` level, so
the pods are admitted in a namespace labelled
`pod-security.kubernetes.io/enforce=restricted`:

- `allowPrivilegeEscalation: false`
- `capabilities.drop: [ALL]` (nothing is added back; compute_ctl and Postgres run
unprivileged on ports above 1024)
- `seccompProfile: RuntimeDefault`
- `runAsNonRoot: true`, with a numeric `runAsUser` / `runAsGroup` (default `1000`, see below to override)

The numeric uid matters: the pinned compute image declares its user by name
(`postgres`) and the init image as `neon`, and the kubelet refuses `runAsNonRoot`
for a named user it cannot resolve to a number. Both are uid/gid 1000 in the pinned
digests, which is the identity the pods already ran as, so file ownership and the
boot sequence are unchanged. `readOnlyRootFilesystem` is not set: Postgres writes
under the container filesystem.

**Using your own compute or init image.** The operator can be pointed at other
images with `APPDB_COMPUTE_IMAGE` and `APPDB_INIT_IMAGE`. The uid above is only
right for the default digests, so an image that runs as a different user needs its
identity supplied with it, on the operator Deployment:

| Variable | Applies to | Default |
| --- | --- | --- |
| `APPDB_COMPUTE_UID` / `APPDB_COMPUTE_GID` | the `compute` container | `1000` / `1000` |
| `APPDB_INIT_UID` / `APPDB_INIT_GID` | the `wait-timeline` and `resolve-lsn` init containers | `1000` / `1000` |

The compute and init images are separate images with their own users, which is why
each has its own pair. Read the real value from the image rather than guessing:
`crane export <image> - | tar -xO etc/passwd`. Each value must be a decimal number
from 1 to 2147483647; `0` is refused because `runAsNonRoot` forbids root, and a
user name is refused because the kubelet cannot verify a name as non-root. A bad
value stops the operator at start-up with a message naming the variable. If the
uid is wrong for the image, the container starts as the wrong user and the compute
fails to boot (typically a permission error on its data directory), so set it
whenever you override an image.

**If you bump the default image digests in the manifests**, record each new
image's uid first: the repository's unit tests keep a hand-maintained table of
every pinned digest and its uid and fail when a digest has no entry.

The break-glass template (`deploy/compute-app.template.yaml`) carries the same
fields. The base-tier manifests (`20-compute.yaml`, `25-compute-warm.yaml`,
`26-compute-ro.yaml`) still set only the pod-level seccomp profile.

## Tenant isolation controls (issue #112 / #113 / #114)

Because NetworkPolicy is **inert on flannel** (above), the tenant boundary must not
Expand Down
51 changes: 49 additions & 2 deletions packages/scale-zero-pg/gateway/cmd/appdb-operator/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import (
"encoding/hex"
"fmt"
"log"
"math"
"net/http"
"os"
"os/signal"
Expand Down Expand Up @@ -49,6 +50,51 @@ func gatewayHostFromEnv() string {
return env("APPDB_GATEWAY_HOST", appdb.DefaultGatewayHost)
}

// applyImageEnv applies the image overrides (APPDB_COMPUTE_IMAGE, APPDB_INIT_IMAGE)
// together with the numeric identity those images run as. The compute pods run
// every container with runAsNonRoot and an explicit runAsUser/runAsGroup, because
// the images declare their user by name and the kubelet cannot verify a name as
// non-root. The compute image and the init image are different images with their
// own users, so each has its own pair: APPDB_COMPUTE_UID/GID and
// APPDB_INIT_UID/GID. Unset keeps the pinned defaults (right only for the pinned
// digests); an image override whose user is not that default MUST set its pair,
// or the container starts as the wrong user. A malformed value is an error here
// (start-up), never a silent fallback.
func applyImageEnv(render *appdb.RenderConfig) error {
render.ComputeImage = env("APPDB_COMPUTE_IMAGE", render.ComputeImage)
render.InitImage = env("APPDB_INIT_IMAGE", render.InitImage)
for _, f := range []struct {
key string
dst *int64
}{
{"APPDB_COMPUTE_UID", &render.ComputeUID},
{"APPDB_COMPUTE_GID", &render.ComputeGID},
{"APPDB_INIT_UID", &render.InitUID},
{"APPDB_INIT_GID", &render.InitGID},
} {
v, err := idFromEnv(f.key, *f.dst)
if err != nil {
return err
}
*f.dst = v
}
return nil
}

// idFromEnv reads a numeric uid/gid: unset or empty -> def; otherwise a decimal
// integer in [1, 2147483647]. 0 is refused because runAsNonRoot forbids root.
func idFromEnv(key string, def int64) (int64, error) {
raw := os.Getenv(key)
if raw == "" {
return def, nil
}
n, err := strconv.ParseInt(raw, 10, 64)
if err != nil || n < 1 || n > math.MaxInt32 {
return 0, fmt.Errorf("%s=%q is invalid: want a numeric id between 1 and %d (runAsNonRoot forbids 0, and the container image's user name cannot be used)", key, raw, math.MaxInt32)
}
return n, nil
}

func envInt(key string, def int) int {
if v := os.Getenv(key); v != "" {
if n, err := strconv.Atoi(v); err == nil {
Expand Down Expand Up @@ -112,8 +158,9 @@ func main() {
render.PageserverHost = pageserverHost
render.PGVersion = strconv.Itoa(pgVersion)
render.RolePrefix = rolePrefix
render.ComputeImage = env("APPDB_COMPUTE_IMAGE", render.ComputeImage)
render.InitImage = env("APPDB_INIT_IMAGE", render.InitImage)
if err := applyImageEnv(&render); err != nil {
logger.Fatalf("[appdb] %v", err)
}

cluster := appdb.NewK8sCluster(cs, dyn, namespace, render, reclaimCM, logger)
// The DB warm-hold actuator (knext #388, ADR-0030 addendum; #777): the
Expand Down
87 changes: 87 additions & 0 deletions packages/scale-zero-pg/gateway/cmd/appdb-operator/main_test.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,98 @@
package main

import (
"os"
"strings"
"testing"

"github.com/alpheya/scale-zero-pg/gateway/internal/appdb"
)

var identityEnv = []string{
"APPDB_COMPUTE_IMAGE", "APPDB_INIT_IMAGE",
"APPDB_COMPUTE_UID", "APPDB_COMPUTE_GID", "APPDB_INIT_UID", "APPDB_INIT_GID",
}

func clearIdentityEnv(t *testing.T) {
t.Helper()
for _, k := range identityEnv {
t.Setenv(k, "")
}
}

// With nothing set, the image overrides and uids stay at the pinned defaults.
func TestApplyImageEnvDefaults(t *testing.T) {
clearIdentityEnv(t)
got := appdb.DefaultRenderConfig("ns")
if err := applyImageEnv(&got); err != nil {
t.Fatalf("applyImageEnv with no env: %v", err)
}
if want := appdb.DefaultRenderConfig("ns"); got != want {
t.Fatalf("no env changed the config:\n got %+v\nwant %+v", got, want)
}
}

// The images and the identity they run as are overridden together, each image
// with its own pair.
func TestApplyImageEnvOverrides(t *testing.T) {
clearIdentityEnv(t)
t.Setenv("APPDB_COMPUTE_IMAGE", "registry.example.com/pg@sha256:aa")
t.Setenv("APPDB_INIT_IMAGE", "registry.example.com/init@sha256:bb")
t.Setenv("APPDB_COMPUTE_UID", "26")
t.Setenv("APPDB_COMPUTE_GID", "27")
t.Setenv("APPDB_INIT_UID", "65532")
t.Setenv("APPDB_INIT_GID", "65533")
c := appdb.DefaultRenderConfig("ns")
if err := applyImageEnv(&c); err != nil {
t.Fatalf("applyImageEnv: %v", err)
}
if c.ComputeImage != "registry.example.com/pg@sha256:aa" || c.InitImage != "registry.example.com/init@sha256:bb" {
t.Errorf("image overrides not applied: %q / %q", c.ComputeImage, c.InitImage)
}
if c.ComputeUID != 26 || c.ComputeGID != 27 || c.InitUID != 65532 || c.InitGID != 65533 {
t.Errorf("identity overrides not applied: compute %d:%d init %d:%d", c.ComputeUID, c.ComputeGID, c.InitUID, c.InitGID)
}
}

// A bad value fails at start-up and names the variable and the value: runAsNonRoot
// forbids uid 0 and a non-numeric id cannot be rendered.
func TestApplyImageEnvRejectsBadIdentity(t *testing.T) {
for _, key := range []string{"APPDB_COMPUTE_UID", "APPDB_COMPUTE_GID", "APPDB_INIT_UID", "APPDB_INIT_GID"} {
for _, bad := range []string{"0", "-1", "postgres", "1000x", "1.5", "2147483648", "99999999999999999999"} {
clearIdentityEnv(t)
t.Setenv(key, bad)
c := appdb.DefaultRenderConfig("ns")
err := applyImageEnv(&c)
if err == nil {
t.Errorf("%s=%q was accepted, want a start-up error", key, bad)
continue
}
if !strings.Contains(err.Error(), key) || !strings.Contains(err.Error(), bad) {
t.Errorf("%s=%q: error %q does not name the variable and the value", key, bad, err)
}
}
}
}

// main() itself is not unit-testable, so scan it: the image overrides must flow
// through applyImageEnv (which also carries their uid/gid), never be read raw
// again, or an overridden image would silently run as the default uid.
func TestMainAppliesImageOverridesOnlyThroughApplyImageEnv(t *testing.T) {
src, err := os.ReadFile("main.go")
if err != nil {
t.Fatal(err)
}
s := string(src)
if n := strings.Count(s, "applyImageEnv(&render)"); n != 1 {
t.Errorf("main() must call applyImageEnv(&render) exactly once, found %d", n)
}
for _, key := range []string{"APPDB_COMPUTE_IMAGE", "APPDB_INIT_IMAGE"} {
if n := strings.Count(s, `"`+key+`"`); n != 1 {
t.Errorf("%s must be read in exactly one place (applyImageEnv), found %d", key, n)
}
}
}

// Half 1: with no APPDB_GATEWAY_HOST set, the operator mints DSNs against the
// ROOTED apps-gateway name (trailing dot) — the only form that skips the ndots:5
// search-path walk on a fresh pod's first resolutions; a 4-dot name without the dot
Expand Down
2 changes: 2 additions & 0 deletions packages/scale-zero-pg/gateway/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ require (
k8s.io/api v0.34.1
k8s.io/apimachinery v0.34.1
k8s.io/client-go v0.34.1
k8s.io/pod-security-admission v0.34.1
sigs.k8s.io/yaml v1.6.0
)

Expand Down Expand Up @@ -47,6 +48,7 @@ require (
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/component-base v0.34.1 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b // indirect
k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
Expand Down
4 changes: 4 additions & 0 deletions packages/scale-zero-pg/gateway/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -154,10 +154,14 @@ k8s.io/apimachinery v0.34.1 h1:dTlxFls/eikpJxmAC7MVE8oOeP1zryV7iRyIjB0gky4=
k8s.io/apimachinery v0.34.1/go.mod h1:/GwIlEcWuTX9zKIg2mbw0LRFIsXwrfoVxn+ef0X13lw=
k8s.io/client-go v0.34.1 h1:ZUPJKgXsnKwVwmKKdPfw4tB58+7/Ik3CrjOEhsiZ7mY=
k8s.io/client-go v0.34.1/go.mod h1:kA8v0FP+tk6sZA0yKLRG67LWjqufAoSHA2xVGKw9Of8=
k8s.io/component-base v0.34.1 h1:v7xFgG+ONhytZNFpIz5/kecwD+sUhVE6HU7qQUiRM4A=
k8s.io/component-base v0.34.1/go.mod h1:mknCpLlTSKHzAQJJnnHVKqjxR7gBeHRv0rPXA7gdtQ0=
k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b h1:MloQ9/bdJyIu9lb1PzujOPolHyvO06MXG5TUIj2mNAA=
k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b/go.mod h1:UZ2yyWbFTpuhSbFhv24aGNOdoRdJZgsIObGBUaYVsts=
k8s.io/pod-security-admission v0.34.1 h1:XsP5eh8qCj69hK0a5TBMU4Ed7Ckn8JEmmbk/iepj+XM=
k8s.io/pod-security-admission v0.34.1/go.mod h1:87yY36Gxc8Hjx24FxqAD5zMY4k0tP0u7Mu/XuwXEbmg=
k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 h1:hwvWFiBzdWw1FhfY1FooPn3kzWuJ8tmbZBHi4zVsl1Y=
k8s.io/utils v0.0.0-20250604170112-4c0f3b243397/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE=
Expand Down
Loading
Loading