Add configurable pod and container securityContext, including readOnlyRootFilesystem support

[mouchar]

Motivation

The chart cannot currently be deployed on a cluster that enforces a restrictive pod security policy (Pod Security Admission, OPA Gatekeeper, Kyverno):

• A pod-level securityContext is exposed for only zookeeper, bookkeeper, broker and oxia.server.
• There is no container-level securityContext anywhere, so allowPrivilegeEscalation, capabilities, seccompProfile and readOnlyRootFilesystem are not settable at any value.
• The autorecovery StatefulSet renders no securityContext and has no probes.
• readOnlyRootFilesystem cannot work unaided, because the Pulsar images rewrite /pulsar/conf on startup and log to /pulsar/logs.

Modifications

Two commits, reviewable independently.

1. securityContext cascade. Adds global podSecurityContext and containerSecurityContext, each merged with a matching per-component override that wins per key:

podSecurityContext        <- <component>.securityContext
containerSecurityContext  <- <component>.containerSecurityContext

Applied to all 18 pod templates, including initContainers and the init/cleanup Jobs. The merge uses mergeOverwrite rather than merge, because merge treats zero values in its destination as absent and would silently discard a per-component fsGroup: 0 or allowPrivilegeEscalation: false.

2. readOnlyRootFilesystem support. When the effective container securityContext sets it, the chart mounts an emptyDir over /pulsar/conf, /pulsar/logs and /tmp and prepends a copy-pulsar-conf initContainer that seeds the conf volume from the image, for the components that run a Pulsar image. writableRootfsVolumes: false opts out in favour of your own extraVolumes/extraVolumeMounts.

This could not be done from values: values-supplied initContainers are appended after the built-in ones so a seeding container cannot run first, and only autorecovery exposes initContainersExtraVolumeMounts, so broker/wait-bookkeeper-ready — which runs apply-config-from-env.py — would fail.

Also adds examples/values-restricted-psp.yaml as a worked example, including the per-component overrides needed to move the four components shipping fsGroup: 0 off GID 0.

Rendered output with default values is unchanged except for three deliberate items:

• The autorecovery StatefulSet gains liveness/readiness probes. It had none and no knob to add them. The daemon does not start BookKeeper's HTTP service, so they target the Prometheus stats endpoint on autorecovery.ports.http — the endpoint the PodMonitor already scrapes.
• The zookeeper and broker sts-cleanup upgrade-hook Jobs gain pod template labels via pulsar.template.labels. They were the only pod templates without them, so .Values.labels never reached their pods.
• Three Jobs now render the fsGroup: 0 of the component they belong to. Those pods mount only ConfigMaps, Secrets and the service account token, so this has no functional effect.

Verifying this change

• Make sure that the change passes the CI checks.

helm lint clean. kubeconform against k8s 1.25 / 1.31 / 1.36 for the default values, every .ci/clusters/* config and .ci/templates-all-values.yaml, with the new feature both off and on.

Rendered output diffed against master across all of the above: identical apart from the three items listed, and byte-identical for commit 2 when readOnlyRootFilesystem is unset.

Runtime-tested on EKS 1.34 with the AWS EBS CSI driver (gp3/ext4, fsGroupPolicy: ReadWriteOnceWithFSType), upgrading in place from published 4.7.0:

• Moving fsGroup from 0 to 10000 triggers the recursive relabel and preserves data: a 5000-message unacked backlog was consumed intact afterwards, with identical storageSize and no pod restarts.
• With readOnlyRootFilesystem: true, all pods reach Ready, copy-pulsar-conf and the built-in init containers complete, apply-config-from-env.py writes to the seeded conf volume, and produce/consume keeps working with no read-only filesystem errors.

Also exercised on a 4-node kind cluster (k8s 1.36.1) with the default values, with the restricted-PSP example applied as an upgrade, and with Oxia as the metadata store.

[lhotari]

Thanks for this — it's careful work and the parts that are right are very right. I verified your central compatibility claim and it holds exactly: with default values the rendered output differs from master in precisely the three items you list, 43 diff lines total. With oxia and pulsar_manager enabled it's byte-identical. kubeconform 1.31 strict passes for both the default render and values-restricted-psp.yaml. The mergeOverwrite + deepCopy choice is correct, and the reasoning in the helper comment about merge discarding zero values is right.

Commit 1 (the securityContext cascade) looks close to mergeable on its own. The issues are in commit 2's edge cases and in the docs.

Blocking

Four issues, all inline: a --reuse-values render failure, and three cases where readOnlyRootFilesystem produces a pod that can't start. The first one is the most urgent since it breaks upgrades for everyone who uses that flag, regardless of whether they touch the new feature.

The fsGroup / GID 0 framing — please reconsider

This is the part I'd most like changed, because I think the example teaches the wrong lesson, and it's the reason fsGroup: 0 is the chart default in the first place.

fsGroup: 0 is not a privilege. Group 0 inside a container is an ordinary group; root power comes from UID 0 and capabilities. It does grant DAC access to group-0-accessible files — which is exactly the mechanism that makes the Pulsar image work under an arbitrary UID — but it is not "running as root".

Concretely: the Kubernetes restricted Pod Security Standard places no constraint on fsGroup, fsGroupChangePolicy or supplementalGroups at all. Its controls are runAsNonRoot, runAsUser != 0, allowPrivilegeEscalation, capabilities, seccompProfile, and the host/volume rules. The PSS-restricted policy sets that Gatekeeper and Kyverno ship mirror PSA and likewise don't restrict fsGroup — both engines can via separately installed policy, but nothing does out of the box.

So the four fsGroup: 10000 overrides in examples/values-restricted-psp.yaml aren't needed for the file's stated purpose, and they're the single most operationally dangerous lines in it — a recursive chown of every bookie ledger volume, which your own warning comment at the top of the file describes.

The background, from Red Hat's UID guide: "the user in the Container always has GID=0, which is the root group", and Red Hat's recommendation is that writable files "should be owned by the root group and be read/writable by GID=0". That's why the chart defaults to fsGroup: 0.

Worth knowing though — I went and checked, and on a stock OpenShift project neither value is admissible. From OpenShift's shipped manifest (openshift/cluster-kube-apiserver-operator, bindata/bootkube/scc-manifests/…_00_scc-restricted-v2.yaml), restricted-v2 is:

runAsUser:          {type: MustRunAsRange}
fsGroup:            {type: MustRunAs}
supplementalGroups: {type: RunAsAny}
allowPrivilegeEscalation: false
requiredDropCapabilities: [ALL]
seccompProfiles: [runtime/default]

and the MustRunAs implementation (openshift/apiserver-library-go, pkg/securitycontextconstraints/group/mustrunas.go) validates any pod-supplied group against the namespace's preallocated range, rejecting with "%d is not an allowed group". On a typical range (~1000620000/10000): fsGroup: 10000 rejected, fsGroup: 0 rejected, runAsUser: 10000 rejected. Omitting them lets admission inject correct values. (fsGroup: 0 works on OpenShift today under anyuid or a custom SCC, which use fsGroup: RunAsAny.)

Nice result worth calling out: your containerSecurityContext block matches restricted-v2 field for field. That half of the example is exactly right.

Suggested changes:

• Keep the component fsGroup: 0 defaults as they are.
• Keep the example's containerSecurityContext block verbatim.
• Drop runAsUser/runAsGroup/fsGroup/supplementalGroups from the example's podSecurityContext, keeping runAsNonRoot: true, and delete the four per-component fsGroup: 10000 blocks.
• Retitle the README's "Moving off GID 0" — it reads as a hardening step, which it isn't — and note that PSA restricted doesn't constrain fsGroup.
• If you want an OpenShift example, it needs to be a separate file that clears the defaults (zookeeper: {securityContext: null} etc.), because simply removing the overrides restores fsGroup: 0, which restricted-v2 also rejects.

Non-blocking

• User-supplied <component>.initContainers are raw toYaml passthrough and never merged with containerSecurityContext — same for oxia.coordinator.extraContainers (oxia-coordinator-deployment.yaml:93) and dekaf…extraContainers (dekaf-deployment.yaml:105). The README says the settings apply to "every container and initContainer the chart renders". Either narrow the wording or merge them.
• autorecovery.probe.liveness.enabled: true adds a default-on livenessProbe to a component that previously had none, targeting an endpoint that comes from the image's BookKeeper stats defaults rather than anything the chart sets. A user who changes statsProviderClass/prometheusStatsHttpPort via autorecovery.configData turns a working deployment into a restart loop, where that config was previously inert. Defaulting liveness false and readiness true would be the safer compatibility choice.
• "No functional effect" for the three Jobs inheriting fsGroup: 0 is true for volume ownership but not for admission — fsGroup is an SCC input on OpenShift. Narrow, but the claim as written is incomplete.

_{Reviewed with Codex gpt-5.6-sol and Claude Opus 5; every finding reproduced locally by rendering the chart, validating with kubeconform -strict, and checking OpenShift's shipped SCC manifests.}

[mouchar]

Hello Lari, thank you for your time spent on reviewing my PR. All four blocking issues are
fixed, and you're right about the fsGroup framing — the confusion was mine, explained below.

Fixed

• --reuse-values. Every autorecovery.probe lookup now goes through dig with a
  default, and pulsar.rootfs.enabled uses hasKey so an absent writableRootfsVolumes
  means "chart default: enabled" rather than being indistinguishable from an explicit
  false. Verified against a real release record rather than --set …=null: install
  published 4.7.0, then helm upgrade --reuse-values --dry-run — nil pointer before, clean
  render after, and the --set …readOnlyRootFilesystem=true variant now emits the seed
  container and volumes instead of nothing.
• bookkeeper waitMetadataTimeout. initContainers: is emitted whenever anything needs
  it — waitMetadataTimeout, the rootfs feature, cacerts, or user-supplied
  bookkeeper.initContainers — with verify-clusterid still gated on the timeout.
• pulsar-manager cluster-initialize. Now gets the rootfs volumes. The README claim is
  corrected: the pulsar_manager StatefulSet runs apachepulsar/pulsar-manager and stays
  excluded, but its cluster-initialize Job runs a Pulsar image and is included.
• runAsGroup / fsGroup. Removed from the README and all three places in
  values.yaml, replaced with the opposite.
• autorecovery probes. Both liveness and readiness now default to false. Readiness
  targets the same /metrics endpoint, so it has the same failure mode — worse, in a way,
  since a permanently unready pod stalls the StatefulSet rollout rather than just
  restarting. With both off this is purely an added knob, and the default-values delta
  against master drops from the 43 lines you measured to 21 — only the sts-cleanup pod
  labels and the three Jobs rendering their component's fsGroup: 0 remain.
• Values-supplied containers. README now says "every container the chart itself
  renders" and names <component>.initContainers, oxia.coordinator.extraContainers and
  dekaf.deployment.extraContainers as verbatim passthrough. I narrowed the wording rather
  than merging, on the grounds that silently injecting into user-provided YAML is the more
  surprising behaviour — happy to change that if you disagree.
• "No functional effect" for the Jobs inheriting fsGroup: 0 — reworded, since it is
  also an SCC input on OpenShift.

One pre-existing bug, found while fixing the second item

The initContainers: guard is identical on master, so this is latent today and unrelated
to this PR:

helm template t charts/pulsar --set bookkeeper.waitMetadataTimeout=0 \
  --set bookkeeper.initContainers[0].name=x --set bookkeeper.initContainers[0].image=busybox
Error: YAML parse error ... did not find expected key

The user's init containers are emitted without their parent key. My change fixes it as a
side effect. Say the word if you'd rather have it as a separate PR.

The "restricted" confusion — my fault

We were talking about different things. You read it as the restricted Pod Security
Standard; I was using "restricted" in the everyday sense, and naming the file
values-restricted-psp.yaml while the README said "Pod Security Admission, OPA Gatekeeper,
Kyverno" made that reading the obvious one. You're right that PSS restricted places no
constraint on fsGroup — I confirmed it on a namespace labelled
pod-security.kubernetes.io/enforce: restricted, where fsGroup: 0 is admitted and runs
as uid=10000 gid=0(root).

What I was actually targeting is a Gatekeeper constraint built on the PSP-derived
K8sPSPAllowedUsers template with

fsGroup: {rule: MustRunAs, ranges: [{min: 1, max: 65535}]}

which rejects fsGroup: 0 at admission. This is not hypothetical or bespoke — regulated
environments, financial-sector ones in particular, deploy exactly this and workloads are
denied rather than merely flagged. So the capability is needed, but calling it "restricted"
was wrong.

The example is therefore split rather than deleted:

• examples/values-psa-restricted.yaml (new) — PSS restricted: runAsNonRoot: true
  plus your containerSecurityContext block verbatim, and the per-component fsGroup: 0
  defaults left completely alone.
• examples/values-restricted-group-ranges.yaml (renamed) — the group-range case,
  opening with an explicit note that this is not what PSS restricted requires and
  pointing at the other file as the safer default.

The README section is retitled "Overriding fsGroup, if a policy constrains group IDs" and
now states that fsGroup: 0 is not a privilege, that it is what lets the images run under
an arbitrary assigned UID, and that restricted does not constrain it.

I have not added an OpenShift example. It needs to clear the chart defaults rather than
override them, and I'd rather test that on a real cluster before shipping it than guess —
happy to follow up separately.

Re-verification

helm lint clean; 129 clean kubeconform runs across k8s 1.25/1.31/1.36 over the default
values, every .ci/clusters/* config and .ci/templates-all-values.yaml, feature off and
on; both example files render and pass kubeconform -strict on all three versions.

[lhotari]

Thanks — this is a big step forward, and the fsGroup discussion is fully settled from my side. I re-ran the whole review against 9e0f9204, rendering rather than reading, and all six previous findings check out. I've resolved those threads.

Verified fixed

Finding	How I checked
--reuse-values nil pointer	--set autorecovery.probe=null renders clean; absent writableRootfsVolumes + readOnlyRootFilesystem still yields copy-pulsar-conf and all three volumes. hasKey semantics are right.
bookkeeper waitMetadataTimeout: 0	Now renders init=[copy-pulsar-conf] with conf/logs/tmp mounted. All 16 guard combinations render correctly — never empty, never omitted when a child renders.
pulsar-manager cluster-initialize Job	Seeds conf and mounts conf/logs/tmp on all four containers, so curl -D headers.txt works.
runAsGroup / fsGroup	Zero stale copies; replaced with the correct opposite.
autorecovery probes	Both default off. Default-values delta is exactly 21 added lines — your number is precise, and no probe delta remains.
Docs over-claim	Narrowed correctly. I agree narrowing beats merging: silently rewriting user-supplied container specs would be the more surprising behaviour.

The pre-existing bookkeeper bug is real on master (waitMetadataTimeout=0 plus user initContainers emits an orphan list item and fails to parse) and fixed here. Please keep it in this PR — it's a side effect of a guard you had to touch anyway, and splitting it out would just create a rebase dependency.

On the Gatekeeper justification: you're right and I was wrong to frame the capability as unnecessary. K8sPSPAllowedUsers is a current template in the official Gatekeeper library exposing exactly fsGroup.rule (enum MustRunAs/MayRunAs/RunAsAny) and ranges[].min/max. The split is well executed — the group-ranges file opens by saying it is not what PSS restricted requires, points at the safer file, and even carries an accurate OpenShift note. Settled.

---

New blockers

The adversarial pass found four things, all verified by rendering. Three are in the new PSA example and the pulsar-manager StatefulSet; the fourth is a design gap in the volume feature itself. Details inline.

Briefly:

1. values-psa-restricted.yaml can't install into an enforce=restricted namespace — the bundled monitoring stack is left enabled, and node-exporter renders with hostNetwork: true, hostPID: true and hostPath mounts.
2. The PSA example's own reasoning breaks the pre-upgrade hooks — runAsNonRoot: true without runAsUser, but the sts-cleanup Jobs run alpine/k8s, which has no USER directive.
3. Six templates get readOnlyRootFilesystem with no writable volumes at all — pulsar-manager is one instance of a wider class.
4. The rootfs volumes collide fatally with any user-supplied mount on the same path.

On the emptyDirVolumes redesign

I like the direction and I think finding 3 forces it — see the inline note on values.yaml. Short version: rename to emptyDirVolumes, make it a list of {path, seedFromImage, sizeLimit}, and allow a per-component override. That turns the six-component gap from "documented exclusion" into something a user can actually fix, and it removes the collision by letting people drop an entry.

Happy to iterate on the shape before you build it — no need to guess.

_{Reviewed with Codex gpt-5.6-sol and Claude Opus 5. Claude Fable was in the reviewer set but failed to return again, so this round is two-model rather than three. Every finding was reproduced locally by rendering the chart.}

[mouchar]

Thanks — all four are addressed. Force-pushed as two commits again.

1. emptyDirVolumes list replaces the boolean. Entries are {path, seedFromImage, sizeLimit}, one volume per entry, name derived from the path, and a single seed container mounting each seeded volume at /mnt/<name>. <component>.emptyDirVolumes replaces the inherited list rather than extending it, and [] opts a component out. An entry whose path the component already mounts through extraVolumeMounts is skipped, so the duplicate-mountPath collision you pointed at no longer happens — I reproduced it against a live API server first, and the same manifest now applies cleanly. Relative paths, duplicate paths, unknown entry keys, colliding volume names and over-long names all fail at render time with an explanatory message.

2. The six templates that had no writable volumes. dekaf gets /tmp. oxia server and coordinator get /tmp as you suggested, though they also started fine without it. The two sts-cleanup hooks run kubectl from alpine/k8s and need nothing writable, so they are left outside the mechanism.

pulsar_manager is the exception: [{path: /run/postgresql}, {path: /tmp}] is not enough. That image also needs /var/log/nginx, /var/lib/nginx and /run/nginx.pid, and writes a supervisord socket into /pulsar-manager — its own install directory, which an emptyDir cannot cover without hiding the application. Since a documented exclusion that yields a non-starting pod is still a trap, enabling both now fails the render with a message naming the reason and the opt-out, following the existing fail precedent in the chart.

3. The PSA example. runAsUser: 10000 is now set explicitly, for the reason you gave — alpine/k8s has no USER directive, so runAsNonRoot alone gets the pre-upgrade hooks refused by the kubelet and blocks helm upgrade. The bundled monitoring stack is disabled, since these values do not reach subchart templates. I applied the same to values-restricted-group-ranges.yaml, which had the identical problem.

4. The pre-existing bookkeeper initContainers: fix stays in this PR, as you asked.

Verification: helm lint clean, kubeconform across k8s 1.25/1.31/1.36 over 17 value sets, and output unchanged against the previous push for the default values and every .ci config — the default render is still the same 21 added lines you counted. Checked at runtime on EKS by installing without the feature and then upgrading with --reuse-values --set containerSecurityContext.readOnlyRootFilesystem=true: messages produced before the upgrade were consumed after it, with no read-only filesystem errors in any container.

One open question: opting pulsar_manager out also opts out its cluster-initialize Job, which shares the pulsar_manager securityContext even though it runs a Pulsar image and would otherwise work. Happy to give that Job its own key if you would rather not have the edge case.

[lhotari]

All four addressed, and the round-1 fixes survived the rewrite. I've resolved the remaining threads. LGTM.

The rewrite touched 21 files and replumbed volumes in every workload, so I re-tested the earlier fixes specifically rather than assuming — everything below is from rendering, not reading.

emptyDirVolumes

The implementation is better than what I sketched. The collision that made this a blocker is gone, and every validation you claim actually fires:

Case	Result
relative path (pulsar/logs)	fails
duplicate paths	fails
unknown entry key	fails
over-long derived name (70 chars)	fails
derived-name collision (/pulsar/logs + /pulsar-logs)	fails
trailing-slash collision (/tmp + /tmp/)	fails
valid custom list	renders
[] opt-out	no volumes

The last two are worth calling out. The sanitiser plus the DNS-1123 regexMatch and 63-char check catches both collision classes I raised — including /tmp vs /tmp/, which I only thought to test after reading the helper.

The pulsar_manager guard

Well built. It keys on the merged effective context rather than the global, and the message names the escape hatch. I hunted false positives specifically:

Scenario	Behaviour
global roRootFS + pulsar_manager enabled	fails (intended)
+ pulsar_manager.containerSecurityContext.readOnlyRootFilesystem: false	renders — escape hatch works
pulsar_manager enabled, roRootFS unset	renders
roRootFS set, pulsar_manager disabled	renders

That distinction — a guard you can escape rather than a wall — is exactly right.

Everything else

• Six templates: oxia server/coordinator and dekaf get /tmp. Leaving the sts-cleanup hooks outside is the right call: the script only runs kubectl with no temp files, and kubectl degrades gracefully when its discovery cache isn't writable.
• Both examples: runAsUser: 10000 set, monitoring stack fully disabled (zero VM/node-exporter references, zero hostNetwork/hostPID/hostPath violations). Every workload in both files carries all four PSS-restricted controls, and both pass kubeconform -strict (43 valid, 0 invalid).
• Default delta: exactly 21 added, 0 removed. Your number is precise.
• No regressions. The one I was most concerned about was the key rename: under --reuse-values both autorecovery.probe and emptyDirVolumes are absent, and the chart correctly falls back to the default list — init=[copy-pulsar-conf, wait-zookeeper-ready, wait-bookkeeper-ready], vols=[pulsar-conf, pulsar-logs, tmp]. Renaming the key didn't reintroduce the original bug. bookkeeper waitMetadataTimeout: 0 still correct too.
• Pre-existing bookkeeper fix retained, as agreed.

Your open question — no need to handle it

Don't give the cluster-initialize Job its own key. pulsar_manager support is going to be dropped from the chart before long: the upstream project is no longer maintained and carries unfixed vulnerabilities, and dekaf is already in the chart as the replacement. Adding values surface for a component on its way out isn't worth it, and the edge case is harmless in the meantime — the Job simply doesn't get the read-only hardening it could have had. Leave it.

That also retroactively justifies not investing in making the pulsar-manager StatefulSet itself work under a read-only root: the fail is the right amount of effort for a component with a limited remaining life.

One optional nit

<component>.emptyDirVolumes: [] combined with readOnlyRootFilesystem: true gives a read-only container with no writable paths and no warning — the same trap class as the old pulsar-manager case, just user-initiated. It's the documented escape hatch so it's defensible; a line in the values comment saying you must then supply the paths yourself via extraVolumes/extraVolumeMounts would close the loop. Not blocking.

Nice work on this one — it went from a broad feature to something with real edges tested.

_{This round was reviewed solo (Claude Opus 5); Codex gpt-5.6-sol and Claude Fable both failed to return within the time budget, so it did not get the usual multi-model cross-check. Every finding above was reproduced locally by rendering the chart.}

[lhotari]

Optional, non-blocking.

<component>.emptyDirVolumes: [] plus readOnlyRootFilesystem: true produces a read-only container with no writable paths and no warning — structurally the same trap as the pulsar-manager case, just chosen by the user rather than inherited. It's the documented escape hatch so this is defensible as-is, but a sentence here would close the loop:

Setting this to [] for a component opts it out entirely; if that component also has readOnlyRootFilesystem: true, you must supply the writable paths yourself via <component>.extraVolumes / <component>.extraVolumeMounts.

The rest of this block reads well — the REPLACES-not-extends note in particular is the thing people would otherwise get wrong.

[mouchar]

Added, folded into the sentence that sets the trap rather than as a separate paragraph:

## `<component>.emptyDirVolumes` REPLACES this list rather than extending it; use `[]`
## for "this component needs none" -- but a component with `readOnlyRootFilesystem: true`
## and an empty list has no writable path at all, so supply them yourself through
## `<component>.extraVolumes` / `<component>.extraVolumeMounts`. A path the component
## already mounts through `extraVolumeMounts` is skipped, since duplicate mountPaths are
## rejected.

Amended into the readOnlyRootFilesystem change; comment-only, so the rendered output is unchanged.

I stopped short of failing the render on [] + readOnlyRootFilesystem: true, since the helper can't tell that mistake from someone who legitimately supplied the paths via extraVolumeMounts -- it only drops list entries whose path is already mounted, and never sees mounts beyond them.