Add nodeAffinity support to all scheduled workloads

[smbecker]

Fixes #247.

Motivation

Users running on clusters with multiple node pools (e.g. GKE) need to pin Pulsar
components to specific nodes. Today the chart exposes nodeSelector and pod
anti-affinity, but no nodeAffinity, forcing workarounds like cordoning nodes.

This revives the work from #349 (by @MonicaMagoniCom), whose source branch was
deleted before it could be merged, and extends it to every other scheduled
workload in the chart.

Modifications

Adds an optional affinity.nodeAffinity to every pod-producing template. The
value is passed through to the pod spec verbatim via toYaml | nindent, so it
is native Kubernetes node affinity syntax — the same convention the chart's
adjacent scheduling knobs (nodeSelector, tolerations,
topologySpreadConstraints) already follow. Unset by default, so no node
affinity is applied and existing deployments are unaffected. It renders as a
sibling of the chart's own podAntiAffinity rather than replacing it.

broker:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: cloud.google.com/gke-nodepool
                operator: In
                values:
                  - pulsar-pool

The value paths, one per component:

Value path	Templates
zookeeper.affinity.nodeAffinity	zookeeper-statefulset.yaml, zookeeper-statefulset-upgrade.yaml
bookkeeper.affinity.nodeAffinity	bookkeeper-statefulset.yaml
broker.affinity.nodeAffinity	broker-statefulset.yaml, broker-statefulset-upgrade.yaml
proxy.affinity.nodeAffinity	proxy-statefulset.yaml
autorecovery.affinity.nodeAffinity	autorecovery-statefulset.yaml
function_worker.affinity.nodeAffinity	function-worker-statefulset.yaml
toolset.affinity.nodeAffinity	toolset-statefulset.yaml
standalone.affinity.nodeAffinity	standalone-deployment.yaml
oxia.server.affinity.nodeAffinity	oxia-server-statefulset.yaml
oxia.coordinator.affinity.nodeAffinity	oxia-coordinator-deployment.yaml
pulsar_manager.affinity.nodeAffinity	pulsar-manager-statefulset.yaml
dekaf.deployment.affinity.nodeAffinity	dekaf-deployment.yaml
pulsar_metadata.affinity.nodeAffinity	bookkeeper-cluster-initialize.yaml, pulsar-cluster-initialize.yaml, pulsar-manager-cluster-initialize.yaml
auth.authentication.jwt.generateSecrets.affinity.nodeAffinity	jwt-secret-init.yaml

Alongside the node affinity, the broker and zookeeper sts-cleanup hook Jobs
pick up the scheduling knobs their component StatefulSets already honour —
imagePullSecrets, nodeSelector, priorityClassName, tolerations,
topologySpreadConstraints, and container imagePullPolicy — so a cluster that
reserves nodes for Pulsar does not leave these hooks unschedulable. They
deliberately get no pod anti-affinity: they are short-lived single-shot
kubectl pods, spreading them across nodes buys nothing, and inheriting
zookeeper's default required rule would make the pre-upgrade hook
unschedulable on a 3-node cluster and hang helm upgrade.

.ci/templates-all-values.yaml and its patch1 overlay set nodeAffinity on
every component so the CI sweep exercises the passthrough, and the README gains
a node-pinning example next to the existing anti-affinity note.

Verifying this change

• helm template test charts/pulsar with default values is byte-identical to
  master apart from the two intended cleanup-Job additions and the chart's
  randomly generated secrets — in particular there are zero scheduling-related
  differences, so default behavior is genuinely unchanged.
• helm lint charts/pulsar clean.
• kubeconform -strict passes on k8s 1.25 / 1.31 / 1.36 across default values
  and all 19 .ci/clusters/*.yaml configs, both with and without nodeAffinity
  set on every component — 120 combinations — plus both
  templates-all-values.yaml renders at 1.36.
• Every one of the 18 pod-producing templates was confirmed to render its own
  component's nodeAffinity, with the expected shape and no copy-paste
  crossover, and to keep nodeAffinity/podAntiAffinity as siblings under a
  single affinity: mapping.
• The jwt-secret-init guard was checked across all four combinations of
  nodeAffinity set/unset × anti_affinity true/false, including the case where
  neither renders and no affinity: key is emitted at all.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency)
• The public API: new affinity.nodeAffinity value per component; additive
  and unset by default
• The schema
• The default values of configurations
• The wire protocol
• The rest endpoints
• The admin cli options
• Anything that affects deployment

Documentation

• doc-not-needed — documented inline in values.yaml and in the README

[lhotari]

Thanks for reviving #349 — the nodeAffinity rendering itself works for the documented case. Two things need to change before this can land, and one design question is worth settling now rather than after release.

The blocker

helm template test charts/pulsar on this branch vs master with default values differs in exactly two places, and both are the sts-cleanup Jobs. The zookeeper one gains a required pod anti-affinity, which makes the pre-upgrade hook unschedulable on a 3-node cluster. Details inline on zookeeper-statefulset-upgrade.yaml.

That directly contradicts "When unset (the default), no node affinity is applied, so existing deployments are unaffected." Commit 6f1ffb3 says the same thing — "All blocks are guarded, so default behavior is unchanged" — and the blocks are guarded, but the podAntiAffinity guard is and .Values.affinity.anti_affinity .Values.<component>.affinity.anti_affinity, and both default to true. So it guards nothing by default.

Scope

The title says "all scheduled workloads", but 6 of 18 pod-producing templates have no support: bookkeeper-cluster-initialize.yaml, dekaf-deployment.yaml, oxia-coordinator-deployment.yaml, pulsar-cluster-initialize.yaml, pulsar-manager-cluster-initialize.yaml, pulsar-manager-statefulset.yaml. Even setting the init Jobs aside, Dekaf / Oxia coordinator / Pulsar Manager are component workloads people will expect to pin.

Separately: both cleanup Jobs also gained imagePullSecrets, nodeSelector, priorityClassName, tolerations, topologySpreadConstraints and container imagePullPolicy. That's described in 6f1ffb3 but not in the PR body — worth calling out there, and it might be cleaner as its own PR.

Checked and clean

All 12 changed templates read their own component's value path — no copy-paste errors. Node and pod affinity correctly render as siblings under a single affinity: mapping everywhere. And Exists/DoesNotExist with no values renders values: null, which is a nil slice and passes both kubeconform and API validation — that one's fine as-is.

_{Reviewed with Codex gpt-5.6-sol and Claude Opus 5; every finding reproduced locally by rendering the chart and validating with kubeconform -strict.}

[smbecker]

All three findings reproduced locally before changing anything. Pushed four commits:

Blocker — 751f834. Reproduced exactly: the default render differed from master in only the two cleanup Jobs, and the and .Values.affinity.anti_affinity .Values.<component>.affinity.anti_affinity guard is true on both sides by default so it guarded nothing. My 6f1ffb3 commit message was wrong. Took your suggested fix: the hooks get node affinity and no pod anti-affinity, with the reasoning recorded in the template. The default render now has zero scheduling-related differences against master, and neither Job renders podAntiAffinity even with affinity.anti_affinity=true.

Design question — bab4985. Went with the native passthrough. The custom schema couldn't express weighted preferred, multiple terms, multiple expressions per term, or matchFields; it broke the chart's own toYaml convention; and it made the quoting bug unfixable-in-principle rather than just unfixed. Nothing shipped with the old shape, so there's no compatibility cost. This also resolves the | quote finding by construction — users write real YAML, and Gt/Lt become usable.

Scope — ab71ff3. All six templates now covered, so the title is accurate. The three init Jobs share pulsar_metadata.affinity.nodeAffinity, matching how they already share its nodeSelector and tolerations; Dekaf, Oxia coordinator and Pulsar Manager get new affinity blocks carrying nodeAffinity only, since they have no pod anti-affinity to configure.

Undocumented extras — the cleanup Jobs' imagePullSecrets / nodeSelector / priorityClassName / tolerations / topologySpreadConstraints / imagePullPolicy are now described in the PR body. I kept them here rather than splitting them out: with the anti-affinity gone they're the same one-idea change as the node affinity — making the upgrade hooks land where the operator has said Pulsar may run — and splitting would leave this PR adding node affinity to hooks that still ignore nodeSelector. Happy to pull them into a follow-up if you'd rather review them separately.

One thing I found while fixing the scope item, worth flagging as pre-existing and out of scope here: pulsar-cluster-initialize.yaml renders nodeSelector twice in the same pod spec (lines 47 and ~185, both under the same if .Values.pulsar_metadata.nodeSelector guard). It's inert today because both emit identical content, but it's a duplicate YAML key and it's how I caught my own first attempt at that template. Want me to fix it here or open a separate issue?

Verification — helm lint clean; kubeconform -strict on k8s 1.25/1.31/1.36 across default values and all 19 .ci/clusters configs, with and without node affinity on every component (120 combinations), plus both templates-all-values renders at 1.36. .ci/templates-all-values.yaml and its patch1 overlay now set nodeAffinity on every component, using shapes the old schema could not express, so CI exercises the passthrough rather than my say-so. Each of the 18 pod templates was confirmed to read its own component's value path and to keep nodeAffinity/podAntiAffinity as siblings; the jwt-secret-init guard was checked across all four set/unset × true/false combinations, including the one where no affinity: key is emitted at all.

[lhotari]

All five addressed. LGTM — thanks for the thorough job on this, particularly for reproducing the blocker before changing anything.

Everything below is from rendering, not reading.

The blocker — fixed

Default render vs master is now 4 added lines, 0 removed, and they're two blank lines plus two imagePullPolicy: "IfNotPresent". Zero scheduling differences. With affinity.anti_affinity=true forced on, neither cleanup Job renders podAntiAffinity — in fact neither renders an affinity: key at all. In the all-values config where node affinity is configured, both hooks get nodeAffinity and no podAntiAffinity, which is exactly the shape we wanted.

Native passthrough — the right call

This resolves the quoting bug by construction, as you say. Verified against a config using shapes the old schema couldn't express:

values: ["true", "123"]     ->  - "true"   - "123"      # quoted, stay strings
operator: Gt, values: ["2"] ->  - "2"                   # Gt/Lt now usable
matchFields                 ->  renders
preferred… + weight: 80     ->  renders
two nodeSelectorTerms       ->  renders (OR)

kubeconform -kubernetes-version 1.31.0 -strict passes on that config, and on the full all-values render (64 valid, 0 invalid). nodeAffinity and podAntiAffinity stay siblings under one affinity: key, and I found no duplicate affinity: keys in any rendered workload.

Scope, and the rest

• All 18 pod-producing templates carry node-affinity handling and each reads its own component path. The three init Jobs sharing pulsar_metadata.affinity.nodeAffinity is right — it matches how they already share nodeSelector and tolerations. dekaf's safe-navigation form renders correctly too.
• jwt-secret-init guard: exactly one affinity: key when node affinity is set, zero when unset, across both anti-affinity values, no errors.
• Undocumented extras: keeping them is fine now that the anti-affinity is gone. Your reasoning holds — with the anti-affinity removed these are one coherent idea (make the hooks land where the operator has said Pulsar may run), and splitting would leave this PR adding node affinity to hooks that still ignore nodeSelector. No need to pull them out.
• CI values genuinely exercise the passthrough rather than your say-so: matchFields, preferredDuringSchedulingIgnoredDuringExecution and weighted entries all appear, none of which the old schema could express. Good addition.

Your question about the duplicate nodeSelector

Confirmed pre-existing: it's on master at pulsar-cluster-initialize.yaml:47 and :185 (:190 here), and the rendered Job really does carry two nodeSelector: keys in one pod spec. Inert today only because both emit identical content and the last key wins.

Please open a separate PR for it. This one is ready to merge and an unrelated fix would restart review on it. That's not inconsistent with my asking you to keep the bookkeeper initContainers fix inside #715 — there the fix was a side effect of a guard that had to be touched anyway, so it was inseparable. Here it's an independent three-line deletion in a file you happened to touch for another reason.

Good catch either way — worth fixing before someone runs the chart through a strict YAML parser that rejects duplicate keys outright.

One non-blocking follow-up

.ci/templates-all-values.yaml configures standalone.affinity.nodeAffinity but sets standalone.enabled: false just below it, so the standalone Deployment never renders in either all-values run — its node affinity is the one path of the eighteen that CI doesn't actually exercise. .ci/clusters/values-standalone.yaml enables standalone but sets no node affinity, so it isn't covered there either.

Not worth holding the merge for, and standalone genuinely is mutually exclusive with the distributed setup so it can't just be flipped on in that file. Adding affinity.nodeAffinity to values-standalone.yaml would close it whenever you're next in there.

_{Reviewed with Codex gpt-5.6-sol and Claude Opus 5; Claude Fable was in the reviewer set but did not return. Every finding reproduced locally by rendering the chart and validating with kubeconform -strict.}

[lhotari]

Non-blocking, for whenever you're next in here.

This configures standalone.affinity.nodeAffinity, but enabled: false on the next line means the standalone Deployment never renders — so neither all-values run exercises this path. Of the eighteen templates that now carry node-affinity handling, standalone is the only one CI doesn't actually cover.

.ci/clusters/values-standalone.yaml does enable standalone, but sets no node affinity, so it isn't covered there either.

I wouldn't enable standalone here — your comment above is right that it's mutually exclusive with the distributed setup. Adding an affinity.nodeAffinity block to .ci/clusters/values-standalone.yaml would be the natural place to close the gap.

Concretely, the risk this leaves open is narrow: a malformed value path, an indentation error, or a duplicate affinity: key in standalone-deployment.yaml would pass both advertised all-values validations because the Deployment simply isn't in the output.

[smbecker]

Is merging this PR waiting on me to resolve this comment?