15.1 — Automatic sharding of targets across ConversionWebhookServers

[vrabbi]

Phase: Phase 15 — Performance and scale
Proposal: docs/proposals/next-phases.md §15
Priority: P2 · Size: XL

---

Problem

internal/assign/assign.go resolves which ConversionWebhookServer serves a config: an explicit reference, or the instance marked default: true. Scaling out today therefore means an operator manually creating instances and manually assigning configs to them — which is fine for tenant isolation (a deliberate choice) but is not a scaling story.

Every replica of a given server compiles and holds every config assigned to that server, so the per-replica cost grows with the number of targets, and the only lever is manual.

Goal

N targets spread across M servers without an operator hand-assigning each one.

Non-goals

• Cross-cluster anything.
• Removing explicit assignment. Deliberate pinning must keep working exactly as it does — automatic balancing is for the configs that do not express a preference.

Design

Add a balancing policy to ConversionWebhookServer, e.g. spec.sharding: {enabled, replicas|weight}, and a resolver that distributes unpinned configs across the servers that opt in — by consistent hashing on the target name, so the assignment is stable as configs come and go and does not reshuffle the world when one server is added.

The hard part is the move, not the assignment. Reassigning a target from server A to server B means the XRD's spec.conversion must point at B only once B is actually serving the plan — otherwise there is a window with no working webhook. The safe sequence is: B compiles and reports ready for the target → patch the XRD to B → A drops it. That needs per-target readiness from the server, which the project currently and deliberately does not have (docs/limitations.md, "Per-pod webhook-server state isn't surfaced back into status").

So this issue probably has a prerequisite: a minimal, bounded form of per-target readiness — enough to answer "is server B serving target X on every replica?" without making the reconcile loop depend on polling pods. Investigate whether an aggregated status.servedTargets written by the servers themselves (each replica updating a lease or a per-replica status) is acceptable, and write up the design before building.

Acceptance criteria

• Design note in this issue covering the move sequence and the readiness question, agreed before implementation
• Consistent-hash assignment; adding a server moves a bounded fraction of targets
• Explicit spec.conversionWebhookServerRef always wins
• No target is ever unserved during a move (asserted by an e2e that reassigns under load)
• Rebalancing is rate-limited, reusing internal/enqueue's paced fan-out
• docs/architecture.md and docs/configuration/conversionwebhookserver.md updated

Notes

Largest item in the phase and the one most likely to need splitting. Start with the design note.

---

Generated by Claude Code

[vrabbi]

Design note: the move sequence and the readiness question

Written before implementation, as this issue asks. It is the design the PR implements; if any of it is wrong, this is the thing to argue with rather than the diff.

1. Assignment: rendezvous, not a ring

spec.sharding: {enabled, weight} per instance. The pool is every instance that opts in. A config's assignment resolves in strict precedence:

1. spec.webhookServerRef — deliberate pinning wins over everything. Sharding can never move a pinned config; tenant isolation depends on that and it is a non-goal to change it.
2. The pool, if non-empty, by weighted rendezvous (highest-random-weight) hashing on the target resource's name.
3. spec.default, exactly as today, when the pool is empty.

Rendezvous rather than a hash ring because it gives optimal disruption with no tuning: adding an instance moves only the keys that instance wins — in expectation 1/(N+1) — and moves nothing between the instances that were already there. A ring approximates that, with a quality that depends on a virtual-node count somebody has to pick correctly. Rendezvous is also a pure function of (key, pool), which is what lets the operator and every webhook-server replica compute the same answer independently — the property the whole internal/assign package exists to preserve. The cost is O(N) per lookup where N is the instance count, i.e. single digits.

Hashing the target name rather than the config name: the target is the registry key and the /convert/{name} path, so renaming a config does not move the resource it converts.

The invariant that makes the precedence safe. While a pool exists it, not spec.default, answers for unpinned configs. Admission therefore rejects a state where the instance marked default is not a pool member — otherwise enabling sharding on one non-default instance would move every unpinned config onto it in a single write, a fleet-wide reassignment produced by what reads as a local change. There is always a valid ordering: enable it on the default instance first (pool of one, nothing moves), then add the others.

2. The readiness question

The issue is right that the move, not the assignment, is the hard part — and right that this needs per-target readiness, which the project deliberately does not have.

What the operator needs to know: "is instance B already serving target X, on every one of its replicas?" Not "is B healthy" — a healthy B that has not compiled X yet answers a ConversionReview for X with a failure, and the apiserver turns that into a failed read or write.

It cannot be a query. docs/limitations.md commits to the operator's reconcile loop making no network calls to webhook-server pods, and that constraint is worth keeping: it is why a wedged pod cannot wedge the operator.

So it is published. Each replica writes the set of targets it holds a compiled, servable plan for (a registry entry with a Router, not an error-only placeholder) into a coordination.k8s.io/v1 Lease of its own, named dco-served-<podName>, labelled with the instance name, in the instance's namespace. The set is sorted, gzipped and base64'd into an annotation; renewTime is refreshed on a 30 s heartbeat and the set is republished immediately on any registry change.

A Lease rather than a ConfigMap or the Pod's own annotations:

• it is owned by the Pod, so it is garbage-collected with it and needs no reaper;
• renewTime is a first-class staleness signal, which is the backstop ownership cannot provide — a pod that is alive but wedged;
• it is a small dedicated object, so a 30 s heartbeat is not rewriting something other controllers watch.

The aggregate is an intersection, not a union. ConversionWebhookServer.status.servedTargets is what every live replica reports, because a target two replicas out of three can serve is a target that fails one request in three. status.reportingReplicas says how many Leases fed it; below readyReplicas the intersection is not yet a statement about the instance.

Costs, stated plainly:

• The operator gains a Lease informer. It must be label-scoped or it holds one Lease per node from kube-node-lease plus every leader election in the cluster. It is scoped to app.kubernetes.io/managed-by, alongside the other owned types.
• The CWS controller gains a Lease watch. A heartbeat every 30 s per replica would otherwise re-SSA a Deployment, Service, HPA and PDB that often, so the watch carries a predicate that only passes an event when the reported target set changes.
• The webhook-server ServiceAccount, documented today as read-only, gains one write. It is a namespaced Role in the instance's namespace with get, create, update, patch on leases — no list, no watch, nothing cluster-wide. Leases are how leader election is implemented across the ecosystem; cluster-wide write on them is not a grant to hand out for a bookkeeping annotation.
• An encoding cap. Past ~50k targets on one replica the annotation would not fit; the replica then marks itself truncated, which readers must treat as "I cannot tell you", not as a partial answer.

3. The move sequence

assignment changes  ──▶  destination replicas compile the plan
                              │  (from the same watch events the operator sees)
                              ▼
                         each publishes its servable set into its own Lease
                              │
                              ▼
   operator waits until EVERY live replica of the destination reports the target
                              │
                              ▼
                    operator patches spec.conversion → destination
                              │
                              ▼
         source replicas see the target stop naming them, and drop the plan

The part that makes waiting safe is on the source side, and it is a behaviour change. Today a replica drops a target the instant the resolver stops assigning it — so during the wait the target's spec.conversion still names the source and the source has already stopped serving it. That is an outage caused by waiting.

So: a replica holds a compiled plan while either the resolver assigns the target to its instance or the live target's spec.conversion still names its Service. The second clause closes the window from the losing end, and it is a fix in its own right — editing webhookServerRef by hand has always had exactly this window.

Match on the Service name only, not namespace: instance names are cluster-unique and the Service name is derived from them, so the name is unambiguous and a replica does not otherwise need to know its own Service's namespace.

Gate only on a move. A first apply has no previous server still covering the target; gating it would delay every new config for nothing.

When the destination publishes nothing at all — a fleet mid-upgrade, or an instance in a namespace whose Lease Role was never created — the move proceeds, with HandoverReady=True and reason HandoverUnverified. Blocking forever would be a regression against every previous release; proceeding restores exactly the old behaviour and says so rather than passing it off as a verified handover. A truncated or undecodable report is different: that is a replica saying "I cannot tell you", and it blocks.

The config carries a HandoverReady condition throughout, so a held move is visible rather than looking like a stall.

4. Rate limiting

Rebalancing reaches the configs through the existing Watches(&ConversionWebhookServer{}, enqueue.PacedMapFuncs(..., enqueue.CWSConfigEnqueueQPS)) on both config controllers — the paced fan-out already in place, at 50 QPS. It needed no change: adding a pool member enqueues exactly the configs that member now wins (the create event's new view), and removing one enqueues exactly what it used to hold (the delete event's old view). A rebalance is the largest burst that watch ever produces, so it is the case the pacing most exists for, and there is now a test that says so.

5. What proves it

hack/e2e-reassign.sh, in the PR e2e matrix: two webhook-server instances, sustained reads and writes at a non-storage version, and three reassignments — an explicit webhookServerRef pin, an unpin, and a sharding-driven move — asserting zero failed requests and zero wrong values, plus that each move was verified (HandoverReady reason HandoverReady, not the unverified fallback) so the test cannot pass with the whole mechanism removed. It also asserts admission rejects a pool the default instance is outside of.

6. Not in scope here

• Balancing by cost. Rendezvous distributes targets evenly by count; it knows nothing about schema size or traffic. Pin the outliers with webhookServerRef, or bias with spec.sharding.weight. Recorded in docs/limitations.md.
• Automatic rebalancing on load. Nothing observes per-instance load and moves work in response. The pool changes when an operator changes it.