Phase: Phase 15 — Performance and scale
Proposal: docs/proposals/next-phases.md §15
Priority: P2 · Size: M
Problem
cmd/webhook-server/main.go:161 runs reconciler.InitialSync(ctx) and only then calls server.SetReady(true). That ordering is correct — a replica must not accept conversion traffic before its registry is populated — but it means every replica compiles every assigned plan before it can serve.
Phase 9 measured compile at ~3 ms for a 1000-leaf schema. With hundreds of targets that is seconds, which is fine; the problem is that nobody knows where the cliff is, because it has never been measured, and a slow cold start shows up as a rolling update that stalls or a startupProbe that kills a pod mid-sync.
Goal
Cold start is measured, bounded, and visible.
Design
- Measure it. A
dco_webhook_initial_sync_duration_seconds gauge (or a histogram observed once), plus a log line stating target count and elapsed time. This alone answers most operational questions.
- Parallelise
InitialSync. Compilation is CPU-bound and independent per target; a bounded worker pool is a small change with a large effect at high target counts.
- Benchmark the curve. Extend the existing benchmark suite with initial-sync time vs target count (10 / 100 / 1000) and publish it in
docs/operations/capacity.md next to the compile numbers.
startupProbe. With a known budget, add one to the webhook-server so a slow sync is not mistaken for a hung pod and killed by the liveness probe. Failure threshold derived from the published budget; configurable via chart values.- Consider a
--registry-ready-timeout after which the replica reports ready anyway with a loud warning and a metric, rather than never becoming ready — an unavailable replica is sometimes worse than a partially-loaded one. Decide deliberately; the current fail-closed behaviour is defensible and should not be changed casually.
Acceptance criteria
Generated by Claude Code
Phase: Phase 15 — Performance and scale
Proposal:
docs/proposals/next-phases.md§15Priority: P2 · Size: M
Problem
cmd/webhook-server/main.go:161runsreconciler.InitialSync(ctx)and only then callsserver.SetReady(true). That ordering is correct — a replica must not accept conversion traffic before its registry is populated — but it means every replica compiles every assigned plan before it can serve.Phase 9 measured compile at ~3 ms for a 1000-leaf schema. With hundreds of targets that is seconds, which is fine; the problem is that nobody knows where the cliff is, because it has never been measured, and a slow cold start shows up as a rolling update that stalls or a
startupProbethat kills a pod mid-sync.Goal
Cold start is measured, bounded, and visible.
Design
dco_webhook_initial_sync_duration_secondsgauge (or a histogram observed once), plus a log line stating target count and elapsed time. This alone answers most operational questions.InitialSync. Compilation is CPU-bound and independent per target; a bounded worker pool is a small change with a large effect at high target counts.docs/operations/capacity.mdnext to the compile numbers.startupProbe. With a known budget, add one to the webhook-server so a slow sync is not mistaken for a hung pod and killed by the liveness probe. Failure threshold derived from the published budget; configurable via chart values.--registry-ready-timeoutafter which the replica reports ready anyway with a loud warning and a metric, rather than never becoming ready — an unavailable replica is sometimes worse than a partially-loaded one. Decide deliberately; the current fail-closed behaviour is defensible and should not be changed casually.Acceptance criteria
InitialSyncparallelised with a bounded pool; results identical to serialstartupProbeon the webhook-server, chart-configurable, defaults derived from the measured budgetdocs/operations/ha-checklist.mdmentions the cold-start budgetGenerated by Claude Code