15.4 — Nightly scale run at a raised envelope

[vrabbi]

Phase: Phase 15 — Performance and scale
Proposal: docs/proposals/next-phases.md §15
Priority: P2 · Size: M

---

Problem

make test-e2e-scale (hack/e2e-scale.sh) generates a fleet of CRDs with 3 versions each and drives parallel Get/List through the apiserver conversion path, up to TARGETS=100 INSTANCES=100. It is explicitly not in the CI matrix.

A scale target nobody runs is a scale target nobody trusts. The first time it is run in anger will be the first time anyone discovers it has bit-rotted.

Goal

The scale envelope is exercised regularly, at a number higher than the current documented ceiling, with results anyone can look at.

Design

• A scheduled workflow (.github/workflows/scale.yml), nightly or weekly, running test-e2e-scale at a raised envelope — target 1000 CRDs, since that is roughly the CRD count of a mature Crossplane cluster and therefore the number people will ask about.
• Publish the results as a workflow artifact and a job summary: latency percentiles, throughput, peak memory, initial-sync time. A trend nobody can see is not a trend.
• Fail on regression, not on absolute numbers. Absolute timings vary too much between runners to gate on. Compare against the previous run's artifact and fail on a large relative regression; that is the signal worth having.
• Runner capacity. 1000 CRDs on a standard GitHub runner may not fit. Determine the achievable ceiling first and set the envelope to the highest number that runs reliably, rather than configuring an aspirational number that always fails. Document the ceiling and why.
• Pair with the soak test from 14.6 — Webhook-server rollout safety, and a soak test that proves it #151 — same schedule, same results artifact.

Acceptance criteria

• Scheduled workflow running the scale target at a documented, reliable envelope
• Results published as an artifact and rendered in the job summary
• Regression detection against the previous run, with a tunable threshold
• The achievable runner ceiling documented, with the reasoning
• docs/operations/capacity.md cites the scheduled run as the source of its envelope
• A failure is actionable: the summary names which measurement regressed

---

Generated by Claude Code