Skip to content

Recalibrate warm P95 budgets after schema-v2 baselines #511

Description

Problem

PR #510 changes the existing full_refresh and time_to_first_env distributions from one cold + nine warm samples to ten independently paired warm samples. The current P95 budgets were calibrated against the old mixed distribution and will become too permissive once schema-v2 baselines land.

Three schema-v2 hosted runs report:

Platform Warm refresh P95 Warm time-to-first P95 Current absolute budgets
Linux 60 / 58 / 69ms 15 / 16 / 15ms 1,000ms / 250ms
Windows 171 / 143 / 206ms 38 / 28 / 47ms 5,000ms / 500ms
macOS 198 / 270 / 171ms 166 / 228 / 136ms 1,000ms / 750ms

After #510, those old limits could allow multi-second warm-cache drift even though cold performance is gated separately.

Scope

After #510 merges and exact schema-v2 main baselines exist:

  • inspect repeated warm refresh/time-to-first baseline variance on all platforms;
  • tighten warm P95 dual budgets with measured hosted-runner headroom;
  • keep P50, server-startup, and cold-refresh budgets unchanged unless independent evidence requires adjustment;
  • update comparator tests and docs/QUALITY_SNAPSHOTS.md;
  • retain the schema-v1 migration behavior only where still required.

Acceptance criteria

  • Repeated unchanged-head schema-v2 runs stay green.
  • Synthetic multi-second warm refresh and warm time-to-first regressions fail on every platform.
  • Cold P50 gating remains independent and unchanged.
  • Missing metrics, inventory mismatches, and malformed snapshots still fail closed.
  • Full CI, coverage, performance, and CodeQL workflows pass.

Depends on #510.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestimportantIssue identified as high-priority

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions