Skip to content

Add body-pressure CI coverage#7

Open
upgle wants to merge 4 commits into
mainfrom
codex/body-pressure-ci
Open

Add body-pressure CI coverage#7
upgle wants to merge 4 commits into
mainfrom
codex/body-pressure-ci

Conversation

@upgle

@upgle upgle commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Summary

  • add deterministic HTTP/2 coverage for concurrent body-budget exhaustion, peer-stream isolation, and full accounting release
  • exercise actual response bytes at the 1 MiB limit and chunked overflow beyond it
  • enable request and response inspection in the qualification profile, add p99.9 and sampled peak-RSS evidence, and parameterize large-body/soak workloads
  • run the full body-pressure stress profile in the existing QA job for every pull request
  • retry completed qualification performance-threshold failures once on a fresh Envoy process, preserve both attempts, and continue body-pressure stress after repeated performance failures
  • render CRS counts and both benchmark reports in the Actions job summary, publish browser-readable Markdown previews, and annotate threshold violations while retaining complete artifacts
  • document which stress expectations are enforced now and which performance thresholds remain diagnostic pending a reviewed Linux baseline

Why

The existing CI covers boundary conditions, short body waves, and atomic budget contention, but it did not hold multiple real HTTP streams against the aggregate body budget or transfer large inspected responses. It also lacked a pull-request stress profile for concurrent 1 MiB request/response traffic and repeated 256 KiB body waves.

The generated benchmark and CRS reports were available only as archived workflow artifacts, so reviewers had to download and unpack them before inspecting the results. A single GitHub-hosted runner tail-latency excursion could also fail the qualification gate even when Envoy CPU per request was unchanged.

Impact

Pull requests gain deterministic aggregate-budget and actual-response-overflow coverage through the existing QA and sanitizer targets. The full stress profile reuses QA's already-built custom Envoy binary, adding the workload without another build job.

Reviewers can inspect CRS result counts and benchmark latency, throughput, CPU, RSS, and terminal-gauge evidence directly in the Actions run summary. A completed qualification measurement that only exceeds performance thresholds is retried once; both attempts must exceed before QA fails, and both reports are retained. Runtime errors, request failures, and incomplete reports fail immediately without a retry. Body-pressure stress still runs after repeated qualification performance failures so the PR retains its stress evidence.

The first qualification threshold failure is emitted as a warning annotation and a repeated final failure as an error. Standalone Markdown reports open in the browser, and complete JSON, Markdown, and diagnostic evidence remains available as 30-day workflow artifacts.

Unexpected responses, timeouts, and non-zero terminal gauges fail QA. Body-pressure performance and RSS thresholds remain diagnostic until stable Linux evidence is reviewed.

Validation

  • bazel test --macos_minimum_os=10.15 //test/integration:filter_protocol_integration_test //test/integration:envoy_http_integration_test
  • python3 -m unittest test.tools.ci_qualification_retry_test test.tools.ci_report_summary_test — 12 tests passed
  • bazel test --macos_minimum_os=10.15 //test/tools:ci_qualification_retry_test //test/tools:ci_report_summary_test
  • full default qualification profile with existing enforced thresholds: PASS
  • reduced 1 MiB request/response benchmark smoke: completed with zero terminal gauges
  • full macOS body-pressure profile: functional expectations and terminal gauges passed; diagnostic results observed 20.27 s large-request p99 and 665.52 MiB sampled peak RSS growth
  • shell syntax, workflow YAML parsing, Make dry-run, and git diff --check

@upgle
upgle force-pushed the codex/body-pressure-ci branch from a3e3307 to 7ec21a6 Compare July 19, 2026 10:15
@upgle
upgle marked this pull request as ready for review July 19, 2026 11:53
@upgle
upgle requested a review from Copilot July 19, 2026 11:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR expands the CI/QA pipeline to add deterministic “body-pressure” coverage (large request/response inspection at 1 MiB and concurrent budget exhaustion), improves benchmark/report visibility in GitHub Actions summaries, and adds a single-retry mechanism for qualification runs that only fail performance thresholds.

Changes:

  • Add a CI wrapper to retry qualification performance-threshold failures once while preserving attempt artifacts.
  • Extend the qualification benchmark to support response inspection, large-body workloads, p99.9 reporting, and sampled peak RSS evidence.
  • Render CRS/benchmark summaries and publish Markdown previews + annotations in the QA workflow, and run the body-pressure stress profile in the existing QA job.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tools/run-ci-qualification.sh Adds one-retry orchestration for performance-threshold-only qualification failures and manages attempt artifact layout.
tools/qualification-benchmark.py Adds body-pressure workload parameters, response inspection config, p99.9 reporting, and peak RSS sampling/recording.
tools/ci-report-summary.py Generates GitHub Actions QA summary content, markdown previews, and threshold annotations from produced artifacts.
tools/ci-envoy-build.sh Runs qualification via the retry wrapper and adds body-pressure stress execution to the QA mode.
tools/BUILD Exports new CI helper scripts for Bazel data dependencies.
test/tools/ci_report_summary_test.py Adds unit tests for summary rendering, preview generation, and annotation formatting.
test/tools/ci_qualification_retry_test.py Adds unit tests validating retry/publish behavior for qualification runs.
test/tools/BUILD Registers the new tool-focused py_tests and their script data dependencies.
test/README.md Documents the new body-pressure target and updated integration coverage scope.
test/performance/qualification_bootstrap.yaml.tmpl Enables response max-bytes limits and increases route timeout for the benchmark profile.
test/integration/filter_protocol_integration_test.cc Adds integration coverage for chunked response overflow and HTTP/2 aggregate-budget exhaustion behavior.
test/integration/envoy_http_integration_test.py Adds upstream endpoints + tests for exact-limit responses and chunked overflow above the inspection limit.
Makefile Adds body-pressure-stress target and includes new tool tests in test/check.
docs/development.md Updates developer docs to describe body-pressure CI evidence, summaries, retry behavior, and new benchmark outputs.
.github/workflows/ci.yml Adds QA summary generation, preview artifact uploads, annotations, and body-pressure artifact publishing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +559 to +562
if rss_sampler_stop is not None:
rss_sampler_stop.set()
if rss_sampler_thread is not None:
rss_sampler_thread.join(timeout=1)
Comment on lines +148 to +152
body = (
b"r" * ARGS.large_body_bytes
if self.path == "/large-response"
else b"ok"
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants