test: benchmark long-lived sessions and inventory scale (Fixes #533)

[Karthik Nadig (karthiknadig)]

Add deterministic session and scale workloads alongside the installed-manager performance benchmark, establishing evidence before concurrency and ownership changes.

• Separate first-process/empty-cache, new-process/after-refresh, and same-process warm refresh measurements with full fixture identity assertions.
• Prove real persisted-cache reuse separately: matching cold/disk-warm/same-process resolves with interpreter probe counts 1/0/0 and a nonempty owned cache.
• Exercise inventory churn and barrier-proven concurrent resolves while configure, refresh, and info remain responsive, retaining the originally configured cache.
• Record receipt-based latency, process-specific resource samples, subprocess counts, and honest cache-use measurements.
• Add fast three-platform PR checks, weekly/manual stress runs, and validated privacy-safe artifacts with explicit failure status.
• Validate request deadlines, graceful shutdown, and metrics integrity. Final follow-up Windows fast/stress and exact CI lint pass; native Linux cache control passed before the final cache-stability assertion. Hosted final-head Linux/macOS validation remains pending.

Fixes #533

[github-actions]

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	1ms	1ms	+0ms	+0.0%	>5ms and >100%	➖
Server startup P95	1ms	1ms	+0ms	+0.0%	>50ms and >200%	➖
Discovery duration P50	56ms	56ms	+0ms	+0.0%	>25ms and >30%	➖
Discovery duration P95	60ms	59ms	+1ms	+1.7%	>50ms and >50%	🔺
Startup-to-first environment P50	14ms	13ms	+1ms	+7.7%	>20ms and >100%	🔺
Startup-to-first environment P95	16ms	16ms	+0ms	+0.0%	>25ms and >100%	➖
Cold discovery duration P50	139ms	137ms	+2ms	+1.5%	>100ms and >50%	🔺
Refresh round-trip P50	57ms	56ms	+1ms	+1.8%	>25ms and >30%	🔺
Refresh round-trip P95	60ms	60ms	+0ms	+0.0%	>50ms and >50%	➖
Request-to-first environment P50	13ms	11ms	+2ms	+18.2%	>20ms and >100%	🔺
Request-to-first environment P95	15ms	15ms	+0ms	+0.0%	>25ms and >100%	➖
Cold refresh round-trip P50	139ms	137ms	+2ms	+1.5%	>100ms and >50%	🔺

Workload	PR	Baseline
Environments	5	5
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

[github-actions]

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	85.589%	85.589%	+0.000pp
Functions	88.400%	88.400%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	9ms	9ms	+0ms	+0.0%	>10ms and >50%	➖
Server startup P95	12ms	12ms	+0ms	+0.0%	>50ms and >100%	➖
Discovery duration P50	136ms	135ms	+1ms	+0.7%	>150ms and >50%	🔺
Discovery duration P95	146ms	138ms	+8ms	+5.8%	>250ms and >100%	🔺
Startup-to-first environment P50	20ms	20ms	+0ms	+0.0%	>25ms and >50%	➖
Startup-to-first environment P95	25ms	26ms	-1ms	-3.8%	>100ms and >100%	✅
Cold discovery duration P50	136ms	134ms	+2ms	+1.5%	>150ms and >50%	🔺
Refresh round-trip P50	136ms	135ms	+1ms	+0.7%	>150ms and >50%	🔺
Refresh round-trip P95	147ms	139ms	+8ms	+5.8%	>250ms and >100%	🔺
Request-to-first environment P50	10ms	10ms	+0ms	+0.0%	>25ms and >50%	➖
Request-to-first environment P95	15ms	16ms	-1ms	-6.2%	>100ms and >100%	✅
Cold refresh round-trip P50	137ms	134ms	+3ms	+2.2%	>150ms and >50%	🔺

Workload	PR	Baseline
Environments	8	8
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

[github-actions]

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	69ms	93ms	-24ms	-25.8%	>100ms and >50%	✅
Server startup P95	605ms	779ms	-174ms	-22.3%	>750ms and >100%	✅
Discovery duration P50	116ms	167ms	-51ms	-30.5%	>100ms and >50%	✅
Discovery duration P95	146ms	213ms	-67ms	-31.5%	>300ms and >100%	✅
Startup-to-first environment P50	101ms	136ms	-35ms	-25.7%	>150ms and >50%	✅
Startup-to-first environment P95	120ms	154ms	-34ms	-22.1%	>250ms and >100%	✅
Cold discovery duration P50	270ms	323ms	-53ms	-16.4%	>250ms and >50%	✅
Refresh round-trip P50	116ms	168ms	-52ms	-31.0%	>250ms and >50%	✅
Refresh round-trip P95	147ms	214ms	-67ms	-31.3%	>300ms and >100%	✅
Request-to-first environment P50	32ms	38ms	-6ms	-15.8%	>50ms and >50%	✅
Request-to-first environment P95	45ms	47ms	-2ms	-4.3%	>100ms and >100%	✅
Cold refresh round-trip P50	271ms	324ms	-53ms	-16.4%	>600ms and >50%	✅

Workload	PR	Baseline
Environments	10	10
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

[github-actions]

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	83.130%	83.130%	+0.000pp
Functions	85.791%	85.791%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unix fixture identities will fail version assertions, and the cache scenarios do not exercise the claimed disk-cache behavior.

Review effort: Balanced
Findings: 1 · 2

Open (3)

• Unix fixture cannot produce the expected virtualenv version · New
• Warm-cache test passes without producing or verifying cache data · New
• Cache directory change is ignored after initial configuration · New

What changed in this PR

Adds deterministic benchmarks for long-lived PET sessions, inventory scaling, concurrent resolves, resource usage, and cache behavior.

Changes:

• Adds cross-platform session and churn benchmarks with concurrent resolve barriers.
• Adds validated metrics extraction and privacy-safe artifacts.
• Adds fast PR and scheduled stress workflows.

File	Description
.github/​workflows/​session-benchmarks.yml	Runs fast and stress benchmarks across three platforms.
crates/​pet/​tests/​session_performance.rs	Implements session, scale, churn, and concurrency workloads.
crates/​pet/​tests/​jsonrpc_client.rs	Adds receipt timing and pending-request support.
crates/​pet/​tests/​jsonrpc_server_test.rs	Tests request timing and deadline behavior.
crates/​pet/​tests/​fixtures/​session_sitecustomize.py	Implements the resolve concurrency barrier.
scripts/​session_metrics.py	Validates and writes benchmark metrics.
scripts/​tests/​test_session_metrics.py	Tests metrics validation and failure handling.
docs/​SESSION_BENCHMARKS.md	Documents workloads, metrics, and CI usage.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Derived resource metrics can be internally inconsistent while still passing artifact validation.

Review effort: Balanced
Findings: None

Resolved since last review (3)

• Unix fixture cannot produce the expected virtualenv version
• Cache directory change is ignored after initial configuration
• Warm-cache test passes without producing or verifying cache data

Previously missed (1)

In code that hasn't changed since last review

Validate derived resource peak and RSS delta values

scripts/​session_metrics.py:342

These derived resource fields are only type-checked, so an internally false artifact is accepted as validated. For example, valid_metrics() reports equal pre/after RSS values but a delta of -10, and changing the peak below every sampled value also passes validation. Recompute/compare observedResourcePeak against all resource samples and require rssDeltaFromPreResolveBytes == resourceAfter.residentBytes - preResolveResources.residentBytes before preserving the artifact.

[Karthik Nadig (karthiknadig)]

Published signed follow-up 8faaabc after another independent clean review.

• Native Linux/macOS exposed an over-strict test assertion: global refresh legitimately adds ambient cache entries. The proof now verifies every original cache entry's relative path and exact bytes remain unchanged, while allowing additions. Regression tests reject same-size mutations and deletions.
• Fixed Linux-only Clippy's map(...).count() finding with error-propagating, allocation-free descriptor counting.
• Addressed the review-body finding: artifact validation now recomputes the exact observed resource peak from every sample category and verifies the signed RSS delta. Tests cover incorrect high/low peaks, every category, null/mixed availability, and positive/negative/zero deltas.

Local validation: 6 Rust session tests, 78 Python tests (1 Windows symlink skip), mandatory precommit checks and exact all-target/all-feature Clippy passed. Windows fast + final-schema extraction passed after one initial cold-resolve timeout; that first attempt is not being represented as a pass. No timeouts were loosened. Hosted final-head checks and fresh Copilot review are now running.

Importantly, the preceding native macOS run got through the real cold/disk-warm cache proof and held concurrent-resolve barrier; it failed only at the now-corrected aggregate-cache assertion. Full final-head native completion is still required before readiness.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Concurrent resolve workloads do not verify that each response belongs to and correctly identifies its submitted environment.

Review effort: Balanced
Findings: None

Previously missed (1)

In code that hasn't changed since last review

Validate barrier responses against their submitted environments

crates/​pet/​tests/​session_performance.rs:1054

These barrier-proven requests are treated as successful for any non-null JSON value. A concurrency bug that returns another request's environment (or a malformed object) would still pass this central workload. Deserialize each response and assert its error, executable, prefix, kind, and version against the corresponding submitted venv so the benchmark proves request/result ownership as well as overlap.

This issue also appears on line 1092 of the same file.

[Karthik Nadig (karthiknadig)]

Published signed de44782, integrating merged coverage main 8320c0f without dropping the subprocess proof. The new review-body finding is addressed across every real resolve: cache-control, warm-up, held overlap, and latency batches now validate typed responses against their own submitted fixture's executable, prefix, Venv kind, authoritative full runtime version, and absence of errors. Regressions reject swapped, malformed, missing-field, error-bearing, wrong-kind, and wrong-version responses. No identities are emitted in metrics.

Independent source review found no code issues. Integrated local validation passed: 20 server tests, 7 session tests, 101 Python tests (1 Windows symlink skip), mandatory precommit and exact all-target/all-feature Clippy. Windows fast plus strict artifact validation passed. The preceding head's native Linux/Windows/macOS workloads all passed; this stricter final head is now undergoing fresh hosted checks and Copilot review.

[Copilot]

Copilot review overview

🟡 Changes recommended

Ambient maximum metrics omit churn observations, and the validator accepts contradictory overlap and maximum counts.

Review effort: Balanced
Findings: 2

Open (2)

• Include replacement refresh counts in ambient maxima · New
• Reject overlap counts exceeding ambient maxima · New

[Karthik Nadig (karthiknadig)]

All three native stress jobs passed at de44782: 1/10/100/1000 environments, 10 warm samples per size, 10 simultaneously held resolves, and 50 cold latency probes. Every platform proved cache probe counts 1/0/0 and exact response ownership. Linux/Windows thread and descriptor counts did not grow after overlap; macOS unavailable counters remain null. Run: https://github.com/microsoft/python-environment-tools/actions/runs/36755908701.

The separate Linux coverage failure was fully attributed: only three test-helper lines in pet-jsonrpc/src/output.rs lost coverage when the writer had already completed before the first poll. Production source, line/function denominators, and all other file coverage totals are unchanged. This is being fixed separately on main in #564, not by relaxing the gate or rerunning until green. The ambient-count review fixes are now published in f8db49d, with fresh CI/review running. This PR stays draft while that independent coverage reliability fix is completed.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The extensive cross-platform concurrency, cache, process, and resource-measurement harness warrants final human review and remaining hosted validation.

Review effort: Balanced
Findings: None

Resolved since last review (2)

• Reject overlap counts exceeding ambient maxima
• Include replacement refresh counts in ambient maxima

[Karthik Nadig (karthiknadig)]

PR #565 has merged and #564 is closed. Signed merge 94d6fa9 imports the exact deterministic coverage repair from main without rewriting any published benchmark commits. The isolated merge passed independent review, 53 JSONRPC tests, mandatory Rust precommit, and exact workspace/all-target/all-feature Clippy. Fresh current-head review and hosted quality checks are requested; this stays draft until those complete.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The large cross-platform concurrency benchmark still requires the pending final-head hosted Linux and macOS validation.

Review effort: Balanced
Findings: None

[Karthik Nadig (karthiknadig)]

Current-head quality inspection (94d6fa9): the deterministic output coverage fix is now merged from main, and Linux/Windows/native macOS coverage all have 0.000pp drift (85.589%, 83.130%, and 80.586%). All three CodeQL analyses have zero results, warnings, and errors. The fresh Copilot review has no findings and no unresolved threads remain.

Native fast session benchmarks passed on Linux, Windows, and macOS. I downloaded and independently revalidated all three artifacts: exact 1/10/100 inventories and measurement counts, persisted-cache interpreter counts 1/0/0, and stable post-overlap/final resource counts. Linux threads/descriptors stayed 7/6; Windows threads/handles stayed 10/92. macOS correctly reports unavailable thread/descriptor measurements as null.

Ordinary matched-inventory discovery P50 is Linux 56/56ms, Windows 136/135ms, and macOS 116/167ms (head/base). The small Windows P95 +8ms drift is accepted within unchanged budgets; this PR contains no production Rust changes. All performance and coverage gates pass. The last native macOS ARM64 full-test job is still running, so this remains draft until it finishes.

Protected auto-merge is currently blocked by the CLI credential lacking workflow scope, which is required for workflow-file changes. No merge protection or required human review will be bypassed.

[Karthik Nadig (karthiknadig)]

The final native macOS ARM64 job passed: all 39 automated checks on 94d6fa9 are now green. This PR is ready, and protected squash auto-merge was accepted and verified (enabled at 21:28:20Z). Required human review remains in place. Correction to my earlier scope warning: GitHub accepted this PR's queued auto-merge request; the workflow-scope denial remains specific to the attempted request for #561.