test: prove subprocess and production coverage (Fixes #534)

[Karthik Nadig (karthiknadig)]

Make coverage evidence demonstrate real server execution and expose production gaps without changing existing whole-workspace regression gates.

• Compare exact idle/info child profiles and require positive execution in the real handler, transport dispatch, and response writer after graceful shutdown.
• Add production/test and changed-line diagnostics with conservative accounting for LLVM summary entries without unique source lines.
• Require profile proof and reporting in Linux/Windows PR and baseline jobs; add native macOS coverage measured against the exact base on the same runner.
• Document classification, branch-instrumentation limits, and artifact semantics.

Validation: 76 Python tests, 19 Windows native cases, workspace formatting/Clippy, and independent review pass. Actual Windows LLVM profiles prove all three execution witnesses increase from 0 to 1. Hosted macOS validation and quality inspection remain pending; no signing/release services run.

Fixes #534

[github-actions]

Performance Report (Linux)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	1ms	1ms	+0ms	+0.0%	>5ms and >100%	➖
Server startup P95	1ms	1ms	+0ms	+0.0%	>50ms and >200%	➖
Discovery duration P50	55ms	65ms	-10ms	-15.4%	>25ms and >30%	✅
Discovery duration P95	59ms	68ms	-9ms	-13.2%	>50ms and >50%	✅
Startup-to-first environment P50	12ms	14ms	-2ms	-14.3%	>20ms and >100%	✅
Startup-to-first environment P95	17ms	18ms	-1ms	-5.6%	>25ms and >100%	✅
Cold discovery duration P50	140ms	150ms	-10ms	-6.7%	>100ms and >50%	✅
Refresh round-trip P50	56ms	65ms	-9ms	-13.8%	>25ms and >30%	✅
Refresh round-trip P95	60ms	68ms	-8ms	-11.8%	>50ms and >50%	✅
Request-to-first environment P50	10ms	13ms	-3ms	-23.1%	>20ms and >100%	✅
Request-to-first environment P95	15ms	16ms	-1ms	-6.2%	>25ms and >100%	✅
Cold refresh round-trip P50	140ms	151ms	-11ms	-7.3%	>100ms and >50%	✅

Workload	PR	Baseline
Environments	5	5
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

[github-actions]

Test Coverage Report (Linux)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	85.535%	85.535%	+0.000pp
Functions	88.368%	88.368%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[github-actions]

Performance Report (Windows)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	5ms	9ms	-4ms	-44.4%	>10ms and >50%	✅
Server startup P95	9ms	14ms	-5ms	-35.7%	>50ms and >100%	✅
Discovery duration P50	71ms	160ms	-89ms	-55.6%	>150ms and >50%	✅
Discovery duration P95	77ms	175ms	-98ms	-56.0%	>250ms and >100%	✅
Startup-to-first environment P50	11ms	22ms	-11ms	-50.0%	>25ms and >50%	✅
Startup-to-first environment P95	13ms	28ms	-15ms	-53.6%	>100ms and >100%	✅
Cold discovery duration P50	71ms	159ms	-88ms	-55.3%	>150ms and >50%	✅
Refresh round-trip P50	71ms	161ms	-90ms	-55.9%	>150ms and >50%	✅
Refresh round-trip P95	77ms	176ms	-99ms	-56.2%	>250ms and >100%	✅
Request-to-first environment P50	5ms	12ms	-7ms	-58.3%	>25ms and >50%	✅
Request-to-first environment P95	7ms	18ms	-11ms	-61.1%	>100ms and >100%	✅
Cold refresh round-trip P50	71ms	159ms	-88ms	-55.3%	>150ms and >50%	✅

Workload	PR	Baseline
Environments	8	8
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

[github-actions]

Performance Report (macOS)

Result: ✅ Within regression budgets

Metric	PR	Baseline	Delta	Change	Blocking budget	Status
Server startup P50	82ms	57ms	+25ms	+43.9%	>100ms and >50%	🔺
Server startup P95	603ms	514ms	+89ms	+17.3%	>750ms and >100%	🔺
Discovery duration P50	144ms	94ms	+50ms	+53.2%	>100ms and >50%	🔺
Discovery duration P95	180ms	104ms	+76ms	+73.1%	>300ms and >100%	🔺
Startup-to-first environment P50	104ms	83ms	+21ms	+25.3%	>150ms and >50%	🔺
Startup-to-first environment P95	131ms	95ms	+36ms	+37.9%	>250ms and >100%	🔺
Cold discovery duration P50	295ms	218ms	+77ms	+35.3%	>250ms and >50%	🔺
Refresh round-trip P50	145ms	95ms	+50ms	+52.6%	>250ms and >50%	🔺
Refresh round-trip P95	181ms	104ms	+77ms	+74.0%	>300ms and >100%	🔺
Request-to-first environment P50	36ms	25ms	+11ms	+44.0%	>50ms and >50%	🔺
Request-to-first environment P95	47ms	35ms	+12ms	+34.3%	>100ms and >100%	🔺
Cold refresh round-trip P50	296ms	219ms	+77ms	+35.2%	>600ms and >50%	🔺

Workload	PR	Baseline
Environments	10	10
Managers	1	1

A regression must exceed both the documented absolute and relative budget. Environment and manager inventories must match exactly within the same inventory schema.

[github-actions]

Test Coverage Report (Windows)

Result: ✅ Within regression budget

Metric	PR	Baseline	Delta
Lines	83.083%	83.075%	+0.009pp
Functions	85.753%	85.753%	+0.000pp

Allowed numerical tolerance: 0.01 percentage points.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The new cross-platform coverage gate includes pending hosted macOS validation.

Review effort: Balanced
Findings: None

What changed in this PR

Adds production-focused coverage diagnostics and proves graceful subprocess profile collection across supported platforms.

Changes:

• Adds LCOV classification, changed-line reporting, and isolated subprocess proof.
• Adds Linux, Windows, and native macOS coverage workflow enforcement.
• Documents coverage semantics and limitations.

File	Description
scripts/​coverage_detail.py	Implements coverage diagnostics and proof verification.
scripts/​tests/​test_coverage_detail.py	Tests parsing, classification, and proof behavior.
scripts/​tests/​test_quality_workflows.py	Validates workflow coverage requirements.
crates/​pet/​tests/​jsonrpc_server_test.rs	Captures graceful-shutdown subprocess profiles.
.github/​workflows/​coverage.yml	Adds PR coverage proof and reports.
.github/​workflows/​coverage-baseline.yml	Adds baseline proof and artifacts.
.github/​workflows/​coverage-macos.yml	Adds native macOS exact-base coverage.
docs/​QUALITY_SNAPSHOTS.md	Documents supplemental coverage evidence.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The new macOS coverage gate still needs hosted validation, and the source scanner has unresolved scaling issues.

Review effort: Balanced
Findings: None

Previously missed (2)

In code that hasn't changed since last review

Avoid quadratic source slicing in raw-string detection

scripts/​coverage_detail.py:118

code_mask passes source[i:] to this regex on nearly every unmasked character, copying the remaining file each time. That makes production/test classification quadratic across the Rust files processed by summarize. Match at offset i without slicing, and use the match's absolute end to find the raw-string terminator.

Avoid quadratic source slicing in character-literal detection

scripts/​coverage_detail.py:137

This character-literal check copies the remaining source for every apostrophe, including Rust lifetime markers. Even after fixing the raw-string scan above, files with many lifetimes or character literals can still take quadratic time to classify. Match at offset i and use the match's absolute end.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The new cross-platform coverage gate has unresolved reporting issues, and hosted macOS validation remains pending.

Review effort: Balanced
Findings: None

Previously missed (2)

In code that hasn't changed since last review

Recognize test-only conditional attributes requiring test

scripts/​coverage_detail.py:155

The matcher handles only #[cfg(test)] and #[test], so it treats helpers inside #[cfg(all(test, unix))] modules as production. For example, the executable get_env_var branch at crates/pet-homebrew/src/lib.rs:193 belongs to such a test-only module but contributes to the new production totals on Unix/macOS. Recognize conditional attributes that require test (including all(test, unix)) when excluding an item, and add a classification regression test; avoid treating any(test, unix) as test-only.

Classify unmapped LLVM entries in test files as test coverage

scripts/​coverage_detail.py:242

For a source under tests/ or benches/, all mapped lines are classified as tests at line 229, but this line puts every unmapped LLVM LF entry into production_found. Such entries cannot be production in these test-only files, so the production gap is overstated. Put their unmapped entries in test_found instead, still uncovered, and add a regression test using a test file with LF larger than its DA count.

[Karthik Nadig (karthiknadig)]

Current-head update (e79fcac): the source-scanner allocation findings and both classification findings are addressed, with 81 Python tests and independent review. Raw coverage gates remain unchanged.

Native macOS validation has completed successfully: lines 80.561% versus the same-runner exact base's 80.504% (+0.058pp), functions unchanged at 83.180%. The isolated idle/info profiles prove 0 -> 1 execution at all three witnesses: handler, transport dispatch, and writer.

I am also investigating the repeated macOS performance drift rather than accepting only the green budget result. A diagnostic-only eight-pass, counterbalanced same-host comparison of exact main and this head is running here: https://github.com/microsoft/python-environment-tools/actions/runs/36749488202 . It retains full inventories, binary/harness provenance, and all samples. No production Rust or Cargo inputs changed in this PR; the control will help distinguish PR effects from runner variation. This PR remains draft until review and quality inspection are complete. The diagnostic branch is not intended for merge.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The new macOS coverage workflow and exact-base comparison still need hosted validation.

Review effort: Balanced
Findings: None

[Karthik Nadig (karthiknadig)]

The current-head macOS performance failure was investigated rather than ignored: discovery P50 was 205 ms vs 94 ms, exceeding the unchanged absolute/relative gate by 111 ms / 118.1%. Linux and Windows performance improved with matching inventories; their coverage deltas are 0.000 pp / +0.009 pp, and native macOS coverage is +0.058 pp with all isolated subprocess witnesses passing.

The exact-ref same-host control completed successfully: https://github.com/microsoft/python-environment-tools/actions/runs/36749488202. It tested base 597b599 and head e79fcac in B-H-H-B-H-B-B-H order, retained every sample, verified binary hashes before each pass, and all eight inventories have the same SHA-256 (10 environments, 1 manager).

Run-level statistic	Base passes	Head passes
Discovery P50 (ms)	105, 90, 146, 137	127, 112, 124, 104
Discovery P95 (ms)	160, 96, 170, 154	207, 194, 149, 116
Refresh RTT P50 (ms)	106, 91, 147, 138	128, 113, 124, 105
Cold RTT P50 (ms)	274, 208, 358, 323	324, 275, 274, 229

Median run-level discovery P50 is 121 ms base / 118 ms head; refresh P50 122 / 118.5 ms. The tail remains noisy (median discovery P95 157 / 171.5 ms) and is not being discarded. Both original jobs used the same macOS image/architecture; this is not an image-transition claim. The controlled distributions support run/host variance, not a stable source regression.

I am making one targeted confirmation rerun of the original failed macOS job, with the exact head, baseline, workflow, and budgets unchanged. The failed sample above remains part of the evidence; no rerun-until-green loop or budget relaxation. Independent review accepted the control methodology and this single confirmation, but could not independently inspect the local raw-artifact directory because of an access boundary. The diagnostic artifact remains attached to the linked run for authorized review. This PR remains draft pending the confirmation and remaining gates.

[Karthik Nadig (karthiknadig)]

Final quality gate at e79fcac: the single unchanged confirmation passed. macOS discovery P50/P95 are 144/180 ms, refresh RTT 145/181 ms, and cold RTT P50 296 ms; all 10 environment and 1 manager identities match. This is consistent with the same-host control's variability, though slower than the historical baseline. The original failing 205 ms sample and both attempts' artifacts remain preserved; no budgets, baseline, or runtime source were changed.

All 36 automated checks now pass. Current-head CodeQL analyses for Rust, Python, and Actions report 0 results and no analysis errors. Coverage is non-regressing on all three platforms and native subprocess attribution is proven. Current-head Copilot has no actionable findings; its pending-macOS-coverage note is superseded by the completed native run.

The accepted noisy metric is macOS host-to-host discovery/RTT variance, supported by the eight-pass exact-ref control rather than merely a green retry. I am marking this ready and enabling protected auto-merge. The remaining VS Code policy gate requires one collaborator approval (0/1); it will not be bypassed. The diagnostic control branch is not part of this PR and will not be merged.