Eliminate macOS cold-refresh 30-second tail latency

[karthiknadig]

Summary

PET's macOS performance baseline has a deterministic cold-refresh tail near 30 seconds that is hidden by healthy P50 values.

Evidence

Across 11 successful main-branch performance baselines from July 24 through August 6, 2026:

• macOS full-refresh P50 median: 137 ms
• macOS full-refresh P95 median: 30,291 ms
• Latest baseline (8bd5fb4, run 31071711968): P50 142 ms, P95 30,291 ms, mean 8,642 ms
• The latest raw iterations were:
  • 30,370 ms (9 environments)
  • 30,195 ms (9 environments)
  • 24,927 ms (10 environments)
  • then 139, 131, 110, 144, 105, 128, and 174 ms (10 environments)

The same cache directory is reused while each iteration starts a fresh PET server. The first three processes are slow and initially miss one environment, after which the disk cache is warm. Environment inventory is otherwise stable (10 environments, 1 manager), so this is not explained by changing runner inventory.

Investigation requirements

• Attribute each slow iteration to a refresh phase and locator using existing RefreshProgress/RefreshPerformance data or equivalent privacy-safe instrumentation.
• Determine whether the wait is interpreter spawning, macOS locator probing, cache population, subprocess/pipe behavior, or another blocking path.
• Verify every file-based detection path before retaining any spawn fallback.
• Measure exact spawn/read operation counts in regression tests.

Acceptance criteria

• Root cause is demonstrated with phase/locator and operation-count evidence.
• A cold PET process discovers the complete expected environment inventory on its first refresh.
• macOS cold-refresh tail no longer approaches the 15/30-second timeout boundaries.
• A realistic macOS CI P95 threshold is established and enforced by the snapshot gate.
• Median performance does not regress on macOS, Windows, or Linux.
• Fix includes behavior tests and performance/operation-count tests.
• No extra Python or manager process spawning is introduced.

Related

• Add end-to-end performance tests for JSONRPC server across platforms #313
• Quality snapshot enforcement issue filed from the same audit.

[karthiknadig]

Diagnostic run 31423551324 on head f9485d0 established the root cause with phase, locator, and operation-count evidence:

• First three iterations each emitted exactly two interpreter-probe timeouts, both classified as usrBinPython3.
• Iterations: ~30.3s, ~30.2s, ~30.3s; each found only 9 environments.
• Path P95: 30,308 ms.
• MacCommandLineTools P95: 15,185 ms.
• Later iterations had no probe timeouts and completed in 182-294ms with 10 environments.

The two /usr/bin/python3 calls share the same interpreter cache entry. MacCommandLineTools::try_from() starts a 15-second spawn to enrich the alias while PATH discovery asks to resolve the same shim. The cache-entry mutex serializes them: PATH waits for the locator probe, then performs its own 15-second probe after the first timeout, producing the ~30-second phase.

The product fix should therefore:

1. Resolve Apple /usr/bin/python* shims from the active developer directory using filesystem state (DEVELOPER_DIR / /var/db/xcode_select_link / standard fallbacks).
2. Let MacXCode or MacCommandLineTools identify the selected shim without spawning it.
3. Skip spawning unresolved Apple system shims; a non-selected or unavailable shim is not a usable Python environment.
4. Remove the duplicate /usr/bin/python3 alias-enrichment spawn from the macOS locators.
5. Add operation-count tests proving zero interpreter spawns for these paths, plus cross-platform behavior tests.

[karthiknadig]

First post-fix validation run 31425828087 on b923035 passed all platform gates.

macOS results:

• 0 interpreter probe timeouts (previously 2 per cold iteration).
• Iteration 1 found all 10 environments in 2,436ms (previously 9 environments in ~30s).
• Full refresh P95: 1,437ms (previously ~30,308ms).
• Path P95: 1,436ms (previously ~30,308ms).
• MacCommandLineTools P95: 137ms (previously ~15,185ms).
• All ten iterations found the complete inventory.

Linux and Windows also passed, retained identical environment inventories, and had no interpreter probe timeouts. A second identical run is in progress to distinguish normal macOS median variance from code drift before opening the PR.

[karthiknadig]

Repeat validation run 31426630835 on the same commit also passed every platform gate:

• macOS iteration 1 found all 10 environments; every iteration retained the complete inventory.
• 0 interpreter probe timeouts.
• Full refresh P95: 2,056ms.
• path P95: 2,055ms.
• MacCommandLineTools P95: 130ms.
• macOS P50 remained within the enforced platform budget; Linux and Windows inventories/performance gates also passed.

Across both fixed runs, the deterministic 15s/30s tail did not recur. The remaining cold-run variance is ordinary runner startup/build-host noise rather than interpreter probing.

[karthiknadig]

Final quality conclusion for PR #506 at c0f026e:

• Full CI, lint, CodeQL, Linux/Windows coverage, and enforced performance gates pass.
• Exact-head Copilot review covered 7/7 files with 0 comments and 0 unresolved threads.
• macOS repeat: P50 120ms vs 99ms baseline; P95 1,256ms vs 30,320ms baseline (95.9% lower); time-to-first P50 84ms vs 99ms.
• Windows repeat: P50 138ms vs 124ms; P95 1,224ms vs 2,533ms. Linux P50 53ms vs 55ms.
• Every measured iteration retained the complete inventory (macOS/Windows 10, Linux 5); interpreter probe timeout maps are empty on all platforms.
• Untimed diagnostics still retain 10 phase/locator samples; macOS MacCommandLineTools P95 is 35ms.
• Coverage increased: Linux lines/functions +0.121pp/+0.015pp; Windows +0.142pp/+0.066pp.

Hosted cold-start variance remains visible (notably first-iteration startup), but two final-head-equivalent gated runs and the exact-head repeat show no material median regression. The deterministic 15s/30s interpreter-probe tail does not recur.