Skip to content

fix: promptly reject in-flight PET RPC requests on process exit/error - #22

Open
StellaHuang95 wants to merge 1 commit into
mainfrom
fix-pet-exit-rpc
Open

fix: promptly reject in-flight PET RPC requests on process exit/error#22
StellaHuang95 wants to merge 1 commit into
mainfrom
fix-pet-exit-rpc

Conversation

@StellaHuang95

Copy link
Copy Markdown
Owner

Problem

When the PET (Python Environment Tools) locator process exits or errors, its in-flight JSON-RPC requests (configure / refresh / resolve / info) did not reject promptly. Callers waited out the full 30–60s request timeouts even though PET was already gone, stalling environment discovery and refresh.

Reproduction

  1. Trigger a refresh/resolve that issues a PET RPC request.
  2. Have the PET child exit (crash) or emit error while the request is in flight.
  3. Observe the request stays pending for its full 30–60s timeout instead of rejecting when the process dies.

Root cause

NativePythonFinderImpl.start() piped PET's stdout into a PassThrough (readable) with end: false, and the child exit/error handlers only flipped flags. Because readable was never ended, StreamMessageReader never saw EOF, the JSON-RPC connection stayed open, and pending requests rejected only after their own timeouts.

A second latent bug: onClose disposed the shared startDisposables. After a restart reassigned that field, a stale old-connection close could dispose the replacement connection.

Fix

  • Give each spawned child its own captured streams, JSON-RPC connection, and localDisposables. On that child's exit/error, idempotently end only those local streams so the connection closes and pending requests reject at once.
  • Guard shared-state mutation (proc / exit flags) and disposal by child/connection identity, so a stale old-child event can never end streams for or dispose a replacement connection.
  • Unpipe the child's stdout from the local readable before ending it, so a late buffered write cannot raise ERR_STREAM_WRITE_AFTER_END.
  • Classify the resulting PendingResponseRejected rejection as a recoverable PET connection loss (not a hard rpc_error), so an exited-child crash still restarts and retries instead of failing immediately — preserving recover-on-crash behavior with prompt rejection instead of a 30–60s wait.

Tests

Deterministic fake-child / fake-RPC unit tests (nativePythonFinder.petExit.unit.test.ts):

  • pending request rejects promptly on child exit
  • pending request rejects promptly on child error
  • duplicate error + exit is harmless (idempotent teardown)
  • stale old-child exit cannot close a replacement connection
  • restart produces a usable connection and resets exit state
  • a current-child crash during refresh promptly restarts and the retry succeeds
  • a connection loss during disposal is NOT treated as a recoverable crash
  • refresh retry limit is preserved for connection-loss errors
  • buffered stdout after exit does not raise a late write error or affect a replacement

Error-classifier tests cover isPetConnectionLostError and PendingResponseRejectedconnection_error.

NativePythonFinderImpl.start() piped PET stdout into a PassThrough with
end:false, and the child exit/error handlers only flipped flags, so
StreamMessageReader never saw EOF. The JSON-RPC connection stayed open and
in-flight configure/refresh/resolve/info requests only rejected after their
30-60s timeouts, long after PET had already gone.

- Give each child its own captured streams, JSON-RPC connection, and
  localDisposables; the child's exit/error idempotently ends only those
  local streams so the connection closes and pending requests reject at once.
- Guard shared-state mutation by child/connection identity so a stale old
  child event can never end streams for or dispose a replacement connection.
- Unpipe the child's stdout from the local readable before ending it so a
  late buffered write cannot raise ERR_STREAM_WRITE_AFTER_END.
- Classify the resulting PendingResponseRejected rejection as a recoverable
  PET connection loss (not a hard rpc_error) so an exited-child crash still
  restarts and retries instead of failing immediately.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@StellaHuang95 StellaHuang95 added the bug Something isn't working label Aug 23, 2026
@StellaHuang95

StellaHuang95 commented Aug 23, 2026

Copy link
Copy Markdown
Owner Author

🔒 Automated review in progress — @StellaHuang95 is auto-reviewing this PR.

ex instanceof rpc.ConnectionError ||
(ex instanceof rpc.ResponseError && ex.code === rpc.ErrorCodes.PendingResponseRejected)
);
}

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning · Non-blocking recommendation

isPetConnectionLostError now controls PET retry behavior but lives in a telemetry module, deepening the manager-to-telemetry dependency. Move this predicate to a PET/RPC domain error module and import it from both telemetry and NativePythonFinderImpl.

[verified]

}

function getState(f: NativePythonFinderImpl): { processExited: boolean; processExitReason: string | undefined } {
return f as unknown as { processExited: boolean; processExitReason: string | undefined };

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning · Non-blocking recommendation

These tests repeatedly cast through unknown to invoke private lifecycle methods and inspect private state, so behavior-preserving refactors can break them. Prefer injecting or extracting a child/RPC session factory that exposes termination and connection behavior through a stable test seam.

[verified]

this.processExitReason =
ex instanceof rpc.ConnectionError ? 'rpc_connection_error' : 'rpc_resolve_timeout';
ex instanceof RpcTimeoutError ? 'rpc_resolve_timeout' : 'rpc_connection_error';
}

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning · Non-blocking recommendation

The new resolve recovery path treats PendingResponseRejected as a recoverable connection loss, but the new tests cover that behavior only through refresh. Add a resolve-specific regression test so its restart behavior does not drift from refresh.

@StellaHuang95 StellaHuang95 added the review-auto:approved Automated review: no blocking findings (approval posted). label Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant