fix: bound end-to-end refresh discovery latency - #25
Conversation
A NativePythonFinder refresh could stall discovery indefinitely: the single-worker WorkerPool queue wait was unbounded, and CLI-fallback enrichment scaled with the environment count. Capture one monotonic operation budget (184s, derived from the existing stage-timeout constants) at enqueue. The WorkerPool expires the item with QueueTaskExpiredError if it is still queued when the budget elapses, and the same Deadline clamps every extension-controlled running stage (configure, refresh, resolve, restart backoff, CLI find + enrichment) to the remaining budget, failing fast with RefreshBudgetExceededError below a 1s floor. The CLI fallback never truncates enumeration: it retains every discovered record and only stops further enrichment when the budget is spent. resolve() and all non-refresh callers pass no deadline, so their behavior is unchanged. Both new errors classify as rpc_timeout via the existing telemetry patterns. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
🔒 Automated review in progress — @StellaHuang95 is auto-reviewing this PR. |
Address review threads on bounded refresh discovery latency (PR #25): - Queue admission/dequeue now consume the same monotonic absolute deadline (performance.now) as the running stages, instead of recapturing a relative budget on a wall clock. A clock rollback can no longer let an already-expired queued item execute. The queue's expiry parameter is now an absolute instant and the pool runs on the finder's monotonic clock. - A late environment notification below MIN_STAGE_BUDGET_MS now retains the discovered record (like the CLI path) instead of dropping it, so a budget-exhausted refresh no longer returns and caches an empty list. - Emit one top-level PET_REFRESH timeout event for queue-expiry and running-budget exhaustion, which previously bypassed refresh telemetry (non-duplicating with per-attempt stage events). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Honor the no-new-code-comments rule: drop the one explanatory comment in the absolute-deadline regression test. No behavior or assertion change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
GitHub cannot anchor PR review comments to unchanged lines in the diff. Falling back to a general PR comment for src/managers/common/nativePythonFinder.ts:L1085.
If a deadline-clamped |
|
GitHub cannot anchor PR review comments to unchanged lines in the diff. Falling back to a general PR comment for src/managers/common/nativePythonFinder.ts:L1085.
A resolve that starts with a clamped timeout still drops the discovered environment when it rejects because this handler only logs. Retain the original [verified] |
…fresh (PR #25) A refresh RPC timeout emits attempt-level PET_REFRESH telemetry in doRefreshAttempt; the retry path could then mint RefreshBudgetExceededError ('refresh_retry') when the budget was spent, which the top-level handler re-emitted as a second terminal PET_REFRESH timeout for the same logical refresh. Route the retry-budget-exhaustion case to surface the original, already-reported attempt error and funnel all terminal-timeout telemetry through a single emitTerminalRefreshTimeout owner that only fires for no-attempt failures (queue expiry, restart/stage/CLI budget). Ordinary per-attempt nonterminal telemetry is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
The new deadline paths can silently omit discovered environments and emit duplicate refresh telemetry for a single logical refresh. |
…(PR #25) A deadline-clamped (or ordinary) resolve that started but then rejected or timed out was only logged, silently dropping the incomplete environment the refresh had already discovered. Route the resolve through resolveOrRetainEnv, which retains the raw discovered record on failure — matching the CLI fallback and the below-floor path — so no discovered environment is silently omitted. Enumeration timeouts still reject rather than returning a truncated list. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Both concerns are addressed on the current head ( 1. Silently omitting discovered environmentsThe bounded-refresh paths now retain every discovered record; nothing is dropped and no truncated enumeration is cached:
Regressions: 2. Duplicate refresh telemetry for a single logical refreshTerminal refresh-timeout telemetry now has exactly one owner:
Regressions: 4 Validation: lint, |
| * @returns A promise that when resolved gets the result from running the worker function. | ||
| */ | ||
| addToQueue(item: T, position?: QueuePosition): Promise<R>; | ||
| addToQueue(item: T, position?: QueuePosition, expiresAt?: number): Promise<R>; |
There was a problem hiding this comment.
Warning · Non-blocking recommendation
expiresAt is a raw timestamp interpreted by the pool's injected clock, but the WorkerPool interface does not couple the two. A caller using a different clock domain (Date.now() versus performance.now()) could expire tasks immediately or never; please make this clock-bound or use a relative duration.
| // calls share a single long-lived process — so unbounded parallelism would cause CPU/memory | ||
| // pressure. Process in batches of CLI_RESOLVE_CONCURRENCY. | ||
| const retainRemainingUnresolved = (fromIndex: number): void => { | ||
| const remaining = toResolve.slice(fromIndex); |
There was a problem hiding this comment.
Warning · Non-blocking recommendation
Please add coverage that drives CLI fallback through budget exhaustion between resolve batches and asserts every remaining discovered environment is returned raw. This branch implements the stated non-truncation guarantee, but current tests exercise only the generic retention helper rather than the CLI fallback path.
Problem
A
NativePythonFinderrefresh could stall discovery indefinitely. Two waits were unbounded:WorkerPool. If PET hangs on a refresh forone cache key, a refresh enqueued behind it (a different key) waited forever for the worker.
pet findandthen resolves each incomplete environment in batches. Enrichment time scales with the environment
count, so on a machine with hundreds of envs the fallback could run for a very long time.
Fix
Capture one monotonic operation budget at enqueue time and use it two ways:
WorkerPoolrejects the item withQueueTaskExpiredErrorif it is still queued when thebudget elapses — it is removed and never executes.
Deadlineclamps every extension-controlled running stage (configure / refresh /resolve / restart backoff / CLI find + enrichment) to the remaining budget. Below a small floor a
stage fails fast with
RefreshBudgetExceededErrorinstead of starting a doomed near-zero timeout.resolve()and every other non-refresh caller pass no deadline, so their timeouts and behaviorare unchanged.
Operation budget — arithmetic (derived from existing constants)
The budget is the worst-case wall-clock of a successful bounded refresh, so the cap can never cut a
valid flow. A successful
doRefreshruns at mostMAX_REFRESH_RETRIES + 1 = 2attempts:restart()callsconfigureRetry.reset(), so a single attempt can never have both a restart andthe 60s extended configure — the two maxima occur on different attempts, so summing them is a true
attained maximum rather than a loose over-approximation. The formula is written in terms of
MAX_REFRESH_RETRIESso it scales if the retry count changes.Floor:
MIN_STAGE_BUDGET_MS = 1_000. Below 1s remaining a stage fails fast (the fastest PETround-trip,
info, is already budgeted a generous 2s).Semantics
Deadlineis absolute and monotonic (performance.now(), injectable for tests).clampTimeoutToRemaining(base, deadline?, stage)returnsbaseunchanged whendeadlineisundefined, throwsRefreshBudgetExceededErrorwhenremaining < floor, elsemin(base, remaining).WorkerPoolpending-task expiration (addToQueue(item, position?, expiresInMs?)): the queuedwrapper stores an absolute
expiresAt(injectablenowclock) in addition to asetTimeout.next()rechecksnow() >= expiresAtbefore dequeuing, so a delayed timer / event-loop stallcan never start a past-deadline item. Transitions are queued → running | expired → settled exactly
once; timers are cleared on dequeue, settle, and stop. Omitting
expiresInMspreserves the originalunbounded queueing, keeping the pool generic.
restart()fails fast on entry if the budget is spent, and after its clamped backoff rechecksthe deadline (
backoffThenCheckBudget) immediately before teardown/spawn — PET is never startedafter the budget is spent.
pet findcannot complete within budget the callrejects; once
findcompletes, every discovered record is retained and running out of budget onlystops further enrichment (unresolved records kept, matching the CLI path's existing behavior).
QueueTaskExpiredErrorandRefreshBudgetExceededErrorare time-budget exhaustions, classified viathe existing telemetry patterns as
rpc_timeout(theinstanceofbranch runs before message-patternmatching so the budget message's "restart" text is not misread as
process_crash).Tests
workerPool.unit.test.ts, deterministic fake timers): queued-behind-never-resolvingexpires and never runs; dequeue clears the timer; expiry/dequeue boundary settles exactly once (both
directions); stop clears the timer; later tasks still run; omitting
expiresInMspreserves unboundedqueueing. Absolute-deadline suite (injected clock decoupled from the faked
setTimeout): anevent-loop stall past the deadline still expires via
next()'s recheck; anow == expiresAtboundary expires;
next()skips a stalled-expired item and continues; an already-expired enqueuerejects without stranding the parked worker.
nativePythonFinder.budget.unit.test.ts, injected clock): the 184000 formula and1s floor;
Deadlinecountdown/isExhausted;clampTimeoutToRemainingpassthrough / clamp / floorthrow and stage-to-stage propagation;
backoffThenCheckBudgetrestart recheck; theRefreshBudgetExceededErrormessage.errorClassifier.unit.test.ts): both new errors classify asrpc_timeout.compile-testsclean.