Skip to content

fix: bound end-to-end refresh discovery latency - #25

Open
StellaHuang95 wants to merge 5 commits into
mainfrom
stellahuang-microsoft-fuzzy-guacamole
Open

fix: bound end-to-end refresh discovery latency#25
StellaHuang95 wants to merge 5 commits into
mainfrom
stellahuang-microsoft-fuzzy-guacamole

Conversation

@StellaHuang95

Copy link
Copy Markdown
Owner

Problem

A NativePythonFinder refresh could stall discovery indefinitely. Two waits were unbounded:

  1. Queue wait. Refreshes run through a single-worker WorkerPool. If PET hangs on a refresh for
    one cache key, a refresh enqueued behind it (a different key) waited forever for the worker.
  2. CLI-fallback enrichment. When server mode is exhausted, the CLI fallback runs pet find and
    then resolves each incomplete environment in batches. Enrichment time scales with the environment
    count, so on a machine with hundreds of envs the fallback could run for a very long time.

Fix

Capture one monotonic operation budget at enqueue time and use it two ways:

  • The WorkerPool rejects the item with QueueTaskExpiredError if it is still queued when the
    budget elapses — it is removed and never executes.
  • The same Deadline clamps every extension-controlled running stage (configure / refresh /
    resolve / restart backoff / CLI find + enrichment) to the remaining budget. Below a small floor a
    stage fails fast with RefreshBudgetExceededError instead of starting a doomed near-zero timeout.

resolve() and every other non-refresh caller pass no deadline, so their timeouts and behavior
are unchanged.

Operation budget — arithmetic (derived from existing constants)

The budget is the worst-case wall-clock of a successful bounded refresh, so the cap can never cut a
valid flow. A successful doRefresh runs at most MAX_REFRESH_RETRIES + 1 = 2 attempts:

Attempt 0 — retryable refresh-RPC timeout (no restart precedes it, so the extended
            configure timeout is NOT reset):
    configure (extended)   MAX_CONFIGURE_TIMEOUT_MS  = 60_000
    refresh RPC            REFRESH_TIMEOUT_MS         = 30_000
                                                     = 90_000

Attempt 1 — restarts the killed process, then succeeds (restart resets configure to base):
    restart backoff  RESTART_BACKOFF_BASE_MS * 2^(MAX_RESTART_ATTEMPTS-1) = 1_000*2^2 =  4_000
    configure (base)       CONFIGURE_TIMEOUT_MS       = 30_000
    refresh RPC            REFRESH_TIMEOUT_MS          = 30_000
    parallel resolve       RESOLVE_TIMEOUT_MS          = 30_000
                                                      = 94_000

REFRESH_OPERATION_BUDGET_MS = 90_000 + 94_000 = 184_000 ms

restart() calls configureRetry.reset(), so a single attempt can never have both a restart and
the 60s extended configure — the two maxima occur on different attempts, so summing them is a true
attained maximum rather than a loose over-approximation. The formula is written in terms of
MAX_REFRESH_RETRIES so it scales if the retry count changes.

Floor: MIN_STAGE_BUDGET_MS = 1_000. Below 1s remaining a stage fails fast (the fastest PET
round-trip, info, is already budgeted a generous 2s).

Semantics

  • Deadline is absolute and monotonic (performance.now(), injectable for tests).
    clampTimeoutToRemaining(base, deadline?, stage) returns base unchanged when deadline is
    undefined, throws RefreshBudgetExceededError when remaining < floor, else min(base, remaining).
  • WorkerPool pending-task expiration (addToQueue(item, position?, expiresInMs?)): the queued
    wrapper stores an absolute expiresAt (injectable now clock) in addition to a setTimeout.
    next() rechecks now() >= expiresAt before dequeuing, so a delayed timer / event-loop stall
    can never start a past-deadline item. Transitions are queued → running | expired → settled exactly
    once; timers are cleared on dequeue, settle, and stop. Omitting expiresInMs preserves the original
    unbounded queueing, keeping the pool generic.
  • restart() fails fast on entry if the budget is spent, and after its clamped backoff rechecks
    the deadline (backoffThenCheckBudget) immediately before teardown/spawn — PET is never started
    after the budget is spent.
  • CLI fallback never truncates enumeration: if pet find cannot complete within budget the call
    rejects; once find completes, every discovered record is retained and running out of budget only
    stops further enrichment (unresolved records kept, matching the CLI path's existing behavior).
  • QueueTaskExpiredError and RefreshBudgetExceededError are time-budget exhaustions, classified via
    the existing telemetry patterns as rpc_timeout (the instanceof branch runs before message-pattern
    matching so the budget message's "restart" text is not misread as process_crash).

Tests

  • WorkerPool (workerPool.unit.test.ts, deterministic fake timers): queued-behind-never-resolving
    expires and never runs; dequeue clears the timer; expiry/dequeue boundary settles exactly once (both
    directions); stop clears the timer; later tasks still run; omitting expiresInMs preserves unbounded
    queueing. Absolute-deadline suite (injected clock decoupled from the faked setTimeout): an
    event-loop stall past the deadline still expires via next()'s recheck; a now == expiresAt
    boundary expires; next() skips a stalled-expired item and continues; an already-expired enqueue
    rejects without stranding the parked worker.
  • Finder budget (nativePythonFinder.budget.unit.test.ts, injected clock): the 184000 formula and
    1s floor; Deadline countdown/isExhausted; clampTimeoutToRemaining passthrough / clamp / floor
    throw and stage-to-stage propagation; backoffThenCheckBudget restart recheck; the
    RefreshBudgetExceededError message.
  • Classifier (errorClassifier.unit.test.ts): both new errors classify as rpc_timeout.
  • Targeted run: 47 passing. Full unit suite: 1719 passing / 5 pending. Lint and compile-tests
    clean.

A NativePythonFinder refresh could stall discovery indefinitely: the
single-worker WorkerPool queue wait was unbounded, and CLI-fallback
enrichment scaled with the environment count.

Capture one monotonic operation budget (184s, derived from the existing
stage-timeout constants) at enqueue. The WorkerPool expires the item with
QueueTaskExpiredError if it is still queued when the budget elapses, and the
same Deadline clamps every extension-controlled running stage (configure,
refresh, resolve, restart backoff, CLI find + enrichment) to the remaining
budget, failing fast with RefreshBudgetExceededError below a 1s floor. The CLI
fallback never truncates enumeration: it retains every discovered record and
only stops further enrichment when the budget is spent. resolve() and all
non-refresh callers pass no deadline, so their behavior is unchanged. Both new
errors classify as rpc_timeout via the existing telemetry patterns.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@StellaHuang95 StellaHuang95 added the bug Something isn't working label Aug 23, 2026
@StellaHuang95

StellaHuang95 commented Aug 23, 2026

Copy link
Copy Markdown
Owner Author

🔒 Automated review in progress — @StellaHuang95 is auto-reviewing this PR.

Comment thread src/common/utils/workerPool.ts
Comment thread src/managers/common/nativePythonFinder.ts
Comment thread src/managers/common/nativePythonFinder.ts
@StellaHuang95 StellaHuang95 added the review-auto:changes-requested Automated review: posted blocking findings to address. label Aug 23, 2026
StellaHuang95 and others added 2 commits August 22, 2026 22:24
Address review threads on bounded refresh discovery latency (PR #25):

- Queue admission/dequeue now consume the same monotonic absolute
  deadline (performance.now) as the running stages, instead of
  recapturing a relative budget on a wall clock. A clock rollback can no
  longer let an already-expired queued item execute. The queue's expiry
  parameter is now an absolute instant and the pool runs on the
  finder's monotonic clock.
- A late environment notification below MIN_STAGE_BUDGET_MS now retains
  the discovered record (like the CLI path) instead of dropping it, so a
  budget-exhausted refresh no longer returns and caches an empty list.
- Emit one top-level PET_REFRESH timeout event for queue-expiry and
  running-budget exhaustion, which previously bypassed refresh telemetry
  (non-duplicating with per-attempt stage events).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Honor the no-new-code-comments rule: drop the one explanatory comment in
the absolute-deadline regression test. No behavior or assertion change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@StellaHuang95

Copy link
Copy Markdown
Owner Author

GitHub cannot anchor PR review comments to unchanged lines in the diff. Falling back to a general PR comment for src/managers/common/nativePythonFinder.ts:L1085.

Warning · Non-blocking recommendation

If a deadline-clamped resolve request starts but then times out or otherwise rejects, the failure path logs the error without retaining the original incomplete environment. Preserve data in nativeInfo on resolve failure as well, and add a deadline-boundary notification test.

Comment thread src/managers/common/nativePythonFinder.ts Outdated
@StellaHuang95 StellaHuang95 added review-auto:approved Automated review: no blocking findings (approval posted). and removed review-auto:changes-requested Automated review: posted blocking findings to address. labels Aug 23, 2026
@StellaHuang95

Copy link
Copy Markdown
Owner Author

GitHub cannot anchor PR review comments to unchanged lines in the diff. Falling back to a general PR comment for src/managers/common/nativePythonFinder.ts:L1085.

Issue · Please address or respond

A resolve that starts with a clamped timeout still drops the discovered environment when it rejects because this handler only logs. Retain the original data on resolve failure, and add coverage for a resolve that starts above the floor but times out at the remaining deadline.

[verified]

Comment thread src/managers/common/nativePythonFinder.ts
@StellaHuang95 StellaHuang95 added review-auto:changes-requested Automated review: posted blocking findings to address. and removed review-auto:approved Automated review: no blocking findings (approval posted). labels Aug 23, 2026
…fresh (PR #25)

A refresh RPC timeout emits attempt-level PET_REFRESH telemetry in
doRefreshAttempt; the retry path could then mint RefreshBudgetExceededError
('refresh_retry') when the budget was spent, which the top-level handler
re-emitted as a second terminal PET_REFRESH timeout for the same logical
refresh. Route the retry-budget-exhaustion case to surface the original,
already-reported attempt error and funnel all terminal-timeout telemetry
through a single emitTerminalRefreshTimeout owner that only fires for
no-attempt failures (queue expiry, restart/stage/CLI budget). Ordinary
per-attempt nonterminal telemetry is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@StellaHuang95

Copy link
Copy Markdown
Owner Author

The new deadline paths can silently omit discovered environments and emit duplicate refresh telemetry for a single logical refresh.

…(PR #25)

A deadline-clamped (or ordinary) resolve that started but then rejected or
timed out was only logged, silently dropping the incomplete environment the
refresh had already discovered. Route the resolve through resolveOrRetainEnv,
which retains the raw discovered record on failure — matching the CLI fallback
and the below-floor path — so no discovered environment is silently omitted.
Enumeration timeouts still reject rather than returning a truncated list.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@StellaHuang95

Copy link
Copy Markdown
Owner Author

Both concerns are addressed on the current head (8547a03). Mapping each to the specific findings:

1. Silently omitting discovered environments

The bounded-refresh paths now retain every discovered record; nothing is dropped and no truncated enumeration is cached:

  • Late notification below the stage floor (resolveTimeoutForRefreshundefined): the raw incomplete record is kept via nativeInfo.push(data) instead of returning/caching [] — the fix requested at nativePythonFinder.ts:1097.
  • A resolve that started (budget above the floor) but then times out / rejects (the L1085 fallback findings): previously this path only logged and dropped the env. It now goes through resolveOrRetainEnv, which retains the raw discovered record on failure — mirroring the CLI fallback's retainRemainingUnresolved, so a discovered environment is never silently omitted.
  • Enumeration itself timing out still rejects with RefreshBudgetExceededError rather than returning a partial list.

Regressions: resolveTimeoutForRefresh (below-floor retains the record) and resolveOrRetainEnv (a timed-out/rejected resolve retains the raw record and reports the error; success returns the resolved env).

2. Duplicate refresh telemetry for a single logical refresh

Terminal refresh-timeout telemetry now has exactly one owner:

  • The retry path (decideRefreshRetryAction) routes a retryable failure that hits an exhausted budget to surface — it rethrows the original, already-reported attempt error instead of minting a new terminal RefreshBudgetExceededError('refresh_retry').
  • emitTerminalRefreshTimeout is the single terminal owner and fires exactly one PET_REFRESH event, and only for no-attempt failures (queue expiry, restart/stage/CLI budget). A surfaced attempt error is neither a queue-expiry nor a budget error, so the owner stays silent — one logical refresh emits one terminal timeout event. Ordinary per-attempt nonterminal telemetry is unchanged.

Regressions: 4 emitTerminalRefreshTimeout spy tests (exactly-one emit for restart-budget / queue-expiry; zero for a surfaced RpcTimeoutError / ConnectionError / generic error) plus 6 decideRefreshRetryAction routing tests.

Validation: lint, compile-tests, and the full unit suite (1736 passing / 5 pending) all green.

* @returns A promise that when resolved gets the result from running the worker function.
*/
addToQueue(item: T, position?: QueuePosition): Promise<R>;
addToQueue(item: T, position?: QueuePosition, expiresAt?: number): Promise<R>;

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning · Non-blocking recommendation

expiresAt is a raw timestamp interpreted by the pool's injected clock, but the WorkerPool interface does not couple the two. A caller using a different clock domain (Date.now() versus performance.now()) could expire tasks immediately or never; please make this clock-bound or use a relative duration.

// calls share a single long-lived process — so unbounded parallelism would cause CPU/memory
// pressure. Process in batches of CLI_RESOLVE_CONCURRENCY.
const retainRemainingUnresolved = (fromIndex: number): void => {
const remaining = toResolve.slice(fromIndex);

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning · Non-blocking recommendation

Please add coverage that drives CLI fallback through budget exhaustion between resolve batches and asserts every remaining discovered environment is returned raw. This branch implements the stated non-truncation guarantee, but current tests exercise only the generic retention helper rather than the CLI fallback path.

@StellaHuang95 StellaHuang95 added review-auto:approved Automated review: no blocking findings (approval posted). and removed review-auto:changes-requested Automated review: posted blocking findings to address. labels Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant