Skip to content

fix(runtime): add bounded retry to managed node version probe - #771

Merged
piorpua merged 2 commits into
mainfrom
aionissue/fix-2.1.45-138401321-003
Aug 4, 2026
Merged

fix(runtime): add bounded retry to managed node version probe#771
piorpua merged 2 commits into
mainfrom
aionissue/fix-2.1.45-138401321-003

Conversation

@piorpua

@piorpua piorpua commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes a false "AionUi installation incomplete / reinstall / check antivirus" alert triggered by a single transient external-process failure during managed Node validation. In the reported incident, npm --version exited with code 7 once; the bundled runtime was actually healthy and activated ~38.8s later, so the alert was misleading and the reinstall/antivirus guidance was useless to the user.

Root cause: command_version ran the managed Node --version probe once with no retry, so a transient exit was latched into a permanent bundled_resource_invalid failure event that AionUi surfaced to the user and reported to Sentry.

Fix: Bounded retry on the --version probe, scoped entirely to AionCore. On retry success, install_and_validate_with_reporter returns Ok and never broadcasts phase=Failed, so the UI never shows the alert and never reports. classify_error and AionUi are intentionally untouched — this is Option 1 from the approved fix spec.

Changes

  • crates/aionui-runtime/src/node_runtime/mod.rs: split command_version into a thin wrapper plus probe_command_version_with_retry (generic, closure-injected, unit-testable, no process spawn) and command_version_once (the original body, verbatim). The public signature of command_version is unchanged, so all node/npm/npx call sites are untouched. New private constants VERSION_PROBE_ATTEMPTS = 3 and VERSION_PROBE_BACKOFFS = [250, 500, 1000]ms (~1.75s worst case), independent of the existing copy-activation retry budget. Adds 4 inline unit tests that inject a closure and never exercise command_version_once.
  • crates/aionui-runtime/src/node_runtime/managed/tests.rs: adds two #[cfg(unix)] tests that spawn real shell-script stand-ins for node/npm/npx (npm exits 7 for the first N invocations, then prints a version) and drive the real call chain command_version -> validate_runtime -> validate_managed_runtime -> bundled activation path. One covers a transient failure inside the retry budget (expects success, zero Failed phases reported); the other covers a persistent failure beyond the budget (expects bundled_resource_invalid still reported). Test-only, no product code changes.

Scope boundaries (intentional)

  • classify_error (managed.rs) and all AionUi files are deliberately unchanged. Persistent failures (probe failing for the full ~1.75s budget) still classify as bundled_resource_invalid and still alert/report, to avoid under-reporting real failures.
  • Retry semantics: every failed attempt (including the last) sleeps its backoff, matching activate_copy_with_retry; a persistent failure spends the full ~1.75s before the verdict. Only transient error classes (process fails to start, non-zero exit, non-semver output) reach the retry loop — "node major too low" is decided in validate_runtime, outside the probe, and is never retried.
  • Logging: one warn only when a retry will follow (fields label/attempt/max_attempts/error); the budget-exhausted final failure is left to the existing caller warn in managed.rs to avoid duplicate logging; no info on retry-success; no sensitive payloads.
  • Known residual (accepted, documented in the fix spec): the false alert is not fully eliminated — 3 consecutive probe failures within ~1.75s plus no node ready within AionUi's 15s reconcile window will still alert/report. Further reduction is out of scope for this fix.

Related repositories

  • AionUi: no changes required for this fix. The existing reconciler, alert allowlist, and RuntimeFailureKind type are intentionally untouched.
  • aionrs: not involved.

Test plan

  • cargo test -p aionui-runtime version_probe — 4 new closure-injected unit tests pass (RED before implementation, GREEN after).
  • cargo test -p aionui-runtime — full crate regression, 94 passed / 0 failed, including unchanged classification tests (classify_error_still_detects_bundled_invalid_by_message, classify_error_detects_bundled_node_runtime_missing, bundled_runtime_missing_reports_bundled_resource_missing).
  • cargo fmt --all -- --check — clean.
  • cargo clippy -p aionui-runtime -- -D warnings — clean.
  • just push — full pre-push gate (migration check, lint, format, workspace test suite: 8119 tests run, 8119 passed) — clean.
  • The two new #[cfg(unix)] subprocess-driven tests (added in the second commit) are not yet covered by a logged CI run against this exact HEAD in the fix archive; they ran as part of the just push workspace suite above but are worth a second look in CI given they are Unix-only and won't run on Windows, where the original incident occurred. The retry logic itself is platform-independent and is covered on every platform by the version_probe_* unit tests.
  • End-to-end confirmation that the user-facing alert and the Sentry runtime-installation-integrity-failure report no longer fire on a transient-but-recoverable probe failure in a real/packaged environment is out of automated scope for this fix and has not been executed.

zynx added 2 commits August 4, 2026 14:10
A single transient external-process failure during managed Node
validation (process fails to start, a non-zero exit such as npm exit
code 7, or non-semver output) was latched into a permanent
bundled_resource_invalid failure event, surfacing a misleading
"installation incomplete / reinstall / check antivirus" alert even
though the bundled runtime was healthy and activated moments later.

Wrap command_version's single probe (command_version_once) in a bounded
retry helper (probe_command_version_with_retry): 3 attempts with
250/500/1000ms backoff (~1.75s worst case), aligned with
activate_copy_with_retry. All three transient error classes are retried;
real failures (node major too low) are decided in validate_runtime,
outside the probe, so they are never retried. A warn is logged only when
a retry will follow; the budget-exhausted final failure is left to the
existing caller warn to avoid duplicate logging. command_version keeps
its signature, so node/npm/npx call sites are unchanged. classify_error
and AionUi are intentionally untouched.
The four version_probe_* tests all inject a closure into
probe_command_version_with_retry, so command_version_once never runs and
the retry is never exercised through the call chain. If the retry wrapper
were detached from command_version (calling command_version_once
directly), all four would still pass — the regression that matters is
invisible to them. AC1's second half ("does not report phase=Failed via
the reporter") had no coverage at all, even though that event is what
makes AionUi show the installation-integrity alert.

Add two #[cfg(unix)] tests driving the real chain — spawn a process, get a
real exit code 7, through command_version -> validate_runtime ->
validate_managed_runtime -> the bundled activation path — sharing one
fixture: a minimal bundled managed-Node tree whose bin/npm exits 7 for
the first N invocations and then prints a version. The attempt counter
lives in a file because every probe attempt is a fresh process, as in
production.

- transient_npm_version_failure_is_absorbed_without_failed_report:
  2 failures inside the budget -> Ok, version 24.11.0, source Bundled,
  exactly 3 npm invocations, and zero Failed phases on the reporter.
  Verified as a real guard: with VERSION_PROBE_ATTEMPTS forced to 1 it
  fails with this issue's own error, "bundled Node runtime failed
  validation ...: npm exited with exit status: 7".
- persistent_npm_version_failure_still_reports_bundled_resource_invalid:
  failure outlasting the budget -> spends exactly VERSION_PROBE_ATTEMPTS,
  still errors, and still reports bundled_resource_invalid, so the fix
  cannot swallow a genuinely broken install.

Unix-only because the fixture needs executable shell scripts as
node/npm/npx stand-ins; the retry logic is platform-independent and the
version_probe_* tests cover it everywhere. Product code is untouched.
@piorpua
piorpua merged commit 1dee9f1 into main Aug 4, 2026
6 checks passed
@piorpua
piorpua deleted the aionissue/fix-2.1.45-138401321-003 branch August 4, 2026 08:57
kaizhou-lab pushed a commit that referenced this pull request Aug 4, 2026
🤖 I have created a release *beep* *boop*
---


##
[0.1.58](v0.1.57...v0.1.58)
(2026-08-04)


### Features

* **conversation:** agent-driven session auto-naming (ACP
session_info_update + claude generate_session_title)
([#768](#768))
([c551c66](c551c66))
* **conversation:** fork a conversation into a new one at a chosen
message ([#772](#772))
([bb7c14e](bb7c14e))
* **fs:** add ChatFileRef content endpoints
([#757](#757))
([e8b6f4c](e8b6f4c))
* **fs:** pdf stream endpoint + office ChatFileRef resolve + retire
fs/resolve & WS fs/read
([#762](#762))
([e4e991a](e4e991a))


### Bug Fixes

* **agent:** keep the thought-level picker on a resumed conversation
([#763](#763))
([dcf015e](dcf015e))
* **runtime:** add bounded retry to managed node version probe
([#771](#771))
([1dee9f1](1dee9f1))
* **session:** keep claude session cost cumulative across process
respawns ([#767](#767))
([f498e9d](f498e9d))
* **session:** settle cards through teardown and resume so no stored row
spins forever ([#766](#766))
([3f61cc4](3f61cc4))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant