Skip to content

Owner-managed update lanes (dsh desktop / pi / profile converge) retry every 3 min with no InstallBackoff — desktop lane re-downloads the tarball on every retry #2192

Closed

Description

@ranxianglei

Summary

The #1603 InstallBackoff machinery only protects the global self-update path. The owner-managed lanes — refreshDshDesktopCopy, refreshPiNpmCopy, refreshDshProfileCopy, and the convergeDshProfileBundles driver — bypass it entirely: a persistently failing lane retries every check cycle (default 3 min) forever, and the desktop lane re-downloads the full tarball on every attempt.

Mechanics

  • checkForUpdate returns early for host-managed installs (src/update.ts:1047) → refreshOwnerManagedCopies (src/update.ts:923) → refreshDshProfileCopy + refreshPiNpmCopy; refreshDshDesktopCopy (src/update.ts:834) is additionally driven from convergeDshProfileBundles (src/update.ts:988 / :819).
  • The backoff key check and recordInstallFailure (src/update.ts:1117 region) sit on the global path only, after the early return — owner lanes never touch them.
  • refreshDshDesktopCopy documents the behavior itself: "failures log and retry next cycle; never throws" (src/update.ts:830-833).
  • Desktop lane is the only one that burns bandwidth per retry: it calls installViaTarball (src/update.ts:854 → :1271), a raw fetchWithEgress into a temp dir — no pnpm store, no cache. A machine whose desktop smoke / egress persistently fails re-downloads the tarball up to ~480×/day at the default 3-min cadence.
    • The dsh profile lanes are bandwidth-safe by accident: dsh plugin add → pnpm add hits the pnpm content-addressable store, so repeat attempts don't re-download. They still spin (subprocess every 3 min) but don't inflate npm stats.
  • dsh中工具会突然消失 #2082's bootSmoke: true (shipped v0.1.184, 2026-10-04) widened the failure surface: a persistent boot-smoke failure now keeps the lane out-of-step, which is exactly the retry-forever precondition.

Trigger conditions: desktop profile dir exists + diskVersion < latest + a persistent failure (smoke fail, egress fail, missing dist.tarball).

Impact

  • User bandwidth: up to ~480 × ~23 MB/day per affected desktop machine.
  • Fleet-visible side effect: repeated registry tarball GETs inflate npm download counts (surfaced while auditing the 10/04 download spike — 101k downloads/day, single-day k ≈ 4.4k/latest-reign-hour vs ~2.0k on 10/02; how much of the 10/04 excess is this loop vs lane propagation is unresolved — 10/05's single-release count is the discriminator: ~45–55k clean vs 70k+ = loop share is real).
  • We reproduced the retry shape locally: a VM whose desktop profile had a broken pnpm layout (ERR_PNPM_VIRTUAL_STORE_DIR_MAX_LENGTH_DIFF) sat out-of-step with the warn log firing every cycle until the layout was repaired by hand.

Proposed fix (design — not yet implemented)

  1. Route owner-lane attempts through the same InstallBackoff, keyed owner:<lane>:<version> (e.g. owner:dsh-desktop:0.1.185); recordInstallFailure on !result.ok, clear on success. Owner lanes are non-critical — a more conservative series (e.g. 15 min → 6 h cap) is fine.
  2. Owner-lane freshness does not need the global 3-min cadence: give host-managed instances their own check interval (30–60 min) with jitter.
  3. After N consecutive failures on one lane, surface one actionable warning (reuse the warnOnce/dedupe machinery from cb1e978) naming the lane and the manual fix, instead of a silent retry loop.
  4. Tests: hermetic suite pinning that (a) a failing desktop refresh is a no-op within the backoff window, (b) success resets the key, (c) profile/pi lanes regress nothing.

Verification after landing: npm daily downloads for the package should drop by the repeat-share; the desktop-lane warn log goes quiet on broken machines.

Related: #1603 (backoff machinery), #1575 (desktop in-place lane), #2082 (boot smoke), #1803/#1804 (profile converge), #1196 (profile self-refresh).

Activity

  1. ranxianglei commented on Oct 5, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 收到,先报验证结论,随后开工。

    Issue 声称的机制已全部对照代码核实(属实):

    • checkForUpdate 对 host-managed install 在 src/update.ts:1041-1049 提前 return → refreshOwnerManagedCopies(:923),而 auto-update installs but never activates on long-lived plugin hosts; unwritable install dir retries every 3-min cycle for 11+ days #1603 的 backoff key 检查在 :1117-1120、recordInstallFailure 在 :1144——都在 early return 之后,owner lanes 完全绕开。
    • refreshDshDesktopCopy(:834)每次尝试都走 installViaTarball(:854)→ 裸 fetchWithEgress 重下 tarball,无 store/缓存;注释自述 "failures log and retry next cycle"。
    • desktop lane 有三个驱动点:profile-copy 自检(:819)、convergeDshProfileBundles(:988)、全局安装成功后(:1141)。
    • 另核实一个连带事实:throttle marker(THROTTLE_FILE)是机器全局共享的,混合机器上 profile copy 自己的周期检查今天就被全局实例的 3-min 写入饿死——所以「owner lane 独立 cadence」需要独立的 throttle marker,不能只改间隔常量。

    实现计划(按 issue 方案 1-4,细节开工后在 PR 里展开):backoff 状态从单键对象改为多键 map(key 形如 owner:dsh-desktop:<version> / owner:dsh-profile:<version> / owner:pi-npm:<version>,全局路径沿用现有 key);三条 owner lane 的「一次尝试」统一过 backoff 门;host-managed 实例独立 throttle marker + 30-min 基线 + jitter(硬编码常量,不加新配置面);lane 专属 remediation hint 复用 cb1e978 的 warnOnce 机制;hermetic 测试按 (a)(b)(c) 三点钉死。

    src/update.ts 属 load-bearing infra(§7.4),本 PR 不会 auto-merge,走人工 review。

  2. ranxianglei commented on Oct 5, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 开发完成,PR 已开:**#2206 #2192)。issue 里的机制描述逐条核实无误(行号引用全部对上了),按「设计」节的四项全部落地。

    结构化七问

    1. 是什么问题:三条 owner 托管更新通道(dsh desktop 原位副本 / dsh profile bundle / pi npm 副本)完全绕过 auto-update installs but never activates on long-lived plugin hosts; unwritable install dir retries every 3-min cycle for 11+ days #1603 的 InstallBackoff —— 持续失败的通道每个 3 分钟周期永远重试,其中 desktop 通道每次重试都重新下载整个 tarball(~23 MB,每台坏机器每天最多约 480 次)。
    2. 什么原因导致的:backoff 键检查与 recordInstallFailure 只存在于全局自更新路径上、且位于 checkForUpdate 的 host-managed 提前返回(src/update.ts:1047)之后;owner 通道走独立函数,从不触碰该状态——原注释甚至明写 "retrying next cycle"。叠加因素:host-managed 实例与全局共用同一个 3 分钟节流标记,没有独立节奏。
    3. 怎么修复的:① backoff 状态由单值改为按 owner:<lane>:<version> 分键的 Map,复用既有退避序列(3 连败 → 5 min 指数翻倍 → 6 h 封顶),每条通道的升级提示改为指明具体通道与手动修法(重建 desktop profile / dsh plugin --profile <name> add … / pi update --extension …+BILI_PI_BIN);所有通道在下载/spawn 前过冷却门,失败记击、成功清键、新版本重新武装;② host-managed 实例改用独立节流标记 + 30 min 基础 + ≤15 min 抖动的慢节奏;③ profile bundle 刷新改为返回 {refreshed, failed},全败时驱动层记一次通道击数(部分成功不算败,不浪费重试预算)。
    4. 是否有回归问题:未发现。审过的既有行为:各通道原有 warn/success 日志行保持字节一致(被既有测试套件钉住)、全局路径行为不变(update-backoff 套件原样通过)、advisory 强制安装路径不变(仅把 bundle 刷新调用点换到包装器)、plugin-install 成功消息仅当 failed>0 才追加 "(N failed)"。rebased head(v0.1.185)上定向验证:typecheck/build 干净,12 个相关套件 100/100 通过;全量回归交给 CI。
    5. 是否破坏缓存:不涉及压缩/前缀缓存。仅新增一个本地节流标记文件(.owner-update-check,在既有 cache 目录下);lastUpdateCheckTime() 现取两个标记的较大值(状态上报更准,不影响节流判定本身)。
    6. 是否改了设计方向:无架构转向 —— 修法是让 owner 通道接入现有 auto-update installs but never activates on long-lived plugin hosts; unwritable install dir retries every 3-min cycle for 11+ days #1603 机制(同一机制的扩展),未引入新子系统;单写者契约(native模式下自动更新问题需要解决 #991/dsh plugin-market installs never auto-update: the refresh chain's only trigger is a global bili self-update #1196)不变。
    7. 是否引入了新的配置:无新增配置项/env/CLI 参数(PR 中已按 config-surface 纪律说明理由);逃生口仍是 "autoUpdate": false / ACP_AUTO_UPDATE=0。

    有意行为变更(旧 → 新,详见 PR 描述表格):a) owner 通道 3 分钟无限重试 → 三连败后指数退避至多 6 h;b) host-managed 实例共享 3 分钟检查 → 独立 30–45 min 抖动节奏;c) 全局安装成功原清除全部 backoff 状态 → 现只清自己的键(各通道相互独立);d) 失败的 owner 通道周期多一行退避进度日志(原有告警行不动);e) lastUpdateCheckTime() 取双标记最大值。

    合并后验证(按 issue 要求):发布后观察包日下载量是否降掉重复份额(判别式即 10/05 单版本发布计数:~45–55k 干净 vs 70k+ = 循环份额真实存在),以及坏机器上 desktop 通道 warn 日志是否安静下来。

    注:改动触及 src/update.ts(load-bearing infra),按 §7.4 需人工 review。CI 跑全量回归中,红了我会定向排查。

  3. ranxianglei commented on Oct 5, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 CI 更新:PR #2206(commit afbdce2,base v0.1.185)19/19 全绿 —— 含全量 test 矩阵(ubuntu/windows 各分片)、hermetic registry e2e、kernel-guard / version-guard / docs-sync / docs-coverage、build-artifact 及全部 real-client lane。无回归发现。

    按 §7.4(src/update.ts 属 load-bearing infra)等待人工 review 后合并;合并后请按上条所述观察 npm 日下载量与坏机器 desktop 通道日志验证效果。

  4. added 3 commits that reference this issue on Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions