fix: harden ip table failover and config verification under extreme networks#12529
fix: harden ip table failover and config verification under extreme networks#12529originalix wants to merge 20 commits into
Conversation
| .filter(([, latency]) => latency !== Infinity) | ||
| .toSorted((a, b) => a[1] - b[1])[0]?.[0]; | ||
| if (bestMeasuredIp) { | ||
| await this.backgroundApi.simpleDb.ipTable.updateLastBestIp( |
There was a problem hiding this comment.
当测速基于旧 domainConfig 运行期间,新的已验签配置可能已经保存并移除了某个 endpoint。这里仍会把旧测速得到的 IP 写回 lastBestIp,后续还可能写成 selection;selectionGeneration 不会因 saveConfig() 和 pruning 而变化,所以无法拦截这次陈旧写入。
请在测速开始时记录配置 hash/version,并在写入前重新读取当前配置;配置已变化或候选不再属于当前 endpoint 集合时,丢弃本轮结果并重新测速。
| outcome: { ok: boolean; startedAtMs: number }, | ||
| ): 'applied' | 'stale' { | ||
| if (outcome.startedAtMs < entry.lastOutcomeAt) { | ||
| return 'stale'; |
There was a problem hiding this comment.
当两个并发请求在同一毫秒启动时,它们会得到相同的 startedAtMs。如果较晚请求先成功、较早请求随后慢失败,这里的严格 < 判断仍会应用旧失败并重新增加计数,导致 outcome ledger 不再与请求顺序收敛,可能误触发 fail-open 或测速。
请在 transport 启动时分配 runtime 内单调递增的请求序号,并使用该序号排序 outcome,不要只依赖毫秒时间戳。
| // A misconfigured CDN edge can return HTML or a truncated body that | ||
| // still parses; reject it here so the log points at the delivery | ||
| // layer instead of a bogus "signature verification failed". | ||
| if (!isValidIpTableRemoteConfigShape(remoteConfig)) { |
There was a problem hiding this comment.
当 captive portal 或异常 CDN edge 以 HTTP 200 返回 HTML、空内容或结构不完整的数据时,请求不会抛异常,而是进入这个分支后直接返回 null。这样 fetchRemoteConfigViaSniFallback() 完全不会执行,即使 builtin SNI 候选能够取得正确配置,首次安装设备仍无法恢复。
请将空响应和 shape validation 失败统一转入 fetchRemoteConfigViaSniFallback();代理是否允许绕行继续由 fallback 方法里的 preflight 决定。
Background
Production incident on 2026-07-16 (Slack #C09JEQE8P2L): on mainland-China Wi-Fi without a proxy, tapping a ticker left every request loading forever. Log analysis confirmed two independent root causes:
@ethersprojectsegment (seg:nm.@ethersprojectis owned bymainin the split-bundle manifest); a catch-all swallowed the error and reported it as "signature verification failed", misleading the investigation toward the CDN. (The segment-ownership fix lands in a separate branch; this PR covers the observability and resilience side.)Changes (by commit)
decideEndpoint: reachability beats latency (real-traffic failures override the 30% threshold), symmetric 30% hysteresis against flapping; includes an incident-scenario regression testcurrentSelection === undefineddeadlock that could block speed tests forever; persists lastBestIp on every speed testno_configgap (60 occurrences in the incident log); HTTP error responses don't count; failing requests are never replayed (wallet POST double-send risk); recovery on first domain successverifier_load_failed/signer_mismatch/malformed_signature/missing_signature) — this log line alone would have pinpointed the incident on day one; canonicalization now uses an allowlist of signed fields (extra server-side metadata no longer breaks verification — validated against the real signer); shape guard on the CDN responseconfig.onekeycn.comis fetched through the adapter-backed client instead of plain axios, giving the config-delivery path the same wall resistance as business trafficendpointSwitched/adapterFailover/configVerifyFailedLogToServer events (no PII), so regional failover and verify-failure rates are visible without asking users to upload logsdisableIpTableFailoverdev setting + Dev Settings UI entry — the new behaviors can be turned off in production without a rollbackTesting
incident regression: domain failing real traffic beats 30% threshold,verifier import failure -> verifier_load_failed (split-bundle incident)yarn agent:check --profile prlocal checks all passOut of scope (tracked separately)
@ethersproject+ the mirrored sentry issue) — separate task/PRRisk & verification suggestions
The largest risk surface is the adapter fail-open (shared by appApiClient / analytics / hardware configFetcher). Constraints: only transport-level errors count, threshold 3 + 5min TTL, exits on first success, kill switch available. Suggested QA: regress the core request paths both with the direct domain blackholed (hosts entry for wallet.onekeycn.com) and on a normal network to confirm no behavior change in the healthy case.