Skip to content

[wire lane] First-sight emergency distillation marathons (5-15 min serial preflight stalls) - wire-mode churn mints a new proxy session per resume/history rewrite #3452

Closed

Description

@MojitoJoe1813

[wire lane] First-sight emergency distillation marathons (5–15 min serial preflight stalls) — wire-mode churn mints a new proxy session per resume/history rewrite

Summary

Running bili in the /bili/ URL-prefix lane in front of a plain-HTTP local OpenAI-compatible engine, driven by Hermes Agent (desktop app), every session resume and every structural history rewrite mints a new proxy session (pfa-…). The new session sees the entire current history at once ("first-sight"), usage lands at 186–398% of the window, and the emergency path runs a serial preflight distillation of all ranges locally, before forwarding — measured 25 s to 14.9 min of total client stall per event, with the request held alive only via SSE keep-alive comments.

Incremental folding never fires in this shape of session, so the emergency path is the only active compression path. This makes the product experience worse than the host's own summarization for interactive use.

Environment

  • bili v0.1.196 (pinned; autoUpdate:false, advisoryCheck:false), npm user-prefix install, daemon on loopback :8787
  • Client: Hermes Agent (desktop), Python httpx, OpenAI chat-completions wire
  • Upstream: plain-HTTP llama.cpp-style server on a LAN host (http://<LAN-host>:8080/v1), model swift-flash, MoE, n_ctx 204800, output headroom 16384 → window denominator 188416
  • Decode speed ~150 tok/s, cold prefill ~4–5K tok/s (typical for local GGUF/MoE rigs)
  • bili config: defaults for the compress block (nudgeGrowthTokens 50000, maxContextLimit "75%", emergencyThresholdPercent "95%"), modelContextLimit 204800

Observed — one day, 7 Hermes sessions

Every resume of a session with history above the hard wall produced one emergency block:

proxy session trigger ranges distilled wall time
pfa-5aab… preflight 1 range (205740→55496) 1 25 s
pfa-8b61… EMERGENCY T2 (389121→60407) 4 5.7 min
pfa-f81f… EMERGENCY T2 (435326→64902) 7 11.0 min
pfa-4768… EMERGENCY T1 (327451→63609) 5 7.7 min
pfa-20b4… EMERGENCY T2 (323342→65881) 8 5.5 min
pfa-df7b… EMERGENCY T2 (416827→61579) 14 8.1 min
pfa-1f12… EMERGENCY T2 (503388→61711) 19 14.9 min

bili.log recorded 17 distinct new session pfa-… mints for ~7 client sessions in one day — churn factor ≈ 2–3 per conversation.

Log evidence (verbatim, host redacted)

16:26:46 nudge INJECT T1: usage=279% (525808/188416), growth=0/22500 (ref=525808, interval=50000),
         pendingT1=126606/50000 model=swift-flash,
         reason="EMERGENCY T1: max effective pending 126606, usage 279% [first-sight mass]"
16:32:29 nudge INJECT T2: usage=279% (525808/188416), growth=374662/22500 (ref=151146, interval=50000),
         pendingT1=11737/50000 model=swift-flash,
         reason="EMERGENCY T2 distill: max pending 12974 (T1 effective 11737, T2 12974, T3 0), usage 279%"
16:32:29 preflight compressed 4 range(s), ~405800 tokens saved (389121 → 60407) in 343380ms
18:56:34 request: 262 msgs, inbound=938.4KiB, outbound=265.8KiB, local=483451ms
18:56:44 stream silent 15000ms — holding client with SSE keep-alive comments past its undici body timeout (#1647)
19:31:59 preflight compressed 19 range(s), ~614799 tokens saved (503388 → 61711) in 893779ms

Between local=<ms> and the first forwarded token the client sees only "waiting for the first stream chunk" — from the user side this is indistinguishable from a hang.

Mechanism as we understand it (please correct)

  1. Growth layer: T1 needs ≥ one full step (50000) of compressible mass in a single foldable range. Agent sessions shaped as many small tool results never reach that per-range gate (pendingT1 observed at 5–17K persistently) → incremental folding goes silent. CONFIGURATION.md ("nudgeGrowthTokens") itself documents this failure mode ("incremental folding goes silent … falls straight into the multi-round preflight summarization marathon").
  2. Wire lane has no stable session identity: every resume / host-side history rewrite (the host's own compaction, compress tool calls, forks) changes the byte-exact prefix → new anonymous proxy session → full-history first-sight.
  3. First-sight with history > hard wall → EMERGENCY T1/T2 → serial preflight distillation of all ranges before forwarding. Each range ≈ cold prefill + summary decode (~35 s on a 150 tok/s model); 14–19 ranges ≈ 8–15 min. The fold state of the previous proxy session is lost when it churns, so the same history pays the marathon again after every rewrite.

Impact

  • Interactive use becomes unusable for long sessions: multi-minute blackouts at exactly the moments users expect continuity (resume after idle).
  • The value proposition inverts: amortized compression turns into concentrated stalls worse than the host's native summarization.

Asks

  1. Is the identity pipeline (Identity pipeline: key sessions by (agent, conversation id) in the unified store — decouple fold ledgers from protocol/host origin (#2938 / #2847 / #2671 defect-1) #2960, R2/R2.1 in v0.1.197) intended to cover the wire-prefix lane (no client headers), i.e. can first-sight marathons be eliminated for header-less clients like httpx-based agents? We are ready to re-measure against a build with a checklist: count new session pfa- per resume, count preflight compressed … in <ms> events.
  2. Graceful degrade for the preflight marathon path ([resilience] Preflight compress/timeout loop on sustained mass fold-coverage loss has no graceful degrade #2438): would a bounded/parallel/staged distillation with a wall-time budget (e.g. forward partially-reconciled requests after N seconds, continue in background) be acceptable direction? Our data suggests serial preflight is the dominant UX failure on slow-decode local engines.
  3. Recommended defaults for slow-decode local rigs: is lowering nudgeGrowthTokens (we tested reasoning toward 25000) the sanctioned lever, and does [Kernel] decideNudge(): growth-path T1-first ordering structurally starves T2/T3 distillation nudges when t1Eff >= nudgeGrowthTokens | [内核] decideNudge(): growth-path T1-first ordering structurally starves T2/T3 distillation nudges when t1Eff >= nudgeGrowthTokens #2513 (T1-first starvation) interact with it? What floor is safe given auto-fold sharp edges: a sub-floor autoFoldTargetTokens causes a fold storm, and externalSummary.* file edits do not take effect until a PUT /__bili/config #3444 (sub-floor autoFoldTargetTokens fold storms)?
  4. Would a per-session "first-sight budget" (cap ranges distilled at first sight, defer the rest to incremental folds) be a reasonable feature request?

Related

#2960 (identity pipeline), #2938 (per-request identity side-channel), #2438 (preflight graceful degrade), #2513 (decideNudge T1 starvation), #2638 (range re-nomination), #3399 (cold start ledger loss + 413), #3434/#3441 (silent stall visibility), #3444 (fold-storm sharp edges).

Happy to provide the full bili.log (redacted) and the stats dumps (/__bili/stats, /__bili/sessions) — and to run verification passes against candidate builds on this exact rig.


Disclosure: This issue was drafted by an AI assistant (Hermes Agent) operating the test rig on behalf of its human owner. All measurements and log excerpts are verbatim from the real bili.log of this setup (private host redacted); the human reviewed and submitted it. Happy to answer follow-ups through either of us.

Activity

  1. ranxianglei commented on Oct 16, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    查重: 不重复。 Searched open+closed issues/PRs by failure signature (first-sight, pfa- minting, preflight marathon duration, prefix-affinity wire lane): no same-symptom+same-mechanism entry exists. Nearest neighbours, all different mechanism: #2438 (graceful-degrade design for the preflight DOOM loop — same symptom family but its failure domain is hopeless substrates ≥8× window; this instance is convergent-but-slow at 1.9–4×), #2960 (storage keying for explicit-ID lanes), #3444 (autoFoldTargetTokens below baseline fold storm — different knob, different path), #2513 (T1-first starvation — already fixed on master via #2519 slot rotation, commit 81f6e1ca1). Per the granularity policy, the sub-facets you list stay managed here as checklist items, not split issues.

    I verified your mechanism analysis against current master (v0.1.197-369-gce9bc90a3). All three load-bearing claims hold in code:

    Churn → first sight (mechanism 1): confirmed. Wire-lane identity IS anonymous prefix affinity (src/prefix-affinity.ts:531+). Reattachment requires a byte-exact head-anchored match over the neutral projection, plus two narrow adoption rungs: append-continuation (#2807, assistant tail-run extension only) and simhash near-identical adoption (#2265, enabled by default, decorative mutations only — same length, small per-position Hamming distance, unique candidate required, 24 h chain TTL). Any structural host-side rewrite (compaction, edit, delete, regenerated system prompt on relaunch) breaks the chain at some position and mints a fresh pfa- session — by design (#1262 fork discipline). Note: a pure byte-exact resume does reattach fine, so your observed churn means hermes mutates something across relaunch; a redacted log around one mint event would pin down which bytes (§Requests below).

    Serial preflight before forwarding (mechanism 2): confirmed. Multi-range batch folding (up to 4 ranges concurrent via the executor pool, MAX_PREFLIGHT_SUMMARY_BATCH = 4) is explicitly "gated on the external lane (the legacy streaming lane has no batch executor)" (src/preflight.ts:2055-2069). With no externalSummary chain configured, every range is summarized strictly serially against the same engine serving the conversation — that is your 25 s–14.9 min wall.

    Answers to the four Asks

    Ask 1 — Does #2960 R2/R2.1 cover the wire prefix lane? No. R2/R2.1 is storage-layer keying for explicit-ID lanes (unified store keyed by agent namespace + conversation id; protocol/host demoted to metadata columns). It cannot change wire-lane identity because a headerless client presents no id to key on — anonymous prefix affinity is the wire-lane identity design, and the churn you see is its intended behaviour for diverging histories. The sanctioned fix specific to Hermes is the native plugin lane: bili plugin install hermes stamps x-bili-plugin-conversation: <hermes session_id> on every LLM request once tools are ready (hermes-plugin/__init__.py:879-880); the proxy then keys the session on hermes' own id, so the fold ledger survives resumes and history rewrites — drift inside a session is handled by reconciliation (worst case it unfolds, but it never starts from zero like a fresh pfa-). Round 1 of a brand-new session still rides wire mode, but at that point history is just system + first turn, so the marathon class disappears in practice. One caveat: whether hermes' session_id itself persists across app restarts is host-side behaviour this repo can't verify — your measurement checklist works unchanged against the plugin lane (count mints after switching; if mints still track restarts, hermes rotates ids and the fix belongs on the hermes side or in the #2938 side-channel). Extending the anonymous adoption ladder (mid-chain anchor matching etc.) is identity-layer design work — owner territory, and not recommended while the plugin lane exists.

    Ask 2 — Graceful degrade direction. Aligned with #2438's F1 bound-segment philosophy (wall-clock budget, guaranteed progress per pass, fast-fail once exhausted) — but note your instance sits below #2438's entry gate (payloads 1.86–3.98× window vs the ≥8× hopeless-substrate gate), so it is a distinct sub-facet of the same symptom family: convergent-but-slow digestion on slow local engines. Options:

    • (a) per-pass wall-clock budget with staged continuation (your proposal) — reasonable; needs a default (~30–60 s for this rig class) and must preserve monotonic convergence (folds persist, so each continued pass strictly shrinks the remainder);
    • (b) cap the number of ranges distilled per first-sight pass (= Ask 4) — the cheaper variant, bounds the stall directly;
    • (c) available today, zero code: point externalSummary at a fast small model (second local slot or a remote API). This unlocks the batch pool (4-way concurrency per round) and a faster decoder — both axes of the marathon shrink at once. On a single-slot llama.cpp rig, decoder speed matters more than concurrency.

    Recommendation: ship (c) as the documented remedy now; treat (a)+(b) as one design decision (owner).

    Ask 3 — nudgeGrowthTokens for slow-decode rigs. Yes, it is the documented lever — CONFIGURATION.md:1218 names exactly this failure mode ("incremental folding goes silent and an over-window request … falls straight into the multi-round preflight summarization marathon instead of folding early"). Verified kernel defaults: growthFloor = max(minGrowthFloor = 20000, minGrowthRatio = 0.45 × step) (kernel/src/config.ts:18-19), T1 mass trigger = step. Two levers:

    • nudgeGrowthTokens: 25000 — halves the T1 mass trigger, the cadence floor (max(20K, 11.25K) = 20K) and the first-sight mass bypass;
    • finer-grained: pin compress.tierNudgeTokens.t1: 25000 alone — lowers only the T1 mass trigger, leaves the cadence floor at 22.5K.

    Interaction with #2513: resolved — slot rotation landed on master (#2519), so count-ready T2/T3 no longer starve under high t1Eff. Safe floor: nothing structurally storms the nudge path the way #3444 does — that one is autoFoldTargetTokens sitting below the irreducible baseline (preflight auto-fold target vs nudge growth step: different mechanism; the nudge path carries its own anti-thrashing — cadence floor, pending-threshold halving, minPressureBenefit gate #198). Practical floor ≈ 20000: below it, minGrowthFloor dominates the growth gate so you buy less cadence while paying more summary-call overhead and prefix-cache churn per fold. For a 188K window I'd start at 25000. Important nuance: the pressure path (usage ≥ 75%) does NOT depend on nudgeGrowthTokens at all — lowering it only tightens the growth path, which is precisely what keeps payloads near the wall so first-sight mass stays smaller.

    Ask 4 — First-sight budget. Reasonable feature request. It is new config surface, so owner-gated — manage it here as a checklist sub-item (same symptom family). Sketch: cap LLM-distilled ranges per first-sight pass (e.g. N oldest full-budget chunks, N ≈ 4–6), defer the rest to subsequent incremental passes; must preserve (i) monotonic convergence and (ii) the EMERGENCY_FOLD_COVERAGE digest fallback when overshoot exceeds 19.2× window (src/preflight.ts:142).

    Requests

    @MojitoJoe1813:

    1. Try the native plugin lane (bili plugin install hermes) and re-run your mint-count checklist against it — that directly tests Ask 1 on your exact rig.
    2. Send a redacted bili.log covering one resume cycle (requests immediately before/after a new session pfa- mint) so we can see exactly which bytes hermes mutates on relaunch (system prompt? tool-result re-render? hermes-side compaction?).
    3. Confirm whether hermes' session_id persists across app restarts (it decides whether the plugin lane fully closes the churn).

    Owner decisions wanted (each sketched with impact):

    • D1 — wall-clock budget per preflight pass (new knob, suggested default ~30–60 s, off-by-default or arm-only-on-first-sight): changes stall behaviour when armed, no regression surface when off.
    • D2 — first-sight range budget (new knob, N ≈ 4–6): tradeoff = smaller stalls vs more total passes; quality-neutral if deferred ranges fold incrementally afterwards.
    • D3 — endorse the externalSummary-fast-model recipe as the documented remedy for slow rigs (docs-only change, zero behaviour risk).

    Bottom line: your diagnosis is correct on every mechanism I could verify in code. Highest-leverage move available today: switch Hermes to the native plugin lane (stable host id kills the churn at the source), plus an externalSummary chain to a fast model if you keep the wire lane; D1/D2/D3 are the follow-on design decisions, and I'll track them as checklist items in this issue.

  2. MojitoJoe1813 commented on Oct 17, 2026

    @MojitoJoe1813
    Author

    Follow-up from the same rig — measured attach-gate rule, and why per-session spawning is our default here

    Companion data for @ranxianglei's diagnosis in this thread (wire-mode churn: minting a proxy session per resume/history rewrite). On a plain-http local engine (llama.cpp behind a forward proxy) we now have the gate pinned down:

    1. The exact installed-lane rule. hermes-plugin/__init__.py:394 (installed as <HERMES_HOME>/plugins/billion-context/__init__.py; byte-identical to hermes-plugin/__init__.py v0.1.198):

    if inst["mitm_domains"] != want_domains: return "mitm-domains-mismatch"

    Compared against mitm_domains_from_config() (same file, line 555) = every https host among top-level providers: + legacy custom_providers: entries in config order. The TS twin in dist/index.js (instanceCompatible) demands length equality then per-index equality. Consequences:

    • an empty mitm.domains does not pass when config has any https provider row (we tested: refusal at 22:14 today on both ports);
    • a superset does not pass;
    • mitm.enabled:false kills the CONNECT handler entirely (if (opts.mitm.enabled) { setupMitm(...) }) → curl: (56) Proxy CONNECT aborted for every https rung, i.e. cloud fallback chains die;
    • there is no hot reload: opts.mitm.domains is read only at daemon start, and mitm_domains_from_config() runs at client registration. A hand-started daemon with a hand-written list goes stale silently.

    2. Why our lane is permanently stale without help. A cron watcher (chain_watcher.py → model_filter/scout_chain.py, hourly) writes fallback_providers and providers.scout-* desktop rows, so the derived list legitimately changes: current live value ["openrouter.ai","integrate.api.nvidia.com"]. Every change flips attach → spawn. Our measurements:

    • 2026-10-17 21:27:48 config write (scout run) ⇒ no new attach line afterwards, refusals resumed (6 refusals after 21:00, last attach 21:19:56).
    • spawned sessions show empty /__bili/stats on the lane port ⇒ measurement blindness for exactly the mint/session_id numbers this thread asks for.
    • first-sight distillation marathons on resume: 14 ranges ≈ 8 min, 483 332 ms of local work, stalls 25 s – 14.9 min, ~17 fresh proxy sessions per day; prefix-affinity invalidated per fold (cold prefills at 4–5K tok/s ⇒ 20–45 s silence on 100–180K histories).

    3. What we shipped as a workaround (/root/.hermes/scripts/bili_whitelist_sync.py, ~110 lines, called from the two chain-writing jobs and via ExecStartPre in the lane unit; derive via the plugin's own mitm_domains_from_config(), compare, write on difference only, restart the lane daemon, fail-open so the scout job never fails; BILI_SYNC_NORESTART=1 prevents recursive restart at unit start). Verified self-healing today: poisoned mitm.domains=["bogus.example"] → next sync run restored ["openrouter.ai","integrate.api.nvidia.com"] + lane restart → next chat attached (22:14:44) and its traffic showed up in lane stats (title "SYNCTEST3…", upstream http://192.168.178.129:8080).

    4. Ask (cheapest fix for everyone, please consider):

    • (a) Accept a superset of want_domains (and an explicitly configured empty list) in instanceCompatible / _attach_reason: a proxy serving more MITM domains cannot blind-tunnel any provider the client uses, so this looks safe to us and it removes the equality race for every daemon-managed setup.
    • (b) If strict equality stays: allow the attach path to reconcile a running daemon over its control surface — e.g. a POST /__bili/mitm/domains (the existing /__bili/* API already carries privileged routes) plus a SIGHUP config watcher, so whitelists can be reconciled without restarting the daemon mid-chat. Our timing gap is exactly this: mitm_domains_from_config() runs at registration, a hand-started daemon has the list from start; a hook that writes config alone leaves the daemon stale until restart, which costs a 2 s port blip.
    • (c) Doc line in README's plugin lane stating the exact equality rule with its failure consequence (spawn churn), so plain-http users don't have to read TS to find it.

    Our lane works now (GPU/ling chains direct, cloud rungs tunnel, 0 Model fallback rows on 2026-10-17), but it works because we added sync hooks on our side. On this rig the gate — not wire-mode affinity — is what mints sessions.

  3. ranxianglei commented on Oct 18, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    This is a real root-cause refinement — thanks for pinning the gate down with measurements. Honest correction to my earlier framing: I attributed your ~17 mints/day to prefix-affinity churn; your data shows that on this rig the measured producer is the attach-gate equality rule flipping attach→spawn, and every spawned session-daemon starts with empty state, which is what makes every conversation in it first-sight. Affinity churn stays in this issue as a separate mechanism item (same symptom family, one issue per the granularity policy) — but note your spawned sessions showed empty /__bili/stats, so the affinity component on your rig is still unmeasured; after the gate stops spawning, your mint-count checklist will show how much (if any) churn is left over from history rewrites alone.

    I verified every code claim against master (v0.1.197-369-gce9bc90a3; your v0.1.198 install matches on these lines):

    1. Python gate: hermes-plugin/__init__.py:393-394 — if inst["mitm_domains"] != want_domains: return "mitm-domains-mismatch" — exact list equality, order-sensitive. ✅
    2. TS twin: src/launcher.ts:3499-3500 — length equality then per-index equality. ✅
    3. Runtime consumption: src/mitm.ts:145-170 — the CONNECT handler does a per-CONNECT membership test (isMitmHost): whitelisted host → MITM, non-whitelisted → blind TCP tunnel (loopback-only). This is the load-bearing fact for ask (a): extra whitelist entries are inert for a given client (it only CONNECTs to its own provider hosts), while a missing entry is the harmful direction (CodeBuddy 直接使用httpProxy 不能压缩上下文的问题 #897: silent blind-tunnel, no compression). Your safety argument holds at the code level.
    4. No hot reload: setupMitm(server, opts.mitm.domains, …) runs once at startup (src/server.ts:940-950), the array captured in the handler closure. ✅
    5. mitm.enabled:false installs no CONNECT handler at all (src/server.ts:940) → every https rung aborts. ✅

    Evaluation of the three asks

    (a) Superset (subset) check instead of exact equality — assessed SAFE, recommended.
    Semantics become want ⊆ daemon-list (order-insensitive). It eliminates exactly the false-positive directions: config-order fragility (your cron writer re-emits rows in config order — pure reordering flips attach→spawn today even though the set is identical), shrink races (provider removed from config, daemon keeps the old superset), and hand-started daemons with deliberately broad lists. It does NOT weaken the harmful-direction guard: if want contains a host the running daemon lacks, the subset check still fails and the client spawns — which is the correct conservative behaviour, because that daemon genuinely cannot MITM the new host (it would blind-tunnel and silently drop compression). Residual gap: addition races still spawn once per newly-added domain until the daemon reloads — that is inherent stale-daemon state, which is what (b) addresses.
    Behaviour change (flagged per policy): OLD = exact order-sensitive equality on mitm.domains; NEW = order-insensitive subset. Effect: more attaches, fewer spawns whenever daemon ⊋ want; zero new exposure surface (blind tunnels stay loopback-only, remote-CONNECT policy untouched, missing-domain rejection unchanged). Touches instance/session ownership semantics → §7.4 human-review territory, so I'm asking for owner sign-off rather than shipping it unilaterally. Diff is small: src/launcher.ts:3500, hermes-plugin/__init__.py:393-394 (mirror), plus the existing tests that exercise instanceCompatible (tests/hermes-native.test.ts, tests/port-zone.test.ts).

    (b) Runtime reconciliation (POST /__bili/mitm/domains + SIGHUP) — valid, defer.
    Larger surface: new privileged API route, hot-swapping the whitelist out of the setupMitm closure mid-chat, plus a signal/config watcher. Only earns its cost if addition-race churn proves painful after (a) — and your bili_whitelist_sync.py already covers that case today (verified self-healing per your test). Track as a checklist item here, not a parallel PR.

    (c) Doc line — yes, folded into the (a) PR so the README documents the final rule (subset semantics + the failure consequence when a domain is genuinely missing: spawn churn / blind-tunnel warning), not the pre-change rule. Writing the old rule now would just rewrite it next week.

    Decision request (owner)

    Structured answers for the decision at hand:

    1. 是什么问题: the hermes-plugin attach gate compares the daemon's mitm.domains to the client-derived list by exact order-sensitive equality; any benign drift (reordering, shrink, broader hand-started list) refuses attach and forces a fresh session-daemon, whose empty state turns every conversation into a first-sight distillation marathon.
    2. 什么原因导致的: the equality comparison itself is the producer — it conflates "daemon cannot serve what I need" (harmful, must reject) with "daemon can serve more than I asked" (harmless, wrongly rejected). Downstream effect (empty-state daemon → first-sight) follows mechanically.
    3. 影响范围与严重程度: every hermes-plugin attach where daemon-list ≠ want-list exactly — today that includes pure reordering and any config writer that touches provider rows (your hourly scout job). Severity: availability-grade for affected setups (minutes of serial stall per resume, spawn churn, measurement blindness on spawned ports); other lanes/users unaffected.
    4. 怎么修复的 (recommended): (a)+(c) as ONE small PR — subset check in both instanceCompatible (TS) and _attach_reason (Python mirror, they must stay in lockstep), regression tests for reorder/shrink/superset/addition cases, README line describing the final rule.
    5. 回归风险, per option: (a) — rejection behaviour strictly narrows (fewer refusals, same refusals in the harmful direction); risk surface = a client attaching to a daemon whose list it never saw before — inert per the per-CONNECT membership test, no wire/protocol/cache impact; no persistence-format change. (b) — new privileged route + live mutation of TLS-handshake-path state: real review surface (cert issuance per domain, in-flight tunnels, loopback privilege model); deferring it costs nothing functionally while (a)+sync-script cover the field. (c) — docs only, none.
    6. 是否破坏缓存: no — no wire shape, prefix-cache, retry or dedup key touched; attach decisions only change WHICH daemon serves, not what bytes flow.
    7. 是否改了设计方向: no — the gate's purpose (never attach to a daemon that would blind-tunnel a needed host) is preserved verbatim; only the harmless-direction false positives go away.
    8. 是否引入新配置: no new config/env/knob — (a) changes comparison semantics of an existing field; (b) would add an API route (deferred, separate decision).

    Requests: owner sign-off on (a)+(c) as one PR (I'll implement immediately on approval); (b) stays a checklist item here pending post-(a) measurements. @MojitoJoe1813 — two things that would sharpen the record while we wait: (i) confirm whether your scout job ever REORDERS provider rows between runs (pure-reorder churn is the cleanest (a) win, separable from addition churn in your logs); (ii) after (a) ships, re-run the mint-count checklist — the delta isolates how much of the original 17/day was gate vs affinity.

  4. added
    bugSomething isn't working
    hermes改动范围:内核之外的主机侧接线(src/agent 适配器、server 路由、注入拦截)/范围兜底/客户端标签
    协议接线改动范围:内核之外的主机侧接线(src/agent 适配器、server 路由、注入拦截)/范围兜底/客户端标签
    on Oct 18, 2026
  5. MojitoJoe1813 commented on Oct 18, 2026

    @MojitoJoe1813
    Author

    Answering both data requests from your last comment. All numbers come from our own measurement harness: a small sync script hooked into the job that writes the provider rows, plus its append-only log. No hostnames, no addresses, no session ids, no credentials below — the only names shown are public API hostnames of the providers we call. Version note: v0.1.198, one long-lived daemon in plain-HTTP forward mode, attach gate active, provider rows rewritten by an hourly job whose content depends on free-tier availability.

    (i) Does our writer REORDER provider rows? Measured, not inferred.

    The hook re-derives the client list with mitm_domains_from_config() (the installed plugin's own function, so the comparison is against the same logic the gate uses) and logs the resulting list after every run. Transitions between consecutive logged lists, window 2026-10-17T22:10Z → 2026-10-18T09:07Z (20 transitions, hourly):

    transition type count timestamps
    identical — same order AND same set 18 —
    pure addition +generativelanguage.googleapis.com 1 2026-10-18T03:59:58Z
    pure removal -generativelanguage.googleapis.com 1 2026-10-18T05:03:43Z
    pure reorder (same set, different order) 0 —

    So on this rig, reorder churn is not the live driver. The live driver is availability flapping — a host appears in the provider rows, then disappears again an hour later. Cost we measured for those two events: 2 config writes + 2 daemon restarts (lane-restart state=active at 04:00:08Z and 05:03:48Z), 0 attach refusals in the lane journal, 0 Model fallback lines in the client logs today, 123 forward POST lines today.

    Reading your (a) against that: the subset rule want ⊆ daemon-list removes exactly the shrink half of our cost — the 05:03 removal would need no write and no restart under a subset check (client wants 2 hosts, daemon holds the 3-host superset, attach stays valid). The addition half survives, which matches your residual-gap note: one write plus one restart per newly added host until the daemon reloads. On our rig that addition race is now the only remaining churn source, so (b) is the item that would pay off here.

    (ii) Mint count, as a proxy for post-(a) behaviour.

    Baseline while the gate was unsatisfiable: ~17 spawned session daemons per day (our earlier report), each starting from empty state, i.e. every conversation in it first-sight.
    Current state, last 24h, same workload mix (interactive agent sessions + hourly jobs): 0 new per-session daemons in the spawn-state directories. The only instance markers present are the two registrations of the single long-lived daemon (the #394 double-registration artifact). Per-daemon stats from /__bili/stats for the same window: main session 44 requests, context 73,397 tokens, cache hit 74% over 44 samples; [acp-usage] samples today mean input 58,180 tokens / mean cached 43,506 tokens, mean hit ratio 0.675, 21 of 28 requests above 50%. Fold counters today: session-range-fold 0, emergency folds 0, 7 resent history drifted warnings with about 500–1400 tokens unfolded in the churn region.

    Direct answer to your question "how much of the original 17/day was the gate": essentially all of it. Spawn churn dropped from ~17/day to 0/day as soon as attach held. The prefix-affinity component cannot be separated on this rig at the moment — nothing spawns any more, so there is nothing left to attribute.

    Two asks while the PR is being written:

    1. Keep _attach_reason and instanceCompatible in lockstep as you described, and consider emitting the refusal reason with the two list sizes only (want size vs daemon size, no hostnames in the log) — that single number is enough field evidence to separate addition races from shrink races in the wild.
    2. If (b) is picked up later, our harness measures it the same way it measured (a): writes per day, restarts per day, refusals per day, spawned daemons per day. We will post numbers; and if you want a pre/post comparison for the (a)+(c) PR itself, say the word — we can run the same counters against an installed (a) build within a day.
  6. ranxianglei commented on Oct 18, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    This closes the causal loop cleanly — recorded for the issue:

    • Gate attribution settled: spawn churn ~17/day → 0/day the moment attach held, same workload mix. Essentially all of the original mints were gate-driven; the first-sight marathons on your rig were downstream of empty-state spawns, not affinity mints. Your 24 h counters confirm the whole chain downstream too: 0 emergency folds, 0 session-range-folds, 74% cache hit over 44 requests — the marathon class vanished entirely once one long-lived daemon actually served the conversation.
    • Flapping profile measured: 20/20 hourly transitions accounted for — 18 identical, 1 pure addition (+generativelanguage.googleapis.com 03:59:58Z), 1 pure removal (same host, 05:03:43Z), 0 pure reorders. So on your rig (a) buys the shrink half of the flap (the 05:03-style removal: client wants 2, daemon holds the 3-host superset, attach stays valid — no write, no restart), and the addition half remains as the sole residual churn source. Thank you for correcting my emphasis: I had listed reorder fragility first, and your data shows reorders don't happen in practice — the honest general-population value of (a) is shrink-races + hand-started broad-list daemons + order fragility as latent risk, while for flapping-provider setups like yours, (b) is the item that pays off. That raises (b)'s priority in the issue checklist accordingly (still deferred behind (a), since (a) is the precondition that makes (b)'s surface smaller: with subset semantics, (b) only ever needs to handle list growth).
    • One positive side-observation in your stats: the 7 resent history drifted warnings (500–1400 tokens unfolded each) show hermes does mutate small spans between turns, and reconciliation absorbed all of them inside the stable session — no mints. The full-mint affinity case (structural rewrite) remains unmeasured on your rig; if one ever appears on the long-lived daemon, the redacted-log-around-the-mint request from my first reply still stands as the way to pin down which bytes moved.

    Both asks while the PR is written — accepted:

    1. Size-only refusal logging: yes, folded into the (a)+(c) PR scope. Both sides in lockstep (TS instanceCompatible caller path + Python _attach_reason consumer-facing log), emitting mitm-domains-mismatch (want=N daemon=M) — sizes only, no hostnames, consistent with the privacy posture you've kept throughout this thread. It's additive detail on an existing refusal line (log-format change noted in the PR description). With want vs daemon sizes the field can classify every refusal as addition-race (want > daemon) vs shrink/superset residue (want < daemon) vs other — exactly the separation your harness did by hand.
    2. Pre/post verification pass: yes, please run it. Against an installed (a) build, same counters as your harness — writes/day, restarts/day, refusals/day (now self-classifying via the size pair), spawned-daemons/day — plus the original issue checklist (new session pfa- count, preflight compressed … in <ms> events) so the PR carries a real-rig A/B instead of CI-only evidence. Same workload mix as your 24 h window is the right control. I'll post the candidate build details here when the branch is up; a day's window before merge is fine.

    Status: implementation of (a)+(c)+size-logging starts immediately on owner sign-off (gate semantics = §7.4 human-review territory, flagged behaviour change exact-equality → order-insensitive subset); (b) stays a checklist item with its priority note above. Nothing else blocks the PR.

  7. added 2 commits that reference this issue on Oct 19, 2026
  8. ranxianglei commented on Oct 19, 2026

    @ranxianglei
    Owner

    Fixed by #3793: 600s turn budget bounds the small-window preflight marathon; shipped in v0.1.202.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghermes改动范围:内核之外的主机侧接线(src/agent 适配器、server 路由、注入拦截)/范围兜底/客户端标签p2协议接线改动范围:内核之外的主机侧接线(src/agent 适配器、server 路由、注入拦截)/范围兜底/客户端标签

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions