Skip to content

[PR] fix(#5199): network.throttle429Scope — model-scoped 429 holds for marker-less gateway bodies - #5889

Merged
ranxianglei merged 1 commit into
masterfrom
2026-11-07_throttle429-scope-knob
Nov 7, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-11-07_throttle429-scope-knob

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Implements #5853 decision 1 = option A (recommended) and closes #5199: a new knob network.throttle429Scope: "credential" | "model" (default "credential", zero behavior change) lets deployments whose gateway fronts many independently-limited models behind one credential (LiteLLM / one-api / opencode.ai-style relays) opt into model-scoped 429 holds for marker-less bodies — the gap PR #5696 deliberately left (it scopes only the keyring_model_temporarily_unavailable marker and bili-relay bili_throttle_hold scope=model markers).

#5853 decision 2 = option A: streak-1 arming unchanged — no code change.

Field evidence for the marker-less shape (both in #5199):

  • {"error":{"code":null,"message":"qnaigc: rate limit reached for RPM (request_id: …)"}} — one model's RPM pool froze every model on the credential; 10-min window: 5 real 429s, 49 local 429 rejects.
  • {"error":{"message":"[prefill_cap_refused] the cell is at capacity; retry shortly (after 30 s)"}} — opencode.ai lane: a big session's prefill-cap 429 froze a fresh session's title-gen sidecar.

How

  • src/knobs.ts: throttle429Scope() — BILI_THROTTLE_429_SCOPE > network.throttle429Scope > "credential"; exact "model" match, everything else falls back to "credential"; read per settlement so it flips live.
  • src/upstream-throttle.ts: throttle429ModelScope(status, requestModel, responseBody) — recognized KeyRing/bili-relay markers win exactly as before; otherwise scope=model + identifiable request model → the request model; model-less or non-429 → legacy credential key. Worst-case protection never widens beyond legacy.
  • Settlement sites switched to the wrapper: src/server/relay.ts (main lane), src/loop/core.ts (replay lane), src/degenerate-retry.ts (provisional settler). Governor internals (gate/peek/resolved/probe ownership) already thread requestModel post-[PR] fix: isolate gateway model-scoped 429 holds #5696 — a model-scoped hold skips sibling-model and model-less sends by construction, and only the matching model's resolved response clears it.

config surface

Tests

tests/issue5199-throttle429-scope.test.ts — 6 rows, both directions per gate discipline: knob read (live flip, exact match, empty/invalid → credential); default mode keeps marker-less bodies credential-wide (#5696 row) while the recognized marker still grants model scope; scope=model settles RPM/prefill-cap/malformed bodies on the request model; governor integration — exhausted model held, sibling + model-less sends eligible, sibling 200 doesn't clear, matching 200 clears; model-less 429 stays credential-wide (storm protection intact); mid-flight scope flip changes only future settlements.

Existing suites pass unchanged (default behavior identical): issue4204 (61 tests incl. 4725/4743/5795/upstream-error-log lanes), typecheck clean.

Docs

CONFIGURATION.md / CONFIGURATION.zh-CN.md + website seed (process.yaml) regenerated via tools/gen-config-docs.mjs generate (coverage check green); LIFECYCLE.md env-table row extended.

No kernel/ or pi-subagents/ changes.

…less gateway 429s

Implements #5853 decision 1 option A (recommended); decision 2 unchanged
(streak-1 arming kept). Default "credential" preserves #5696 semantics —
zero behavior change without opt-in.

- knobs: BILI_THROTTLE_429_SCOPE > network.throttle429Scope > "credential",
  read per settlement (hot flip)
- upstream-throttle: throttle429ModelScope() — recognized KeyRing/bili-relay
  markers win as before; scope=model settles marker-less 429s (RPM/quota
  text, opaque relay rewrites) on the request model; model-less sends stay
  credential-wide
- relay/loop/degenerate-retry lanes settle through the wrapper
- docs: CONFIGURATION en/zh + website seed regenerated, LIFECYCLE env row
- tests: tests/issue5199-throttle429-scope.test.ts (6 rows, both directions)
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-07_throttle429-scope-knob (eb8bd3c)

Latest build — install this exact version

npm install -g billion-context@0.0.8519-pr.5889.20eee93

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
eb8bd3c billion-context@0.0.8519-pr.5889.20eee93 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5889.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix(#5199): network.throttle429Scope — model-scoped 429 holds for marker-less gateway bodies [PR] fix(#5199): network.throttle429Scope — model-scoped 429 holds for marker-less gateway bodies Nov 7, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

CI: the 9 failing test (…) lanes are pre-existing on master — not from this diff

Evidence trail (all reproducible locally):

  1. master itself is red: run 42376409811's failures are the same file that already failed master's own CI run 42358603178 (head 5af5f9fed, 2026-11-07T09:10Z — hours before this PR).
  2. Local repro on clean master (git checkout master = 5af5f9fed, no diff): node --import tsx --test tests/fork-nudge-trigger.test.ts → pass 1 / fail 2, identical signature to CI (expected trailing user nudge, tests/fork-nudge-trigger.test.ts:177 and :313).
  3. Culprit pinned to a7cbfd1 (kernel 0.0.184 phantom-pressure guard: estimate-grade readings hold pressure-band injection ONE round). Setting the escape hatch compress.requireUsageSampleForPressure: false in the fixture makes the file pass 3/3 — the two failing rows (anonymous zero-baseline fork → nudge injects, \#1137 image-heavy fork → nudge injects) pin the pre-a7cbfd12f semantics and were not updated with it.
  4. This PR's diff touches only the 429 governor surface (knobs/config/upstream-throttle + 4 call sites + docs) — zero intersection with the nudge/estimate path; its own suite (tests/issue5199-throttle429-scope.test.ts 6/6, 4204/4725/4743/5795 61/61) is green.

Suggested follow-up (separate PR, one concern per PR): update the two rows to pin the new contract — round 1 (estimate-grade, no billing sample) holds, a round-2 send with a confirming billing sample fires the nudge; a small billing sample never fires it. Happy to send it if wanted.

@ranxianglei ranxianglei added bug Something isn't working p2 scope/wiring labels Nov 7, 2026
@ranxianglei
ranxianglei merged commit 6945a35 into master Nov 7, 2026
55 of 117 checks passed
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1075-master.6945a35 (rolling master tag).

npm install -g billion-context@0.0.1075-master.6945a35  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Nov 7, 2026
PR #5905 review follow-up: the regression test named "one warn" only pinned
the no-phantom-hold behavior. Capture the logger via setLogCapture and assert
the saturation is reported exactly once, so a future change that drops or
duplicates the warn fails here (#5199/#5889).
Louie0806 pushed a commit to Louie0806/billion-context that referenced this pull request Nov 8, 2026
…-value guard, media-refs carry-forward filter & cloud prefix families, oversize refusal messages that name the real fix, dsh durable lines with readable numbers, website streamKeepAliveMs typo, 5755 ord-crossing guard kept with regression tests

- upstream-throttle: a saturated hold map (64/64 account-wide, no model-scoped
  hold to evict) silently dropped the next model-scoped 429 — now it warns
  once and the send is documented as ungoverned (ranxianglei#5199/ranxianglei#5889)
- knobs: network.throttle429Scope matches its value EXACTLY; a typo now warns
  once and falls back to credential instead of silently doing so (ranxianglei#5199)
- kernel media-refs: block summaries written by older builds can carry refs
  looksEphemeralSignedUrl would reject today; the carry-forward into merged
  blocks now applies the same filter so a dead signed URL cannot survive
  forever (ranxianglei#5865 review). EPHEMERAL_QUERY_PARAM also gains the Huawei OBS
  x-obs-*, Tencent COS x-cos-* and Baidu BCE x-bce-* prefix families.
- native-intercept/dsh-native oversize refusals now say what to DO (restart
  host / fix the model window declaration), and dsh durable lines format
  numbers with thousands separators (ranxianglei#2924/ranxianglei#4068)
- website: streamKeepaliveMs → streamKeepAliveMs (the real knob name) across
  main.js, docs/index.html, config-reference/process.yaml
- CONFIGURATION.md/.zh-CN.md: throttle429Scope exact-match + model-string
  aliasing caveat documented
- store/persist: ranxianglei#5755 refs ord-crossing guard (touching rows stay in the
  old-refs set until the new-refs commit lands) kept as defense-in-depth
  with a regression test

kernel-change: 0.0.186 → 0.0.187 (media-refs carry-forward filter + cloud
prefix families; compress.ts starvation-bound comment corrected)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/wiring

Projects

None yet

1 participant