Skip to content

[PR] batch-A: billing & registry perf — 5 mechanical fixes absorbed (original PRs superseded) - #7448

Open
ranxianglei wants to merge 7 commits into
masterfrom
batch-A-2026-11-21
Open

ranxianglei wants to merge 7 commits into
masterfrom
batch-A-2026-11-21

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

What this batch is

One batch PR that absorbs five small mechanical billing/registry fixes by cherry-picking each original PR's commits with -x — original authorship preserved, source commits cited in every message. No new work was written here; every line comes from the PRs listed below. The point is a single review surface and a single merge decision instead of five.

Overlap note: #7363 is the root-cause aggregation for the billing-lookup-perf family. Its third commit (relay suffix index, #7284) was already merged to master separately as #7285, so cherry-picking it landed empty and it was skipped — it is not re-applied here. #7279 and #7280 are superseded by #7363 (see Skipped below).

Absorbed original PRs

Original PR Author One-line description
#7363 ranxianglei [PR] perf: root-cause fix for billing-registry-lookup-perf (Fixes #7276, #7278, #7284)
#7279 Afloat16 [PR] perf: reuse normalized billing override rows between reloads
#7280 ranxianglei [PR] fix(billing): build the normalized cost table once per config load, not per lookup (#7276)
#7395 wstczyw [PR] fix(registry): bound the request-path wait on models.dev and back off after a failed fetch (#7394)
#7411 CentillionJinx [PR] fix: ignore non-model top-level dsh_* extension fields in raw body token estimates

What each one actually changes (what merges into master with this batch):

Fixes

(#7284 — the third issue in the #7363 family — is already closed: its relay-suffix-index fix landed on master separately via #7285. Related but NOT closed by this batch: #6307, the off-switch feature for the models.dev prefetch — #7395 deliberately keeps 10 min / 3 s as constants, not config knobs, and leaves that request open.)

Skipped (record)

原 PR 将在本批合并后关单注明收编。

Afloat16 and others added 7 commits November 21, 2026 15:20
(cherry picked from commit 4ebcc22)
(cherry picked from commit e0ba9e5)
…ling path scans (#7278)

The billing price path scans two cost tables per lookup (the overrides
table, then the registry fallback). costLowerIndex kept only ONE cached
index, so an overrides-miss lookup evicted the overrides index for the
registry index and back again — rebuilding the (large) registry cost
index on every lookup, O(registry rows).

Make costLowerIndex a small LRU Map (cap 2: the two tables the path
scans), refreshing recency on hit and evicting the oldest on overflow.
The registry index is now built once and reused; bounded memory.

The model-registry index (regLowerIdx) stays a single slot — it has only
one table. Regression test counts index builds via a Proxy ownKeys trap
and pins the registry index to exactly one build across repeated
overrides-miss lookups (fails on the old single-slot cache).

(cherry picked from commit 4521b10)
(cherry picked from commit 2a37304)
The committed test exercised stable identities for both cost tables (the
post-#7276 shape). At the PR head the real path (overrideCosts, pre-#7276)
hands scanRegistryCostRow a fresh overrides identity per lookup; a FIFO
cache would evict the stable registry index every other lookup and rebuild
it, yet the stable-identity test cannot see that. Add a fresh-identity case
that pins the recency-refresh: passes on the LRU, fails (25x rebuilds) on a
FIFO variant.

(cherry picked from commit a5b32a2)
(cherry picked from commit accf68f)
A failed fetch leaves the cache on the bundled snapshot (or stale disk
cache), so loadRegistry() fetches again on the next call. server/handle.ts
awaits contextFromRegistry() for any model the warm cache misses, so with
models.dev slow or unreachable every such request stalled for the full
15 s fetch timeout.

contextFromRegistry() now serves the fallback for 10 minutes after a
failed fetch. loadRegistry() keeps its per-call retry (the boot pre-warm
and the existing memo tests are unchanged).

(cherry picked from commit 2a5006e)
The fetch watchdog is an idle timer, so a models.dev that trickles bytes
holds the load open far past its 15 s budget (measured: 8 s to first byte,
then about 11 B/s). contextFromRegistry() now waits at most 3 s and answers
from the current cache; the load keeps running and warms the cache for
later requests.

(cherry picked from commit f935f9b)
The stalled-fetch test pins a 50 ms request-path budget via the test
seam; the wall-clock backstop was 5_000 ms (100x the budget), so a
regression that let the request path wait up to ~4 s would still pass.
1_000 ms keeps a 20x margin for scheduler jitter while catching a
multi-second budget regression. No behavior change.

(cherry picked from commit a654516)
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: batch-A-2026-11-21 (f6e891d)

Latest build — install this exact version

npm install -g billion-context@0.0.11179-pr.7448.3c701ca

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
f6e891d billion-context@0.0.11179-pr.7448.3c701ca build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr7448.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Verifying the batch PR #7448 (head batch-A-2026-11-21) — not redoing the work, per [PR] handling. Status so far:

Verified so far:

Still in progress: CI check status (mirror cache had no entries yet — doing a single check-runs fetch now), reading the 4 test files, and confirming the skipped items (relay-suffix index already on master as cee5b4e08). Will post the final verdict shortly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment