Repository navigation
[PR] batch-A: billing & registry perf — 5 mechanical fixes absorbed (original PRs superseded) - #7448
Open
ranxianglei wants to merge 7 commits into
Open
ranxianglei wants to merge 7 commits into
ranxianglei wants to merge 7 commits into
Conversation
…ling path scans (#7278) The billing price path scans two cost tables per lookup (the overrides table, then the registry fallback). costLowerIndex kept only ONE cached index, so an overrides-miss lookup evicted the overrides index for the registry index and back again — rebuilding the (large) registry cost index on every lookup, O(registry rows). Make costLowerIndex a small LRU Map (cap 2: the two tables the path scans), refreshing recency on hit and evicting the oldest on overflow. The registry index is now built once and reused; bounded memory. The model-registry index (regLowerIdx) stays a single slot — it has only one table. Regression test counts index builds via a Proxy ownKeys trap and pins the registry index to exactly one build across repeated overrides-miss lookups (fails on the old single-slot cache). (cherry picked from commit 4521b10) (cherry picked from commit 2a37304)
The committed test exercised stable identities for both cost tables (the post-#7276 shape). At the PR head the real path (overrideCosts, pre-#7276) hands scanRegistryCostRow a fresh overrides identity per lookup; a FIFO cache would evict the stable registry index every other lookup and rebuild it, yet the stable-identity test cannot see that. Add a fresh-identity case that pins the recency-refresh: passes on the LRU, fails (25x rebuilds) on a FIFO variant. (cherry picked from commit a5b32a2) (cherry picked from commit accf68f)
A failed fetch leaves the cache on the bundled snapshot (or stale disk cache), so loadRegistry() fetches again on the next call. server/handle.ts awaits contextFromRegistry() for any model the warm cache misses, so with models.dev slow or unreachable every such request stalled for the full 15 s fetch timeout. contextFromRegistry() now serves the fallback for 10 minutes after a failed fetch. loadRegistry() keeps its per-call retry (the boot pre-warm and the existing memo tests are unchanged). (cherry picked from commit 2a5006e)
The fetch watchdog is an idle timer, so a models.dev that trickles bytes holds the load open far past its 15 s budget (measured: 8 s to first byte, then about 11 B/s). contextFromRegistry() now waits at most 3 s and answers from the current cache; the load keeps running and warms the cache for later requests. (cherry picked from commit f935f9b)
The stalled-fetch test pins a 50 ms request-path budget via the test seam; the wall-clock backstop was 5_000 ms (100x the budget), so a regression that let the request path wait up to ~4 s would still pass. 1_000 ms keeps a 20x margin for scheduler jitter while catching a multi-second budget regression. No behavior change. (cherry picked from commit a654516)
…ken estimates (cherry picked from commit 97f5969)
Contributor
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.11179-pr.7448.3c701caThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr7448.tgz
npm install -g packageInstall instructions are refreshed on each push. |
Owner
Author
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Verifying the batch PR #7448 (head Verified so far:
Still in progress: CI check status (mirror cache had no entries yet — doing a single check-runs fetch now), reading the 4 test files, and confirming the skipped items (relay-suffix index already on master as |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this batch is
One batch PR that absorbs five small mechanical billing/registry fixes by cherry-picking each original PR's commits with
-x— original authorship preserved, source commits cited in every message. No new work was written here; every line comes from the PRs listed below. The point is a single review surface and a single merge decision instead of five.Overlap note: #7363 is the root-cause aggregation for the billing-lookup-perf family. Its third commit (relay suffix index, #7284) was already merged to master separately as #7285, so cherry-picking it landed empty and it was skipped — it is not re-applied here. #7279 and #7280 are superseded by #7363 (see Skipped below).
Absorbed original PRs
What each one actually changes (what merges into master with this batch):
resolveBillingPriceno longer rebuilds the normalized override cost table on every lookup — it is built once insetBillingOverridesand reused until the next config reload; and the registry cost-row index cache becomes a small LRU (cap 2) so the overrides table and the registry snapshot stop evicting each other's index on every lookup.contextFromRegistry) now waits at most 3 s on the models.dev load (the load keeps warming the cache in the background) and serves the fallback for 10 minutes after a failed fetch instead of re-fetching for every cache-miss request.dsh_*fields (e.g.dsh_session_log) no longer count toward raw-body token estimation, so they can't trip the egress window fuse (HTTP 413) and block a session's models.Fixes
(#7284 — the third issue in the #7363 family — is already closed: its relay-suffix-index fix landed on master separately via #7285. Related but NOT closed by this batch: #6307, the off-switch feature for the models.dev prefetch — #7395 deliberately keeps 10 min / 3 s as constants, not config knobs, and leaves that request open.)
Skipped (record)
4ebcc2298is patch-identical (same patch-id) to the [PR] perf: root-cause fix for billing-registry-lookup-perf (Fixes #7276, #7278, #7284) #7363 commit this batch already applied ase0ba9e547. Nothing left to absorb.ef8f9aa9(relay suffix index) — already on master ascee5b4e08(merged via [PR] perf(registry): index relay model suffix candidates #7285); cherry-pick landed empty, skipped.原 PR 将在本批合并后关单注明收编。