Skip to content

[PR] fix(billing): build the normalized cost table once per config load, not per lookup (#7276) - #7280

Closed
ranxianglei wants to merge 1 commit into
masterfrom
2026-11-19_cache-billing-cost-rows
Closed

ranxianglei wants to merge 1 commit into
masterfrom
2026-11-19_cache-billing-cost-rows

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

resolveBillingPrice is a hot path (per-request billing stamping + the pricing UI), but it called overrideCosts() on every invocation, allocating a fresh CostsShape object each time. That new table identity defeated costLowerIndex's identity cache (src/registry.ts:564), so the lowercase lookup index was rebuilt on every lookup too — even for an exact override hit. A price table with N configured models paid for N full row conversions plus a full re-index on each of its lookups.

Fix

Build the normalized cost table once in setBillingOverrides (which every config-reload path already funnels through — startup, web Apply onChanged, /__bili/config reload) and reuse it until the next load:

  • overrideCostRows (module-level) is rebuilt in the setter, unconditionally — even for a repeated input object — so in-place mutation between sets can't leak stale rows.
  • An empty/cleared set yields null, so the registry fallback applies immediately.
  • resolveBillingPrice now reads the cached table instead of rebuilding it.

Evidence

Synthetic exact-override-hit workload (1000 configured models, 5000 lookups, fresh process, 100-lookup warmup), this host:

before after
process CPU ~1189 ms ~15 ms

Identical result hashes before/after. (The reporter's 4850 ms is the same mechanism on a shared box.)

Regression coverage

Two new tests in tests/billing-prices.test.ts:

  • replace/clear: replacing the table drops the old override and exposes the registry row; emptying and clearing (undefined) both fall back to the registry; a previously resolved price object is never rewritten.
  • reused table + zero rates: re-setting the same object rebuilds the table (mutated row + explicit zero output/cacheRead/cacheWrite exposed immediately); case-folded model names still hit.

21/21 billing tests pass; typecheck clean.

Scope / no behavior change

No config field, pricing formula, matching order, request wire, or kernel behavior changes. The resolved prices are byte-identical to before — only the per-lookup rebuild is removed.

Related

Credit

Approach and regression tests follow the implementation proposed in #7276 by @Afloat16 (branch 2026-11-19_cache-billing-cost-rows); independently verified and re-based onto master.

…ot per lookup (#7276)

resolveBillingPrice is hot (per-request stamping + the pricing UI) but called
overrideCosts() on every invocation, allocating a fresh CostsShape each time.
That new table identity defeated costLowerIndex's identity cache, so the
lowercase index was rebuilt on every lookup too — even an exact override hit.

Build the normalized table once in setBillingOverrides (which every config
reload path already funnels through) and reuse it until the next load. The
setter rebuilds unconditionally (even for a repeated object) so in-place
mutation between sets can't leak stale rows; an empty/cleared set yields null
so the registry fallback applies immediately.

No config field, pricing formula, matching order, request wire, or kernel
behavior changes.
@ranxianglei ranxianglei added bug Something isn't working p2 scope/other labels Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-19_cache-billing-cost-rows (abc252d)

Latest build — install this exact version

npm install -g billion-context@0.0.10959-pr.7280.79ce547

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
abc252d billion-context@0.0.10959-pr.7280.79ce547 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr7280.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix(billing): build the normalized cost table once per config load, not per lookup (#7276) [PR] fix(billing): build the normalized cost table once per config load, not per lookup (#7276) Nov 19, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Verified: the fix is correct and CI is in progress (0 failures) — but this PR duplicates the still-open PR #7279 by the original contributor, and per the no-replace policy #7279 should be the one that gets merged. Your call is needed.

This is a [PR] pickup, so I verified rather than re-did:

Code (head abc252d) — correct, no behavior change

  • Mechanism confirmed: costLowerIndex (src/registry.ts:564) is a single-slot identity cache; the old code handed it a fresh CostsShape on every resolveBillingPrice call, so the lowercase index was rebuilt on every lookup. The fix caches overrideCostRows and rebuilds it only in setBillingOverrides (src/billing.ts:120-123); empty/cleared → null → immediate registry fallback.
  • Single funnel confirmed: the only production caller of setBillingOverrides is loadOptions (src/config.ts:1670) — every config-load path (startup, web Apply, /__bili/config reload) funnels through it, and each load parses a fresh object, so snapshot-at-set is safe.
  • No mutation hazard: the only consumer of the raw table (billingOverrides()) is read-only (src/web/api.ts:532).
  • Same conversion, matching, finalize, and fallback order as before — resolved prices are byte-identical; only the per-lookup rebuild is removed. The two new regression tests (replace/clear; reused-table rebuild with zero rates + case folding) pin the reload semantics.

CI (live, mirror cache): 30 checks — 0 failed, 13 in progress (11 test shards + 2 static-gates); every completed lane green. Verdict pending.

Labels: applied to the mirror issue on pickup — bug, p2, scope/other (plus the existing pr).

⚠️ Duplicate PR — the decision you need to make

Recommendation: merge #7279 and close this PR (+ delete the upstream branch 2026-11-19_cache-billing-cost-rows). Cost: #7279's body lacks the perf table / regression detail / #7278 reference — I've posted those as a comment on #7279 so nothing is lost. If you'd rather keep this PR (richer body + labels), close #7279 instead — but that keeps the authorship replacement. I can't close either PR (human-only). Default if undecided: both stay open; merging one leaves the other as a stale duplicate.

结构化八问

  1. 是什么问题:resolveBillingPrice 是热路径(每请求计费盖章 + 定价 UI),但每次查找都重建整张归一化成本表——N 个配置模型 = 每次查找付 N 行转换 + 全表小写索引重建。纯性能缺陷,功能不受影响。
  2. 什么原因导致的:overrideCosts() 每次调用都分配全新 CostsShape,新表身份让 costLowerIndex 的单槽身份缓存(src/registry.ts:564)永远 miss,索引逐次重建。
  3. 影响范围与严重程度:配置了 billing.prices 覆盖的实例,每次计费查找都触发;模型配置越多越慢(报告者 4850ms/5000 次查找;本 PR 合成负载 1189ms→15ms)。能用但性能受损 → p2。
  4. 怎么修复的:归一化表只在 setBillingOverrides(所有配置加载路径的唯一漏斗,src/config.ts:1670)重建一次,缓存于模块级 overrideCostRows;查找只读缓存;空表/清空 → null 立即回退 registry;同一对象重设也无条件重建,防原地变更泄漏旧行。
  5. 是否有回归问题:无。转换/匹配/finalize/回退顺序全部不变,解析价格字节级一致;唯一语义差异是"set 时快照"而非"查找时实时读"——已核对裸表唯一消费者(src/web/api.ts:532)只读、配置路径每次都是新对象,无原地变更路径。
  6. 是否破坏缓存:不破坏,修复目标正是让 costLowerIndex 身份缓存对 override 表开始命中。registry 自身索引在 override-miss 路径的单槽抖动是既有问题,已另立 [Drive-by] Registry cost-row index rebuilt per lookup on overrides-miss (single-slot cache thrash) — separate from #7276 #7278,本 PR 不碰。
  7. 是否改了设计方向:没有。无架构/边界/数据流变化,无新配置面。
  8. 是否引入了新的配置:没有。

判定:代码本身可合(等 CI 收尾转绿);合并路径卡在重复 PR 决策上——建议合 #7279、关本 PR。

@ranxianglei

Copy link
Copy Markdown
Owner Author

已收编进 batch-A PR #7448(已合并)。本 PR 与 #7279 为同一修复(重复工单):代码改动相同,仅注释与测试名不同;#7448 采用 #7363 的版本以避免重复测试,本 PR 无增量内容,关闭。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Billing price lookups rebuild every override row and lowercase index

1 participant