Skip to content

[PR] perf: reuse normalized billing override rows between reloads - #7279

Closed
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-19_cache-billing-cost-rows
Closed

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-19_cache-billing-cost-rows

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Fixes #7276

Problem and change

resolveBillingPrice converts every configured override into a new registry cost row on every lookup. The new table identity also defeats the registry's existing lowercase-index reuse. Exact override hits therefore allocate and scan the complete configured table before looking up one model; the pricing UI repeats that work for each model it lists.

Build the normalized cost rows once in setBillingOverrides, the existing entry point for startup and every configuration reload. Rebuild on every setter call, including reuse of the same input object, and release the table when overrides are cleared. Price matching, fallback order, and the resolved value objects use their existing paths.

The only production setter is loadOptions; the web API's billingOverrides() consumer only reads the configured rows. No supported configuration field or persistence format changes, and no request-wire or kernel code changes.

Validation

Base: 04ecbb42dd38543e9c407bd40eb773ae5ec8b60b (billion-context 0.1.216).

  • 21/21 billing parser, pricing API, stamping and accounting tests passed, including two new reload regressions. They cover replacing, emptying and clearing overrides; reusing the same input object; zero optional rates; case-folded keys; unchanged previously resolved prices; and registry fallback.
  • npm run typecheck passed.
  • npm run build passed, including kernel/pi-subagents builds and dist import/external checks.
  • git diff --check passed.
  • Independent review checked all setter/getter consumers and registry-index invalidation.

The authoritative full-suite verdict remains remote CI on this head. The new correctness tests protect cache invalidation; they are not claimed as pre-fix failures, since the previous implementation returned the same prices at a higher cost.

Performance evidence

Fresh Linux processes, Node.js 24.22.3, base/change/change/base. Configure 1,000 distinct override models, warm 100 exact-key lookups, then resolve 5,000 exact-key lookups. The timed loop excludes setup, module loading, and output hashing.

Revision Process CPU time in lookup loop Wall time
Base, first run 4,850 ms 4,857 ms
Change, first run 32 ms 14 ms
Change, second run 48 ms 73 ms
Base, second run 8,679 ms 11,463 ms

All four complete output hashes were 5eec8f4988f60f50e270e5eed29a7276e649190fa93be950220ac5c3aa5379d4. The shared test host produced variable timings; the structural reduction is one table conversion per configuration load instead of one per lookup. These measurements describe exact override hits. Alternating override misses and registry lookups can still rebuild the registry's single-slot index; this change does not claim to remove that separate cost or improve end-to-end request latency by the same factor.

Reproduce from the repository root with its installed dependencies:

// node --expose-gc --import tsx --input-type=module
import { createHash } from "node:crypto";
import { performance } from "node:perf_hooks";
import { setBillingOverrides, resolveBillingPrice } from "./src/billing.ts";
const prices = Object.fromEntries(Array.from({ length: 1000 }, (_, i) => [
    `custom/model-${i}`,
    { input: i + 1, output: (i + 1) * 3, cacheRead: (i + 1) / 10, cacheWrite: (i + 1) * 1.25 },
]));
setBillingOverrides(prices);
const models = Object.keys(prices);
for (let i = 0; i < 100; i++) resolveBillingPrice(models[i % models.length]);
global.gc?.();
const results = [];
const cpuStart = process.cpuUsage();
const start = performance.now();
for (let i = 0; i < 5000; i++) results.push(resolveBillingPrice(models[i % models.length]));
const wallMs = performance.now() - start;
const cpu = process.cpuUsage(cpuStart);
console.log({ wallMs, cpuMs: (cpu.user + cpu.system) / 1000,
    digest: createHash("sha256").update(JSON.stringify(results)).digest("hex") });

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Duplicate screening: not a duplicate. This PR is the fix for its own source issue #7276. Related thread #7278 (single-slot index thrash on overrides-MISS lookups) is the same family but a different mechanism: #7276 = per-lookup override-table conversion (fixed here); #7278 = pre-existing registry index thrash on alternating miss/registry workloads (out of scope, correctly disclosed in the PR body as not claimed).

Labels: perf + p2 + scope/other applied to this PR (judged on its own diff: performance degradation on a per-request hot path; server-side billing lookup, no client lane applies). Mirror-side labeling is unavailable in this environment (the shim's label endpoints return "not implemented"), noted for the record.

Review

Direction (source issue #7276): verified against code, not just the report. resolveBillingPrice (src/billing.ts:131) called overrideCosts() per lookup, which allocates a fresh CostsShape every time; costLowerIndex (src/registry.ts:564) is a single-slot cache keyed on table identity, so the fresh identity can never hit and the lowercase index is rebuilt on every lookup. The fix moves normalization to the right layer — the config-load point — instead of masking the cost at the lookup site. Root cause, not symptom.

Consumer audit (regression review):

  • Only production setter: loadOptions (src/config.ts:1670). All three reload paths funnel through it: startup (src/cli.ts), web Apply (src/server/admin.ts:414 → loadOptions()), admin config reload (src/server/admin.ts:930). The cache therefore rebuilds exactly when the table changes.
  • Only billingOverrides() consumer: the pricing UI (src/web/api.ts:532-544) — strictly read-only (Object.keys, index reads, JSON serialization). No in-place mutation anywhere, so the new cache cannot go stale via that face.
  • Empty/clear semantics preserved: overrideCosts() returns null for an empty map; the setter stores null; the resolver's !== null check is unchanged → registry fallback intact.
  • Test isolation preserved: resetState() (tests/billing-prices.test.ts:37) routes through setBillingOverrides(undefined), so overridePrices and overrideCostRows can never diverge (both written atomically in the one setter; consistent initial values {} / null).
  • Residual single-slot thrash (alternating overrides-miss vs registry lookups): cost unchanged, never worse — before, the slot was rebuilt on every call regardless; the PR body discloses this honestly and [Drive-by] Registry cost-row index rebuilt per lookup on overrides-miss (single-slot cache thrash) — separate from #7276 #7278 tracks it.
  • No config schema, persistence, request-wire, or kernel changes — the diff touches exactly src/billing.ts (+3/-2) and the test file.

Code quality (external-submission bar): passes. overrideCostRows: CostsShape | null mirrors the existing costCache pattern in registry.ts (the module comment at src/billing.ts:106-108 already names that pattern); naming consistent with overridePrices; no dead code, no as any, no new dependencies, style matches.

Test validity: the two new tests have teeth against the plausible wrong implementation: the reused-table test (mutate the same input object, re-set, assert new values) fails any "skip if same reference" shortcut; it also pins case-folded keys and zero optional rates (explicit 0 must not fall back to the input rate via ??). First test pins replace/empty/clear invalidation and immutability of previously resolved objects. Honestly positioned as correctness guards for the new cache, not pre-fix failures — correct for a perf PR.

Independent verification on head 4ebcc229859:

  • Targeted suites: tests/billing-prices.test.ts 12/12 pass; adjacent tests/billing-web.test.ts + tests/registry.test.ts + tests/registry-price-profile.test.ts + tests/issue2189-preflight-billing-block.test.ts 49/49 pass.
  • npm run typecheck clean.
  • Ran the PR's repro (1,000 overrides, 5,000 exact-key lookups) on base 04ecbb42dd3 and head: base 1,679 ms CPU → head 25 ms CPU on this host (Node 22.23.2), and the output digest is identical on both (5eec8f49…), matching the PR's claimed digest. Structural reduction confirmed: one table conversion per config load instead of per lookup.

CI: in progress on the head sha as of this review (all test lanes in_progress); the full-suite verdict is remote CI per repo discipline. Will follow up when it lands.

Auto-merge gate: not eligible — perf: (not a pure fix:) and it adds module state; human merge required.

八问审核(中文)

  1. 是什么问题:resolveBillingPrice 每次查找都把整张 billing.prices 覆盖表重新归一化成新的 cost-row 对象,新对象身份又让 registry 小写索引的单槽缓存永远 miss、每次重建——按请求的价格盖章(stampBillingPrice)和定价 UI 逐模型枚举都在为一次查找付 O(N) 转换 + O(N) 索引重建。
  2. 什么原因导致的:归一化 overrideCosts() 被放在查找路径里,每次返回新表身份;costLowerIndex(src/registry.ts:564)以表身份为缓存键,身份每变必重建。成本所在层错了——应在配置加载时做,不在查找时做。
  3. 影响范围与严重程度:配置了 billing.prices 覆盖的用户,每次请求的盖章路径 + 定价 UI 枚举受影响。性能降级而非功能缺陷——解析出的价格前后完全一致(5000 次解析结果 sha256 摘要一致,独立复测确认);覆盖表越大影响越大,小表感知不到。
  4. 怎么修复的:归一化上提到 setBillingOverrides(唯一生产写入点,三条 reload 路径全部收敛于此),结果缓存进模块级 overrideCostRows,每次 setter 调用必重建(含同一输入对象复用);查找路径读稳定表,索引缓存命中。
  5. 是否有回归问题:逐项审过——解析优先级(override→registry)保持;盖章语义保持;web 定价 UI(唯一 billingOverrides() 消费者,只读)保持;空表/清空语义保持(空表 → null → registry 兜底);测试隔离保持(resetState 走同一 setter,两变量不可能分叉);配置 schema/持久化/wire/kernel 无触碰。唯一行为差异:billingOverrides() 返回对象的就地修改不再影响解析——代码库中无此用法(唯一消费者只读),不构成回归。交替 override-miss/registry 工作负载的单槽索引抖动成本不变(PR 正文已如实披露,[Drive-by] Registry cost-row index rebuilt per lookup on overrides-miss (single-slot cache thrash) — separate from #7276 #7278 跟踪)。
  6. 是否破坏缓存:本 PR 正是修缓存——costLowerIndex 单槽缓存对覆盖表从「永不命中」变「命中」(稳定身份);不影响任何既有缓存/重试/游标/去重失效策略。
  7. 是否改了设计方向:没有。沿用既有 registry costCache 模式(src/billing.ts:106-108 注释本就声明本模块镜像该模式)。
  8. 是否引入了新的配置:没有。

判定:代码审核通过,CI 绿后可合(本 PR 不满足 auto-merge 条件——perf 类且新增模块状态,需人工合并)。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI follow-up — green on head 4ebcc229859. Verified live just now: the main CI workflow (full test matrix ubuntu/windows × node 22/24 × shards + static gates: kernel-guard, pi-subagents-guard, lock-sync, version-guard, config-surface-guard, architecture-docs, test-import-graph, unused-exports, …) completed success, and every e2e lane (fake-codex, codex-persona, native-pi, native-dsh, native-opencode, pi-codex-ws, local-registry, Windows codex overlay) completed success. The cancelled entries in the run list are superseded runs from the 09:09Z re-run batch, not failures; red-signal is schedule-only and correctly cancelled on a PR run.

Nothing outstanding on my side: review verdict stands (mergeable, human merge required — perf class, not auto-merge eligible), labels applied (perf/p2/scope/other), residual overrides-miss thrash tracked separately in #7278.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Verification notes from the review of the duplicate agent PR #7280 (same fix, opened ~2 minutes after this one, re-committed under the agent identity with only comment/test-name differences):

@Afloat16

Afloat16 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

(ai: OpenAI/GPT-6 Astra Pro)

Update, 2026-11-19 13:15 UTC: thanks for enabling ci:full. The full CI run 44502838417 passed all 16 test legs and all 4 static-gate legs on the unchanged head 4ebcc229859339c9a1b9f42459385b922ec81ad7. The guard jobs also passed; only the schedule-only red-signal job was skipped. All eight e2e/Windows-overlay workflows from that trigger completed successfully.

The missing Windows/Node 24 coverage raised in my earlier request is now resolved. Could the owner review this for the human merge, preserving #7279 as the canonical contribution for #7276 as noted in the review?

Original CI request — resolved

The original run 44492227517 passed 11 core test shards and both core static-gate jobs. A later label-triggered run 44493839591 skipped those matrices. The requested ci:full trigger supplied the missing full matrix in the successful run linked above.

@ranxianglei ranxianglei added the ci:full Run the FULL CI matrix (all e2e lanes + 12-leg test matrix) on this PR label Nov 19, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Both claims verified against the run jobs:

  • Original run 44492227517: all 11 CORE test legs (ubuntu 22×3, ubuntu 24×3, windows 22×5) + both CORE static-gate legs (ubuntu 24, windows 22) + all guard jobs success. Correct — that is the CORE matrix for a plain PR push (three-layer CI, feat!: remove image pre-compression and the image_full tool (#3266) #3295); the windows node-24 legs exist only in FULL scope.
  • Label-triggered run 44493839591 (fired by my perf/p2/scope/other label adds): plan job correctly scoped it to skip — test and static-gate matrices skipped, guard jobs re-ran green.

So you're right that the 16-leg FULL matrix (windows node-24 × 5 included) had not run on this head, and I should have said "CORE matrix" in my CI follow-up rather than "full test matrix" — corrected here.

Done: ci:full label added. The label-triggered full-matrix run is queued now (created 10:08Z, plan job will scope it FULL: 16 test legs + 4 static-gate legs). I'll post a follow-up with the result once it completes.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Full matrix result — green. The ci:full-triggered run (44502838417) completed success on head 4ebcc229859: all 16 test legs (ubuntu 22×3, ubuntu 24×3, windows 22×5, windows 24×5) + all 4 static-gate legs (ubuntu 22/24, windows 22/24) + all guard jobs success; the single skipped job is red-signal (schedule-only, correct on a PR run).

So the head now has: CORE matrix (original run) + FULL matrix (this run) + all e2e lanes, all green. Review verdict stands — ready for the human merge whenever the owner is.

@ranxianglei

Copy link
Copy Markdown
Owner

已收编进 batch-A PR #7448(已合并)。本 PR 的 billing 查价复用修复与 #7363 中的 commit 逐字节相同(patch-id 一致),已随 #7448 合入 master 并保留原作者署名。本 PR 无需再合,关闭。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:full Run the FULL CI matrix (all e2e lanes + 12-leg test matrix) on this PR p2 perf root-cause scope/other

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Billing price lookups rebuild every override row and lowercase index

2 participants