Skip to content

✨ Gateway spend: price models from the gateway, show unknown prices explicitly - #18463

Merged
kubestellar-prow[bot] merged 3 commits into
v5from
feat/gateway-model-pricing
Nov 16, 2026
Merged

kubestellar-prow[bot] merged 3 commits into
v5from
feat/gateway-model-pricing

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Summary

Item 2 of the gateway model-discovery work: spend tracking for gateway models. Follows #18449, which added the cached and priced OpenRouter catalog this PR builds on.

How it worked before

pkg/tokens estimated cost from a static per-model list-price table, falling back to a coarse tier price for any model it didn't know. Gateway ids such as z-ai/glm-5.3 or a self-hosted Qwen/... model almost never match the table. They were costed at the mid tier ($1.25 in / $10 out per 1M), which is 10 to 100 times too high for cheap OpenRouter models, and the result looked like a real price. The gateway path also recorded prompt_tokens as uncached input, so cached prompts were billed at the full input rate.

Price sources (pkg/dashboard/gateway_prices.go)

For the first configured gateway that serves the model (by gateway default, by an enabled agent on that gateway selecting it, or because the gateway's price list names it):

  1. prices: on the gateway (new GatewayConfig.Prices, USD per 1M tokens: input, cached_input, output, plus optional per-model models: overrides). This wins over the sources below and is the only source for vllm, llm-d, watsonx and custom gateways. cached_input defaults to input. The upsert and scan-to-fund paths keep it when they rewrite the gateway.
  2. openrouter: the pricing block of the cached public catalog (per token, converted to per 1M; input_cache_read is used as the cache-read rate).
  3. litellm: /model/info (input_cost_per_token, output_cost_per_token, cache_read_input_token_cost, cache_creation_input_token_cost), read with the gateway key.
  4. Otherwise the model is unknown.

Remote tables are fetched in the background and cached for 15 minutes; a failed refresh keeps the last good table. They are never fetched on the request path, so an offline hive doesn't stall /api/cost. Server.Start installs the pricer process-wide through tokens.SetGatewayPricer. RegisterAPI does not, so unit tests keep the static table.

Accounting (pkg/tokens)

  • ResolvePrice / EstimateCostWithSource return a PriceSource: gateway, table, tier or unknown. A model seen on the inference (gateway) path, or served by a configured gateway, is never tier-guessed. It is priced from the gateway or the table, or else marked unknown.
  • EstimateFromSummary leaves unknown-priced tokens out of TotalUSD and lists them in unknown_price_models. It also adds price_source and unknown_price_models to model and agent rows. EstimateCostUSD keeps its signature and its behaviour when no pricer is installed.

Cached tokens (crux/proxy)

  • The gateway path now reads usage.prompt_tokens_details.cached_tokens, for both non-streaming responses and the final SSE usage chunk. Sinks that implement RecordWithCache get uncached input = prompt − cached, plus cache reads (clamped). *tokens.InferenceSink and the crux callback sink implement it through a new gateway_cache usage kind. A sink without it gets the previous behaviour.

Where the numbers show up

  • Usage gauge: /api/providers/headroom adds gateway_spend, one row per gateway with enabled agents: agents, USD, tokens and unknown_price_models. The gauge shows a dollar row per gateway ("$1.23 at gateway prices", "… plus price unknown for X (not counted)", or "Price unknown (X)…"), never $0. The rows show even when no plan is measured.
  • Cost card: rows with source: "unknown" show "price unknown" instead of $0, and a note lists the unknown models and says the total is a lower bound.
  • Docs: docs/inference-backends.md gets a "Gateway spend and prices" section with a prices: example. crux.md lists the new callback kind.

Not changed: repo-cost and launch-cost attribution still use EstimateCostUSD. Gateway prices reach them automatically, but an unknown-priced model there counts as $0 with the existing approximate (estimated_tier) marking.

Tests

  • pkg/tokens: the gateway price wins over the table; cached input is billed at the cache rate; unknown is excluded from the total and listed (model and agent rows); gateway-path models are never tier-guessed; CLI models still get the tier price.
  • pkg/dashboard: OpenRouter catalog to price table (cache read, unknown skipped), LiteLLM /model/info parse and keyed fetch, /model/info URL, the config override (flat, per-model, defaults), the vllm unknown path, and end to end through tokens.SetGatewayPricer.
  • crux/proxy: cached tokens from JSON and SSE; split, clamp and fallback in recordGatewayUsage. pkg/cruxhost: the gateway_cache kind and its fallback.
  • Node: gateway_spend_gauge.test.js covers the gauge spend rows and the priced, partial and unknown text.

🤖 Generated with Claude Code

@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18471 ✨ inbox: mail-in files as email items with an owner approval queue (Betty C2 part 5) same files src/pkg/dashboard/static/index.html
#18432 ✨ Inbox: auto-transcribe hive recordings same files src/pkg/dashboard/static/index.html

@hivecommons-hive

hivecommons-hive Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the review swarm requested changes

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing

⏱️ The 900s mutation budget ran out, so 1 package(s) were not fully mutated and the results below do not cover them:

  • pkg/tokens: skipped (not started)

❌ gremlins did not produce a report for 1 package(s), so no verdict can be derived from this run (see the job log):

  • pkg/dashboard: gremlins exited 137

gremlins found 8 surviving mutant(s) in changed files: the code ran under tests, but changing it did not fail any test.

File Line Mutation
src/crux/proxy/anthropic_translate.go 395 CONDITIONALS_BOUNDARY
src/crux/proxy/github_proxy_inference.go 563 CONDITIONALS_NEGATION
src/crux/proxy/github_proxy_inference.go 761 CONDITIONALS_NEGATION
src/crux/proxy/host_seams.go 94 CONDITIONALS_BOUNDARY
src/crux/proxy/host_seams.go 94 CONDITIONALS_BOUNDARY
src/crux/proxy/host_seams.go 94 CONDITIONALS_NEGATION
src/crux/proxy/host_seams.go 94 CONDITIONALS_NEGATION
src/pkg/cruxhost/cruxhost.go 291 ARITHMETIC_BASE

Verdict: error (#17349).

…citly

- tokens: gateway price source hook (ResolvePrice/EstimateCostWithSource);
  gateway-path models are never tier-guessed; unknown prices are excluded
  from totals and listed
- dashboard: OpenRouter catalog prices, LiteLLM /model/info, and a new
  per-gateway prices: override, cached in the background
- proxy: record cached prompt tokens (prompt_tokens_details.cached_tokens)
  so the cache-read rate applies; new gateway_cache callback kind
- usage gauge: one spend row per gateway; cost card says price unknown

Signed-off-by: Andy Anderson <andy@clubanderson.com>
Signed-off-by: Andy Anderson <andy@clubanderson.com>
…arity

Signed-off-by: Andy Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 3465a98. I read the diff and src/pkg/dashboard/openrouter.go at that head. I did not run the tests.

1. A failed or empty OpenRouter catalog fetch leaves every OpenRouter model "price unknown" for 15 minutes (src/pkg/dashboard/gateway_prices.go, remoteGatewayPrices and cachedGatewayPrices)

  • cachedGatewayPrices sets attemptedAt before it fetches. It keeps the old table only when err == nil && len(t) > 0, and it won't refetch until gatewayPriceTTL (15 min) has passed.
  • The OpenRouter fetch closure is return openRouterPriceTable(or.PublicModels()), nil. It always returns a nil error.
  • PublicModels() returns an empty list when the catalog is unreadable. availableOpenRouterSuggested in openrouter.go says so: "An empty live list means the catalog was unreadable".
  • So a cold or offline start, or a transient catalog failure, gives an empty table. After that, gatewayModelPrice returns (_, known=false, served=true) for any model that is an agent's or the gateway's default.
  • ResolvePrice then reports PriceSourceUnknown, so spend for those models shows as "price unknown" for the full 15 minutes even if the catalog recovers seconds later.
  • The first request after boot hits the same path, because the table is nil until the async fetch finishes.

Suggested fix: return an error when the catalog is empty, and shorten the retry delay after a failed attempt. The first snippet changes the fetch closure in remoteGatewayPrices. The second adds a failure-retry interval to cachedGatewayPrices.

 		return cachedGatewayPrices(key, func() (gatewayPriceTable, error) {
-			return openRouterPriceTable(or.PublicModels()), nil
+			t := openRouterPriceTable(or.PublicModels())
+			if len(t) == 0 {
+				return nil, errors.New("openrouter catalog empty")
+			}
+			return t, nil
 		})
 		if err == nil && len(t) > 0 {
 			e.table = t
+		} else {
+			e.attemptedAt = gatewayPriceNow().Add(-gatewayPriceTTL + gatewayPriceRetry) // retry soon
 		}

gatewayPriceRetry would be a new constant, for example 30 seconds. Add the errors import. Add a test where the first fetch returns empty and the second succeeds after the retry interval.

Everything else I checked looks correct:

  • recordGatewayUsage subtracts cached tokens from input, and cached tokens are clamped to the prompt size.
  • The sink falls back to full-input recording when it lacks RecordWithCache.
  • The unknown-price handling in EstimateFromSummary and the cost-source tagging.

— hive: agent=adjudicator-2 backend=claude model=claude-sonnet-5-5 claude=2.1.325

@clubanderson clubanderson added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 16, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow
kubestellar-prow Bot merged commit 6980a26 into v5 Nov 16, 2026
98 of 100 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/gateway-model-pricing branch November 16, 2026 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant