Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/added-gateway-model-pricing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Gateway spend is priced from the gateway: OpenRouter's catalog prices, LiteLLM `/model/info`, or a new per-gateway `prices:` block (input, cached input, output per 1M tokens). Cached prompt tokens are billed at the cache-read rate. Gateway models with no price show as "price unknown" instead of $0 or a tier guess, and the usage gauge gets a spend row per gateway.
29 changes: 29 additions & 0 deletions docs/inference-backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,35 @@ A spoke with an OpenRouter key proxies OpenRouter's `/api/v1/key` credit endpoin

Hive probes `/v1/models` on OpenAI-compatible gateways. LiteLLM discovery includes bearer auth when a key is configured. If discovery fails, the UI falls back to static or configured model lists and marks entries as unverified; fallback data should not be treated as proof that the endpoint is healthy.

## Gateway spend and prices

The cost card and the Overview usage gauge price gateway models from the gateway itself, in this order:

1. A `prices:` block on the gateway in `hive.yaml` (USD per 1M tokens). It wins over everything else and is the only source for `vllm`, `llm-d`, `watsonx` and `custom` gateways, which publish no prices.
2. For `openrouter`, the `pricing` block of the public `/models` catalog that model discovery already reads (input, output and cache-read prices per token).
3. For `litellm`, the proxy's `/model/info` (`input_cost_per_token`, `output_cost_per_token`, `cache_read_input_token_cost`), read with the gateway key.
4. The static list-price table, for gateway ids it knows.

If a gateway model has none of these, it shows as **price unknown**. Its tokens are left out of the total (which is then a lower bound) and listed by name. It is never counted as $0 or as a tier guess. When the gateway reports cached prompt tokens (`usage.prompt_tokens_details.cached_tokens`), they are billed at the cache-read price. A cached-input price you leave unset defaults to the input price.

```yaml
governor:
gateways:
- name: local-vllm
kind: vllm
endpoint: http://vllm:8000
prices: # USD per 1M tokens
input: 0.20
cached_input: 0.02
output: 0.60
models: # optional per-model overrides
Qwen/Qwen3-235B:
input: 0.90
output: 2.70
```

The gauge shows gateways as dollar rows, not dials, because a gateway has no plan to measure. Each row covers the enabled agents whose backend names that gateway, over the same token window as the cost card.

## Guided example: Watsonx via gateway

The watsonx path uses the same gateway machinery: configure the gateway endpoint and credentials, verify `/v1/models`, then assign an agent to that backend/gateway. Use it as the guided flow for any enterprise gateway:
Expand Down
7 changes: 6 additions & 1 deletion src/cmd/hive/boot_launch_deps.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ func defaultBootLaunchDeps() bootLaunchDeps {
}
},
startAgent: func(ctx context.Context, mgr crux.Runtime, name string) error { return mgr.Start(ctx, name) },
serve: (*dashboard.Server).Start,
// Gateway pricing is process-wide (tokens.SetGatewayPricer), so the
// production boot installs it, not Server.Start, which tests call.
serve: func(s *dashboard.Server) error {
s.InstallGatewayPricer()
return s.Start()
},
}
}
5 changes: 4 additions & 1 deletion src/crux/api/callback/callback.go
Original file line number Diff line number Diff line change
Expand Up @@ -268,11 +268,14 @@ type RateLimitSnapshot struct {
OK bool `json:"ok"`
}

// Usage kinds, the three UsageSink record paths.
// Usage kinds, the UsageSink record paths.
const (
UsageKindGateway = "gateway"
UsageKindCopilot = "copilot"
UsageKindCopilotCache = "copilot_cache"
// UsageKindGatewayCache is a gateway response whose prompt was partly
// served from cache: InputTokens excludes CacheRead.
UsageKindGatewayCache = "gateway_cache"
)

// UsageRecord is the body of POST /v1/usage.
Expand Down
4 changes: 2 additions & 2 deletions src/crux/api/callback/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -241,9 +241,9 @@ func (s *Server) handleUsage(w http.ResponseWriter, r *http.Request) {
return
}
switch rec.Kind {
case UsageKindGateway, UsageKindCopilot, UsageKindCopilotCache:
case UsageKindGateway, UsageKindGatewayCache, UsageKindCopilot, UsageKindCopilotCache:
default:
s.writeError(w, r, http.StatusBadRequest, CodeBadRequest, "kind must be gateway, copilot or copilot_cache")
s.writeError(w, r, http.StatusBadRequest, CodeBadRequest, "kind must be gateway, gateway_cache, copilot or copilot_cache")
return
}
if err := s.Caps.RecordUsage(r.Context(), rec); err != nil {
Expand Down
25 changes: 25 additions & 0 deletions src/crux/config/agent_config.go
Original file line number Diff line number Diff line change
Expand Up @@ -647,6 +647,31 @@ type GatewayConfig struct {
// round-trip; an absent name is a normal, backwards-compatible state the
// dashboard renders as "(unnamed)" rather than an error.
KeyName string `yaml:"key_name,omitempty" json:"key_name,omitempty"`
// Prices optionally sets what this gateway charges, in USD per 1M tokens,
// so spend reports can cost its models. It is the only price source for
// vllm, llm-d, watsonx and custom gateways (which publish none) and wins
// over the OpenRouter catalog and LiteLLM /model/info when set. Without a
// price the dashboard reports the gateway's spend as "price unknown"
// rather than $0. omitempty keeps existing hive.yaml files byte-identical.
Prices *GatewayPrices `yaml:"prices,omitempty" json:"prices,omitempty"`
}

// GatewayPrices is a gateway's configured price, USD per 1M tokens. The
// top-level rates apply to every model the gateway serves; Models overrides
// them for specific model ids. CachedInput defaults to Input when unset (cached
// prompt tokens are never assumed free).
type GatewayPrices struct {
Input *float64 `yaml:"input,omitempty" json:"input,omitempty"`
CachedInput *float64 `yaml:"cached_input,omitempty" json:"cached_input,omitempty"`
Output *float64 `yaml:"output,omitempty" json:"output,omitempty"`
Models map[string]GatewayModelPrice `yaml:"models,omitempty" json:"models,omitempty"`
}

// GatewayModelPrice is one model's configured price, USD per 1M tokens.
type GatewayModelPrice struct {
Input *float64 `yaml:"input,omitempty" json:"input,omitempty"`
CachedInput *float64 `yaml:"cached_input,omitempty" json:"cached_input,omitempty"`
Output *float64 `yaml:"output,omitempty" json:"output,omitempty"`
}

// TrajectoryConfig governs the trajectory-review lane: a periodic,
Expand Down
6 changes: 6 additions & 0 deletions src/crux/hivecap/hivecap.go
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,12 @@ func (u UsageSink) Record(agent, model string, in, out int64) {
u.send(callback.UsageRecord{Agent: agent, Model: model, Kind: callback.UsageKindGateway, InputTokens: in, OutputTokens: out})
}

// RecordWithCache records a gateway response with its cached prompt tokens
// split out (in excludes cacheRead).
func (u UsageSink) RecordWithCache(agent, model string, in, out, cacheRead, cacheCreation int64) {
u.send(callback.UsageRecord{Agent: agent, Model: model, Kind: callback.UsageKindGatewayCache, InputTokens: in, OutputTokens: out, CacheRead: cacheRead, CacheCreation: cacheCreation})
}

func (u UsageSink) RecordCopilot(agent, model string, in, out int64) {
u.send(callback.UsageRecord{Agent: agent, Model: model, Kind: callback.UsageKindCopilot, InputTokens: in, OutputTokens: out})
}
Expand Down
52 changes: 46 additions & 6 deletions src/crux/proxy/anthropic_translate.go
Original file line number Diff line number Diff line change
Expand Up @@ -254,14 +254,22 @@ func translateAnthropicToolChoice(raw json.RawMessage) json.RawMessage {
// usage block is present. Used to attribute inference-agent token consumption
// into the token store.
func extractOpenAIUsage(body []byte) (inputTokens, outputTokens int64) {
u := extractOpenAIUsageDetailed(body)
return u.Input, u.Output
}

// extractOpenAIUsageDetailed is extractOpenAIUsage plus the cached part of
// the prompt (usage.prompt_tokens_details.cached_tokens).
func extractOpenAIUsageDetailed(body []byte) gatewayUsage {
var resp openaiResponse
if err := json.Unmarshal(body, &resp); err != nil {
return 0, 0
if err := json.Unmarshal(body, &resp); err != nil || resp.Usage == nil {
return gatewayUsage{}
}
if resp.Usage == nil {
return 0, 0
return gatewayUsage{
Input: int64(resp.Usage.PromptTokens),
Output: int64(resp.Usage.CompletionTokens),
Cached: int64(resp.Usage.cachedTokens()),
}
return int64(resp.Usage.PromptTokens), int64(resp.Usage.CompletionTokens)
}

// translateOpenAIResponseToAnthropic converts a non-streaming OpenAI Chat
Expand Down Expand Up @@ -332,6 +340,13 @@ func translateOpenAIResponseToAnthropic(body []byte, model string) ([]byte, erro
// stream_options.include_usage). Returns (0, 0) when the gateway did not emit
// usage.
func translateOpenAISSEToAnthropic(r io.Reader, w io.Writer, model string) (inputTokens, outputTokens int64, err error) {
u, err := translateOpenAISSEToAnthropicUsage(r, w, model)
return u.Input, u.Output, err
}

// translateOpenAISSEToAnthropicUsage is translateOpenAISSEToAnthropic that
// also reports the cached part of the prompt from the final usage chunk.
func translateOpenAISSEToAnthropicUsage(r io.Reader, w io.Writer, model string) (gatewayUsage, error) {
msgID := fmt.Sprintf("msg_%d", time.Now().UnixNano())

writeSSE(w, "message_start", anthropicSSEMessageStart{
Expand All @@ -352,6 +367,7 @@ func translateOpenAISSEToAnthropic(r io.Reader, w io.Writer, model string) (inpu

var totalInputTokens int
var totalOutputTokens int
var totalCachedTokens int
var contentBlockIndex int
textBlockStarted := false
toolCallState := make(map[int]*sseToolCallAccumulator)
Expand All @@ -376,6 +392,9 @@ func translateOpenAISSEToAnthropic(r io.Reader, w io.Writer, model string) (inpu
if chunk.Usage.PromptTokens > 0 {
totalInputTokens = chunk.Usage.PromptTokens
}
if c := chunk.Usage.cachedTokens(); c > 0 {
totalCachedTokens = c
}
}

if len(chunk.Choices) > 0 {
Expand Down Expand Up @@ -473,7 +492,7 @@ func translateOpenAISSEToAnthropic(r io.Reader, w io.Writer, model string) (inpu
}

writeSSE(w, "message_stop", map[string]string{"type": "message_stop"})
return int64(totalInputTokens), int64(totalOutputTokens), scanner.Err()
return gatewayUsage{Input: int64(totalInputTokens), Output: int64(totalOutputTokens), Cached: int64(totalCachedTokens)}, scanner.Err()
}

type sseToolCallAccumulator struct {
Expand Down Expand Up @@ -697,6 +716,27 @@ type openaiUsage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
// PromptTokensDetails.CachedTokens is the part of PromptTokens served
// from the provider's prompt cache (OpenAI, OpenRouter, LiteLLM, vLLM
// with prefix caching). It bills at the cache-read rate.
PromptTokensDetails *struct {
CachedTokens int `json:"cached_tokens"`
} `json:"prompt_tokens_details,omitempty"`
}

// cachedTokens is the cached part of the prompt, 0 when not reported.
func (u *openaiUsage) cachedTokens() int {
if u == nil || u.PromptTokensDetails == nil {
return 0
}
return u.PromptTokensDetails.CachedTokens
}

// gatewayUsage is one upstream response's token usage on the gateway path.
// Input is the FULL prompt as the gateway reports it (prompt_tokens); Cached
// is the part of it read from the prompt cache.
type gatewayUsage struct {
Input, Output, Cached int64
}

type openaiSSEChunk struct {
Expand Down
76 changes: 76 additions & 0 deletions src/crux/proxy/gateway_cached_usage_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
package proxy

import (
"bytes"
"log/slog"
"strings"
"testing"
)

// cacheSink records gateway usage with the cached split, like
// *tokens.InferenceSink.
type cacheSink struct {
memUsageSink
in, out, cacheRead int64
calls int
}

func (c *cacheSink) RecordWithCache(agent, model string, in, out, cacheRead, cacheCreation int64) {
c.calls++
c.in, c.out, c.cacheRead = in, out, cacheRead
}

func TestExtractOpenAIUsageDetailedReadsCachedTokens(t *testing.T) {
body := []byte(`{"choices":[],"usage":{"prompt_tokens":1000,"completion_tokens":50,"prompt_tokens_details":{"cached_tokens":800}}}`)
u := extractOpenAIUsageDetailed(body)
if u.Input != 1000 || u.Output != 50 || u.Cached != 800 {
t.Fatalf("usage = %+v", u)
}
if in, out := extractOpenAIUsage(body); in != 1000 || out != 50 {
t.Fatalf("legacy extract = %d %d", in, out)
}
if u := extractOpenAIUsageDetailed([]byte(`{"usage":{"prompt_tokens":5,"completion_tokens":1}}`)); u.Cached != 0 {
t.Fatalf("no details must mean no cached tokens: %+v", u)
}
}

func TestSSEUsageReportsCachedTokens(t *testing.T) {
stream := strings.Join([]string{
`data: {"choices":[{"delta":{"content":"hi"}}]}`,
`data: {"choices":[],"usage":{"prompt_tokens":1200,"completion_tokens":30,"prompt_tokens_details":{"cached_tokens":1000}}}`,
`data: [DONE]`,
}, "\n") + "\n"
var out bytes.Buffer
u, err := translateOpenAISSEToAnthropicUsage(strings.NewReader(stream), &out, "deepseek/deepseek-v4.1-flash")
if err != nil {
t.Fatal(err)
}
if u.Input != 1200 || u.Output != 30 || u.Cached != 1000 {
t.Fatalf("usage = %+v", u)
}
}

// OpenAI-shaped usage counts cached tokens INSIDE prompt_tokens: the sink gets
// the uncached remainder as input and the cached part as cache reads.
func TestRecordGatewayUsageSplitsCachedPrompt(t *testing.T) {
p := &GitHubProxy{logger: slog.Default()}
sink := &cacheSink{}
p.SetTokenSink(sink)
p.recordGatewayUsage("a", "m", gatewayUsage{Input: 1000, Output: 50, Cached: 800})
if sink.calls != 1 || sink.in != 200 || sink.out != 50 || sink.cacheRead != 800 {
t.Fatalf("split = %+v", sink)
}
// Cached larger than the prompt is clamped, never a negative input.
p.recordGatewayUsage("a", "m", gatewayUsage{Input: 10, Output: 1, Cached: 99})
if sink.in != 0 || sink.cacheRead != 10 {
t.Fatalf("clamp = %+v", sink)
}
// No cached tokens: the plain Record path, as before.
p.recordGatewayUsage("b", "m", gatewayUsage{Input: 7, Output: 3})
if recs := sink.forAgent("b"); len(recs) != 1 {
t.Fatalf("plain record = %+v", recs)
}
// Nothing to record, and a nil sink, are no-ops.
p.recordGatewayUsage("c", "m", gatewayUsage{})
(&GitHubProxy{logger: slog.Default()}).recordGatewayUsage("c", "m", gatewayUsage{Input: 1, Cached: 1})
}
20 changes: 6 additions & 14 deletions src/crux/proxy/github_proxy_inference.go
Original file line number Diff line number Diff line change
Expand Up @@ -559,13 +559,11 @@ func (p *GitHubProxy) inferenceTranslatorHandler() http.Handler {
if f, ok := w.(http.Flusher); ok {
f.Flush()
}
in, out, err := translateOpenAISSEToAnthropic(resp.Body, w, route.Model)
usage, err := translateOpenAISSEToAnthropicUsage(resp.Body, w, route.Model)
if err != nil {
p.logger.Error("inference SSE translation failed", "agent", agentName, "error", err)
}
if in > 0 || out > 0 {
p.recordUsage(agentName, route.Model, in, out)
}
p.recordGatewayUsage(agentName, route.Model, usage)
return
}

Expand All @@ -581,9 +579,7 @@ func (p *GitHubProxy) inferenceTranslatorHandler() http.Handler {
"preview", truncateBytes(respBody, 500),
)

if in, out := extractOpenAIUsage(respBody); in > 0 || out > 0 {
p.recordUsage(agentName, route.Model, in, out)
}
p.recordGatewayUsage(agentName, route.Model, extractOpenAIUsageDetailed(respBody))

translated, err := translateOpenAIResponseToAnthropic(respBody, route.Model)
if err != nil {
Expand Down Expand Up @@ -761,13 +757,11 @@ func (p *GitHubProxy) handleInferenceRequest(conn net.Conn, req *http.Request, a
if _, err := conn.Write([]byte(hdr)); err != nil {
return
}
in, out, err := translateOpenAISSEToAnthropic(resp.Body, conn, route.Model)
usage, err := translateOpenAISSEToAnthropicUsage(resp.Body, conn, route.Model)
if err != nil {
p.logger.Error("inference SSE translation failed", "agent", agentName, "error", err)
}
if in > 0 || out > 0 {
p.recordUsage(agentName, route.Model, in, out)
}
p.recordGatewayUsage(agentName, route.Model, usage)
return
}

Expand All @@ -777,9 +771,7 @@ func (p *GitHubProxy) handleInferenceRequest(conn net.Conn, req *http.Request, a
return
}

if in, out := extractOpenAIUsage(respBody); in > 0 || out > 0 {
p.recordUsage(agentName, route.Model, in, out)
}
p.recordGatewayUsage(agentName, route.Model, extractOpenAIUsageDetailed(respBody))

translated, err := translateOpenAIResponseToAnthropic(respBody, route.Model)
if err != nil {
Expand Down
24 changes: 24 additions & 0 deletions src/crux/proxy/host_seams.go
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,30 @@ func (p *GitHubProxy) recordUsage(agent, model string, in, out int64) {
}
}

// cacheUsageSink is the optional UsageSink extension that records the
// cached part of a gateway prompt separately, so it is priced at the
// cache-read rate. *tokens.InferenceSink and the crux callback sink implement
// it; a sink without it gets the full prompt as input, as before.
type cacheUsageSink interface {
RecordWithCache(agent, model string, inputTokens, outputTokens, cacheRead, cacheCreation int64)
}

// recordGatewayUsage records one gateway response. OpenAI-shaped usage counts
// cached tokens INSIDE prompt_tokens, so the uncached input is the
// difference; recording the full prompt as input would bill cached tokens at
// the input rate.
func (p *GitHubProxy) recordGatewayUsage(agent, model string, u gatewayUsage) {
if u.Input <= 0 && u.Output <= 0 {
return
}
cached := min(max(u.Cached, 0), max(u.Input, 0))
if cs, ok := p.tokenSink.(cacheUsageSink); ok && cached > 0 {
cs.RecordWithCache(agent, model, u.Input-cached, u.Output, cached, 0)
return
}
p.recordUsage(agent, model, u.Input, u.Output)
}

func (p *GitHubProxy) recordCopilotUsage(agent, model string, in, out int64) {
if p.tokenSink != nil {
p.tokenSink.RecordCopilot(agent, model, in, out)
Expand Down
2 changes: 1 addition & 1 deletion src/docs/design/crux.md
Original file line number Diff line number Diff line change
Expand Up @@ -881,7 +881,7 @@ it links nothing of hive.
| Mint scoped token | `POST /v1/tokens` | agent, repo (`owner/name`) or repos (names), tier, requested TTL | token, `expires_at` |
| GitHub REST accounting | `POST /v1/github/rest` | caller, method, path, status, injected (bool), the `X-RateLimit-*` response headers | `204` |
| Core rate-limit snapshot | `GET /v1/github/rate-limit` | — | remaining, reset, ok |
| Inference usage | `POST /v1/usage` | agent, model, kind (`gateway` / `copilot` / `copilot_cache`), input, output, cache read, cache creation | `204` |
| Inference usage | `POST /v1/usage` | agent, model, kind (`gateway` / `gateway_cache` / `copilot` / `copilot_cache`), input, output, cache read, cache creation | `204` |
| Canary scan | `POST /v1/scan/canary` | agent, text, source | detected, agent, source (**never the canary token**) |
| Text scan | `POST /v1/scan/text` | direction (`input` / `output`), text | blocked, critical_injection, findings (kind, masked snippet, rule, high) |
| Condition | `POST /v1/conditions` | action (`observe` / `clear`), class, key, message, diagnostics | `204` |
Expand Down
Loading
Loading