cost.md

hookprobe — what a run costs, and what it reports about the cost

An agent run is the expensive part of hookstack, so the levers on that cost and the record of having spent it are one subject. Nothing here is required to operate the service; all of it is required to argue about the bill.

What a run costs, and why reuse is the lever

Each turn shows what it cost: model, tokens in and out, cache reads, dollars, seconds. The session's running total sits beside the session key, and the header chip carries the window's spend and its context reuse — of everything the prompts were made of (fresh input, tokens read back from the cache, tokens written to it), the share that was read back. Writes count as paid, because they are the dearer tokens: one turn on a GPT-class model behind an Anthropic-dialect gateway wrote 123k tokens and read 69k, and a ratio that ignored the writes called it 98% cached. The dollar figures are the CLI's estimate at its own price table — the shape of the bill, not the bill, for a model it does not know the price of.

The other half of that arithmetic is whether any of it helped, which nothing measured. Cost was countable to the cent from the first commit and worth was countable nowhere, so the first question anyone asks about adopting this — you want me to pay a model per alert? — had a dollar figure and nothing to set against it. A delivered report now carries Found the cause / Missed it buttons, GET /v1/budget reports investigations, ruled_useful, ruled_useless and the sentence assembled from them, and the console prints it under the spend bars: 12 investigations, $3.42, 5 found the cause. Only a person can supply that side, so an investigation nobody ruled on stays unrated — never counted as a miss.

That number matters more than it looks, because the part of the bill you cannot negotiate is large. An investigation carries roughly 29k input tokens before its alert is even mentioned, and measured on a running container it does not shrink:

configuration	input tokens
preset claude_code + 12 tools	26,920
no preset + 12 tools	25,569
no preset + 4 tools	25,569

The preset is worth ~1,350 tokens and cutting the tool list changes nothing — allowed_tools gates execution, not what is sent. Everything of ours (memory, appended methodology, skills, roles) came to under 3% of the prefix on the deployment this was measured on.

So the difference between an expensive week and a cheap one is reuse, and three habits decide it:

• Follow up in the session rather than opening a new investigation. Measured pair on the same context: $0.1490 fresh, $0.0156 on the follow-up.
• Let bursts run together. Concurrent runs share a warm prefix, so lowering HOOKPROBE_MAX_CONCURRENT to save money does the opposite.
• Batch edits to memory, prompt and skills. Each change starts the prefix over; adjusting them between alerts pays the full entry fee every time.

Whether to pay that entry fee at all is decided upstream — HOOKPROBE_ESCALATE_LEVELS, the event door's idempotency, and the budget breaker. Caching only sets the discount. The reasoning and the rejected alternative (a timer to keep the cache warm — it costs about ten times what it saves at this volume) are in .agents/notes.

Loop hygiene — the run's context and bill

Two advisory guards, neither a security boundary. They exist because an unattended run cannot notice it is wasting itself.

Repeat reminder. Running an identical call again returns the same answer at the same price. On the HOOKPROBE_REPEAT_REMINDER_AT-th identical call (and every multiple after) the result carries a note telling the agent to change approach or record what stays unknown and move on. The budget breaker stops spending after the fact; this is the nudge before it costs.

Where the dollar figure comes from. Set HOOKPROBE_PRICE_IN_PER_1M, HOOKPROBE_PRICE_CACHE_READ_PER_1M and HOOKPROBE_PRICE_OUT_PER_1M to this gateway's own rates and a turn's cost becomes measured tokens times a stated rate — the same arithmetic on every runtime. HOOKPROBE_PRICE_CACHE_WRITE_PER_1M defaults to the input rate, because writes cost at least fresh input wherever they are charged and a fourth number should not be invented silently. Unset, cost_usd stays whatever the runtime reported: the CLI's own table for a model it is not the one billing, or — on codex — nothing at all, which is what leaves a ceiling unable to bind. It is still priced and not billed: a rate goes stale and a gateway can charge for something these four numbers do not name.

Declaring rates late reprices the past, not just the future. Once a node declares its rates, every recorded turn is read at them from its recorded tokens — the budget window, the run list and its spend bars, the run page, the waterfall and the work board all go through one function (runs.turn_cost), and a figure that changed carries recorded_cost_usd beside it; the run files are not rewritten. GET /v1/budget says which table its spend came from in priced_by: declared rates or runtime estimate. Measured on the local work stack on 2026-09-23: weeks of turns recorded at the CLI's list price for the gateway's model read 30 to 60 times the gateway's rate, and with only the NEXT turn priced correctly, lowering a ceiling to its real-dollar size the same day would have tripped the breaker on money that was never spent.

The gateway's own ledger is not the reference either. LiteLLM's per-call cost headers on the Anthropic-shaped path billed neither cache writes nor cache reads — two identical 10,424-token cacheable calls were each charged for 3 input tokens — while the upstream charges an uncached prompt token at the input rate. The default of pricing writes at the input rate is the one that matches what the upstream bills.

What the ceiling can see. GET /v1/budget reports spend_visibility in three states, because two of them are not the same answer: measured (nothing in the window went unpriced, so the spend is the spend), floor (some turns priced and some not — the breaker binds, on a number that understates), and blind (nothing priced at all, so the breaker cannot bind). blind is every window on a runtime that reports tokens and never money: measured on a codex node on 2026-09-09, three turns and 93,388 input tokens read as spent_usd 0.0 with remaining_usd 1.0 and exhausted false — each figure honest alone, and together a dollar of headroom on a node whose spend nothing can see. So remaining_usd is now null there and ceiling_binds is false; the console chip goes dashed and says unmeasured. The fix is deliberately not a price table: a breaker acting on an estimated bill is worse than one that admits it cannot see, for the same reason the weekly page stopped calling any dollar billed.

Per-command deadlines. HOOKPROBE_BASH_TIMEOUT_MS and HOOKPROBE_BASH_MAX_TIMEOUT_MS become the CLI's BASH_DEFAULT_TIMEOUT_MS / BASH_MAX_TIMEOUT_MS, so a curl at an unreachable host or a kubectl at a wedged API server cannot hold a run slot until the whole run times out. The defaults match the CLI's own — this is a lever to tighten, not a change of behaviour.

Every run also records what the model was actually given: model, skill layers and their names, subagent roles, MCP servers, and content digests of the environment memory and the appended methodology. Those files live on a mutable volume, so without that record a report cannot be explained after the volume moves on — a stale line in CLAUDE.md once made every report come back in the wrong language while the request looked identical. It sits on each run and each turn (inputs), and the console's session view shows it.

Both guards and the inputs record are borrowed from DeepSeek Harness (packages/guard, and its "model-visible means logged" rule). Its third idea, spilling oversized tool output to a file, was built here and then removed: the Claude Code harness already does exactly that, and a second layer on top wrote a file that only held the harness's already-truncated copy while claiming to hold everything — see the rejected note.

The week on one page

scripts/cost_report.py joins the three services' read APIs into a governance page: what the judge billed and what its free routes avoided, what the investigator billed, how many re-fires a runbook answered for $0, what people ruled and what they have not, the budget, and the pipe's priced chains and incidents.

Since 2026-10-08 the attention section opens with the two numbers step 4 of the 90-day plan has to move, read from the pipe's own ledger through GET /attention. Reached a person counts the cards sent on a bridge channel, the one a person reads, and how many went between 23:00 and 07:00 in the operator's zone (--utc-offset; weekly_page.sh passes the host's). Pressed by a person counts card presses by kind and how many distinct people made them, never who. Both print beside pilot zero's five weeks: 138, 13, 167, 354 and 59 cards with about a third at night, and nine presses in the first week, then none. The pipe serves times and kinds; the page does the arithmetic.

Since 2026-09-15 it arrives by itself: scripts/weekly_page.sh runs the report from inside the compose network on a timer (the pipe publishes no host port, so a host-side run always printed the pipe section as unread), writes a dated page under WEEKLY_PAGE_DIR, and posts the headline lines through the pipe's watch door at level low — the operator reads it where the cards are, and the investigator declines it by level.

Since 2026-09-14 two gate verdicts are on it as well. What the decline list did — events declined at the event door, distinct conditions, priced at the week's average paid run and labelled as such — read from GET /v1/declines, and kept three-valued: a node with no list, a list that matched nothing and a node too old to answer are three different sentences. And the judge's golden gate at the last deploy, dated, with its firing-row count and the same thin qualifier the console prints, read from the judge's /status.

Since 2026-09-09 --probe is repeatable (or comma-separated), and a deployment running several investigators gets one line per node saying how much of its work waits on a person and how much was abandoned — with a node that could not be read printed as unread rather than dropped, since an unknown board is not an empty one. One probe, and the section is absent.

Since 2026-09-08 it also reports the work — the product figures rather than the bill. How much work opened in the window and how much of it completed; what share was verified and by what; how much closed with nobody stepping in; the median time to a first useful result and the median time a person took to answer an approval; how much re-fired; and how the work survived — interrupted by a restart and finished anyway, retried automatically after a provider blip, or handed back to a person. Every one of those was already recorded somewhere and had never been added up: the board answers "what is happening now" and this answers "how did the week go", which is a different question and belongs on a different page. Windowed by when the work OPENED, so a piece of work and the outcome it reached are counted in the same week. Medians, not means: one approval that waited three days would move a mean far enough to say nothing true about the other nine. Every avoided figure is a count times this week's average paid call and is labelled a counterfactual; a service that was not read gets a sentence, not a zero. Deterministic — no model is paid to do arithmetic on a ledger.

How full the context was, and when it was folded

Every turn's record now carries how full the model's context was when it ended — tokens, the limit, the percentage, and whether the runtime folds it away on its own — and any compaction that happened while it ran, with what triggered it. The console prints both on the turn line, in amber past 80% or whenever a fold happened.

The reason is one failure. A patrol on this deployment died on "the model has reached its context window limit", and nothing anywhere had said it was close. The number existed the whole time; nobody asked for it.

Asking is bounded twice, and the second bound was bought the hard way. It is an extra request to the runtime on the hot path of every turn, so it has a three second timeout — an observation that can hang is not an observation. And it is asked once per process if it fails: the first version had a ten second timeout and no latch, and CLI 2.1.259 does not answer the request at all, so a trivial turn that should have taken two and a half seconds took twelve and a half, every time, buying a number that was never going to arrive. The latch is per process rather than per run, so a deployment that upgrades its CLI gets the number back on its next restart — which is when its runtime changed anyway. On a runtime that does not answer, context stays null and the log says so once.

Compaction is not free and it is not visible: what it drops, it drops, and a report with a gap in the middle of a long investigation has no other explanation available afterwards. The PreCompact hook refuses nothing — it is the only way to know a fold happened at all.

Only the facts that keep their shape are stored. The per-kind categories breakdown follows the runtime and is dropped; a record is worth more when it holds the same keys next year. A runtime that answers nothing leaves null rather than zeros, because "we were not told" and "the context was empty" are different facts.

Asking for a compaction is not built. The runtime compacts on its own and exposes no entry point to demand one, so the honest surface is observing it. If that changes, the place to add it is the runtime contract on service.Engine, where the PRD's compact(session) belongs.

Model-call telemetry (on by default, received here, forwarded if you say where)

Every run's totals are already on its record — cost, tokens, per-model breakdown, duration, and the tool steps. What that cannot show is the shape of a run: how many model calls it took, which one was slow, where the context went, that one investigation spent on two different models.

The bundled CLI emits all of that itself over OpenTelemetry, and the SDK merges this container's environment into the CLI's, so it is configuration rather than code. It is on by default, and since 2026-09-08 the first place it goes is this service: the engine points the CLI at POST /otel/v1/{logs,metrics} on its own port, and the receiver keeps a per-run timing record under {workdir}/telemetry/ — one line per model call, tool result and counter, numbers and names only. The sessions page draws it as a waterfall: every model call and tool call on one time axis with its duration, tokens and cost, and the gaps between them, which are the run thinking or waiting on the harness — the part no bill itemises. GET /v1/runs/{key}/telemetry is the same shape as JSON, and the run's /audit record carries the totals. Nothing has to be deployed beside the service to see it.

Three things about that record. Only the CLI this service launched can write it — the subprocess carries a per-process header the receiver checks, and events are addressed to a run by the hookstack.session_key attribute below, so a body for a run this service does not know is dropped and counted. What is kept is a timing record, not a transcript: the free-text attributes the CLI can attach — prompts, tool content, responses, the account identity it stamps on every event — never reach the file, whatever the OTEL_LOG_* switches say. And it never raises into a run: a malformed body or a full disk is logged and the investigation carries on. HOOKPROBE_TELEMETRY_RECEIVER=off restores the old behaviour, where the CLI's exporter goes wherever the container's OTEL_* say, or nowhere.

A collector is still where fleet-wide questions get answered, and naming one is still the whole setup:

OTEL_EXPORTER_OTLP_ENDPOINT=http://your-collector:4318   # that is the whole setup

Set on the service, it is read by the receiver, which forwards every body it accepts to {endpoint}/v1/{signal} untouched — content switches and all, with OTEL_EXPORTER_OTLP_HEADERS if you set them — on a thread, after the response, so a slow collector cannot slow a run. http/json (the 4318 side of a collector) is the one dialect forwarded; the local record and the Grafana one are the same events, not two configurations. No default endpoint, no vendor. Point it at whatever you already run, whenever you get to it.

One event per model call, claude_code.api_request:

model: "deepseek-v4-pro"      input_tokens: 25853     output_tokens: 2
cost_usd: 0.129315            duration_ms: 1752       cache_read_tokens: 0
session.id: 1a09123d…         prompt.id: bcf7d8b9…    event.sequence: 1
query_source: "sdk"           effort: "high"

Alongside it: assistant_response, tool_decision, and the counters claude_code.token.usage / cost.usage / tool.execution / subagent.spawn / mcp.rpc / compaction / active_time.total.

This is where per-model detail appears that a run record cannot show: one investigation was observed spending on two models — a fast one for cheap turns and the main one for the reasoning — while the run's own total is a single number. Anyone re-pricing usage needs both names.

Who sent it, and which operation it belongs to

Every event carries resource attributes in three layers, set per container in the compose file rather than from one shared variable (three probes on one image would otherwise be indistinguishable): service.namespace is the deployment (work, shadow), service.name the component (hookprobe), hookstack.node the instance (watch, plan, work, probe). At the start of each run the engine adds the pipe's key beside them — hookstack.session_key=probe:watch:111 and hookstack.event_id=111 when the key embeds one — so a filter on one event id returns every model call of that operation across every node that worked on it, the same key /timeline groups the chain by. Patrol runs carry a session key and no event id; they are their own origin.

What is redacted, and by whom

The CLI substitutes <REDACTED> for content in these events — nothing in this repository does that — under five independent switches. Measured on 2.1.229:

switch	exposes	default here
OTEL_LOG_ASSISTANT_RESPONSES	the model's answers (assistant_response.response)	on
OTEL_LOG_USER_PROMPTS	the prompt — and the answers with it	on
OTEL_LOG_TOOL_DETAILS	tool arguments	on
OTEL_LOG_TOOL_CONTENT	whole tool outputs	on
OTEL_LOG_RAW_API_BODIES	raw request/response bodies per call	off
CLAUDE_CODE_OTEL_CONTENT_MAX_LENGTH	lifts the 60 KiB cap on those raw bodies	off

Four of the five are on, which is a deliberate posture and not a default to drift into: everything the agent was given and everything it produced leaves the container in cleartext once an endpoint is set — including whatever a tool happened to print, credentials included. The fifth is off because it is the one that pays most and delivers least (below). It buys the material that cost and token counts cannot give you: what the model was actually asked, and what it actually answered. Note the asymmetry between the first two: enabling prompts also enables answers, but not the reverse.

The raw bodies are capped at 60 KiB. The CLI truncates the attribute and says so in the value ([TRUNCATED - Content exceeds 60KB limit]) while body_length keeps reporting the true size, so the loss is visible rather than silent. Every real investigation exceeds it — one call here carried 113 KB. Because a request body is {model, messages, system, tools}, what falls off the end is the tool schemas first and the conversation only after that. CLAUDE_CODE_OTEL_CONTENT_MAX_LENGTH lifts the cap (measured: 113426 declared, 113425 received, valid JSON through the closing brace) and is left unset here — uncapping on top of five open switches would push the whole conversation on every call unconditionally. Set it when you want byte-exact replay. Note the unprefixed OTEL_CONTENT_MAX_LENGTH also exists and does not affect these bodies.

Two things to know before you build on it, both measured rather than assumed (CLI 2.1.229):

• These are events, not spans. OTEL_TRACES_EXPORTER is accepted but no spans were emitted for a plain run, and the events carry no traceId. A call tree is reconstructable from session.id + prompt.id + event.sequence; it does not arrive as one.
• cost_usd prices Claude models. On another provider reached through the Anthropic dialect it is an over-estimate — the run above reports $0.129 for a DeepSeek call. The event carries the real model name and the raw token counts, so re-price from those and treat the field as a relative signal.

The trace id a tool inside a turn can pick up

Because there are no spans, a kubectl or a script that runs inside a turn and happens to speak OpenTelemetry starts a brand-new trace with no parent, and lands in a collector as an orphan nobody can join back to the alert. So every runtime this service drives gets a TRACEPARENT in the environment it hands the agent — one W3C trace context, version 00, sampled.

It is derived, not propagated, and that word is the design. There is nothing to propagate: the id is a pure function of the session key (hookprobe/gate.py, trace_environment), which makes it computable by any node that knows which investigation this is, with nothing to pass along and nothing to lose on the way. The same value reaches all three adapters — the two that spawn their gate get it from gate.environment, the in-process one from _subprocess_env — because a derivation that differs between them would file two nodes' work on one alert under two traces and say nothing about it.

TRACEPARENT rather than a header because that is the name the tooling reads (otel-cli, the OpenTelemetry shell wrappers, several SDKs); W3C itself specifies a header and says nothing about environments.

What this does not do, before anybody builds on it: the parent span it names was never emitted by anything. A collector that receives a child of it shows a trace whose root is missing. That is accepted — the alternative is this service generating spans, which is a tracing backend's worth of work for a picture /telemetry already draws from event timestamps — and it is why nothing is required to read the variable. A tool that ignores it behaves exactly as before.