Skip to content

Commit f2bc2cb

Browse files
arunSunnyKVSclaude
andcommitted
docs(pricing): document testing cost and scale display precision
Cost precision now scales in three bands rather than two. Between a cent and a dime the third decimal is what distinguishes one evaluator from another — $0.037 vs $0.049 is a third more expensive, but both round to the same two-decimal figure — so that band keeps it. From a dime up the precision is noise, so a run total reads as money ($0.18, not $0.177). Sub-cent amounts keep two significant figures as before, since a fixed decimal count renders them "$0.0000" and they read as free. docs/cli.md said "No cost estimation is performed", which this feature made false. That section is now "Token usage and testing cost" and covers the per-model split, where prices come from, and the accuracy caveats — chiefly that multi-turn runs read high, because providers discount repeated context and opfor prices every input token at full rate. Users budgeting off this number should know it is conservative. Also documents that "testing cost" is opfor's own spend and excludes the target's inference cost, and that an unpriced model is never counted as free. Same summary added to the README and referenced from the browser extension guide. Note: the SDK's report type exposes neither tokenUsage nor cost. That gap predates this work and is left alone here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 0e5c360 commit f2bc2cb

5 files changed

Lines changed: 69 additions & 14 deletions

File tree

README.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,26 @@ When you run a scan, opfor:
111111

112112
Each run lands in its own subfolder under `.opfor/reports/run-report-<compactTs>-<slug>-<shortId>/` containing `<slug>-report.html` and `<slug>-report.json`. Autonomous `opfor hunt` runs use the same layout under `hunt-report-<compactTs>-<slug>-<shortId>/`.
113113

114+
### Testing cost
115+
116+
Every run reports what it cost to run, broken down by model:
117+
118+
```
119+
Token usage: 51,323 input / 6,057 output (57,380 total)
120+
Testing cost: ≈$0.18
121+
deepseek/deepseek-v4-pro [attacker]: ≈$0.037
122+
anthropic/claude-opus-5 [judge]: ≈$0.14
123+
```
124+
125+
This is **opfor's own spend** — the attacker and judge LLMs. It excludes your target's inference cost, which opfor cannot see from the outside. The per-model split is the useful part: the judge is often the bigger share, and pointing it at a cheaper model is usually the easiest saving.
126+
127+
Prices come from a snapshot of LiteLLM's public price map that ships with the package, so runs work offline and a report re-rendered later produces the same figure. Two caveats worth knowing:
128+
129+
- **Multi-turn runs read high.** Providers discount repeated context, and a multi-turn attack re-sends the conversation each turn — opfor prices every input token at full rate, so the real bill is usually lower.
130+
- **Unknown models are never counted as free.** A model missing from the price table is reported as unpriced and the total is marked a lower bound, rather than silently reading as $0.
131+
132+
[Token usage and testing cost](docs/cli.md#token-usage-and-testing-cost)
133+
114134
## Evaluator coverage
115135

116136
Opfor ships with curated suites that map to industry standards. Pick a suite or run individual evaluators.

core/src/pricing/estimateCost.ts

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -68,20 +68,24 @@ export function estimateRunCost(breakdown?: ModelTokenUsage[]): RunCost | undefi
6868
}
6969

7070
/**
71-
* Format a USD amount for display.
71+
* Format a USD amount for display, with precision that scales to the amount.
7272
*
73-
* Fractions of a cent are the norm here — a smoke suite on a cheap model lands
74-
* around $0.00003 — and a fixed number of decimals renders those as "$0.0000",
75-
* which reads as free. So small amounts are shown to two significant figures
76-
* instead, and anything below a millionth of a dollar is labelled as such rather
77-
* than rounded away.
73+
* Three bands, each solving a different problem:
74+
* - Below a cent, a fixed decimal count renders "$0.0000" and reads as free,
75+
* so these show two significant figures instead ("$0.0034").
76+
* - Between a cent and a dime, the third decimal is what distinguishes one
77+
* evaluator from another — "$0.037" vs "$0.049" is a third more expensive,
78+
* but both round to the same two-decimal figure.
79+
* - From a dime up, that precision is noise; show it as money ("$0.18").
80+
*
81+
* Anything below a millionth of a dollar is labelled rather than rounded away.
7882
*/
7983
export function formatUsd(usd: number): string {
8084
if (usd === 0) return "$0.00";
8185
if (usd < 0.000001) return "<$0.000001";
8286
// Number() strips the trailing zeros toPrecision leaves behind (0.0034 stays
8387
// "0.0034" rather than becoming "0.0034000").
8488
if (usd < 0.01) return `$${Number(usd.toPrecision(2))}`;
85-
if (usd < 1) return `$${usd.toFixed(3)}`;
89+
if (usd < 0.1) return `$${usd.toFixed(3)}`;
8690
return `$${usd.toFixed(2)}`;
8791
}

core/tests/pricing.test.ts

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -216,10 +216,24 @@ test("sub-cent runs keep enough precision not to read as free", () => {
216216
assert.equal(formatUsd(0.000001), "$0.000001");
217217
assert.equal(formatUsd(0.0034), "$0.0034");
218218
assert.equal(formatUsd(0.0099), "$0.0099");
219-
assert.equal(formatUsd(0.25), "$0.250");
220219
assert.equal(formatUsd(12.3456), "$12.35");
221220
});
222221

222+
test("cent-to-dime amounts keep the digit that separates one evaluator from another", () => {
223+
// Real per-evaluator figures from a run. At two decimals these collapse to
224+
// $0.04 / $0.05 / $0.06 and you can no longer see which evaluator is dearest.
225+
assert.equal(formatUsd(0.037), "$0.037");
226+
assert.equal(formatUsd(0.049), "$0.049");
227+
assert.equal(formatUsd(0.055), "$0.055");
228+
});
229+
230+
test("a dime and up reads as plain money, not false precision", () => {
231+
assert.equal(formatUsd(0.1), "$0.10");
232+
assert.equal(formatUsd(0.177), "$0.18");
233+
assert.equal(formatUsd(0.25), "$0.25");
234+
assert.equal(formatUsd(0.999), "$1.00");
235+
});
236+
223237
test("amounts too small to show are labelled, never rounded to zero", () => {
224238
assert.equal(formatUsd(0.0000000001), "<$0.000001");
225239
});

docs/browser-extension.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ The extension uses a **single LLM configuration** for all operations (attack gen
8888

8989
The extension runs up to **20 turns per evaluator** (default 10). It stops a given evaluator early when the judge returns a definitive verdict.
9090

91-
**Token usage** is tracked per evaluator and shown on the Done screen and in the downloadable HTML report.
91+
**Token usage and testing cost** are tracked per evaluator and shown on the Done screen and in the downloadable HTML report. Cost covers the attacker and judge LLMs you configured in Options — the chat UI you are testing is driven through the browser, so there is nothing to bill for the target itself. See [Token usage and testing cost](cli.md#token-usage-and-testing-cost) for how the figure is derived and its caveats.
9292

9393
---
9494

docs/cli.md

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -215,21 +215,38 @@ Partial reports include all completed evaluator results and are marked with `sto
215215

216216
---
217217

218-
## Token usage tracking
218+
## Token usage and testing cost
219219

220-
Every LLM call (attacker generation, adaptive follow-ups, judge) is metered. After the run completes, the CLI prints a summary line:
220+
Every LLM call (attacker generation, adaptive follow-ups, judge) is metered, and each is attributed to the model that made it. After the run completes, the CLI prints:
221221

222222
```
223223
Results: 5 passed, 2 failed, 0 errors
224224
Safety score: 71%
225225
Token usage: 51,323 input / 6,057 output (57,380 total)
226+
Testing cost: ≈$0.18
227+
deepseek/deepseek-v4-pro [attacker]: ≈$0.037
228+
anthropic/claude-opus-5 [judge]: ≈$0.14
226229
```
227230

228-
Token usage is also included in the JSON report (`summary.tokenUsage` and per-evaluator `tokenUsage` fields) and in the HTML report's executive summary card. When using `--events`, the `run_finish` event includes token counts in its `summary` payload.
231+
The per-model split is the actionable part — the judge is frequently the larger share, and switching it to a cheaper model is usually the easiest saving.
229232

230-
The browser extension shows a `Tokens` stat on its Done screen.
233+
Both are included in the JSON report (`summary.tokenUsage`, `summary.tokenUsageByModel`, `summary.cost`, plus the same fields per evaluator) and in the HTML report's executive summary. When using `--events`, the `run_finish` event includes them in its `summary` payload.
231234

232-
> Token counts reflect raw model usage (input + output tokens). No cost estimation is performed — provider pricing varies and changes frequently.
235+
The browser extension shows the same figures on its Done screen.
236+
237+
### What the cost figure covers
238+
239+
**"Testing cost" is opfor's own spend — the attacker and judge LLM calls.** It excludes your target's inference cost, which opfor cannot observe from the outside.
240+
241+
Prices come from a snapshot of LiteLLM's public price map, vendored into the package. Nothing is downloaded at runtime, so runs work offline and a report re-rendered months later produces the same figure. The snapshot version is recorded in the JSON as `summary.cost.priceTableVersion`.
242+
243+
Maintainers refresh it with `npm run build:pricing` (`-- --check` reports whether it has drifted from upstream).
244+
245+
### Accuracy caveats
246+
247+
- **Multi-turn runs are over-estimated.** Providers discount repeated context — and multi-turn attacks re-send the whole conversation each turn — but opfor prices every input token at the full rate. The more turns, the more conservative the figure.
248+
- **List prices only.** Negotiated rates, credits, and proxy markup are not reflected.
249+
- **Unknown models are never counted as free.** If a model isn't in the price table, the report says so and marks the total a lower bound (`` instead of ``, or `unpriced` when nothing could be priced). Treat `totalUsd` as a floor whenever `summary.cost.complete` is `false`.
233250

234251
---
235252

0 commit comments

Comments
 (0)