You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(pricing): document testing cost and scale display precision
Cost precision now scales in three bands rather than two. Between a cent
and a dime the third decimal is what distinguishes one evaluator from
another — $0.037 vs $0.049 is a third more expensive, but both round to
the same two-decimal figure — so that band keeps it. From a dime up the
precision is noise, so a run total reads as money ($0.18, not $0.177).
Sub-cent amounts keep two significant figures as before, since a fixed
decimal count renders them "$0.0000" and they read as free.
docs/cli.md said "No cost estimation is performed", which this feature
made false. That section is now "Token usage and testing cost" and
covers the per-model split, where prices come from, and the accuracy
caveats — chiefly that multi-turn runs read high, because providers
discount repeated context and opfor prices every input token at full
rate. Users budgeting off this number should know it is conservative.
Also documents that "testing cost" is opfor's own spend and excludes the
target's inference cost, and that an unpriced model is never counted as
free. Same summary added to the README and referenced from the browser
extension guide.
Note: the SDK's report type exposes neither tokenUsage nor cost. That
gap predates this work and is left alone here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: README.md
+20Lines changed: 20 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -111,6 +111,26 @@ When you run a scan, opfor:
111
111
112
112
Each run lands in its own subfolder under `.opfor/reports/run-report-<compactTs>-<slug>-<shortId>/` containing `<slug>-report.html` and `<slug>-report.json`. Autonomous `opfor hunt` runs use the same layout under `hunt-report-<compactTs>-<slug>-<shortId>/`.
113
113
114
+
### Testing cost
115
+
116
+
Every run reports what it cost to run, broken down by model:
This is **opfor's own spend** — the attacker and judge LLMs. It excludes your target's inference cost, which opfor cannot see from the outside. The per-model split is the useful part: the judge is often the bigger share, and pointing it at a cheaper model is usually the easiest saving.
126
+
127
+
Prices come from a snapshot of LiteLLM's public price map that ships with the package, so runs work offline and a report re-rendered later produces the same figure. Two caveats worth knowing:
128
+
129
+
-**Multi-turn runs read high.** Providers discount repeated context, and a multi-turn attack re-sends the conversation each turn — opfor prices every input token at full rate, so the real bill is usually lower.
130
+
-**Unknown models are never counted as free.** A model missing from the price table is reported as unpriced and the total is marked a lower bound, rather than silently reading as $0.
131
+
132
+
→ [Token usage and testing cost](docs/cli.md#token-usage-and-testing-cost)
133
+
114
134
## Evaluator coverage
115
135
116
136
Opfor ships with curated suites that map to industry standards. Pick a suite or run individual evaluators.
Copy file name to clipboardExpand all lines: docs/browser-extension.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -88,7 +88,7 @@ The extension uses a **single LLM configuration** for all operations (attack gen
88
88
89
89
The extension runs up to **20 turns per evaluator** (default 10). It stops a given evaluator early when the judge returns a definitive verdict.
90
90
91
-
**Token usage**is tracked per evaluator and shown on the Done screen and in the downloadable HTML report.
91
+
**Token usage and testing cost**are tracked per evaluator and shown on the Done screen and in the downloadable HTML report. Cost covers the attacker and judge LLMs you configured in Options — the chat UI you are testing is driven through the browser, so there is nothing to bill for the target itself. See [Token usage and testing cost](cli.md#token-usage-and-testing-cost) for how the figure is derived and its caveats.
Copy file name to clipboardExpand all lines: docs/cli.md
+22-5Lines changed: 22 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -215,21 +215,38 @@ Partial reports include all completed evaluator results and are marked with `sto
215
215
216
216
---
217
217
218
-
## Token usage tracking
218
+
## Token usage and testing cost
219
219
220
-
Every LLM call (attacker generation, adaptive follow-ups, judge) is metered. After the run completes, the CLI prints a summary line:
220
+
Every LLM call (attacker generation, adaptive follow-ups, judge) is metered, and each is attributed to the model that made it. After the run completes, the CLI prints:
Token usage is also included in the JSON report (`summary.tokenUsage` and per-evaluator `tokenUsage` fields) and in the HTML report's executive summary card. When using `--events`, the `run_finish` event includes token counts in its `summary` payload.
231
+
The per-model split is the actionable part — the judge is frequently the larger share, and switching it to a cheaper model is usually the easiest saving.
229
232
230
-
The browser extension shows a `Tokens` stat on its Done screen.
233
+
Both are included in the JSON report (`summary.tokenUsage`, `summary.tokenUsageByModel`, `summary.cost`, plus the same fields per evaluator) and in the HTML report's executive summary. When using `--events`, the `run_finish` event includes them in its `summary` payload.
231
234
232
-
> Token counts reflect raw model usage (input + output tokens). No cost estimation is performed — provider pricing varies and changes frequently.
235
+
The browser extension shows the same figures on its Done screen.
236
+
237
+
### What the cost figure covers
238
+
239
+
**"Testing cost" is opfor's own spend — the attacker and judge LLM calls.** It excludes your target's inference cost, which opfor cannot observe from the outside.
240
+
241
+
Prices come from a snapshot of LiteLLM's public price map, vendored into the package. Nothing is downloaded at runtime, so runs work offline and a report re-rendered months later produces the same figure. The snapshot version is recorded in the JSON as `summary.cost.priceTableVersion`.
242
+
243
+
Maintainers refresh it with `npm run build:pricing` (`-- --check` reports whether it has drifted from upstream).
244
+
245
+
### Accuracy caveats
246
+
247
+
-**Multi-turn runs are over-estimated.** Providers discount repeated context — and multi-turn attacks re-send the whole conversation each turn — but opfor prices every input token at the full rate. The more turns, the more conservative the figure.
248
+
-**List prices only.** Negotiated rates, credits, and proxy markup are not reflected.
249
+
-**Unknown models are never counted as free.** If a model isn't in the price table, the report says so and marks the total a lower bound (`≥` instead of `≈`, or `unpriced` when nothing could be priced). Treat `totalUsd` as a floor whenever `summary.cost.complete` is `false`.
0 commit comments