Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/current-functionality.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,14 @@ understudy workloads route <workload-id> --project-id <project-id> --model-id gl
understudy workloads route <workload-id> --project-id <project-id> --clear
understudy gateway health
understudy gateway probe --provider anthropic --project rehearsal --workload classify
understudy reporting summary --window 7d --group-by workload
understudy reporting usage --project rehearsal --window 7d --group-by workload,day
understudy reporting workload-status --project rehearsal --window 24h
understudy reporting cost <request-or-upstream-id>
understudy reporting cost-breakdown --project rehearsal --window 7d
understudy billing balance
understudy billing summary --from 2026-08-01T00:00:00Z --to 2026-09-01T00:00:00Z
understudy billing trend --from 2026-08-01T00:00:00Z --to 2026-09-01T00:00:00Z
understudy captures list --project rehearsal --workload classify
understudy captures get <request-id> --project rehearsal --workload classify
understudy captures export <request-id> --out .understudy/captures/<request-id>.json
Expand Down Expand Up @@ -97,6 +105,12 @@ traces, or secret values.
.understudy/capture-import/workload-card.json
```

The reporting and billing commands are authenticated, read-only hosted calls.
They expose customer-safe usage and customer pricing, plus the authoritative
billing ledger balance; they do not expose provider cost or margin. Call-cost
lookups preserve `unpriced` as `null`, and cost breakdowns surface the API's
coverage and known-gap fields instead of presenting incomplete data as final.

## What Is Skill-Led Now

The public workflow now lives in the skill tree. The authoritative,
Expand Down
41 changes: 20 additions & 21 deletions skills/check-routing-health/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,9 @@ does not write routes or change traffic.

These endpoints are read-only and carry no side effects. They do not change
routes, traffic percentages, or provider configuration. Do not print the full
`sk_*` key in output — mask it to the last 4 characters. Use `understudy run`
to inject credentials into child processes instead of pasting keys into shell
commands.
`sk_*` key in output — mask it to the last 4 characters. Prefer the first-class
`understudy reporting` commands, which resolve the signed-in organization and
project without exposing credentials to a child process.

## Vocabulary — use these words, exactly

Expand Down Expand Up @@ -77,32 +77,28 @@ node dist/bin.js status --json

2. **Ground in volume first — one org-wide call.** Before analyzing or
recommending anything, rank workloads by spend and request count across
the whole org. `understudy run` spawns the child **without a shell**, so
wrap the command in `sh -c '...'` — the injected `$UNDERSTUDY_API_KEY` /
`$UNDERSTUDY_ORG_ID` only expand inside that child shell (bare
`understudy run -- curl` would send the literal string
`$UNDERSTUDY_API_KEY` as the bearer):
the whole org:

```sh
understudy run -- sh -c 'curl -s \
-H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
"https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/reporting?window=7d&group_by=workload"'
understudy reporting summary --window 7d --group-by workload --json
```

Rank by `customer_cost_usd` and `requests` (`group_by` defaults to
`project` — use that first when the org has many projects, then
`workload`). For a tight incident window add `granularity=minute` (ranges
`workload`). For a tight incident window add `--granularity minute` (ranges
up to 24h); drill down with the `project_id` / `workload_id` filters.
Every statement you make must be grounded in that volume ranking — lead
with the workloads that carry the spend and traffic. Do not anchor on
low-leverage generic advice about workloads that barely run.

3. Pull the unified per-workload view (same `sh -c` wrapper):
Organization reporting has no pricing-coverage block. Use
`understudy reporting cost-breakdown` before treating the cost ranking as
complete enough for an exact billing claim.

3. Pull the unified per-workload view:

```sh
understudy run -- sh -c 'curl -s \
-H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
"https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/projects/<project-id>/workload-status?window=24h"'
understudy reporting workload-status --project <project> --window 24h --json
```

One row per workload: `status` (healthy | degraded | idle), the declared
Expand Down Expand Up @@ -143,9 +139,7 @@ node dist/bin.js status --json
([`reference.md`](reference.md) § Captures metadata list):

```sh
understudy run -- sh -c 'curl -s \
-H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
"https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/projects/<project-id>/captures?limit=25"'
understudy captures list --project <project> --limit 25 --json
```

6. If any call fails or a number looks wrong, capture the
Expand All @@ -160,8 +154,13 @@ node dist/bin.js status --json
within one project when the org-wide view isn't enough: it carries what
org reporting doesn't — the computed `cache_read_pct` share, a per-group
`error_rate`, and multi-dimension `group_by` (e.g. `workload,day`). Raw
cache token counts are in both. See [`reference.md`](reference.md) § Usage
summary. The
cache token counts are in both:

```sh
understudy reporting usage --project <project> --window 7d --group-by workload,day --json
```

See [`reference.md`](reference.md) § Usage summary. The
older `routing-status`, `provider-health`, and `status` endpoints still
exist but are deprecated — see the legacy section of
[`reference.md`](reference.md). Only fall back to them if `workload-status`
Expand Down
80 changes: 61 additions & 19 deletions skills/check-routing-health/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@ across all projects — the grounding call) followed by **workload-status**
hand the Understudy team) complete the surface. The three older endpoints
are deprecated — see [Legacy endpoints](#legacy-endpoints-deprecated).

For cost-specific investigations, use **call cost** for one request,
**cost breakdown** for a project's workload/category rollup, and the
read-only **billing** commands for the organization ledger position and
customer-cost history.

## Vocabulary

All responses use the canonical vocabulary — use the same words with the user:
Expand Down Expand Up @@ -53,9 +58,7 @@ endpoints).
| `project_id`, `workload_id` | — | Optional filters (max 255 chars). |

```sh
understudy run -- sh -c 'curl -s \
-H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
"https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/reporting?window=7d&group_by=workload"'
understudy reporting summary --window 7d --group-by workload --json
```

The response carries `totals` (`requests`, `input_tokens`,
Expand Down Expand Up @@ -83,14 +86,9 @@ metrics by hand.
| `window` | `24h` | Lookback window. Accepts `30m`, `1h`, `6h`, up to `24h`. |

```sh
understudy run -- sh -c 'curl -s \
-H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
"https://api.understudylabs.com/admin/v1/orgs/<org-id>/projects/<project-id>/workload-status?window=24h"'
understudy reporting workload-status --project <project> --window 24h --json
```

The `sh -c` wrapper is required: `understudy run` spawns the child without a
shell, so `$UNDERSTUDY_API_KEY` only expands inside the child shell.

```jsonc
{
"project_id": "proj_...",
Expand Down Expand Up @@ -164,9 +162,7 @@ within one project.
| `group_by` | — | Comma-separated subset of the server-side allowlist: `workload`, `model`, `day`. |

```sh
understudy run -- sh -c 'curl -s \
-H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
"https://api.understudylabs.com/admin/v1/orgs/<org-id>/projects/<project-id>/usage-summary?window=7d&group_by=workload,day"'
understudy reporting usage --project <project> --window 7d --group-by workload,day --json
```

```jsonc
Expand Down Expand Up @@ -203,6 +199,55 @@ understudy run -- sh -c 'curl -s \
| `customer_cost_usd` | What the customer is billed for the group over the window. Rank workloads by this + `requests` before making any recommendation. |
| `error_rate` | 5xx error rate within the group (0..1). |

## Cost reporting

Look up one call by its Understudy request id, upstream request id, or upstream
response/message id:

```sh
understudy reporting cost <request-or-upstream-id> --json
```

`customer_cost_usd` and `cost_categories` are customer pricing, not supplier
cost or margin. An event not yet covered by pricing returns
`pricing_status: "unpriced"` and `customer_cost_usd: null`; never translate
that into zero. The response's `coverage.known_gaps` explains incomplete
category or source data.

For a project-wide rollup by workload and token category:

```sh
understudy reporting cost-breakdown --project <project> --window 7d --json
```

The output separates `requests` from `priced_requests` and preserves
`coverage.data_completeness` plus `coverage.known_gaps`. A workload filter is
available as `--workload-id <id>`.

Organization reporting does not carry a coverage block. Treat its cost ranking
as directional until a project cost breakdown confirms the applicable pricing
coverage.

## Billing reads

These commands are read-only. Balance reads the authoritative billing ledger;
summary and trend read metered tokens plus derived customer pricing:

```sh
understudy billing balance --json
understudy billing summary --from <inclusive-timestamp-with-timezone> --to <exclusive-timestamp-with-timezone> --json
understudy billing trend --from <inclusive-timestamp-with-timezone> --to <exclusive-timestamp-with-timezone> --json
```

Keep the sources distinct: balance is the ledger position, while summary and
trend are usage-derived estimates. None is a provider-cost or margin report.
Metered request and priced-event counts are different units: complete zero-cost
traffic or pricing lag can make them differ, so their difference is not itself
an exact missing-price count.
The CLI intentionally does not expose the lower-level billing model-breakdown
endpoint because its raw provider/model fields do not use the customer-safe
scrubbing contract of the reporting surface.

## Captures metadata list

```
Expand All @@ -217,9 +262,7 @@ defaults to 25 (max 100); `truncated: true` comes with an opaque R2 `cursor`
to pass back verbatim for the next page.

```sh
understudy run -- sh -c 'curl -s \
-H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
"https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/projects/<project-id>/captures?limit=25"'
understudy captures list --project <project> --limit 25 --json
```

Capture **content** (request/response bodies) is not readable with an `sk_*`
Expand All @@ -244,10 +287,9 @@ Every error envelope carries `request_id`, and every response carries the
## Calling without the CLI

The endpoints accept a standard `Authorization: Bearer sk_*` header. Prefer
`understudy run -- sh -c '...'` so credentials come from the CLI's credential
store (see the examples above). To call curl directly instead, first export
the key from `~/.understudy/credentials.json` in your own shell — the variable
below expands in *your* shell, not via `understudy run`:
the first-class `understudy reporting` commands so credentials come from the
CLI's credential store. To call curl directly instead, first export the key in
your own shell:

```sh
curl -s -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
Expand Down
4 changes: 4 additions & 0 deletions skills/understudy/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,10 @@ Identify the developer's current stage and load exactly one:
provider health", "are there 500s", "what's our error rate", or wants
self-service diagnostics without asking the team →
[`../check-routing-health/SKILL.md`](../check-routing-health/SKILL.md).
- **Understudy usage, customer cost, or billing position** — the developer asks
for organization/project usage, the customer-priced cost of a call or
workload, pricing coverage, billing balance, or cost history →
[`../check-routing-health/SKILL.md`](../check-routing-health/SKILL.md).
- **Share savings / leaderboard receipt** — the developer wants to share how
much money they saved, submit an anonymous lower-Anthropic-bill result, or
send metrics back for the coming leaderboard →
Expand Down
140 changes: 140 additions & 0 deletions src/commands/billing.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
import { Command } from "commander";
import kleur from "kleur";

import { request } from "../internal/http.js";
import { isJsonMode, runAction } from "../internal/output.js";
import { resolveOrganizationAuth } from "../internal/projects.js";
import {
BillingBalanceResponseSchema,
BillingSummaryResponseSchema,
BillingTrendResponseSchema,
formatCount,
formatUsd,
parseBillingWindow,
sanitizeForTerminal,
} from "../internal/reporting-contracts.js";

interface OrgOptions {
org?: string;
}

interface WindowOptions extends OrgOptions {
from: string;
to: string;
}

export function registerBillingCommand(program: Command): void {
const billing = program
.command("billing")
.description("Read the hosted billing position and customer-cost history.");

billing
.command("balance")
.description("Show the authoritative organization balance and billing status.")
.option("--org <id>", "Org id to use (default: active or only signed-in org).")
.action(async function (this: Command, opts: OrgOptions) {
await runAction(this, () => runBalance(this, opts));
});

billing
.command("summary")
.description("Show metered tokens, priced events, and estimated customer cost.")
.requiredOption("--from <timestamp>", "Inclusive ISO UTC start timestamp.")
.requiredOption("--to <timestamp>", "Exclusive ISO UTC end timestamp.")
.option("--org <id>", "Org id to use.")
.action(async function (this: Command, opts: WindowOptions) {
await runAction(this, () => runSummary(this, opts));
});

billing
.command("trend")
.description("Show daily token and estimated customer-cost points.")
.requiredOption("--from <timestamp>", "Inclusive ISO UTC start timestamp.")
.requiredOption("--to <timestamp>", "Exclusive ISO UTC end timestamp.")
.option("--org <id>", "Org id to use.")
.action(async function (this: Command, opts: WindowOptions) {
await runAction(this, () => runTrend(this, opts));
});
}

async function runBalance(cmd: Command, opts: OrgOptions): Promise<void> {
const auth = resolveOrganizationAuth(opts.org);
const res = await request(
{
url: `/admin/v1/orgs/${auth.orgId}/billing/balance`,
orgId: auth.orgId,
},
BillingBalanceResponseSchema,
);
if (isJsonMode(cmd)) {
process.stdout.write(`${JSON.stringify(res.data)}\n`);
return;
}
const { balance } = res.data;
process.stdout.write(`${kleur.bold("billing balance")} · ${sanitizeForTerminal(balance.billing_mode)}\n`);
process.stdout.write(`status ${sanitizeForTerminal(balance.status)}\n`);
process.stdout.write(`balance ${formatUsd(balance.balance_usd)} ${sanitizeForTerminal(balance.currency)}\n`);
process.stdout.write(`low threshold ${formatUsd(balance.low_balance_threshold_usd)}\n`);
process.stdout.write(`grant remaining ${formatUsd(balance.grants.total_remaining_usd)}\n`);
if (balance.grants.soonest_expiry) {
process.stdout.write(`next expiry ${sanitizeForTerminal(balance.grants.soonest_expiry)}\n`);
}
process.stdout.write(`${kleur.gray("authoritative ledger position; not a provider-cost report")}\n`);
}

async function runSummary(cmd: Command, opts: WindowOptions): Promise<void> {
const window = parseBillingWindow(opts.from, opts.to);
const auth = resolveOrganizationAuth(opts.org);
const search = new URLSearchParams(window);
const res = await request(
{
url: `/admin/v1/orgs/${auth.orgId}/billing/summary?${search.toString()}`,
orgId: auth.orgId,
},
BillingSummaryResponseSchema,
);
if (isJsonMode(cmd)) {
process.stdout.write(`${JSON.stringify(res.data)}\n`);
return;
}
const { summary } = res.data;
process.stdout.write(`${kleur.bold("billing summary")} · ${sanitizeForTerminal(summary.from)} to ${sanitizeForTerminal(summary.to)}\n`);
process.stdout.write(`estimated cost ${formatUsd(summary.estimated_cost_usd)}\n`);
process.stdout.write(`metered ${formatCount(summary.metered_requests)} requests\n`);
process.stdout.write(`priced ${formatCount(summary.priced_events)} events\n`);
process.stdout.write(`total tokens ${formatCount(summary.tokens.total_tokens)}\n`);
process.stdout.write(`blended price ${formatUsd(summary.blended_price_per_mtok)}/MTok\n`);
if (summary.priced_events !== summary.metered_requests) {
process.stdout.write(
`${kleur.yellow("coverage note")} request and event counts are different units; complete zero-cost traffic or pricing lag can make them differ\n`,
);
}
process.stdout.write(`${kleur.gray("derived customer pricing; use billing balance for the ledger position")}\n`);
}

async function runTrend(cmd: Command, opts: WindowOptions): Promise<void> {
const window = parseBillingWindow(opts.from, opts.to);
const auth = resolveOrganizationAuth(opts.org);
const search = new URLSearchParams(window);
const res = await request(
{
url: `/admin/v1/orgs/${auth.orgId}/billing/trend?${search.toString()}`,
orgId: auth.orgId,
},
BillingTrendResponseSchema,
);
if (isJsonMode(cmd)) {
process.stdout.write(`${JSON.stringify(res.data)}\n`);
return;
}
process.stdout.write(`${kleur.bold("billing trend")} · ${sanitizeForTerminal(window.from)} to ${sanitizeForTerminal(window.to)}\n`);
for (const point of res.data.points) {
process.stdout.write(
`${sanitizeForTerminal(point.day)}: ${formatUsd(point.cost_usd)} · ${formatCount(point.tokens.total_tokens)} tokens\n`,
);
}
if (res.data.points.length === 0) {
process.stdout.write(`${kleur.gray("No metered traffic in this window.")}\n`);
}
process.stdout.write(`${kleur.gray("daily customer pricing; not provider cost")}\n`);
}
Loading