diff --git a/docs/current-functionality.md b/docs/current-functionality.md index d70e5505..eb6d7fb2 100644 --- a/docs/current-functionality.md +++ b/docs/current-functionality.md @@ -38,6 +38,14 @@ understudy workloads route --project-id --model-id gl understudy workloads route --project-id --clear understudy gateway health understudy gateway probe --provider anthropic --project rehearsal --workload classify +understudy reporting summary --window 7d --group-by workload +understudy reporting usage --project rehearsal --window 7d --group-by workload,day +understudy reporting workload-status --project rehearsal --window 24h +understudy reporting cost +understudy reporting cost-breakdown --project rehearsal --window 7d +understudy billing balance +understudy billing summary --from 2026-08-01T00:00:00Z --to 2026-09-01T00:00:00Z +understudy billing trend --from 2026-08-01T00:00:00Z --to 2026-09-01T00:00:00Z understudy captures list --project rehearsal --workload classify understudy captures get --project rehearsal --workload classify understudy captures export --out .understudy/captures/.json @@ -97,6 +105,12 @@ traces, or secret values. .understudy/capture-import/workload-card.json ``` +The reporting and billing commands are authenticated, read-only hosted calls. +They expose customer-safe usage and customer pricing, plus the authoritative +billing ledger balance; they do not expose provider cost or margin. Call-cost +lookups preserve `unpriced` as `null`, and cost breakdowns surface the API's +coverage and known-gap fields instead of presenting incomplete data as final. + ## What Is Skill-Led Now The public workflow now lives in the skill tree. The authoritative, diff --git a/skills/check-routing-health/SKILL.md b/skills/check-routing-health/SKILL.md index ca1551e8..8b49e085 100644 --- a/skills/check-routing-health/SKILL.md +++ b/skills/check-routing-health/SKILL.md @@ -24,9 +24,9 @@ does not write routes or change traffic. These endpoints are read-only and carry no side effects. They do not change routes, traffic percentages, or provider configuration. Do not print the full -`sk_*` key in output — mask it to the last 4 characters. Use `understudy run` -to inject credentials into child processes instead of pasting keys into shell -commands. +`sk_*` key in output — mask it to the last 4 characters. Prefer the first-class +`understudy reporting` commands, which resolve the signed-in organization and +project without exposing credentials to a child process. ## Vocabulary — use these words, exactly @@ -77,32 +77,28 @@ node dist/bin.js status --json 2. **Ground in volume first — one org-wide call.** Before analyzing or recommending anything, rank workloads by spend and request count across - the whole org. `understudy run` spawns the child **without a shell**, so - wrap the command in `sh -c '...'` — the injected `$UNDERSTUDY_API_KEY` / - `$UNDERSTUDY_ORG_ID` only expand inside that child shell (bare - `understudy run -- curl` would send the literal string - `$UNDERSTUDY_API_KEY` as the bearer): + the whole org: ```sh - understudy run -- sh -c 'curl -s \ - -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \ - "https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/reporting?window=7d&group_by=workload"' + understudy reporting summary --window 7d --group-by workload --json ``` Rank by `customer_cost_usd` and `requests` (`group_by` defaults to `project` — use that first when the org has many projects, then - `workload`). For a tight incident window add `granularity=minute` (ranges + `workload`). For a tight incident window add `--granularity minute` (ranges up to 24h); drill down with the `project_id` / `workload_id` filters. Every statement you make must be grounded in that volume ranking — lead with the workloads that carry the spend and traffic. Do not anchor on low-leverage generic advice about workloads that barely run. -3. Pull the unified per-workload view (same `sh -c` wrapper): + Organization reporting has no pricing-coverage block. Use + `understudy reporting cost-breakdown` before treating the cost ranking as + complete enough for an exact billing claim. + +3. Pull the unified per-workload view: ```sh - understudy run -- sh -c 'curl -s \ - -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \ - "https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/projects//workload-status?window=24h"' + understudy reporting workload-status --project --window 24h --json ``` One row per workload: `status` (healthy | degraded | idle), the declared @@ -143,9 +139,7 @@ node dist/bin.js status --json ([`reference.md`](reference.md) § Captures metadata list): ```sh - understudy run -- sh -c 'curl -s \ - -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \ - "https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/projects//captures?limit=25"' + understudy captures list --project --limit 25 --json ``` 6. If any call fails or a number looks wrong, capture the @@ -160,8 +154,13 @@ node dist/bin.js status --json within one project when the org-wide view isn't enough: it carries what org reporting doesn't — the computed `cache_read_pct` share, a per-group `error_rate`, and multi-dimension `group_by` (e.g. `workload,day`). Raw -cache token counts are in both. See [`reference.md`](reference.md) § Usage -summary. The +cache token counts are in both: + +```sh +understudy reporting usage --project --window 7d --group-by workload,day --json +``` + +See [`reference.md`](reference.md) § Usage summary. The older `routing-status`, `provider-health`, and `status` endpoints still exist but are deprecated — see the legacy section of [`reference.md`](reference.md). Only fall back to them if `workload-status` diff --git a/skills/check-routing-health/reference.md b/skills/check-routing-health/reference.md index c5b99caa..208b6888 100644 --- a/skills/check-routing-health/reference.md +++ b/skills/check-routing-health/reference.md @@ -17,6 +17,11 @@ across all projects — the grounding call) followed by **workload-status** hand the Understudy team) complete the surface. The three older endpoints are deprecated — see [Legacy endpoints](#legacy-endpoints-deprecated). +For cost-specific investigations, use **call cost** for one request, +**cost breakdown** for a project's workload/category rollup, and the +read-only **billing** commands for the organization ledger position and +customer-cost history. + ## Vocabulary All responses use the canonical vocabulary — use the same words with the user: @@ -53,9 +58,7 @@ endpoints). | `project_id`, `workload_id` | — | Optional filters (max 255 chars). | ```sh -understudy run -- sh -c 'curl -s \ - -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \ - "https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/reporting?window=7d&group_by=workload"' +understudy reporting summary --window 7d --group-by workload --json ``` The response carries `totals` (`requests`, `input_tokens`, @@ -83,14 +86,9 @@ metrics by hand. | `window` | `24h` | Lookback window. Accepts `30m`, `1h`, `6h`, up to `24h`. | ```sh -understudy run -- sh -c 'curl -s \ - -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \ - "https://api.understudylabs.com/admin/v1/orgs//projects//workload-status?window=24h"' +understudy reporting workload-status --project --window 24h --json ``` -The `sh -c` wrapper is required: `understudy run` spawns the child without a -shell, so `$UNDERSTUDY_API_KEY` only expands inside the child shell. - ```jsonc { "project_id": "proj_...", @@ -164,9 +162,7 @@ within one project. | `group_by` | — | Comma-separated subset of the server-side allowlist: `workload`, `model`, `day`. | ```sh -understudy run -- sh -c 'curl -s \ - -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \ - "https://api.understudylabs.com/admin/v1/orgs//projects//usage-summary?window=7d&group_by=workload,day"' +understudy reporting usage --project --window 7d --group-by workload,day --json ``` ```jsonc @@ -203,6 +199,55 @@ understudy run -- sh -c 'curl -s \ | `customer_cost_usd` | What the customer is billed for the group over the window. Rank workloads by this + `requests` before making any recommendation. | | `error_rate` | 5xx error rate within the group (0..1). | +## Cost reporting + +Look up one call by its Understudy request id, upstream request id, or upstream +response/message id: + +```sh +understudy reporting cost --json +``` + +`customer_cost_usd` and `cost_categories` are customer pricing, not supplier +cost or margin. An event not yet covered by pricing returns +`pricing_status: "unpriced"` and `customer_cost_usd: null`; never translate +that into zero. The response's `coverage.known_gaps` explains incomplete +category or source data. + +For a project-wide rollup by workload and token category: + +```sh +understudy reporting cost-breakdown --project --window 7d --json +``` + +The output separates `requests` from `priced_requests` and preserves +`coverage.data_completeness` plus `coverage.known_gaps`. A workload filter is +available as `--workload-id `. + +Organization reporting does not carry a coverage block. Treat its cost ranking +as directional until a project cost breakdown confirms the applicable pricing +coverage. + +## Billing reads + +These commands are read-only. Balance reads the authoritative billing ledger; +summary and trend read metered tokens plus derived customer pricing: + +```sh +understudy billing balance --json +understudy billing summary --from --to --json +understudy billing trend --from --to --json +``` + +Keep the sources distinct: balance is the ledger position, while summary and +trend are usage-derived estimates. None is a provider-cost or margin report. +Metered request and priced-event counts are different units: complete zero-cost +traffic or pricing lag can make them differ, so their difference is not itself +an exact missing-price count. +The CLI intentionally does not expose the lower-level billing model-breakdown +endpoint because its raw provider/model fields do not use the customer-safe +scrubbing contract of the reporting surface. + ## Captures metadata list ``` @@ -217,9 +262,7 @@ defaults to 25 (max 100); `truncated: true` comes with an opaque R2 `cursor` to pass back verbatim for the next page. ```sh -understudy run -- sh -c 'curl -s \ - -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \ - "https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/projects//captures?limit=25"' +understudy captures list --project --limit 25 --json ``` Capture **content** (request/response bodies) is not readable with an `sk_*` @@ -244,10 +287,9 @@ Every error envelope carries `request_id`, and every response carries the ## Calling without the CLI The endpoints accept a standard `Authorization: Bearer sk_*` header. Prefer -`understudy run -- sh -c '...'` so credentials come from the CLI's credential -store (see the examples above). To call curl directly instead, first export -the key from `~/.understudy/credentials.json` in your own shell — the variable -below expands in *your* shell, not via `understudy run`: +the first-class `understudy reporting` commands so credentials come from the +CLI's credential store. To call curl directly instead, first export the key in +your own shell: ```sh curl -s -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \ diff --git a/skills/understudy/SKILL.md b/skills/understudy/SKILL.md index d36c64bb..1d44c19d 100644 --- a/skills/understudy/SKILL.md +++ b/skills/understudy/SKILL.md @@ -197,6 +197,10 @@ Identify the developer's current stage and load exactly one: provider health", "are there 500s", "what's our error rate", or wants self-service diagnostics without asking the team → [`../check-routing-health/SKILL.md`](../check-routing-health/SKILL.md). +- **Understudy usage, customer cost, or billing position** — the developer asks + for organization/project usage, the customer-priced cost of a call or + workload, pricing coverage, billing balance, or cost history → + [`../check-routing-health/SKILL.md`](../check-routing-health/SKILL.md). - **Share savings / leaderboard receipt** — the developer wants to share how much money they saved, submit an anonymous lower-Anthropic-bill result, or send metrics back for the coming leaderboard → diff --git a/src/commands/billing.ts b/src/commands/billing.ts new file mode 100644 index 00000000..bfb9840a --- /dev/null +++ b/src/commands/billing.ts @@ -0,0 +1,140 @@ +import { Command } from "commander"; +import kleur from "kleur"; + +import { request } from "../internal/http.js"; +import { isJsonMode, runAction } from "../internal/output.js"; +import { resolveOrganizationAuth } from "../internal/projects.js"; +import { + BillingBalanceResponseSchema, + BillingSummaryResponseSchema, + BillingTrendResponseSchema, + formatCount, + formatUsd, + parseBillingWindow, + sanitizeForTerminal, +} from "../internal/reporting-contracts.js"; + +interface OrgOptions { + org?: string; +} + +interface WindowOptions extends OrgOptions { + from: string; + to: string; +} + +export function registerBillingCommand(program: Command): void { + const billing = program + .command("billing") + .description("Read the hosted billing position and customer-cost history."); + + billing + .command("balance") + .description("Show the authoritative organization balance and billing status.") + .option("--org ", "Org id to use (default: active or only signed-in org).") + .action(async function (this: Command, opts: OrgOptions) { + await runAction(this, () => runBalance(this, opts)); + }); + + billing + .command("summary") + .description("Show metered tokens, priced events, and estimated customer cost.") + .requiredOption("--from ", "Inclusive ISO UTC start timestamp.") + .requiredOption("--to ", "Exclusive ISO UTC end timestamp.") + .option("--org ", "Org id to use.") + .action(async function (this: Command, opts: WindowOptions) { + await runAction(this, () => runSummary(this, opts)); + }); + + billing + .command("trend") + .description("Show daily token and estimated customer-cost points.") + .requiredOption("--from ", "Inclusive ISO UTC start timestamp.") + .requiredOption("--to ", "Exclusive ISO UTC end timestamp.") + .option("--org ", "Org id to use.") + .action(async function (this: Command, opts: WindowOptions) { + await runAction(this, () => runTrend(this, opts)); + }); +} + +async function runBalance(cmd: Command, opts: OrgOptions): Promise { + const auth = resolveOrganizationAuth(opts.org); + const res = await request( + { + url: `/admin/v1/orgs/${auth.orgId}/billing/balance`, + orgId: auth.orgId, + }, + BillingBalanceResponseSchema, + ); + if (isJsonMode(cmd)) { + process.stdout.write(`${JSON.stringify(res.data)}\n`); + return; + } + const { balance } = res.data; + process.stdout.write(`${kleur.bold("billing balance")} · ${sanitizeForTerminal(balance.billing_mode)}\n`); + process.stdout.write(`status ${sanitizeForTerminal(balance.status)}\n`); + process.stdout.write(`balance ${formatUsd(balance.balance_usd)} ${sanitizeForTerminal(balance.currency)}\n`); + process.stdout.write(`low threshold ${formatUsd(balance.low_balance_threshold_usd)}\n`); + process.stdout.write(`grant remaining ${formatUsd(balance.grants.total_remaining_usd)}\n`); + if (balance.grants.soonest_expiry) { + process.stdout.write(`next expiry ${sanitizeForTerminal(balance.grants.soonest_expiry)}\n`); + } + process.stdout.write(`${kleur.gray("authoritative ledger position; not a provider-cost report")}\n`); +} + +async function runSummary(cmd: Command, opts: WindowOptions): Promise { + const window = parseBillingWindow(opts.from, opts.to); + const auth = resolveOrganizationAuth(opts.org); + const search = new URLSearchParams(window); + const res = await request( + { + url: `/admin/v1/orgs/${auth.orgId}/billing/summary?${search.toString()}`, + orgId: auth.orgId, + }, + BillingSummaryResponseSchema, + ); + if (isJsonMode(cmd)) { + process.stdout.write(`${JSON.stringify(res.data)}\n`); + return; + } + const { summary } = res.data; + process.stdout.write(`${kleur.bold("billing summary")} · ${sanitizeForTerminal(summary.from)} to ${sanitizeForTerminal(summary.to)}\n`); + process.stdout.write(`estimated cost ${formatUsd(summary.estimated_cost_usd)}\n`); + process.stdout.write(`metered ${formatCount(summary.metered_requests)} requests\n`); + process.stdout.write(`priced ${formatCount(summary.priced_events)} events\n`); + process.stdout.write(`total tokens ${formatCount(summary.tokens.total_tokens)}\n`); + process.stdout.write(`blended price ${formatUsd(summary.blended_price_per_mtok)}/MTok\n`); + if (summary.priced_events !== summary.metered_requests) { + process.stdout.write( + `${kleur.yellow("coverage note")} request and event counts are different units; complete zero-cost traffic or pricing lag can make them differ\n`, + ); + } + process.stdout.write(`${kleur.gray("derived customer pricing; use billing balance for the ledger position")}\n`); +} + +async function runTrend(cmd: Command, opts: WindowOptions): Promise { + const window = parseBillingWindow(opts.from, opts.to); + const auth = resolveOrganizationAuth(opts.org); + const search = new URLSearchParams(window); + const res = await request( + { + url: `/admin/v1/orgs/${auth.orgId}/billing/trend?${search.toString()}`, + orgId: auth.orgId, + }, + BillingTrendResponseSchema, + ); + if (isJsonMode(cmd)) { + process.stdout.write(`${JSON.stringify(res.data)}\n`); + return; + } + process.stdout.write(`${kleur.bold("billing trend")} · ${sanitizeForTerminal(window.from)} to ${sanitizeForTerminal(window.to)}\n`); + for (const point of res.data.points) { + process.stdout.write( + `${sanitizeForTerminal(point.day)}: ${formatUsd(point.cost_usd)} · ${formatCount(point.tokens.total_tokens)} tokens\n`, + ); + } + if (res.data.points.length === 0) { + process.stdout.write(`${kleur.gray("No metered traffic in this window.")}\n`); + } + process.stdout.write(`${kleur.gray("daily customer pricing; not provider cost")}\n`); +} diff --git a/src/commands/reporting.ts b/src/commands/reporting.ts new file mode 100644 index 00000000..816e7ca9 --- /dev/null +++ b/src/commands/reporting.ts @@ -0,0 +1,470 @@ +import { Command } from "commander"; +import kleur from "kleur"; + +import { request } from "../internal/http.js"; +import { isJsonMode, runAction } from "../internal/output.js"; +import { resolveOrganizationAuth, resolveProject } from "../internal/projects.js"; +import { + CallCostResponseSchema, + CostBreakdownResponseSchema, + OrganizationReportingResponseSchema, + ReportingGranularitySchema, + ReportingGroupBySchema, + UsageGroupBySchema, + UsageSummaryResponseSchema, + WorkloadStatusResponseSchema, + formatCount, + formatPercent, + formatUsd, + parseCustomReportingRange, + parseDuration, + sanitizeForTerminal, + type CallCostResponse, + type CostBreakdownResponse, + type Coverage, + type OrganizationReportingResponse, + type UsageSummaryResponse, + type WorkloadStatusResponse, +} from "../internal/reporting-contracts.js"; + +interface OrgOptions { + org?: string; +} + +interface ProjectOptions extends OrgOptions { + project?: string; + projectId?: string; +} + +interface SummaryOptions extends OrgOptions { + window: string; + from?: string; + to?: string; + granularity?: string; + groupBy: string; + projectId?: string; + workloadId?: string; + excludeProjectId: string[]; +} + +interface UsageOptions extends ProjectOptions { + window: string; + groupBy: string; +} + +interface CostBreakdownOptions extends ProjectOptions { + window: string; + workloadId?: string; +} + +interface WorkloadStatusOptions extends ProjectOptions { + window: string; +} + +function collect(value: string, previous: string[]): string[] { + return [...previous, value]; +} + +export function registerReportingCommand(program: Command): void { + const reporting = program + .command("reporting") + .description("Read customer-safe hosted usage and cost reporting."); + + reporting + .command("summary") + .description("Show organization-wide requests, tokens, and estimated customer cost.") + .option("--window ", "Preset reporting window: 24h, 7d, or 30d.", "7d") + .option("--from ", "Inclusive UTC start date (YYYY-MM-DD); requires --to.") + .option("--to ", "Inclusive UTC end date (YYYY-MM-DD); requires --from.") + .option("--granularity ", "Bucket size: minute, hour, or day.") + .option("--group-by ", "Group by project, workload, or model.", "project") + .option("--project-id ", "Filter to one project id.") + .option("--workload-id ", "Filter to one workload id.") + .option( + "--exclude-project-id ", + "Exclude a project id; may be repeated.", + collect, + [], + ) + .option("--org ", "Org id to use (default: active or only signed-in org).") + .action(async function (this: Command, opts: SummaryOptions) { + await runAction(this, () => runSummary(this, opts)); + }); + + reporting + .command("usage") + .description("Show project usage, cache, error, and estimated cost groups.") + .option("--project ", "Project slug (default: active project).") + .option("--project-id ", "Project id.") + .option("--window ", "Usage window up to 30d.", "7d") + .option( + "--group-by ", + "Comma-separated subset of workload, model, and day.", + "workload", + ) + .option("--org ", "Org id to use.") + .action(async function (this: Command, opts: UsageOptions) { + await runAction(this, () => runUsage(this, opts)); + }); + + reporting + .command("workload-status") + .description("Show declared routing and observed health for each project workload.") + .option("--project ", "Project slug (default: active project).") + .option("--project-id ", "Project id.") + .option("--window ", "Health window up to 24h.", "24h") + .option("--org ", "Org id to use.") + .action(async function (this: Command, opts: WorkloadStatusOptions) { + await runAction(this, () => runWorkloadStatus(this, opts)); + }); + + reporting + .command("cost ") + .description("Show one call's priced customer cost by request or upstream id.") + .option("--org ", "Org id to use.") + .action(async function (this: Command, correlationId: string, opts: OrgOptions) { + await runAction(this, () => runCallCost(this, correlationId, opts)); + }); + + reporting + .command("cost-breakdown") + .description("Show project customer cost by workload and token category.") + .option("--project ", "Project slug (default: active project).") + .option("--project-id ", "Project id.") + .option("--window ", "Cost window up to 30d.", "7d") + .option("--workload-id ", "Filter to one workload id.") + .option("--org ", "Org id to use.") + .action(async function (this: Command, opts: CostBreakdownOptions) { + await runAction(this, () => runCostBreakdown(this, opts)); + }); +} + +async function runSummary(cmd: Command, opts: SummaryOptions): Promise { + const groupBy = ReportingGroupBySchema.safeParse(opts.groupBy); + if (!groupBy.success) { + throw new Error("--group-by must be one of: project, workload, model."); + } + const granularity = opts.granularity === undefined + ? null + : ReportingGranularitySchema.safeParse(opts.granularity); + if (granularity && !granularity.success) { + throw new Error("--granularity must be one of: minute, hour, day."); + } + const projectId = normalizeFilter(opts.projectId, "--project-id"); + const workloadId = normalizeFilter(opts.workloadId, "--workload-id"); + const excludedProjectIds = [ + ...new Set(opts.excludeProjectId.map((value) => normalizeFilter(value, "--exclude-project-id"))), + ]; + if (excludedProjectIds.length > 20) { + throw new Error("--exclude-project-id may be repeated at most 20 times."); + } + + const custom = parseCustomReportingRange(opts.from, opts.to); + const search = new URLSearchParams(); + if (custom) { + search.set("from", custom.from); + search.set("to", custom.to); + } else { + const window = opts.window.trim(); + if (window !== "24h" && window !== "7d" && window !== "30d") { + throw new Error("--window must be one of: 24h, 7d, 30d."); + } + search.set("window", window); + } + if (granularity?.success) search.set("granularity", granularity.data); + search.set("group_by", groupBy.data); + if (projectId) search.set("project_id", projectId); + if (workloadId) search.set("workload_id", workloadId); + for (const excludedProjectId of excludedProjectIds) { + search.append("exclude_project_id", excludedProjectId); + } + + const auth = resolveOrganizationAuth(opts.org); + const res = await request( + { + url: `/admin/v1/orgs/${auth.orgId}/reporting?${search.toString()}`, + orgId: auth.orgId, + }, + OrganizationReportingResponseSchema, + ); + if (isJsonMode(cmd)) { + process.stdout.write(`${JSON.stringify(res.data)}\n`); + return; + } + renderOrganizationSummary(res.data); +} + +async function runUsage(cmd: Command, opts: UsageOptions): Promise { + const window = parseDuration(opts.window, "--window", 30 * 1_440); + const groupBy = parseUsageGroupBy(opts.groupBy); + const project = await resolveProject(opts); + const search = new URLSearchParams({ window, group_by: groupBy.join(",") }); + const res = await request( + { + url: + `/admin/v1/orgs/${project.auth.orgId}/projects/${encodeURIComponent(project.projectId)}` + + `/usage-summary?${search.toString()}`, + orgId: project.auth.orgId, + }, + UsageSummaryResponseSchema, + ); + if (res.data.groups.length === 5_000) { + throw new Error( + "The usage response reached the 5,000-group server limit and may be incomplete. Narrow --window or query fewer --group-by dimensions.", + ); + } + if (isJsonMode(cmd)) { + process.stdout.write(`${JSON.stringify(res.data)}\n`); + return; + } + renderUsage(res.data); +} + +async function runWorkloadStatus( + cmd: Command, + opts: WorkloadStatusOptions, +): Promise { + const window = parseDuration(opts.window, "--window", 24 * 60); + if (window.endsWith("d")) { + throw new Error("--window for workload status must use minutes or hours, up to 24h."); + } + const project = await resolveProject(opts); + const search = new URLSearchParams({ window }); + const res = await request( + { + url: + `/admin/v1/orgs/${project.auth.orgId}/projects/${encodeURIComponent(project.projectId)}` + + `/workload-status?${search.toString()}`, + orgId: project.auth.orgId, + }, + WorkloadStatusResponseSchema, + ); + if (isJsonMode(cmd)) { + process.stdout.write(`${JSON.stringify(res.data)}\n`); + return; + } + renderWorkloadStatus(res.data); +} + +async function runCallCost( + cmd: Command, + correlationId: string, + opts: OrgOptions, +): Promise { + const trimmed = correlationId.trim(); + if (!trimmed || trimmed.length > 256) { + throw new Error("correlation-id must be between 1 and 256 characters."); + } + const auth = resolveOrganizationAuth(opts.org); + const res = await request( + { + url: + `/admin/v1/orgs/${auth.orgId}/calls/${encodeURIComponent(trimmed)}/cost`, + orgId: auth.orgId, + }, + CallCostResponseSchema, + ); + if (isJsonMode(cmd)) { + process.stdout.write(`${JSON.stringify(res.data)}\n`); + return; + } + renderCallCost(res.data); +} + +async function runCostBreakdown( + cmd: Command, + opts: CostBreakdownOptions, +): Promise { + const window = parseDuration(opts.window, "--window", 30 * 1_440); + const workloadId = normalizeFilter(opts.workloadId, "--workload-id"); + const project = await resolveProject(opts); + const search = new URLSearchParams({ window }); + if (workloadId) search.set("workload_id", workloadId); + const res = await request( + { + url: + `/admin/v1/orgs/${project.auth.orgId}/projects/${encodeURIComponent(project.projectId)}` + + `/cost-breakdown?${search.toString()}`, + orgId: project.auth.orgId, + }, + CostBreakdownResponseSchema, + ); + if (isJsonMode(cmd)) { + process.stdout.write(`${JSON.stringify(res.data)}\n`); + return; + } + renderCostBreakdown(res.data); +} + +function parseUsageGroupBy(value: string): Array<"workload" | "model" | "day"> { + const values = value.split(",").map((entry) => entry.trim()).filter(Boolean); + if (values.length === 0 || new Set(values).size !== values.length) { + throw new Error("--group-by must contain unique values from: workload, model, day."); + } + const parsed = UsageGroupBySchema.array().safeParse(values); + if (!parsed.success) { + throw new Error("--group-by must contain only: workload, model, day."); + } + return parsed.data; +} + +function normalizeFilter(value: string, label: string): string; +function normalizeFilter(value: string | undefined, label: string): string | undefined; +function normalizeFilter(value: string | undefined, label: string): string | undefined { + if (value === undefined) return undefined; + const normalized = value.trim(); + if (normalized === "" || normalized.length > 255) { + throw new Error(`${label} must be between 1 and 255 characters.`); + } + return normalized; +} + +function renderOrganizationSummary( + data: OrganizationReportingResponse, +): void { + process.stdout.write(`${kleur.bold("organization reporting")} · ${data.window} · by ${data.group_by}\n`); + process.stdout.write(`requests ${formatCount(data.totals.requests)}\n`); + process.stdout.write(`total tokens ${formatCount(data.totals.total_tokens)}\n`); + process.stdout.write(`estimated cost ${formatUsd(data.totals.customer_cost_usd)}\n`); + + const groups = new Map(); + for (const point of data.series) { + const identity = data.group_by === "project" + ? point.project_id ?? point.project ?? "unknown" + : data.group_by === "workload" + ? point.workload_id ?? point.workload ?? "unknown" + : point.model ?? "unknown"; + const label = data.group_by === "project" + ? point.project ?? point.project_id ?? "unknown" + : data.group_by === "workload" + ? point.workload ?? point.workload_id ?? "unknown" + : point.model ?? "unknown"; + const current = groups.get(identity) ?? { label, requests: 0, tokens: 0, cost: 0 }; + current.requests += point.requests; + current.tokens += point.total_tokens; + current.cost += point.customer_cost_usd; + groups.set(identity, current); + } + const rows = [...groups.values()].sort((left, right) => right.cost - left.cost).slice(0, 20); + if (rows.length > 0) { + process.stdout.write("\n"); + for (const row of rows) { + process.stdout.write( + `${sanitizeForTerminal(row.label)}: ${formatUsd(row.cost)} · ${formatCount(row.requests)} requests · ${formatCount(row.tokens)} tokens\n`, + ); + } + } + if (groups.size > rows.length) { + process.stdout.write(`${kleur.gray(`showing ${rows.length} of ${groups.size} groups; use --json for all rows`)}\n`); + } + process.stdout.write(`${kleur.gray(`generated ${sanitizeForTerminal(data.generated_at)}; customer pricing, not provider cost`)}\n`); + process.stdout.write(`${kleur.gray("organization reporting does not include pricing coverage; use cost-breakdown for qualified coverage")}\n`); +} + +function renderUsage(data: UsageSummaryResponse): void { + process.stdout.write(`${kleur.bold("project usage")} · ${sanitizeForTerminal(data.window)} · by ${data.group_by.join(", ")}\n`); + const groups = [...data.groups] + .sort((left, right) => right.customer_cost_usd - left.customer_cost_usd) + .slice(0, 20); + for (const group of groups) { + const label = [group.workload, group.model, group.day].filter(Boolean).join(" · ") || "all usage"; + process.stdout.write( + `${sanitizeForTerminal(label)}: ${formatUsd(group.customer_cost_usd)} · ${formatCount(group.requests)} requests` + + ` · cache ${formatPercent(group.cache_read_pct)} · errors ${formatPercent(group.error_rate)}\n`, + ); + } + if (data.groups.length > groups.length) { + process.stdout.write(`${kleur.gray(`showing ${groups.length} of ${data.groups.length} groups; use --json for all rows`)}\n`); + } + process.stdout.write(`${kleur.gray(`generated ${sanitizeForTerminal(data.generated_at)}; customer pricing, not provider cost`)}\n`); +} + +function renderWorkloadStatus(data: WorkloadStatusResponse): void { + process.stdout.write(`${kleur.bold("workload status")} · ${sanitizeForTerminal(data.window)}\n`); + for (const workload of data.workloads) { + const route = workload.declared.routed === "none" + ? "none" + : `${workload.declared.routed} @ ${workload.declared.split_pct}%`; + process.stdout.write( + `${sanitizeForTerminal(workload.display_name)}: ${sanitizeForTerminal(workload.status)}` + + ` · ${formatCount(workload.requests)} requests · errors ${formatPercent(workload.error_rate)}` + + ` · declared ${sanitizeForTerminal(route)} · observed ${formatPercent(workload.rerouted_pct)} understudy\n`, + ); + if (workload.route_shares.fallback > 0) { + process.stdout.write(` fallback ${formatPercent(workload.route_shares.fallback)}\n`); + } + if (workload.example_request_ids.length > 0) { + process.stdout.write( + ` request ids ${workload.example_request_ids.map(sanitizeForTerminal).join(", ")}\n`, + ); + } + } + if (data.workloads.length === 0) { + process.stdout.write(`${kleur.gray("No workloads are configured for this project.")}\n`); + } + process.stdout.write(`${kleur.gray(`generated ${sanitizeForTerminal(data.generated_at)}`)}\n`); +} + +function renderCallCost(data: CallCostResponse): void { + process.stdout.write(`${kleur.bold("call cost")} · ${sanitizeForTerminal(data.request_id)}\n`); + process.stdout.write(`pricing status ${sanitizeForTerminal(data.pricing_status)}\n`); + if (data.pricing_status === "priced" && data.customer_cost_usd !== null) { + process.stdout.write(`customer cost ${formatUsd(data.customer_cost_usd)}\n`); + } else { + const reason = data.unpriced_reason?.replaceAll("_", " ") ?? "pricing pending"; + process.stdout.write(`customer cost pending (${sanitizeForTerminal(reason)})\n`); + } + process.stdout.write(`model ${sanitizeForTerminal(data.served_model)} (${sanitizeForTerminal(data.provider)})\n`); + process.stdout.write( + `tokens ${formatCount( + data.tokens.input_tokens + + data.tokens.cache_creation_input_tokens + + data.tokens.cache_read_input_tokens + + data.tokens.output_tokens + + data.tokens.reasoning_output_tokens, + )}\n`, + ); + renderCoverage(data.coverage); +} + +function renderCostBreakdown(data: CostBreakdownResponse): void { + process.stdout.write(`${kleur.bold("cost breakdown")} · ${sanitizeForTerminal(data.window)}\n`); + process.stdout.write(`customer cost ${formatUsd(data.totals.total_usd)}\n`); + process.stdout.write(`priced ${formatCount(data.totals.priced_requests)}/${formatCount(data.totals.requests)} requests\n`); + renderCostCategories("total categories", data.totals); + for (const workload of [...data.workloads].sort((left, right) => right.cost.total_usd - left.cost.total_usd)) { + process.stdout.write( + `${sanitizeForTerminal(workload.workload ?? workload.workload_id)}: ${formatUsd(workload.cost.total_usd)}` + + ` · ${formatCount(workload.priced_requests)}/${formatCount(workload.requests)} priced\n`, + ); + renderCostCategories(" categories", workload.cost); + } + renderCoverage(data.coverage); +} + +function renderCostCategories( + label: string, + cost: { + uncached_input_usd: number; + cache_write_usd: number; + cache_read_usd: number; + output_usd: number; + }, +): void { + process.stdout.write( + `${label.padEnd(16)}uncached ${formatUsd(cost.uncached_input_usd)}` + + ` · cache write ${formatUsd(cost.cache_write_usd)}` + + ` · cache read ${formatUsd(cost.cache_read_usd)}` + + ` · output ${formatUsd(cost.output_usd)}\n`, + ); +} + +function renderCoverage(data: Coverage): void { + process.stdout.write(`coverage ${formatPercent(data.data_completeness)}\n`); + for (const gap of data.known_gaps) { + process.stdout.write(`${kleur.yellow("gap")} ${sanitizeForTerminal(gap)}\n`); + } + if (data.source_timestamp) { + process.stdout.write(`${kleur.gray(`data as of ${sanitizeForTerminal(data.source_timestamp)}`)}\n`); + } +} diff --git a/src/index.ts b/src/index.ts index 9aa8fffe..3f7c0f51 100644 --- a/src/index.ts +++ b/src/index.ts @@ -45,6 +45,8 @@ import { registerLoginCommand } from "./commands/login.js"; import { registerLogoutCommand } from "./commands/logout.js"; import { registerModelsCommand } from "./commands/models.js"; import { registerProjectsCommand } from "./commands/projects.js"; +import { registerReportingCommand } from "./commands/reporting.js"; +import { registerBillingCommand } from "./commands/billing.js"; import { registerRoutesCommand } from "./commands/routes.js"; import { registerRunCommand } from "./commands/run.js"; import { registerRunsCommand } from "./commands/runs.js"; @@ -920,6 +922,8 @@ export function buildProgram(): Command { registerKeysCommand(program); registerModelsCommand(program); registerProjectsCommand(program); + registerReportingCommand(program); + registerBillingCommand(program); registerWorkloadsCommand(program); registerCapturesCommand(program); registerTracesCommand(program); diff --git a/src/internal/projects.ts b/src/internal/projects.ts index 3c5928c6..03c85124 100644 --- a/src/internal/projects.ts +++ b/src/internal/projects.ts @@ -33,6 +33,10 @@ export interface ResolvedProject { project: Project | null; } +export function resolveOrganizationAuth(org?: string): ResolvedAuth { + return resolveAuth(org ?? readProjectConfig()?.org_id); +} + export async function listProjects(auth: ResolvedAuth): Promise { const projects: Project[] = []; let cursor: string | null = null; @@ -50,7 +54,7 @@ export async function listProjects(auth: ResolvedAuth): Promise { export async function resolveProject(opts: ProjectResolutionOptions): Promise { const config = readProjectConfig(); - const auth = resolveAuth(opts.org ?? config?.org_id); + const auth = resolveOrganizationAuth(opts.org ?? config?.org_id); if (opts.projectId) { return { diff --git a/src/internal/reporting-contracts.ts b/src/internal/reporting-contracts.ts new file mode 100644 index 00000000..d985bc1a --- /dev/null +++ b/src/internal/reporting-contracts.ts @@ -0,0 +1,360 @@ +import { z } from "zod"; + +// Customer-safe response contracts mirrored from the hosted admin API schemas. +// The public CLI validates this wire boundary locally so it can remain a +// standalone package. + +export const ReportingWindowSchema = z.enum(["24h", "7d", "30d", "custom"]); +export const ReportingGranularitySchema = z.enum(["minute", "hour", "day"]); +export const ReportingGroupBySchema = z.enum(["project", "workload", "model"]); +export const UsageGroupBySchema = z.enum(["workload", "model", "day"]); + +export const ReportingTotalsSchema = z.object({ + requests: z.number().int(), + input_tokens: z.number().int(), + cache_read_input_tokens: z.number().int(), + cache_creation_input_tokens: z.number().int(), + output_tokens: z.number().int(), + total_tokens: z.number().int(), + customer_cost_usd: z.number(), +}).passthrough(); + +export const OrganizationReportingResponseSchema = z.object({ + org_id: z.string(), + window: ReportingWindowSchema, + window_start: z.string(), + window_end: z.string(), + granularity: ReportingGranularitySchema, + group_by: ReportingGroupBySchema, + filters: z.object({ + project_id: z.string().nullable(), + workload_id: z.string().nullable(), + exclude_project_ids: z.array(z.string()), + }).passthrough(), + totals: ReportingTotalsSchema, + series: z.array(ReportingTotalsSchema.extend({ + bucket: z.string(), + project_id: z.string().nullable(), + project: z.string().nullable(), + workload_id: z.string().nullable(), + workload: z.string().nullable(), + model: z.string().nullable(), + }).passthrough()), + generated_at: z.string(), +}).passthrough(); +export type OrganizationReportingResponse = z.infer< + typeof OrganizationReportingResponseSchema +>; + +export const UsageSummaryResponseSchema = z.object({ + project_id: z.string(), + window: z.string(), + window_start: z.string(), + window_end: z.string(), + group_by: z.array(UsageGroupBySchema), + groups: z.array(z.object({ + workload_id: z.string().nullable(), + workload: z.string().nullable(), + model: z.string().nullable(), + day: z.string().nullable(), + requests: z.number().int(), + input_tokens: z.number().int(), + output_tokens: z.number().int(), + cache_read_input_tokens: z.number().int(), + cache_creation_input_tokens: z.number().int(), + cache_read_pct: z.number(), + customer_cost_usd: z.number(), + error_rate: z.number(), + }).passthrough()), + generated_at: z.string(), +}).passthrough(); +export type UsageSummaryResponse = z.infer; + +export const CoverageSchema = z.object({ + source_timestamp: z.string().nullable(), + data_completeness: z.number().min(0).max(1), + known_gaps: z.array(z.string()), +}).passthrough(); +export type Coverage = z.infer; + +export const TokenCountsSchema = z.object({ + input_tokens: z.number().int(), + cache_creation_input_tokens: z.number().int(), + cache_read_input_tokens: z.number().int(), + output_tokens: z.number().int(), + reasoning_output_tokens: z.number().int(), +}).passthrough(); + +export const CostCategoriesSchema = z.object({ + uncached_input_usd: z.number(), + cache_write_usd: z.number(), + cache_read_usd: z.number(), + output_usd: z.number(), +}).passthrough(); + +const CallCostResponseBaseSchema = z.object({ + org_id: z.string(), + request_id: z.string(), + ts: z.string(), + project_id: z.string(), + workload_id: z.string(), + provider: z.enum(["anthropic", "openai", "managed"]), + served_model: z.string(), + tokens: TokenCountsSchema, + unpriced_reason: z.string().nullable(), + cost_categories: CostCategoriesSchema.nullable(), + coverage: CoverageSchema, + generated_at: z.string(), +}); + +export const CallCostResponseSchema = z.discriminatedUnion("pricing_status", [ + CallCostResponseBaseSchema.extend({ + pricing_status: z.literal("priced"), + customer_cost_usd: z.number(), + }).passthrough(), + CallCostResponseBaseSchema.extend({ + pricing_status: z.literal("unpriced"), + customer_cost_usd: z.null(), + }).passthrough(), +]); +export type CallCostResponse = z.infer; + +const CostBreakdownCategoriesSchema = CostCategoriesSchema.extend({ + total_usd: z.number(), +}).passthrough(); + +export const CostBreakdownResponseSchema = z.object({ + project_id: z.string(), + window: z.string(), + window_start: z.string(), + window_end: z.string(), + workload_id: z.string().nullable(), + workloads: z.array(z.object({ + workload_id: z.string(), + workload: z.string().nullable(), + requests: z.number().int(), + priced_requests: z.number().int(), + cost: CostBreakdownCategoriesSchema, + }).passthrough()), + totals: CostBreakdownCategoriesSchema.extend({ + requests: z.number().int(), + priced_requests: z.number().int(), + }).passthrough(), + coverage: CoverageSchema, + generated_at: z.string(), +}).passthrough(); + +export const WorkloadStatusResponseSchema = z.object({ + project_id: z.string(), + window: z.string(), + window_start: z.string(), + window_end: z.string(), + workloads: z.array(z.object({ + workload_id: z.string(), + display_name: z.string(), + status: z.enum(["healthy", "degraded", "idle"]), + mode: z.enum(["anthropic", "openai", "managed"]).nullable(), + declared: z.object({ + routed: z.enum(["pin", "steer", "none"]), + split_pct: z.number().int().min(0).max(100), + }).passthrough(), + requests: z.number().int(), + route_shares: z.object({ + primary: z.number(), + understudy: z.number(), + fallback: z.number(), + }).passthrough(), + error_rate: z.number(), + last_error_at: z.string().nullable(), + example_request_ids: z.array(z.string()), + served_models: z.array(z.object({ + model: z.string(), + provider_label: z.enum(["anthropic", "openai", "managed"]), + requests: z.number().int(), + share: z.number(), + }).passthrough()), + rerouted_pct: z.number(), + }).passthrough()), + workload_count: z.number().int(), + generated_at: z.string(), +}).passthrough(); +export type WorkloadStatusResponse = z.infer; +export type CostBreakdownResponse = z.infer; + +export const BillingTokenBreakdownSchema = z.object({ + input_tokens: z.number(), + cache_creation_input_tokens: z.number(), + cache_read_input_tokens: z.number(), + output_tokens: z.number(), + reasoning_output_tokens: z.number(), + total_tokens: z.number(), +}).passthrough(); + +export const BillingBalanceResponseSchema = z.object({ + balance: z.object({ + org_id: z.string(), + billing_mode: z.string(), + status: z.string(), + balance_usd: z.number(), + currency: z.string(), + low_balance_threshold_usd: z.number(), + grants: z.object({ + total_granted_usd: z.number(), + total_remaining_usd: z.number(), + soonest_expiry: z.string().nullable(), + }).passthrough(), + }).passthrough(), +}).passthrough(); +export type BillingBalanceResponse = z.infer; + +export const BillingSummaryResponseSchema = z.object({ + summary: z.object({ + org_id: z.string(), + from: z.string(), + to: z.string(), + tokens: BillingTokenBreakdownSchema, + metered_requests: z.number(), + priced_events: z.number(), + estimated_cost_usd: z.number(), + blended_price_per_mtok: z.number(), + }).passthrough(), +}).passthrough(); +export type BillingSummaryResponse = z.infer; + +export const BillingTrendResponseSchema = z.object({ + points: z.array(z.object({ + day: z.string(), + tokens: BillingTokenBreakdownSchema, + cost_usd: z.number(), + }).passthrough()), +}).passthrough(); +export type BillingTrendResponse = z.infer; + +const USD_FORMATTER = new Intl.NumberFormat("en-US", { + style: "currency", + currency: "USD", + minimumFractionDigits: 2, + maximumFractionDigits: 2, +}); +const SMALL_USD_FORMATTER = new Intl.NumberFormat("en-US", { + style: "currency", + currency: "USD", + minimumFractionDigits: 2, + maximumFractionDigits: 6, +}); +const COUNT_FORMATTER = new Intl.NumberFormat("en-US", { + maximumFractionDigits: 0, +}); +const PERCENT_FORMATTER = new Intl.NumberFormat("en-US", { + style: "percent", + maximumFractionDigits: 1, +}); + +export function parseDuration(value: string, label: string, maxMinutes: number): string { + const normalized = value.trim().toLowerCase(); + const match = normalized.match(/^(\d+)(m|h|d)$/); + if (!match) { + throw new Error(`${label} must be a duration such as 6h, 24h, 7d, or 30d.`); + } + const count = Number(match[1]); + const unit = match[2]; + const minutes = count * (unit === "m" ? 1 : unit === "h" ? 60 : 1_440); + if (count < 1 || minutes > maxMinutes) { + throw new Error(`${label} must be greater than zero and no longer than ${Math.floor(maxMinutes / 1_440)}d.`); + } + return normalized; +} + +export function parseBillingWindow(from: string, to: string): { from: string; to: string } { + const explicitTimezone = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})(?::(\d{2})(?:\.(\d{1,9}))?)?(?:Z|[+-](\d{2}):(\d{2}))$/; + const normalizedFrom = from.trim(); + const normalizedTo = to.trim(); + const fromMatch = normalizedFrom.match(explicitTimezone); + const toMatch = normalizedTo.match(explicitTimezone); + if (!fromMatch || !toMatch) { + throw new Error("--from and --to must be ISO timestamps with Z or a numeric UTC offset."); + } + const hasValidCalendarComponents = (match: RegExpMatchArray): boolean => { + const year = Number(match[1]); + const month = Number(match[2]); + const day = Number(match[3]); + const hour = Number(match[4]); + const minute = Number(match[5]); + const second = Number(match[6] ?? "0"); + const offsetHour = Number(match[8] ?? "0"); + const offsetMinute = Number(match[9] ?? "0"); + const leapYear = year % 4 === 0 && (year % 100 !== 0 || year % 400 === 0); + const daysInMonth = [31, leapYear ? 29 : 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31][month - 1]; + return daysInMonth !== undefined + && day >= 1 + && day <= daysInMonth + && hour <= 23 + && minute <= 59 + && second <= 59 + && offsetHour <= 23 + && offsetMinute <= 59; + }; + if (!hasValidCalendarComponents(fromMatch) || !hasValidCalendarComponents(toMatch)) { + throw new Error("--from and --to must be valid ISO UTC timestamps."); + } + const fromMs = Date.parse(normalizedFrom); + const toMs = Date.parse(normalizedTo); + if (!Number.isFinite(fromMs) || !Number.isFinite(toMs)) { + throw new Error("--from and --to must be valid ISO UTC timestamps."); + } + if (toMs <= fromMs) { + throw new Error("--from must be strictly before --to."); + } + return { from: new Date(fromMs).toISOString(), to: new Date(toMs).toISOString() }; +} + +export function parseCustomReportingRange( + from: string | undefined, + to: string | undefined, +): { from: string; to: string } | null { + if (from === undefined && to === undefined) return null; + if (!from || !to) { + throw new Error("Pass both --from and --to for a custom reporting range."); + } + if (!/^\d{4}-\d{2}-\d{2}$/.test(from) || !/^\d{4}-\d{2}-\d{2}$/.test(to)) { + throw new Error("--from and --to must use YYYY-MM-DD for custom reporting ranges."); + } + const fromMs = Date.parse(`${from}T00:00:00Z`); + const toMs = Date.parse(`${to}T00:00:00Z`); + if ( + !Number.isFinite(fromMs) || + !Number.isFinite(toMs) || + new Date(fromMs).toISOString().slice(0, 10) !== from || + new Date(toMs).toISOString().slice(0, 10) !== to + ) { + throw new Error("--from and --to must be valid UTC calendar dates."); + } + const inclusiveDays = Math.floor((toMs - fromMs) / 86_400_000) + 1; + if (!Number.isFinite(inclusiveDays) || inclusiveDays < 1 || inclusiveDays > 366) { + throw new Error("Custom reporting ranges must be ordered and no longer than 366 days."); + } + return { from, to }; +} + +export function formatUsd(value: number): string { + if (value > 0 && value < 0.000001) return "<$0.000001"; + const formatter = value !== 0 && Math.abs(value) < 0.01 + ? SMALL_USD_FORMATTER + : USD_FORMATTER; + return formatter.format(value); +} + +export function formatCount(value: number): string { + return COUNT_FORMATTER.format(value); +} + +export function formatPercent(value: number): string { + return PERCENT_FORMATTER.format(value); +} + +export function sanitizeForTerminal(value: string): string { + return value.replace(/[\u0000-\u001f\u007f-\u009f]/g, (character) => { + const codePoint = character.codePointAt(0) ?? 0; + return `\\u${codePoint.toString(16).padStart(4, "0")}`; + }); +} diff --git a/tests/cli.test.mjs b/tests/cli.test.mjs index cef13a8b..2b7bc079 100644 --- a/tests/cli.test.mjs +++ b/tests/cli.test.mjs @@ -193,6 +193,230 @@ async function withHostedFixture(fn) { if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/projects") return send(200, { projects: state.projects, cursor: null }); if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/api_keys") return send(200, { keys: [{ id: "key_1", name: "default", obfuscated_value: "sk_...test", last_used_at: null, permissions: [], created_at: "2026-06-01T00:00:00Z" }] }); if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/models") return send(200, { models: [{ id: "glm-5.1", display_name: "GLM 5.1", capabilities: ["chat"], context_window: 128000 }] }); + if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/reporting") { + return send(200, { + org_id: "org_1", + window: url.searchParams.has("from") ? "custom" : (url.searchParams.get("window") ?? "7d"), + window_start: "2026-08-19T00:00:00.000Z", + window_end: "2026-08-26T00:00:00.000Z", + granularity: url.searchParams.get("granularity") ?? "day", + group_by: url.searchParams.get("group_by") ?? "project", + filters: { + project_id: url.searchParams.get("project_id"), + workload_id: url.searchParams.get("workload_id"), + exclude_project_ids: url.searchParams.getAll("exclude_project_id"), + }, + totals: { + requests: 120, + input_tokens: 12000, + cache_read_input_tokens: 4000, + cache_creation_input_tokens: 1000, + output_tokens: 3000, + total_tokens: 20000, + customer_cost_usd: 4.25, + future_total: "preserved", + }, + series: [{ + bucket: "2026-08-25", + project_id: "proj_1", + project: "Rehearsal", + workload_id: "usp_classify", + workload: "classify", + model: null, + requests: 120, + input_tokens: 12000, + cache_read_input_tokens: 4000, + cache_creation_input_tokens: 1000, + output_tokens: 3000, + total_tokens: 20000, + customer_cost_usd: 4.25, + }], + generated_at: "2026-08-26T00:00:01.000Z", + future_field: "preserved", + }); + } + if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/projects/proj_1/usage-summary") { + const groups = url.searchParams.get("window") === "29d" + ? Array.from({ length: 5_000 }, (_, index) => ({ + workload_id: `usp_${index}`, + workload: `workload-${index}`, + model: null, + day: null, + requests: 1, + input_tokens: 1, + output_tokens: 1, + cache_read_input_tokens: 0, + cache_creation_input_tokens: 0, + cache_read_pct: 0, + customer_cost_usd: 0, + error_rate: 0, + })) + : [{ + workload_id: "usp_classify", + workload: "classify", + model: null, + day: null, + requests: 120, + input_tokens: 12000, + output_tokens: 3000, + cache_read_input_tokens: 4000, + cache_creation_input_tokens: 1000, + cache_read_pct: 0.2353, + customer_cost_usd: 4.25, + error_rate: 0.01, + }]; + return send(200, { + project_id: "proj_1", + window: url.searchParams.get("window") ?? "7d", + window_start: "2026-08-19T00:00:00.000Z", + window_end: "2026-08-26T00:00:00.000Z", + group_by: (url.searchParams.get("group_by") ?? "workload").split(","), + groups, + generated_at: "2026-08-26T00:00:01.000Z", + }); + } + if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/projects/proj_1/workload-status") { + return send(200, { + project_id: "proj_1", + window: url.searchParams.get("window") ?? "24h", + window_start: "2026-08-25T00:00:00.000Z", + window_end: "2026-08-26T00:00:00.000Z", + workloads: [{ + workload_id: "usp_classify", + display_name: "classify", + status: "healthy", + mode: "managed", + declared: { routed: "pin", split_pct: 10 }, + requests: 120, + route_shares: { primary: 0.89, understudy: 0.1, fallback: 0.01 }, + error_rate: 0.01, + last_error_at: "2026-08-25T12:00:00.000Z", + example_request_ids: ["req_123"], + served_models: [{ model: "glm-5.1", provider_label: "managed", requests: 12, share: 0.1 }], + rerouted_pct: 0.1, + }], + workload_count: 1, + generated_at: "2026-08-26T00:00:01.000Z", + }); + } + const callCost = url.pathname.match(/^\/admin\/v1\/orgs\/org_1\/calls\/([^/]+)\/cost$/); + if (req.method === "GET" && callCost) { + const correlationId = decodeURIComponent(callCost[1]); + const priced = correlationId !== "req_pending" && correlationId !== "req_unpriced_with_cost"; + const customerCost = correlationId === "req_pending" || correlationId === "req_priced_without_cost" + ? null + : correlationId === "req_tiny" ? 0.0000001 : 0.012345; + return send(200, { + org_id: "org_1", + request_id: priced ? "req_123" : "req_pending", + ts: "2026-08-25T12:00:00.000Z", + project_id: "proj_1", + workload_id: "usp_classify", + provider: "managed", + served_model: "glm-5.1", + tokens: { + input_tokens: 100, + cache_creation_input_tokens: 10, + cache_read_input_tokens: 20, + output_tokens: 30, + reasoning_output_tokens: 0, + }, + pricing_status: priced ? "priced" : "unpriced", + unpriced_reason: priced ? null : "pricing_pending", + customer_cost_usd: customerCost, + cost_categories: priced ? { + uncached_input_usd: 0.004, + cache_write_usd: 0.001, + cache_read_usd: 0.000345, + output_usd: 0.007, + } : null, + coverage: { + source_timestamp: "2026-08-25T12:00:00.000Z", + data_completeness: priced ? 1 : 0, + known_gaps: priced ? [] : ["Pricing is still pending for this call."], + }, + generated_at: "2026-08-26T00:00:01.000Z", + }); + } + if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/projects/proj_1/cost-breakdown") { + const cost = { + uncached_input_usd: 2, + cache_write_usd: 0.5, + cache_read_usd: 0.25, + output_usd: 1.5, + total_usd: 4.25, + }; + return send(200, { + project_id: "proj_1", + window: url.searchParams.get("window") ?? "7d", + window_start: "2026-08-19T00:00:00.000Z", + window_end: "2026-08-26T00:00:00.000Z", + workload_id: url.searchParams.get("workload_id"), + workloads: [{ workload_id: "usp_classify", workload: "classify", requests: 120, priced_requests: 118, cost }], + totals: { ...cost, requests: 120, priced_requests: 118 }, + coverage: { + source_timestamp: "2026-08-25T23:59:00.000Z", + data_completeness: 0.9833, + known_gaps: ["2 requests are awaiting pricing.\nforged\u001b]8;;https://example.com\u0007"], + }, + generated_at: "2026-08-26T00:00:01.000Z", + future_field: "preserved", + }); + } + if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/billing/balance") { + return send(200, { + balance: { + org_id: "org_1", + billing_mode: "prepaid", + status: "active", + balance_usd: 95.75, + currency: "USD", + low_balance_threshold_usd: 10, + grants: { + total_granted_usd: 100, + total_remaining_usd: 95.75, + soonest_expiry: null, + }, + }, + }); + } + if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/billing/summary") { + return send(200, { + summary: { + org_id: "org_1", + from: url.searchParams.get("from"), + to: url.searchParams.get("to"), + tokens: { + input_tokens: 12000, + cache_read_input_tokens: 4000, + cache_creation_input_tokens: 1000, + output_tokens: 3000, + reasoning_output_tokens: 0, + total_tokens: 20000, + }, + metered_requests: 120, + priced_events: 118, + estimated_cost_usd: 4.25, + blended_price_per_mtok: 212.5, + }, + }); + } + if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/billing/trend") { + return send(200, { + points: [{ + day: "2026-08-25", + tokens: { + input_tokens: 12000, + cache_read_input_tokens: 4000, + cache_creation_input_tokens: 1000, + output_tokens: 3000, + reasoning_output_tokens: 0, + total_tokens: 20000, + }, + cost_usd: 4.25, + }], + }); + } if (req.method === "GET" && url.pathname === "/admin/v1/orgs/org_1/projects/proj_1/workloads") return send(200, { workloads: state.workloads, cursor: null }); if (req.method === "POST" && url.pathname === "/admin/v1/orgs/org_1/projects/proj_1/workloads") { const workload = { id: `usp_${body.name}`, project_id: "proj_1", name: body.name, capture_enabled: Boolean(body.capture_enabled), route_model_id: null, route_traffic_pct: null, is_default: false, created_at: "2026-06-07T00:00:00Z" }; @@ -2830,6 +3054,345 @@ class ScoreWithFeedback: }); }); + it("reads customer-safe reporting and billing data through first-class commands", async () => { + await withHostedFixture(async ({ home, repo, requests }) => { + const env = { HOME: home, USERPROFILE: home }; + const credentialsPath = join(home, ".understudy", "credentials.json"); + const credentials = JSON.parse(readFileSync(credentialsPath, "utf8")); + credentials.orgs.org_2 = { + api_key: "sk_test_org_2", + gateway_url: credentials.gateway_url, + }; + writeFileSync(credentialsPath, `${JSON.stringify(credentials, null, 2)}\n`); + + const reporting = await runWithEnvAsync([ + "--json", + "reporting", + "summary", + "--window", + "7d", + "--group-by", + "workload", + "--exclude-project-id", + "proj_internal", + "--exclude-project-id", + " proj_internal ", + ], env, repo); + assert.equal(reporting.status, 0, reporting.stderr); + const reportingPayload = JSON.parse(reporting.stdout); + assert.equal(reportingPayload.totals.customer_cost_usd, 4.25); + assert.equal(reportingPayload.totals.future_total, "preserved"); + assert.equal(reportingPayload.future_field, "preserved"); + assert.equal(requests.at(-1).path, "/admin/v1/orgs/org_1/reporting"); + assert.equal(requests.at(-1).search, "?window=7d&group_by=workload&exclude_project_id=proj_internal"); + assert.equal(requests.at(-1).headers.authorization, "Bearer sk_test_hosted"); + + const usage = await runWithEnvAsync([ + "--json", + "reporting", + "usage", + "--project", + "rehearsal", + "--window", + "30d", + "--group-by", + "workload,day", + ], env, repo); + assert.equal(usage.status, 0, usage.stderr); + assert.equal(JSON.parse(usage.stdout).groups[0].cache_read_pct, 0.2353); + assert.equal(requests.at(-1).path, "/admin/v1/orgs/org_1/projects/proj_1/usage-summary"); + assert.equal(requests.at(-1).search, "?window=30d&group_by=workload%2Cday"); + + const workloadStatus = await runWithEnvAsync([ + "--json", + "reporting", + "workload-status", + "--project", + "rehearsal", + "--window", + "6h", + ], env, repo); + assert.equal(workloadStatus.status, 0, workloadStatus.stderr); + assert.equal(JSON.parse(workloadStatus.stdout).workloads[0].route_shares.understudy, 0.1); + assert.equal(requests.at(-1).path, "/admin/v1/orgs/org_1/projects/proj_1/workload-status"); + assert.equal(requests.at(-1).search, "?window=6h"); + + const callCost = await runWithEnvAsync([ + "--json", + "reporting", + "cost", + "upstream_message_123", + ], env, repo); + assert.equal(callCost.status, 0, callCost.stderr); + assert.equal(JSON.parse(callCost.stdout).request_id, "req_123"); + assert.equal(JSON.parse(callCost.stdout).customer_cost_usd, 0.012345); + assert.equal(requests.at(-1).path, "/admin/v1/orgs/org_1/calls/upstream_message_123/cost"); + + const pending = await runWithEnvAsync([ + "--json", + "reporting", + "cost", + "req_pending", + ], env, repo); + assert.equal(pending.status, 0, pending.stderr); + assert.equal(JSON.parse(pending.stdout).pricing_status, "unpriced"); + assert.equal(JSON.parse(pending.stdout).customer_cost_usd, null); + + const breakdown = await runWithEnvAsync([ + "--json", + "reporting", + "cost-breakdown", + "--project-id", + "proj_1", + "--window", + "7d", + "--workload-id", + "usp_classify", + ], env, repo); + assert.equal(breakdown.status, 0, breakdown.stderr); + const breakdownPayload = JSON.parse(breakdown.stdout); + assert.equal(breakdownPayload.totals.total_usd, 4.25); + assert.equal(breakdownPayload.coverage.data_completeness, 0.9833); + assert.equal(breakdownPayload.future_field, "preserved"); + assert.equal(requests.at(-1).search, "?window=7d&workload_id=usp_classify"); + + const balance = await runWithEnvAsync(["--json", "billing", "balance"], env, repo); + assert.equal(balance.status, 0, balance.stderr); + assert.equal(JSON.parse(balance.stdout).balance.balance_usd, 95.75); + + const summary = await runWithEnvAsync([ + "--json", + "billing", + "summary", + "--from", + "2026-08-19T00:00:00Z", + "--to", + "2026-08-26T00:00:00Z", + ], env, repo); + assert.equal(summary.status, 0, summary.stderr); + assert.equal(JSON.parse(summary.stdout).summary.priced_events, 118); + assert.equal(requests.at(-1).search, "?from=2026-08-19T00%3A00%3A00.000Z&to=2026-08-26T00%3A00%3A00.000Z"); + + const trend = await runWithEnvAsync([ + "--json", + "billing", + "trend", + "--from", + "2026-08-19T00:00:00Z", + "--to", + "2026-08-26T00:00:00Z", + ], env, repo); + assert.equal(trend.status, 0, trend.stderr); + assert.equal(JSON.parse(trend.stdout).points[0].cost_usd, 4.25); + + const humanSummary = await runWithEnvAsync([ + "billing", + "summary", + "--from", + "2026-08-19T00:00:00Z", + "--to", + "2026-08-26T00:00:00Z", + ], env, repo); + assert.equal(humanSummary.status, 0, humanSummary.stderr); + assert.match(humanSummary.stdout, /coverage note/i); + assert.match(humanSummary.stdout, /zero-cost traffic or pricing lag/i); + assert.doesNotMatch(humanSummary.stdout, /pricing gap/i); + + const humanBreakdown = await runWithEnvAsync([ + "reporting", + "cost-breakdown", + "--project-id", + "proj_1", + ], env, repo); + assert.equal(humanBreakdown.status, 0, humanBreakdown.stderr); + assert.match(humanBreakdown.stdout, /uncached \$2\.00/); + assert.match(humanBreakdown.stdout, /cache write \$0\.50/); + assert.match(humanBreakdown.stdout, /cache read \$0\.25/); + assert.match(humanBreakdown.stdout, /output \$1\.50/); + assert.doesNotMatch(humanBreakdown.stdout, /\u001b|\u0007/); + assert.match(humanBreakdown.stdout, /\\u000a/); + assert.match(humanBreakdown.stdout, /\\u001b/); + + const human = await runWithEnvAsync([ + "reporting", + "cost", + "req_pending", + ], env, repo); + assert.equal(human.status, 0, human.stderr); + assert.match(human.stdout, /unpriced/i); + assert.match(human.stdout, /pricing pending/i); + assert.doesNotMatch(human.stdout, /\$0(?:\.00+)?\b/); + + const tiny = await runWithEnvAsync([ + "reporting", + "cost", + "req_tiny", + ], env, repo); + assert.equal(tiny.status, 0, tiny.stderr); + assert.match(tiny.stdout, /<\$0\.000001/); + }); + }); + + it("rejects invalid reporting dimensions and billing windows before the API call", async () => { + await withHostedFixture(async ({ home, repo, requests }) => { + const env = { HOME: home, USERPROFILE: home }; + const initialRequests = requests.length; + + const groupBy = await runWithEnvAsync([ + "--json", + "reporting", + "summary", + "--group-by", + "provider", + ], env, repo); + assert.notEqual(groupBy.status, 0); + assert.match(groupBy.stderr, /group-by/i); + + const window = await runWithEnvAsync([ + "--json", + "billing", + "summary", + "--from", + "2026-08-26T00:00:00Z", + "--to", + "2026-08-19T00:00:00Z", + ], env, repo); + assert.notEqual(window.status, 0); + assert.match(window.stderr, /before/i); + + const naiveTimestamp = await runWithEnvAsync([ + "--json", + "billing", + "trend", + "--from", + "2026-08-19T00:00:00", + "--to", + "2026-08-26T00:00:00", + ], env, repo); + assert.notEqual(naiveTimestamp.status, 0); + assert.match(naiveTimestamp.stderr, /Z or a numeric UTC offset/i); + + const calendarDate = await runWithEnvAsync([ + "--json", + "reporting", + "summary", + "--from", + "2026-02-30", + "--to", + "2026-03-01", + ], env, repo); + assert.notEqual(calendarDate.status, 0); + assert.match(calendarDate.stderr, /calendar dates/i); + + const usageGroup = await runWithEnvAsync([ + "--json", + "reporting", + "usage", + "--project-id", + "proj_1", + "--group-by", + "workload,provider", + ], env, repo); + assert.notEqual(usageGroup.status, 0); + assert.match(usageGroup.stderr, /workload, model, day/i); + + const usageDuration = await runWithEnvAsync([ + "--json", + "reporting", + "usage", + "--project-id", + "proj_1", + "--window", + "31d", + ], env, repo); + assert.notEqual(usageDuration.status, 0); + assert.match(usageDuration.stderr, /no longer than 30d/i); + assert.equal(requests.length, initialRequests); + }); + }); + + it("rejects calendar-invalid billing timestamps before the API call", async () => { + await withHostedFixture(async ({ home, repo, requests }) => { + const env = { HOME: home, USERPROFILE: home }; + const initialRequests = requests.length; + const result = await runWithEnvAsync([ + "--json", + "billing", + "summary", + "--from", + "2026-02-30T00:00:00Z", + "--to", + "2026-03-03T00:00:00Z", + ], env, repo); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /valid ISO UTC timestamps/i); + assert.equal(requests.length, initialRequests); + }); + }); + + it("fails closed when call pricing status contradicts customer cost", async () => { + await withHostedFixture(async ({ home, repo }) => { + const env = { HOME: home, USERPROFILE: home }; + + const pricedWithoutCost = await runWithEnvAsync([ + "--json", + "reporting", + "cost", + "req_priced_without_cost", + ], env, repo); + assert.notEqual(pricedWithoutCost.status, 0); + assert.match(pricedWithoutCost.stderr, /customer_cost_usd/i); + + const unpricedWithCost = await runWithEnvAsync([ + "--json", + "reporting", + "cost", + "req_unpriced_with_cost", + ], env, repo); + assert.notEqual(unpricedWithCost.status, 0); + assert.match(unpricedWithCost.stderr, /customer_cost_usd/i); + }); + }); + + it("fails closed when usage reaches the server group limit", async () => { + await withHostedFixture(async ({ home, repo }) => { + const env = { HOME: home, USERPROFILE: home }; + const result = await runWithEnvAsync([ + "--json", + "reporting", + "usage", + "--project-id", + "proj_1", + "--window", + "29d", + "--group-by", + "workload,day", + ], env, repo); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /5,000-group server limit/i); + }); + }); + + it("normalizes billing timezone offsets before sending them", async () => { + await withHostedFixture(async ({ home, repo, requests }) => { + const env = { HOME: home, USERPROFILE: home }; + const result = await runWithEnvAsync([ + "--json", + "billing", + "summary", + "--from", + "2026-08-18T17:00:00-07:00", + "--to", + "2026-08-25T17:00:00-07:00", + ], env, repo); + assert.equal(result.status, 0, result.stderr); + assert.equal( + requests.at(-1).search, + "?from=2026-08-19T00%3A00%3A00.000Z&to=2026-08-26T00%3A00%3A00.000Z", + ); + }); + }); + it("creates a workload from a local workload card", async () => { await withHostedFixture(async ({ home, repo, requests }) => { const env = { HOME: home, USERPROFILE: home };