diff --git a/CLAUDE.md b/CLAUDE.md index 92a697143..34be585c1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -139,7 +139,7 @@ Key CLI flags for control-plane mode: - Config store: `--config-store`, `--config-poll-interval`, `--internal-secret` - K8s pool: `--k8s-worker-image`, `--k8s-worker-namespace`, `--k8s-control-plane-id`, `--k8s-worker-port`, `--k8s-worker-secret`, `--k8s-worker-configmap`, `--k8s-worker-image-pull-policy`, `--k8s-worker-service-account` (no global worker cap — per-org `Org.MaxWorkers`, 0=unbounded, is the only cap) - AWS / STS: `--aws-region` - - Compute-usage billing needs no config: metering is always on for the remote backend and billing PULLS usage over the internal-secret-authed HTTP API (`GET /api/v1/billing/usage` + `POST /api/v1/billing/ack`). See `docs/design/billing-pull-api.md` and "Compute-Usage Billing" below. + - Billing reports native Trino scan bytes and unchanged storage GiB-seconds through retained batches; see `docs/design/billing-pull-api.md` and "Scan-byte and storage billing" below. - Pod scheduling knobs (CPU/memory requests, node selector, tolerations) are env-only — see `config_resolution.go`. Key CLI flags for duckdb-service mode: @@ -1118,166 +1118,31 @@ not the CP's. The CP wires the suffix from its first configured caller-side minter in the PostHog repo (`products/managed_warehouse/backend/service_credentials.py`). -## Compute-Usage Billing (managed-warehouse, remote backend only) - -duckgres meters per-org compute usage of worker pods into 60s buckets in the -config store; the billing service **pulls** the accumulated usage over an HTTP -API and acks a watermark, at which point duckgres deletes the acked buckets. -Full design + decisions: `docs/design/billing-pull-api.md` (supersedes the -push/capture reporting hop of `billing-compute-seconds-plan.md`; the metering -side of that doc still applies). Scope is **only** the remote/k8s backend -(per-org worker pod with a known `WorkerProfile` size). Pipeline: - -``` -compute: conn end → in-proc counter keyed (org, informational team, query_source, worker size) - │ flusher (~15s) UPSERT-increment → config-store buffer (cross-CP sum) - ▼ duckgres_org_compute_usage (+ duckgres_compute_billing_cursor) -storage: leader sampler (~30m) → org's DuckLake metadata Postgres - SUM(data+delete file sizes) × interval → duckgres_org_storage_usage -billing: GET /api/v1/billing/usage (usage + storage arrays, per key per UTC day, watermarks) - → POST /api/v1/billing/ack {watermark_high} → cursor advance + delete ≤ it (BOTH tables) -safety: leader-only GC hard-deletes buckets older than 30 days (WARN, alertable) -``` - -Two raw metrics per connection over its full lifetime, using the **provisioned** -worker size: `cpu_seconds = vCPU × ceil(conn_secs)`, `memory_seconds = GiB × -ceil(conn_secs)`. Counted internally in integer **millicore-seconds** / -**MiB-seconds** (`compute_meter.go`) to avoid truncating a fractional-core / -sub-GiB worker; worker size is stored in the bucket key as exact NUMERIC -decimals (vCPU / GiB). `team_id` is **informational only** (an integer — -PostHog's `Team.id`; a JSON NUMBER on every API surface): duckgres does NOT -own team-level billing attribution — the external billing service maps -org → team(s) itself. The stamp is resolved from the config snapshot at -record time: compute buckets get the CONNECTING USER's team -(`duckgres_org_users.team_id`, e.g. a project-reader login) when it has one, -else the org's OLDEST team (min `created_at`, ties broken by the smaller -`team_id` — in practice the provision-time first team; `ConfigStore.OrgUsageTeamID`); -storage buckets always get the oldest team (`OrgOldestTeamID`). 0 appears -only defensively (unknown org / stale snapshot — a committed org always has -at least one team). Team changes/deletions NEVER re-attribute existing -buckets; `query_source` is the -`duckgres.query_source` session GUC (`standard` unless set; a mid-connection -change bills the whole connection under the final value). The GUC is a **closed -enum validated at SET time** (`transform.NormalizeQuerySource`): only -`standard` | `endpoints` (case-insensitive, normalized to lowercase; empty = -reset to default) — anything else is rejected with `22023` on every set path -(simple/batched SET, extended Parse, and the `-c` startup option, which rejects -the connection like invalid `duckgres.worker_*` options), and -`server.ConnectionBilling` clamps a non-canonical value to `standard` as -defense in depth so client junk can never become a billing bucket key. -Invariants for anyone -touching this path: - -- **Metering is strictly best-effort and off the hot path.** A metering error - (counter, flush) must NEVER block or fail a query or connection teardown. The - connection-end record is added to an in-process counter (map+mutex, - microseconds, no I/O); the flush is async. `cp.computeMeter` is nil outside - the remote backend — every call site is nil-safe. There is no enable knob: - the remote backend always meters. -- **Worker size is plumbed onto the connection** (`server.SetConnectionWorkerSize` - → `clientConn.workerMillicores/workerMiB`, set in `control.go::handleConnection` - from `workerBillingSize(workerProfile)`, remote-only). `workerMillicores==0` - (non-remote / unknown) → metering skipped. The metric is computed once at the - SAME teardown point as `CloseConnectionMetrics` (the `#841` lifetime defer), - via `server.ConnectionBilling` (which also carries the query source). -- **Bucket = connection-end time floored to 60s.** Flush carries the sub-unit - remainder forward so rounding never loses counts across flushes. Buffer flush - is UPSERT-increment so all CP pods sum into one row per key. -- **Serve only closed buckets.** `watermark_high` = the newest bucket with - `bucket_start ≤ now − 60s − 30s grace` (grace > flush interval, so every - CP's contribution has landed before a minute is served). The GET aggregates - the window `(cursor, watermark_high]` into one row per - `(org, team, query_source, cpu, mem_gib)` per **UTC day** — response size is - bounded by active keys × days, so billing downtime can't make it explode. -- **Ack is the only deletion path (plus the 30d GC).** `POST /billing/ack` - advances the single global cursor monotonically and deletes buckets - `≤ watermark_high` in one TXN (`AckComputeUsage`). Idempotent — re-acks and - stale acks are no-ops. An ack beyond the latest closed bucket is rejected - (400) so it can never delete buckets that were never served. Auth is the - admin internal secret (`RequireAdmin` on both routes, registered inside the - audited `/api/v1` group in `multitenant.go`). -- **Safety GC is leader-only** (`runComputeUsageGC`, attached under the janitor - lease): hard-deletes buckets older than 30 days regardless of ack and logs a - WARN with the dropped count — nonzero means billing stopped pulling (alert). -- **Graceful shutdown does a final flush** after connections drain to their - natural end (`shutdown`/`drainAndShutdown`), so a departing CP pod lands its - last interval before exit. -- **Org team CRUD (`duckgres_org_teams`)**: the PostHog backend manages an - org's team rows via `GET/POST /api/v1/orgs/:id/teams` + - `DELETE /api/v1/orgs/:id/teams/:team_id` (internal secret, - `controlplane/provisioning`). The POST is the **grandfather upsert**: it MAY - overwrite an existing row's `schema_name` and the legacy - `events_table_name`/`persons_table_name`/`schema_data_imports_name` - overrides (NULL = derive from `schema_name`: `.events`, - `.persons`, `_data_imports`), because the PostHog backfill - replaces migration 000024's `team_` placeholder through it. Two teams in - one org can never share a schema (unique `(org_id, schema_name)`, migration - 000025 → 409). Provisioning a warehouse for a NEW org REQUIRES `team_id` - (`ErrProvisionTeamRequired` → 400; `default_team_id` is accepted as a - transitional alias) and creates the org's first plain team row — a - warehouse cannot exist without a team. DELETE removes CONFIG only (never - warehouse data) and never touches usage buckets; the org's LAST team is - undeletable (409 — an org must always have at least one team; delete the - org instead). The admin console mirrors this on a user-facing surface - (`GET /teams`, `POST /teams`, `PUT /orgs/:id/teams/:team_id`) where - `schema_name` is immutable. Shared rules live in - `configstore.UpsertOrgTeamTx` / `DeleteOrgTeamTx`; tests: - `tests/configstore/org_teams_postgres_test.go`, the provisioning/admin API - tests, and `org_teams_crud` in the e2e harness. -- **Storage metric** (`managed_warehouse_storage_gib_seconds`, - `storage_meter.go`): a LEADER-ONLY sampler (double writers would - double-bill — the UPSERT is additive) visits each Ready warehouse's DuckLake - metadata Postgres every 30m (env-only `DUCKGRES_STORAGE_SAMPLE_INTERVAL`; - e2e uses 60s) and credits exactly `tracked_bytes × interval` byte-seconds — - no elapsed-time tracking, a missed sample under-bills one interval. The SUM - is over `ducklake_data_file` + `ducklake_delete_file` with NO snapshot - filter (never `ducklake_table_info()`/`ducklake_table_stats` — current- - snapshot-only / approximate). byte-seconds are NUMERIC (BIGINT overflows); - served as exact-decimal GiB-seconds (÷2³⁰ terminates; - `byteSecondsToGiBSeconds` big-int math). Connection resolution reuses the - cross-org activator (`MetadataPostgresURL`: duckling pgbouncer → sslmode - disable, direct RDS → require). Drift gauges: - `duckgres_org_storage_pending_delete_files` (alert on sustained nonzero) + - `duckgres_org_storage_tracked_bytes`. -- **The admin console usage views read the SAME buffer** — - `GET /api/v1/usage/monthly` (the **Usage** page) and - `GET /api/v1/orgs/:id/usage/daily` (the org detail page's **Usage** charts) - in `controlplane/admin/usage_api.go`, backed by - `configstore.Aggregate{Compute,Storage}Usage{Monthly,Daily}`, sum retained - buckets per UTC month / per UTC day per (org, team), merging the compute and - storage families and joining the team schema name for display. Both - self-gate with `RequireAdmin` (per-team cost data across all orgs is as - sensitive as the raw billing families — viewers get 403, and the UI hides - the nav item / fires no query for them). The daily endpoint's org scope is - the `:id` path segment flowing into the queries' WHERE clause — one org's - usage must never leak into another org's page (the e2e asserts - `.org_id == $o` on the response). These are operations views, NOT invoices: - acked buckets are already deleted and >30d buckets are - GC'd, so responses carry the ack cursor as `watermark_low` and the UI - shows the retention caveat instead of implying all-time totals. They add NO - second accounting pipeline — keep them pure reads over the buffer. - The Usage page also carries a **client-side pricing-sensitivity calculator** - (`ui/src/pages/UsagePricing.tsx` + `lib/pricing.ts`): named unit-price - scenarios ($/CPU-min, $/GiB·min, $/GiB·h) priced against each org's month - totals. It is pure browser math over the monthly rows — no endpoint, no - persistence beyond the operator's own localStorage — so it inherits the - page's admin-only gate and needs no server-side access control of its own - (a PM gets it by holding the console admin role; a lighter pricing-viewer - role is a named follow-up, not implemented). -- Touching the meter/flush/API/GC, the worker-size or query-source plumbing, - the storage sampler, or the bucket keys → update - `controlplane/compute_meter_test.go`, `compute_billing_api_test.go`, - `compute_size_test.go`, `storage_meter_test.go`, - `configstore/storage_usage_test.go`, the migration assertion in - `tests/configstore/migrations_postgres_test.go`, and the - `compute_usage_pull_api` assertion (compute + storage, incl. the - `usage-monthly` checks) in - `tests/mw-dev/e2e/harness.sh`. Touching the monthly/daily aggregation or - the usage views → update `controlplane/admin/usage_api_test.go`, - `tests/configstore/usage_monthly_postgres_test.go` + - `usage_daily_postgres_test.go`, - `ui/src/pages/Usage.test.tsx` + `OrgUsage.test.tsx`, and the - `usage-monthly` / `usage-daily` harness checks. +## Scan-byte and storage billing + +The current accounting contract and operational runbook are in +`docs/design/billing-pull-api.md`. Trino native completion events are committed +through `POST /api/v1/trino/usage` using the dedicated per-cell usage token. All +query outcomes count at `physicalInputBytes`; `(cell, query ID)` deduplicates. +Historical principal ownership is remembered before authentication projection. +Unknown principals remain durable but unexported until resolved. + +Billing uses immutable `POST /billing/batches/next`, exact-ID `POST +/billing/batches/:batch_id/ack`, and retained `GET /billing/batches/:batch_id`. +There is one outstanding batch, no destructive watermark, no retention GC, and +no DuckDB compute meter. Preserve exact numeric amounts and transactional batch +membership. Storage still samples tracked DuckLake data+delete files at the +existing cadence and reports GiB-seconds; additive same-minute samples must not +be lost during export. Ack never removes usage or changes admin history. + +Admin daily/monthly views show scan volume and storage usage. Storage price +calculations remain storage-only; no scan price is invented. Query/row dates +represent occurrence while each batch's UTC `billing_month` controls invoicing. + +Changes must update HTTP tests, PostgreSQL ledger/aggregation tests and the +applicable real-stack harness: `tests/mw-dev/e2e/harness.sh` for storage, +`tests/mw-dev/e2e/trino.sh` for native completion delivery. Document rollout +ordering (RBAC, CP, listener, consumer) and keep SQL/secrets out of usage records. ## Discovery Endpoints (external-writer tenant listing) diff --git a/README.md b/README.md index c443016c1..bb9541038 100644 --- a/README.md +++ b/README.md @@ -548,8 +548,7 @@ The org name is duckgres-internal, so the query events additionally carry a user's team, else the org's oldest team; 0 when unknown or standalone). This is the PostHog-native key that joins duckgres usage to the rest of PostHog (e.g. product-intent cohorts for managed-warehouse activation). It is a config-snapshot -read stamped once per connection, and mirrors the informational team id the -compute-usage meter records. +read stamped once per connection. Billing uses the separate Trino usage ledger. Events never include SQL text, credentials, or secret values — only metadata. @@ -1175,3 +1174,22 @@ The full, authoritative breakdown — every PostgreSQL feature with its support ## License MIT + +## Scan bytes and storage billing + +Trino completion events report native `physicalInputBytes`, including failed and +cancelled queries, to `POST /api/v1/trino/usage`. The provisioner generates a +dedicated immutable `trino-usage-token` Secret. In the Trino chart, +`queryUsage.enabled` defaults to `false`; enable it after the new Duckgres API and +Secret RBAC are deployed. Ingestion accepts at most 16 MiB and waits up to 10s for +a durable database write. The built-in HTTP listener retries transient failures; +rare loss before ingestion is accepted. + +Billing uses `POST /api/v1/billing/batches/next` (optional `limit`, default/max +10,000 per metric family), `POST /api/v1/billing/batches/:batch_id/ack`, and +`GET /api/v1/billing/batches/:batch_id` for retained replay. The API reports exact +scan bytes and storage GiB-seconds; DuckDB CPU/memory metering has been removed. +Usage and batches are kept indefinitely. Storage measurement and its default +30-minute sampling cadence remain unchanged. See the +[billing contract and rollout/recovery runbook](docs/design/billing-pull-api.md) +for downstream idempotency, month boundaries and deployment ordering. diff --git a/controlplane/admin/README.md b/controlplane/admin/README.md index 92451c665..bbbcfa617 100644 --- a/controlplane/admin/README.md +++ b/controlplane/admin/README.md @@ -76,8 +76,8 @@ Added for the console: | `POST /api/v1/orgs/:id/users/:username/disable` | admin | persist `disabled=true` (refused at pgwire connect), reload the snapshot cluster-wide so the block is immediate, AND kill the user's live sessions. Returns `{disabled, killed, …}` | | `POST /api/v1/orgs/:id/users/:username/enable` | admin | persist `disabled=false` + reload cluster-wide so the user can reconnect at once | | `GET /api/v1/metrics/panels`, `/metrics/query_range` | viewer | Prometheus proxy (allow-listed panels only) | -| `GET /api/v1/usage/monthly` | admin | cumulative per-team usage per UTC month (CPU-seconds, memory GiB-seconds, S3 GiB-seconds), backing the **Usage** page. The response also identifies the control plane's effective `aws_region` and its derived `customer_pricing_region`. Self-gates with `RequireAdmin` (not just RoleGate's method check) because per-team cost data across all orgs is as sensitive as the raw billing families. Reads the SAME billing buffer as `GET /billing/usage`, so retention is the buffer's: acked buckets are deleted, >30d buckets GC'd — `watermark_low` in the response marks where billed data was removed. `?months=N` (default 6, max 36) sets the window | -| `GET /api/v1/orgs/:id/usage/daily` | admin | one org's daily per-team usage series (same families), backing the org detail page's **Usage** charts. Same RequireAdmin gate and buffer-retention semantics; the org scope is the `:id` path segment flowing into the queries' WHERE clause. `?days=N` (default 14, max 31 — the buffer's 30d GC bounds useful range) | +| `GET /api/v1/usage/monthly` | admin | retained per-team usage per UTC month (`bytes_scanned`, S3 `gib_seconds`), backing the **Usage** page. The response also identifies the control plane's effective `aws_region` and its derived `customer_pricing_region`. Self-gates with `RequireAdmin` (not just RoleGate's method check) because per-team cost data across all orgs is as sensitive as the raw billing families. Reads the retained query ledger and storage samples; billing acknowledgements never remove history. Scan bytes include failed and cancelled queries, grouped by UTC completion time. `?months=N` (default 6, max 36) sets the window | +| `GET /api/v1/orgs/:id/usage/daily` | admin | one org's daily per-team usage series (same families), backing the org detail page's **Usage** charts. Same RequireAdmin gate and indefinite retention; the org scope is the `:id` path segment flowing into the queries' WHERE clause. `?days=N` (default 14, max 31 to bound response size) | | `GET /api/v1/orgs/:id/monitoring/snapshot` | internal secret | Customer-safe org warehouse state, resource limits, workers, sessions, queue depth, and CP coverage. Omits user, pod, image, SQL, client, trace, and control-plane identifiers | | `GET /api/v1/orgs/:id/monitoring/series` | internal secret | Customer-safe, org-forced Prometheus range query. Requires an allow-listed `metric`; `window` is one of `1h`, `6h`, `24h` (default), `7d`, `30d` | | `GET /api/v1/orgs/:id/users/:username/secrets`, `DELETE .../:name` | viewer/admin | list/delete stored persistent secrets (ciphertext never returned) | @@ -94,7 +94,11 @@ Added for the console: | `POST /api/v1/operators` | admin | add/update an operator (`{email, role}`; last-admin demotion → 409) | | `DELETE /api/v1/operators/:email` | admin | remove an operator (removing the last admin → 409) | -The Usage page currently presents storage only. It converts retained S3 GiB·h +The Usage page presents query scan volume alongside storage economics. Scan +bytes have no price estimate; cost, price and margin totals cover storage only. +The daily organization view shows both scan bytes and storage-time. API byte +aggregates preserve exact decimal digits; charts use rounded display values. +It converts retained S3 GiB·h to GiB-month using the actual number of hours in the selected UTC calendar month. Customer pricing is selected automatically from the control plane's effective `DUCKGRES_AWS_REGION`: `us-*` uses the US schedule and `eu-*` uses diff --git a/controlplane/admin/api_postgres_test.go b/controlplane/admin/api_postgres_test.go index cce4510da..e1137823d 100644 --- a/controlplane/admin/api_postgres_test.go +++ b/controlplane/admin/api_postgres_test.go @@ -384,9 +384,9 @@ func resetConfigStoreTables(t *testing.T, db *gorm.DB) { t.Fatalf("delete %T: %v", model, err) } } - // Billing usage buffers have no gorm model (raw goose-migrated tables) but + // Billing usage tables have no gorm model (raw goose-migrated tables) but // leak across tests on the shared schema all the same. - for _, table := range []string{"duckgres_org_compute_usage", "duckgres_org_storage_usage"} { + for _, table := range []string{"duckgres_trino_query_usage", "duckgres_trino_usage_principals", "duckgres_org_storage_usage"} { if err := db.Exec("DELETE FROM " + table).Error; err != nil { t.Fatalf("delete %s: %v", table, err) } diff --git a/controlplane/admin/ui/src/pages/OrgUsage.test.tsx b/controlplane/admin/ui/src/pages/OrgUsage.test.tsx index 0a6d48b8e..c7055bec4 100644 --- a/controlplane/admin/ui/src/pages/OrgUsage.test.tsx +++ b/controlplane/admin/ui/src/pages/OrgUsage.test.tsx @@ -5,7 +5,7 @@ import { TooltipProvider } from "@/components/ui/tooltip"; import type { DailyUsageResponse } from "@/types/api"; // Mock data + identity hooks: render OrgUsageSection with a controlled daily -// response and assert the chart cards, window totals, period switch, caveat, +// response and assert the chart cards, window totals, period switch, history, // and the viewer gate. (Recharts draws nothing in jsdom — the assertions are // on the surrounding card chrome, which is where the totals live.) const hooks = vi.hoisted(() => ({ @@ -24,12 +24,11 @@ const RESPONSE: DailyUsageResponse = { org_id: "acme", days: 14, from: "2026-08-01T00:00:00Z", - watermark_low: null, rows: [ - // 7200 CPU-seconds = 120 CPU-min; 3600 mem-seconds = 60 GiB·min; 3600 gib-seconds = 1 GiB·h. - { date: "2026-08-13", team_id: 5, schema_name: "team_5", cpu_seconds: 7200, memory_seconds: 3600, gib_seconds: 3600 }, - { date: "2026-08-13", team_id: 6, schema_name: "team_6", cpu_seconds: 60, memory_seconds: 60, gib_seconds: 0 }, - { date: "2026-08-14", team_id: 5, schema_name: "team_5", cpu_seconds: 600, memory_seconds: 600, gib_seconds: 7200 }, + // Scan bytes sum independently of storage time; 3600 GiB-seconds = 1 GiB·h. + { date: "2026-08-13", team_id: 5, schema_name: "team_5", bytes_scanned: 7200, gib_seconds: 3600 }, + { date: "2026-08-13", team_id: 6, schema_name: "team_6", bytes_scanned: 60, gib_seconds: 0 }, + { date: "2026-08-14", team_id: 5, schema_name: "team_5", bytes_scanned: 600, gib_seconds: 7200 }, ], }; @@ -54,9 +53,11 @@ describe("OrgUsageSection", () => { hooks.useOrgDailyUsage.mockReturnValue(ok(RESPONSE)); }); - it("renders one org-level storage chart without compute or team series", () => { + it("renders organization scan and storage charts without CPU, memory or team series", () => { renderSection(); expect(screen.getByText("S3 GiB·hours")).toBeInTheDocument(); + expect(screen.getByText("Bytes scanned")).toBeInTheDocument(); + expect(screen.getByText("7.7 KB total in window")).toBeInTheDocument(); expect(screen.getByText(/3 total/)).toBeInTheDocument(); expect(screen.queryByText("CPU-minutes")).not.toBeInTheDocument(); expect(screen.queryByText("Memory GiB·minutes")).not.toBeInTheDocument(); @@ -97,10 +98,9 @@ describe("OrgUsageSection", () => { expect(screen.getByText(/no usage recorded/i)).toBeInTheDocument(); }); - it("shows the retention caveat when billing has acked inside the window", () => { - hooks.useOrgDailyUsage.mockReturnValue(ok({ ...RESPONSE, watermark_low: "2026-08-12T00:00:00Z" })); + it("retains billed usage without a deletion caveat", () => { renderSection(); - expect(screen.getByText(/billed and removed/i)).toBeInTheDocument(); + expect(screen.queryByText(/billed and removed/i)).not.toBeInTheDocument(); expect(screen.queryByText(/garbage-collected/i)).not.toBeInTheDocument(); }); diff --git a/controlplane/admin/ui/src/pages/OrgUsage.tsx b/controlplane/admin/ui/src/pages/OrgUsage.tsx index d941c27a3..fe5fd86af 100644 --- a/controlplane/admin/ui/src/pages/OrgUsage.tsx +++ b/controlplane/admin/ui/src/pages/OrgUsage.tsx @@ -6,7 +6,7 @@ import { EmptyState, ErrorState, LoadingState } from "@/components/states"; import { InfoTooltip } from "@/components/InfoTooltip"; import { useIdentity } from "@/components/IdentityProvider"; import { useOrgDailyUsage } from "@/hooks/useApi"; -import { fmtTime, fmtUnits } from "@/lib/format"; +import { fmtBytes, fmtUnits } from "@/lib/format"; import { GIB_HOURS_TOOLTIP } from "@/lib/pricing"; import { cn } from "@/lib/utils"; import type { DailyUsageRow } from "@/types/api"; @@ -32,27 +32,28 @@ const PERIODS: { key: PeriodKey; label: string }[] = [ // The API retains an informational team stamp, but storage belongs to the org. // Collapse all stamps into one value per UTC date. -function dailyStorage(rows: DailyUsageRow[]) { +function dailyUsage(rows: DailyUsageRow[], metric: "storage" | "scan") { const byDate = new Map(); for (const r of rows) { - byDate.set(r.date, (byDate.get(r.date) ?? 0) + Number(r.gib_seconds) / 3600); + byDate.set(r.date, (byDate.get(r.date) ?? 0) + (metric === "storage" ? Number(r.gib_seconds) / 3600 : Number(r.bytes_scanned))); } return [...byDate.entries()] - .map(([date, storage]) => ({ date, storage })) + .map(([date, value]) => ({ date, value })) .sort((a, b) => a.date.localeCompare(b.date)); } -function UsageChart({ rows }: { rows: DailyUsageRow[] }) { - const data = useMemo(() => dailyStorage(rows), [rows]); - const total = useMemo(() => data.reduce((sum, row) => sum + row.storage, 0), [data]); +function UsageChart({ rows, metric }: { rows: DailyUsageRow[]; metric: "storage" | "scan" }) { + const data = useMemo(() => dailyUsage(rows, metric), [rows, metric]); + const total = useMemo(() => data.reduce((sum, row) => sum + row.value, 0), [data]); + const format = metric === "storage" ? fmtUnits : fmtBytes; return ( - S3 GiB·hours - + {metric === "storage" ? "S3 GiB·hours" : "Bytes scanned"} + {metric === "storage" && } -

{fmtUnits(total)} total in window

+

{format(total)} total in window

 {data.length === 0 ? ( @@ -67,7 +68,7 @@ function UsageChart({ rows }: { rows: DailyUsageRow[] }) { stroke="hsl(var(--muted-foreground))" fontSize={10} /> - + [`${fmtUnits(v)} GiB·h`, "S3 storage"]} + formatter={(v: number) => metric === "storage" ? [`${fmtUnits(v)} GiB·h`, "S3 storage"] : [fmtBytes(v), "Bytes scanned"]} /> - + )} @@ -86,7 +87,7 @@ function UsageChart({ rows }: { rows: DailyUsageRow[] }) { ); } -// OrgUsageSection renders the org's daily S3 usage over a selectable window. +// OrgUsageSection renders retained daily scan and S3 usage. // Cost data is admin-only — // viewers get nothing at all (the API 403s them anyway; this keeps the page // clean and avoids the wasted request). @@ -105,7 +106,7 @@ export function OrgUsageSection({ orgId }: { orgId: string }) {
Usage

- Daily S3 storage-time (GiB·h) for this organization, summed over the retained billing buffer. + Daily query scan bytes and S3 storage-time (GiB·h). Failed and cancelled queries are included.

@@ -123,20 +124,17 @@ export function OrgUsageSection({ orgId }: { orgId: string }) {
- {usage.data?.watermark_low && ( -

- Usage at or before {fmtTime(usage.data.watermark_low)} has been billed and removed from the buffer, so the - left edge of the selected period may be partial. -

- )} {usage.isError ? ( usage.refetch()} /> ) : usage.isLoading ? ( ) : rows.length === 0 ? ( - + ) : ( - +
+ + +
)} diff --git a/controlplane/admin/ui/src/pages/Usage.test.tsx b/controlplane/admin/ui/src/pages/Usage.test.tsx index 85eac0b53..8a557a6a2 100644 --- a/controlplane/admin/ui/src/pages/Usage.test.tsx +++ b/controlplane/admin/ui/src/pages/Usage.test.tsx @@ -26,13 +26,12 @@ const RESPONSE: MonthlyUsageResponse = { months: 3, aws_region: "us-east-1", customer_pricing_region: "US", - watermark_low: "2026-07-20T00:00:00Z", rows: [ // Two historical team stamps for the same org must become one storage row. - { month: "2026-08", org_id: "acme", team_id: 5, schema_name: "team_5", cpu_seconds: 1, memory_seconds: 1, gib_seconds: gibSeconds(200) }, - { month: "2026-08", org_id: "acme", team_id: 6, schema_name: "team_6", cpu_seconds: 1, memory_seconds: 1, gib_seconds: gibSeconds(400) }, - { month: "2026-08", org_id: "globex", team_id: 9, schema_name: "team_9", cpu_seconds: 1, memory_seconds: 1, gib_seconds: gibSeconds(50) }, - { month: "2026-07", org_id: "acme", team_id: 5, schema_name: "team_5", cpu_seconds: 600, memory_seconds: 600, gib_seconds: 0 }, + { month: "2026-08", org_id: "acme", team_id: 5, schema_name: "team_5", bytes_scanned: 1, gib_seconds: gibSeconds(200) }, + { month: "2026-08", org_id: "acme", team_id: 6, schema_name: "team_6", bytes_scanned: 1, gib_seconds: gibSeconds(400) }, + { month: "2026-08", org_id: "globex", team_id: 9, schema_name: "team_9", bytes_scanned: 1, gib_seconds: gibSeconds(50) }, + { month: "2026-07", org_id: "acme", team_id: 5, schema_name: "team_5", bytes_scanned: 600, gib_seconds: 0 }, ], }; @@ -62,17 +61,18 @@ describe("Usage page", () => { expect(screen.queryByText(/per-team/i)).not.toBeInTheDocument(); }); - it("shows storage only and aggregates historical team rows into one org row", () => { + it("shows scan volume and aggregates historical team rows into one org row", () => { hooks.useMonthlyUsage.mockReturnValue(ok(RESPONSE)); renderPage(); - expect(screen.getByText(/storage-time.*retained billing buffer/i)).toBeInTheDocument(); + expect(screen.getByText(/scan.*storage-time/i)).toBeInTheDocument(); expect(screen.queryByRole("columnheader", { name: "Team" })).not.toBeInTheDocument(); expect(screen.queryByRole("columnheader", { name: /CPU/i })).not.toBeInTheDocument(); expect(screen.queryByRole("columnheader", { name: /Memory/i })).not.toBeInTheDocument(); expect(screen.queryByText("team_5")).not.toBeInTheDocument(); expect(screen.queryByText("team_6")).not.toBeInTheDocument(); + expect(screen.getByTestId("stat-Bytes scanned")).toBeInTheDocument(); expect(screen.getAllByRole("table")).toHaveLength(1); const usageTable = screen.getByRole("table"); expect(within(usageTable).getByRole("columnheader", { name: /allocated aws cost/i })).toBeInTheDocument(); @@ -84,11 +84,11 @@ describe("Usage page", () => { hooks.useMonthlyUsage.mockReturnValue(ok(RESPONSE)); renderPage(); - expect(within(screen.getByTestId("stat-Total cost")).getByText("$14.95")).toBeInTheDocument(); - expect(within(screen.getByTestId("stat-Total price")).getByText("$19.50")).toBeInTheDocument(); - expect(within(screen.getByTestId("stat-Total price")).getByText("US progressive tiers · us-east-1")).toBeInTheDocument(); - expect(within(screen.getByTestId("stat-Total gross margin")).getByText("23.3%")).toBeInTheDocument(); - expect(within(screen.getByTestId("stat-Total gross margin")).getByText("$4.55 gross profit")).toBeInTheDocument(); + expect(within(screen.getByTestId("stat-Storage cost")).getByText("$14.95")).toBeInTheDocument(); + expect(within(screen.getByTestId("stat-Storage price")).getByText("$19.50")).toBeInTheDocument(); + expect(within(screen.getByTestId("stat-Storage price")).getByText("US progressive tiers · us-east-1")).toBeInTheDocument(); + expect(within(screen.getByTestId("stat-Storage gross margin")).getByText("23.3%")).toBeInTheDocument(); + expect(within(screen.getByTestId("stat-Storage gross margin")).getByText("$4.55 gross profit")).toBeInTheDocument(); expect(screen.queryByTestId("stat-S3 GiB·h")).not.toBeInTheDocument(); expect(screen.queryByTestId("stat-CPU-min")).not.toBeInTheDocument(); expect(screen.queryByTestId("stat-Memory GiB·min")).not.toBeInTheDocument(); @@ -100,16 +100,16 @@ describe("Usage page", () => { ); renderPage(); - expect(within(screen.getByTestId("stat-Total price")).getByText("$21.45")).toBeInTheDocument(); - expect(within(screen.getByTestId("stat-Total price")).getByText("EU progressive tiers · eu-central-1")).toBeInTheDocument(); + expect(within(screen.getByTestId("stat-Storage price")).getByText("$21.45")).toBeInTheDocument(); + expect(within(screen.getByTestId("stat-Storage price")).getByText("EU progressive tiers · eu-central-1")).toBeInTheDocument(); expect(screen.getByRole("columnheader", { name: /customer price.*eu/i })).toBeInTheDocument(); expect(screen.queryByRole("button", { name: /us pricing|eu pricing/i })).not.toBeInTheDocument(); }); - it("renders the retention caveat when billing has acked a watermark", () => { + it("retains billed usage without a deletion caveat", () => { hooks.useMonthlyUsage.mockReturnValue(ok(RESPONSE)); renderPage(); - expect(screen.getByText(/billed and removed/i)).toBeInTheDocument(); + expect(screen.queryByText(/billed and removed/i)).not.toBeInTheDocument(); expect(screen.queryByText(/garbage-collected/i)).not.toBeInTheDocument(); }); @@ -120,7 +120,6 @@ describe("Usage page", () => { months: 3, aws_region: "us-east-1", customer_pricing_region: "US", - watermark_low: null, rows: [], }), ); diff --git a/controlplane/admin/ui/src/pages/Usage.tsx b/controlplane/admin/ui/src/pages/Usage.tsx index f9c98dc79..ca78c33a9 100644 --- a/controlplane/admin/ui/src/pages/Usage.tsx +++ b/controlplane/admin/ui/src/pages/Usage.tsx @@ -4,12 +4,11 @@ import { PageBody, PageHeader } from "@/components/AppShell"; import { InfoTooltip } from "@/components/InfoTooltip"; import { StatCard } from "@/components/StatCard"; import { UsagePricing } from "@/pages/UsagePricing"; -import { Card } from "@/components/ui/card"; import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select"; import { EmptyState, ErrorState, TableSkeleton } from "@/components/states"; import { useIdentity } from "@/components/IdentityProvider"; import { useMonthlyUsage, useOrgLabels } from "@/hooks/useApi"; -import { fmtTime } from "@/lib/format"; +import { fmtBytes } from "@/lib/format"; import { AWS_COST_TOOLTIP, GROSS_MARGIN_TOOLTIP, @@ -62,7 +61,7 @@ export function Usage() { if (!isAdmin) { return ( <> - + } @@ -78,7 +77,7 @@ export function Usage() { <>