Skip to content

Replace DuckDB compute billing with Trino scan usage - #1159

Draft
benben wants to merge 1 commit into
mainfrom
feat/trino-scan-billing
Draft

Replace DuckDB compute billing with Trino scan usage#1159
benben wants to merge 1 commit into
mainfrom
feat/trino-scan-billing

Conversation

@benben

@benben benben commented Sep 7, 2026

Copy link
Copy Markdown
Member

Trino executes tenant queries outside Duckgres's connection path, so the old DuckDB connection CPU/memory meter no longer describes query usage. This replaces it with native completed-query physicalInputBytes and preserves the existing storage sampler and GiB-second units.

  • Authenticate native HTTP completion events with a dedicated per-cell token; commit before acknowledging and deduplicate by cell/query ID. Include successful, failed and cancelled queries at their reported byte count.
  • Retain tenant attribution, normalized query usage and immutable billing batches in PostgreSQL. Unknown identities wait for resolution. Batch creation safely claims committed queries and concurrent storage increments; acknowledgement deletes nothing.
  • Replace watermark endpoints with next/ack/replay batch APIs. Assign the billing month when creating a batch; retain original completion/sample dates. Downstream billing must persist charges idempotently by batch ID before acknowledging.
  • Replace admin CPU/memory reporting with scan volume and unchanged storage economics. Remove the old compute meter and retention GC. Document rollout, recovery and accepted HTTP listener loss boundaries.

Validation: new APIs exercised red/green; full PostgreSQL integration suite and Kubernetes control-plane suite pass; shared control-plane/configstore/provisioning packages pass; all UI tests (186), typecheck and production build pass. Real storage and Trino-listener E2E assertions are included but await the cloud CI lane. The full process-controlplane recipe passed its process/handover tests and could not run its DuckLake infrastructure test because Docker is unavailable locally. just lint was run: remaining staticcheck failures reproduce on untouched source; two additional govet findings are in an installed, untracked UI dependency. No remaining findings point to this change.

Requires the DuckLake delete-read-statistics fix (PostHog/trino#21) for delete-file reads to enter native bytes. Deploy companion chart RBAC before this CP version, then enable the listener and switch the billing consumer. Pause the old consumer and remove old CP replicas before the new billing flow; old destructive ack/GC code must not touch retained storage. The downstream billing consumer was not found in the workspace or GitHub code search and is not modified here.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area Added Changed Deleted
Test files 4 12 2
E2E/journey files 0 0 0
Workflow files 0 0 0

Signals

  • Test cases: +20 / -22
  • Assertions: +133 / -105
  • Skips or known failures added: 0
  • Workflow continue-on-error added: 0
  • Workflow path filters added: 0
  • Test commands removed from justfile: 0
  • E2E/journey retry lines added: 0

Coverage risk: likely reduced

Warnings

  • Test files deleted (likely reduced)
    • controlplane/compute_billing_api_test.go
    • controlplane/compute_meter_test.go
  • Test cases removed (likely reduced)
    • 22 removed vs 20 added

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant