Skip to content

feat(observability): durable tenant CostStore with Postgres backend - #1835

Merged
EmersonBraun merged 3 commits into
mainfrom
EmersonBraun/chat-t04
Oct 10, 2026
Merged

EmersonBraun merged 3 commits into
mainfrom
EmersonBraun/chat-t04

Conversation

@EmersonBraun

@EmersonBraun EmersonBraun commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Track 04, stage 4. Covers RF-25, RF-26, RF-28 and the ledger of RF-29 in @agentskit/observability.

Not in this PR, tracked elsewhere:

  • RF-27 (cap read from entitlements, 402 QUOTA_EXCEEDED, warning at 80%) is application behavior and lives in the chat handler: feat(chat): action decisions across requests and turn cost policy agentskit-chat#205. Here the cap is an argument of reserve.
  • RF-29, divergence of at most 5% against the provider invoice: not measured. The ledger is written and summed against the window in the contract; comparing it with a real invoice needs a provider account and a test batch.
  • RF-26 on Neon: the PRD gate asks for Neon. Everything below ran on local PostgreSQL 16.

What changes

  • CostStore port: reserve, commit, release, window, all tenant-scoped and idempotent per reservationId. reserve checks an estimate against a cap passed by the caller; commit writes real spend and usage ledger rows; release returns the hold. reserve with the id of a released reservation throws AK_COST_RESERVATION_RELEASED.
  • createInMemoryCostStore for development and tests.
  • @agentskit/observability/postgres: postgresCostStore({ db }) on BYO Drizzle and node-postgres, with exported tables and idempotent DDL. Each operation is one statement; admission is the conditional update spent + reserved + amount <= cap. expirePostgresCostReservations({ db, olderThan }) releases reservations that were never settled.
  • @agentskit/observability/cost-store-contract: costStoreContract, a framework-free suite for custom stores. It matches errors by code.
  • Guards: costGuard, multiTenantCostGuard and createAdvancedCostGuard accept an optional store. With it, each priced call is written to the store and the budget is checked against the stored window total. Without it, behavior is unchanged except for a one-time warning in production runtimes.
  • ADR 0044, README and agent docs, minor changeset, and three steps in the PostgreSQL CI job (test:postgres, test:cjs, test:workerd).

Decisions a reviewer should check

  • The cap is an argument of reserve, not stored state. The host reads it from its plan data.
  • commit records real spend even above the reservation. The cap is a hard ceiling on reservations only.
  • Guards record after the fact. They bound overshoot to in-flight calls; a host that must refuse a request before it spends calls reserve itself.
  • With a store, a guard budget applies to the store's accounting window (UTC month by default). The advanced guard's rolling window caps stay process-local.
  • No reaper. A reservation abandoned by a crashed process stays held until the host runs expirePostgresCostReservations. A cutoff shorter than a call in flight releases its hold, and that call's commit is then rejected.
  • reserve and host transactions. A concurrent duplicate id ends in a unique violation that the store absorbs. Inside a host transaction that still aborts the transaction, so reserve belongs outside it or in a savepoint. Documented, not changed: admitting the hold and writing the reservation row in one statement is what makes the cap atomic.
  • Missing-store warning. Fires when NODE_ENV=production, and on Cloudflare Workers when NODE_ENV is not set. warnWithoutStore: true | false overrides it.
  • Store failures. They go to onError. A guard without onError logs the first failure once.
  • Size budgets raised: @agentskit/observability (ESM) 16.5 → 17 KB, ./cost-guard 2.5 → 3 KB, ./postgres 3 → 3.5 KB.
  • drizzle-orm and pg are new optional peers, used only by /postgres.

Review follow-up (second commit set)

Item Change
M1 instanceof across CommonJS entries The contract compares error.code. tests/cjs-smoke.cjs runs the contract through require against the in-memory and the PostgreSQL store; with the old check it fails on "commit after release is rejected".
M2 leak in recordCostSpend The hold is released when the commit fails. expirePostgresCostReservations covers holds no process can release.
M3 silent store failure One console.warn per guard when onError is absent.
M4 workerd not in CI, single client test:workerd and test:cjs added to the PostgreSQL job. The worker fixture uses a pool of 5, so the 50-reservation race runs on several connections.
M5 warning in workerd Runtime detection plus warnWithoutStore.
M6 this description Rewritten; RF-27 and the RF-29 invoice check are called out as outside this PR.
Lower items Ledger token columns are bigint. release and window validate tenant and id in the PostgreSQL store. Replay of reserve after release is rejected. The Drizzle schema carries the status CHECK and the ledger index. The integration test loads the built package only when it runs. The 23505-in-transaction case is documented.

Local validation

Run on macOS, Node 26.8.1, pnpm 10.34.4, with the real pre-push hook (no HUSKY=0). Outputs are in the comments.

Check Result
sh .husky/pre-push (52 quality gates, lint, build) exit 0
pnpm test (whole repository) 77/77 turbo tasks (run before the review follow-up)
pnpm --filter @agentskit/observability test:coverage 387 tests passed; lines 94.58%
Contract on PostgreSQL 16.15, Node pool passed, including ledger totals, replay race and expiry
Contract through CommonJS (test:cjs) in-memory and PostgreSQL passed
Contract in workerd, pool of 5 (test:workerd) passed
Contract mutation tests a racy, a double-counting, a leaking, a tenant-sharing, a silent and a re-admitting store each fail the contract
size-limit within budget

Not validated

Neon and Hyperdrive, database row-level security, throughput, and the provider-invoice comparison of RF-29. The two new CI steps have not run on GitHub yet; the first CI run of this push is their first.

No release or publication from this PR.

Add the CostStore port (reserve, commit, release, window) with an in-memory
store, an optional Drizzle/Postgres subpath with atomic cap admission and a
usage ledger, and an exported contract suite. The three cost guards accept
an optional store and warn in production when spend is process-local.
@github-actions

Copy link
Copy Markdown
Contributor

Lighthouse

URL Performance Accessibility Best practices SEO
http://127.0.0.1:3000/ 67 100 96 100
http://127.0.0.1:3000/docs 84 100 96 100
http://127.0.0.1:3000/docs/get-started/getting-started/quickstart 83 100 96 100
http://127.0.0.1:3000/stack 96 100 96 100
http://127.0.0.1:3000/showcase 96 100 96 100

Run by .github/workflows/lighthouse.yml.

@EmersonBraun

Copy link
Copy Markdown
Member Author

Local gates, re-run with the real hooks (no HUSKY=0)

The first push of this branch bypassed the pre-push hook. Everything below was run again on this machine at 88722d4d (macOS, Node v26.8.1, pnpm 10.34.4), and the follow-up commit was pushed through the hook.

Gate Command Result
Pre-push hook, standalone sh .husky/pre-push (doc-bridge index, check:quality-gates, lint, build) exit 0, 12m57s
Pre-push hook, on the real push git push origin EmersonBraun/chat-t04 exit 0, 7m10s, 3f2e242c..88722d4d
Quality gates pnpm check:quality-gates ✓ all 52 quality gates passed
Lint pnpm lint 60/60 turbo tasks
Build pnpm build 44/44 turbo tasks
Repository tests pnpm test 77/77 turbo tasks, exit 0
Package tests + coverage pnpm --filter @agentskit/observability test:coverage 373 passed, 1 skipped (the Postgres test, run separately below); lines 94.44%
CostStore contract, PostgreSQL 16.15 AK_POSTGRES_TEST_PORT=… pnpm --filter @agentskit/observability test:postgres 1/1, then 10 more consecutive runs, 10/10
CostStore contract, workerd pnpm --filter @agentskit/observability test:workerd all 8 criteria passed over wrangler dev --local against the same PostgreSQL 16
Size budget size-core-closure, gen-size-limit-subpaths --check/--build, size-limit exit 0, 110 entries within budget

One thing the hook run turned up: the docs build regenerates apps/docs-next/lib/api-symbol-routes.json with the 17 new public symbols of this PR, and the file was not committed. 88722d4d adds it. No other file changed after the full run.

Quality gates (tail)

▶ Math.random identifier rule tests

 RUN  v4.1.11 /Users/rebecabraun/orca/workspaces/agentskit/chat-t04


 Test Files  1 passed (1)
      Tests  5 passed (5)
   Start at  22:21:39
   Duration  161ms (transform 34ms, setup 0ms, import 49ms, tests 3ms, environment 0ms)

  Math.random identifier rule tests: 0.7s

────────────────────────────────────────────────────────
✓ all 52 quality gates passed
Lint and build summaries
 Tasks:    60 successful, 60 total
Cached:    15 cached, 60 total
  Time:    43.961s 
 Tasks:    44 successful, 44 total
Cached:    27 cached, 44 total
  Time:    3m25.511s 
sh .husky/pre-push  737.87s user 249.96s system 127% cpu 12:57.42 total
EXIT=0
pnpm test (tail)
 Tasks:    77 successful, 77 total
Cached:    43 cached, 77 total
  Time:    2m54.149s 
pnpm test  274.49s user 128.23s system 229% cpu 2:55.40 total
EXIT=0
Observability tests and coverage
 Test Files  29 passed | 1 skipped (30)
      Tests  373 passed | 1 skipped (374)
   Start at  22:26:08
   Duration  3.16s (transform 1.93s, setup 0ms, import 3.78s, tests 3.21s, environment 2ms)

 % Coverage report from v8
-------------------|---------|----------|---------|---------|-------------------
File               | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-------------------|---------|----------|---------|---------|-------------------
All files          |   93.28 |    86.84 |   94.71 |   94.44 |                   
 audit-log.ts      |   97.67 |      100 |   94.44 |     100 |                   
CostStore contract on PostgreSQL 16
16.15 (Debian 16.15-1.pgdg13+2)
 Test Files  1 passed (1)
      Tests  1 passed (1)
   Start at  22:26:25
   Duration  1.62s (transform 70ms, setup 0ms, import 389ms, tests 1.13s, environment 0ms)
EXIT=0
CostStore contract on workerd
workerd + PostgreSQL: CostStore contract, ledger and replay race passed
EXIT=0
Size budget, observability entries
  @agentskit/observability (ESM)
  Size limit: 17 kB
  Size:       16.57 kB gzipped
  @agentskit/observability (ESM export .)
  Size limit: 22.5 kB
  Size:       21.87 kB gzipped
  @agentskit/observability (ESM export ./trace-tracker)
  Size limit: 2 kB
  Size:       1.4 kB gzipped
  @agentskit/observability (ESM export ./cost-guard)
  Size limit: 2.5 kB
  Size:       2.37 kB gzipped
  @agentskit/observability (ESM export ./langfuse)
  Size limit: 3 kB
  Size:       2.45 kB gzipped
  @agentskit/observability (ESM export ./postgres)
  Size limit: 3 kB
  Size:       2.89 kB gzipped
  @agentskit/observability (ESM export ./cost-store-contract)
  Size limit: 2.5 kB
  Size:       2.08 kB gzipped

Turbo served part of pnpm test (43 of 77 tasks) and of the hook's lint/build from its local cache, for inputs unchanged since an earlier run on this machine; the observability tasks and both contract runs executed fresh.

Not covered here: Neon and Hyperdrive (the PRD gate for RF-26 asks for Neon; this run is local PostgreSQL 16 only).

@EmersonBraun
EmersonBraun marked this pull request as ready for review October 10, 2026 01:38
@github-actions

Copy link
Copy Markdown
Contributor

Lighthouse

URL Performance Accessibility Best practices SEO
http://127.0.0.1:3000/ 62 100 96 100
http://127.0.0.1:3000/docs 82 100 96 100
http://127.0.0.1:3000/docs/get-started/getting-started/quickstart 82 100 96 100
http://127.0.0.1:3000/stack 95 100 96 100
http://127.0.0.1:3000/showcase 94 100 96 100

Run by .github/workflows/lighthouse.yml.

- match reservation errors by code in the contract suite; each CommonJS
  subpath bundles its own copy of the class. Add a CommonJS smoke test.
- recordCostSpend releases its hold when the commit fails
- expirePostgresCostReservations returns holds that were never settled
- a guard without onError warns once when the store fails
- warnWithoutStore, and Cloudflare Workers detection for the missing-store warning
- reserve rejects a released reservation id instead of replaying it as admitted
- validate release and window inputs in the PostgreSQL store
- ledger token columns are bigint; Drizzle schema carries the CHECK and index
- run the contract in workerd on a pool, and in CI with the CommonJS smoke
@EmersonBraun

Copy link
Copy Markdown
Member Author

Review follow-up at fdb1e758: local gates

Pushed through the pre-push hook again (no HUSKY=0).

Gate Result
sh .husky/pre-push standalone exit 0, 15m01s: ✓ all 52 quality gates passed, lint 60/60, build 44/44
Same hook on git push exit 0, 7m58s, 88722d4d..fdb1e758
pnpm --filter @agentskit/observability test:coverage 387 passed, lines 94.58%
test:postgres (PostgreSQL 16.15 (Debian 16.15-1.pgdg13+2)) 1/1, nine criteria incl. ledger, replay race, expiry
test:cjs in-memory and PostgreSQL stores pass the contract through require
test:workerd pool of 5 connections, nine criteria
size-limit exit 0 after raising ./cost-guard to 3 KB and ./postgres to 3.5 KB

The first hook run on these changes failed three gates, all expected for new exports and an edited README: public API JSDoc coverage (snapshot needed the two new /postgres symbols), README Standard freshness, and its test. I updated the snapshot and refreshed only the changed source hash (--preserve-review-dates), then re-ran.

To check that the CommonJS smoke test catches M1, I restored the instanceof comparison, rebuilt, and ran it: it fails with CostStore contract: commit after release is rejected. With the fix it passes.

The repository-wide pnpm test (77/77) was run on 88722d4d, before these changes; after them I ran the observability suite and the three contract runs, not the whole repository again. The two new CI steps (test:cjs, test:workerd) run on GitHub for the first time with this push.

Outputs
 Test Files  30 passed (30)
      Tests  387 passed (387)
All files          |   93.42 |     86.9 |   95.07 |   94.58 |                   
### EXIT 0 (pnpm --filter @agentskit/observability test:coverage)
 Test Files  1 passed (1)
      Tests  1 passed (1)
### EXIT 0 (pnpm --filter @agentskit/observability test:postgres)
CommonJS: in-memory CostStore passed the contract
CommonJS: PostgreSQL CostStore passed the contract
### EXIT 0 (pnpm --filter @agentskit/observability test:cjs)
workerd + PostgreSQL (pool of 5): CostStore contract, ledger, replay race and expiry passed
### EXIT 0 (pnpm --filter @agentskit/observability test:workerd)
✓ all 52 quality gates passed
sh .husky/pre-push  611.20s user 223.81s system 92% cpu 15:01.83 total
EXIT=0
   88722d4d..fdb1e758  EmersonBraun/chat-t04 -> EmersonBraun/chat-t04
git push origin EmersonBraun/chat-t04  341.83s user 89.10s system 90% cpu 7:58.22 total
EXIT=0

@github-actions

Copy link
Copy Markdown
Contributor

Lighthouse

URL Performance Accessibility Best practices SEO
http://127.0.0.1:3000/ 64 100 96 100
http://127.0.0.1:3000/docs 82 100 96 100
http://127.0.0.1:3000/docs/get-started/getting-started/quickstart 82 100 96 100
http://127.0.0.1:3000/stack 95 100 96 100
http://127.0.0.1:3000/showcase 94 100 96 100

Run by .github/workflows/lighthouse.yml.

@EmersonBraun
EmersonBraun merged commit ccbb1e6 into main Oct 10, 2026
34 checks passed
@EmersonBraun
EmersonBraun deleted the EmersonBraun/chat-t04 branch October 10, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant