Skip to content

feat(usage): meter every model call's tokens per tenant, agent and operation, and GET /api/v1/spend - #2070

Closed
erni-a wants to merge 3 commits into
mainfrom
CAURA-000-usage-metering
Closed

feat(usage): meter every model call's tokens per tenant, agent and operation, and GET /api/v1/spend#2070
erni-a wants to merge 3 commits into
mainfrom
CAURA-000-usage-metering

Conversation

@erni-a

@erni-a erni-a commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Usage metering: every model call Caura makes records its tokens per tenant, agent and operation, and the savings Caura can stand behind are recorded next to them.

  • common/usage: a scope (@scoped) and an aggregator; provider hooks read usage from OpenAI, Vertex/Gemini (usage_metadata) and the rest.
  • Migration 066: llm_usage_minutes, value_usage_minutes, usage_flush_batches.
  • Savings recorded: recall's source-vs-returned tokens, cross-agent reuse, crystallizer and near-duplicate retirements. Nothing compared with "the whole corpus", no hours estimates.
  • Storage: POST /usage/flush (idempotent per batch id) and /usage/query (tenant-bound).
  • GET /api/v1/spend (tenant or admin credentials only). Named spend because the enterprise gateway already owns /api/v1/usage for plan counters.
  • observability.usage_metering_enabled gates it; docs/usage-metering.md documents what is and isn't metered.

Type of Change

  • New feature (non-breaking change that adds functionality)

How Has This Been Tested?

Root suite 10,094 passed; core-api + core-storage 1,035 passed; migrations 065 → 066 on a fresh database. Python and TypeScript clients pass.

Additional Notes

The tenant-scope gate will report that its allowlist grew by method:usage_flush and route:POST /api/v1/storage/usage/flush. Both are cross-tenant by design: one aggregator batch carries minute totals for many tenants, and the batch id makes a retried flush a no-op. This needs a maintainer override.

caura-ops' Model Spend page reads these tables once 066 is in production. recall() in recall_service.py was removed upstream (#2042), so it carries no @scoped here.

erni-a added 3 commits October 13, 2026 11:42
@erni-a
feat(usage): record every model call's tokens per tenant, agent and o…
388fb76 
…peration

Every LLM and embedding call the server makes is now counted with the
provider's own token figures (input, output, reasoning) and attributed to
the tenant, agent, fleet and operation it was made for.

- common/usage: a usage scope (ContextVar, plus a @Scoped decorator for
  entry points), record_model_call / record_value, and a per-minute
  in-memory aggregator flushed every 15 s.
- Providers report usage: OpenAI-compatible chat and embeddings (the
  counts were already read, then only logged), Vertex and Gemini
  (usage_metadata was discarded). The retry layer counts failed attempts
  separately, so a billed-then-rejected response is never two calls.
- Attribution at the entry points: writes, search, recall, enrichment,
  entity extraction, contradiction, crystallization, forge, ingest,
  evolve, interview, digests, queue consumers and lifecycle actions;
  tracked background tasks inherit their tenant. Request paths fall back
  to the signed-in tenant and the MCP agent. A call with no tenant is
  recorded as _unattributed rather than dropped.
- Storage (migration 066): llm_usage_minutes and value_usage_minutes as
  additive per-minute upserts, and usage_flush_batches so a re-sent
  flush is counted once. POST /usage/flush (cross-tenant by design, like
  the capability and tenant usage flushes) and POST /usage/query, bound
  to one tenant. Both tables join the tenant and fleet purges.
- observability.usage_metering_enabled (default on) turns it off per
  workspace; storage drops that tenant's rows at flush time, for
  core-api and core-worker alike.

Counts only, never content, kept in the deployment's own database.

Signed-off-by: erni <erni@caura.ai>
@erni-a
feat(usage): record the recall and dedup savings Caura can stand behind
517bbbf 
Only measures with both sides known exactly:

- Recall (RecordRecallValue, the search pipeline's last step, and the
  legacy path so the hotfix lever keeps it): recalls, results, tokens
  returned, results written by another agent, and for results that are a
  piece of a larger memory (auto-chunks and atomic facts, which name
  their parent through the reserved parent_memory_id) the tokens returned
  and the full size of the distinct parents. A whole memory returned as
  itself claims no saving; a deleted parent claims nothing; diagnostic
  searches are not recalls. Runs after the response, one storage read
  for the parent sizes.
- Duplicate merges: crystallization records the memories and tokens it
  retired and the tokens it wrote back, once the sources are archived;
  the write-path near-duplicate merge records the row it retired (the
  agent's own new write is not offset against it).

Not recorded: savings against sending every memory, or anything not
observed. Imported documents are left out: the server keeps no
verifiable size for the source, and a client-supplied one would be
forgeable. Tokens are tiktoken cl100k_base, which core-api already ships.

Signed-off-by: erni <erni@caura.ai>
@erni-a
feat(usage): GET /api/v1/spend, and document what is metered
df9e54e 
GET /api/v1/spend returns one workspace's model calls (provider-reported
tokens in, out and reasoning, failed attempts) and recall savings over a
window, grouped by day, agent, fleet, operation, kind, provider, model or
metric, with a summary: total tokens, context saved (parent tokens less
returned), cross-agent results and duplicates retired.

Spend per agent is an admin-plane report, gated like tenant settings:
tenant and admin credentials read it; agent keys and org members get 403;
a tenant outside the caller's readable set is refused.

docs/usage-metering.md says what is recorded and what is deliberately
not, how it is stored and counted, and how a workspace turns it off.

Signed-off-by: erni <erni@caura.ai>
@erni-a
erni-a requested a review from a team as a code owner October 13, 2026 09:34
@github-actions

github-actions Bot commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Claude Code Review — skipped: PR author 'erni-a' is not a public member of the 'caura-ai' org

@erni-a erni-a mentioned this pull request Oct 13, 2026
Merged
1 task done
@erni-a

erni-a commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Combined into #2071 to save serial CI cycles on main (same commits, now the middle three of #2071).

@erni-a erni-a closed this Oct 13, 2026
erni-a added a commit that referenced this pull request Oct 13, 2026
@erni-a
feat: memory map, usage metering and value metering — GET /memories/f…
da9fc57 
…ield, /spend and /value (#2071)

## Summary

The memory map and metering for the enterprise Prism › Map, Home and
Insights › Value, in one PR to save serial CI cycles (it replaces #2069
and #2070). Eleven commits in three parts, each readable on its own:

### 1. `GET /api/v1/memories/field` — the memory map (2 commits)

- Two-level PCA over the memory embeddings, k-means clusters labelled
from the entities their memories mention, percentile normalisation so
outlying clusters don't squeeze the rest into a corner.
- core-storage gains `field-page` / `field-signature` reads; the layout
is cached in-process for 10 minutes, keyed by that signature.
- `docs/api-reference.md` documents the route.

### 2. Usage metering — migration 066, `GET /api/v1/spend` (3 commits)

- `common/usage`: a scope (`@scoped`) and an aggregator; provider hooks
read usage from OpenAI, Vertex/Gemini (`usage_metadata`) and the rest.
- Migration **066**: `llm_usage_minutes`, `value_usage_minutes`,
`usage_flush_batches`.
- Savings recorded: recall's source-vs-returned tokens, cross-agent
reuse, crystallizer and near-duplicate retirements. Nothing compared
with "the whole corpus", no hours estimates.
- Storage: `POST /usage/flush?tenant_id=` and `POST
/usage/query?tenant_id=`, both bound to one tenant. A writer sends one
flush per tenant (`common.usage.per_tenant`); each tenant's batch id is
a `uuid5` of the batch's, so a re-sent batch, even one that half landed,
is counted once.
- `GET /api/v1/spend` (tenant or admin credentials only). Named `spend`
because the enterprise gateway already owns `/api/v1/usage` for plan
counters.
- `observability.usage_metering_enabled` gates it;
`docs/usage-metering.md` documents what is and isn't metered.

### 3. Value metering — migration 067, `GET /api/v1/value` (6 commits)

- Sessions: MCP issues an `Mcp-Session-Id` label, REST takes
`X-Caura-Session-Id`, and a broker's `metadata.session_id` wins.
- Migration **067**: `agent_sessions`, `memory_origins`, `memory_reuse`
(tenant-bound routes, no allowlist growth).
- Origin of a memory: minutes since the session started or last wrote,
and calls (a lower bound), plus tokens where a hook, the OpenClaw plugin
or the SDKs report them (`MemoryCreate.origin`).
- Reuse is counted once per (session, memory), never in the memory's own
session. "Sessions helped" counts only reuse of memories with a recorded
origin; "mistakes prevented" counts resolved contradictions and retired
outdated memories, with pending ones shown as waiting.
- Evolve outcomes confirm reuse (`memory_reuse.outcome`).
- `GET /api/v1/value`, read by the enterprise Insights › Value page and
Home.
- The plugin sends `X-Caura-Session-Id` and per-turn origin; the Python
and TypeScript SDKs take `session_id` / `origin`.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

## How Has This Been Tested?

All of CI's static checks pass locally on this branch, the tenant-scope
gate included (lock files, broker OpenAPI, every ruff and format scope,
the MCP guards, mypy ×6, the legacy-name ratchet, do-not-touch, the
evidence registry). Each part's tests passed on its own branch: root
suite 10,087 / 10,094 / 10,110, core-api + core-storage 1,035 / 1,044,
migrations 065 → 066 → 067 on a fresh database, plugin 574/576 (the two
resolveAgentId fallback warn tests fail the same way on main on the test
machine), Python client 229, TypeScript client 68. The three parts
combined without a conflict.

## Additional Notes

**No tenant-scope allowlist growth.** Every new storage path is bound to
one tenant: the flush takes a required `tenant_id` and writes every row
under it, so a flush can't reach another tenant's totals
(`test_a_flush_writes_only_the_tenant_it_names`).

- The memory field is computed per process with no table yet. Before
broad rollout it needs a bound on concurrent builds and persisted
coordinates (prod core-api is 4 CPU / 8 GiB, concurrency 40). The
enterprise map is behind `features.prism_field`, off by default.
- Named `/spend` because the enterprise gateway already owns
`/api/v1/usage`. caura-ops' Model Spend page reads these tables once 066
is in production.
- Fixed while rebasing: `plugin/src/origin-usage.ts` wasn't in the
plugin install lists (`_plugin_files`, the install script's `SRC_FILES`,
`FALLBACK_SRC_FILES`), so fresh installs would fail with TS2307;
`test_plugin_source_manifest.py` covers it.
- Deferred: tokens from the Claude Code / Codex broker hook need
`origin` on `BulkMemoryItem`, part of the frozen broker contract.

---------

Signed-off-by: erni <erni@caura.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@erni-a