Repository navigation
feat(aio): honest per-tool spend attribution and cost-by-context-size - #73367
pauldambra wants to merge 12 commits into
Conversation
Fixes double-counting in by_tool (each generation's full cost was attributed to every tool it called, summing to ~122% of real spend) by adding cost_attributed_usd / share_attributed, which split cost evenly across the distinct tools a generation called and reconcile to scoped spend. Adds by_input_size, bucketing spend by $ai_input_tokens, since context size -- not tool choice -- is the primary cost driver. Removes the unused top_traces field. Generated-By: PostHog Code Task-Id: 68fa35a8-56d2-4be3-a0c9-d870e966acd2
…end-honest-attribution # Conflicts: # services/mcp/src/generated/ai_observability/api.ts
Collapse the three repeated `(value / scoped) if scoped > 0 else 0.0` share computations (by_tool.share_of_scoped, by_tool.share_attributed, by_input_size.share_of_scoped) into one guarded helper. Generated-By: PostHog Code Task-Id: 68fa35a8-56d2-4be3-a0c9-d870e966acd2
pauldambra
left a comment
pauldambra
left a comment
There was a problem hiding this comment.
QA Swarm review complete. See inline comment.
|
Note 🤖 Automated comment by QA Swarm — not written by a human Multi-perspective review: router (cheap-first pass) + delegated reviewers as warranted Verdict: 💬 APPROVE WITH NITS (round 1 @ a00ad9f)Additive, read-only analytics change. The router hand-traced the fractional cost-attribution SQL and the input-size bucket boundaries and found them correct; one low-severity NULL-handling nit in the bucket Key findings
ConvergenceNone — single-reviewer (router) pass. Reviewer summaries
Automated by QA Swarm — not a human review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a00ad9f473
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Merged by `tool` rather than requeried per-row: the attribution query isn't | ||
| # itself truncated, so a tool absent from the (possibly truncated) row set above | ||
| # just has no row to attach `cost_attributed_usd` to. | ||
| attribution = _fetch_tool_cost_attribution(team, email, from_dt, to_dt, product) |
There was a problem hiding this comment.
The new attribution query here and input-size query in _compute_spend_analysis change the SELECT sequence captured by @snapshot_clickhouse_queries on test_empty_result_when_no_events, but test_personal_spend.ambr still contains only the previous five queries. The actual third query is now the attribution query while snapshot .2 expects the by-product query, so this test deterministically fails until the ClickHouse snapshots are regenerated.
Useful? React with 👍 / 👎.
| # Merged by `tool` rather than requeried per-row: the attribution query isn't | ||
| # itself truncated, so a tool absent from the (possibly truncated) row set above | ||
| # just has no row to attach `cost_attributed_usd` to. | ||
| attribution = _fetch_tool_cost_attribution(team, email, from_dt, to_dt, product) | ||
| for row in truncated["items"]: | ||
| row["cost_attributed_usd"] = attribution.get(row["tool"], 0.0) |
There was a problem hiding this comment.
The rows are ordered and truncated by the overcounting cost_usd before attributed costs are attached. For example, with limit=1, a $100 generation calling two tools ranks either tool above a separate $90 single-tool generation, even though their attributed costs are $50 and $90 respectively. Thus the tool with the largest honest attribution can be omitted from the new attributed breakdown; compute or join attribution before applying the requested limit and rank by the advertised metric.
Useful? React with 👍 / 👎.
# Conflicts: # services/mcp/src/generated/ai_observability/api.ts
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
🤖 CI report
|
| Root | Eager (shipped) | Δ vs base | Budget |
|---|---|---|---|
entry (logged-out pages, app bootstrap)src/index.tsx |
1.36 MiB · 22 files | no change | ███░░░░░░░ 30.3% of 4.51 MiB |
authenticated shell (every logged-in page)src/scenes/AuthenticatedShell.tsx |
8.79 MiB · 3,239 files | no change | █████████░ 90.5% of 9.71 MiB |
🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
Largest files eagerly shipped from src/index.tsx
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 306.9 KiB | ../node_modules/.pnpm/posthog-js@1.420.0_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 260.9 KiB | ../node_modules/.pnpm/posthog-js@1.420.0_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/module.js |
| 250.4 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 154.2 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 104.7 KiB | src/lib/api.ts |
| 95.2 KiB | ../packages/quill/packages/quill/dist/index.js |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Toolbar bundle — eager 2.25 MiB within budget
What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.
| Metric | Size | Δ vs base | Budget |
|---|---|---|---|
| Eager (shipped) entry + static imports |
2.25 MiB · 17 files | no change | ████░░░░░░ 39.4% of 5.72 MiB |
| Deferred (lazy) | 2.10 MiB · 33 files | no change | n/a — loads on demand |
Loader dist/toolbar.js |
1.1 KiB | no change | █░░░░░░░░░ 5.8% of 19.5 KiB |
Largest eagerly-shipped chunks
| Size | File |
|---|---|
| 746.7 KiB | dist/toolbar/toolbar-app-RBPKVAJT.css |
| 585.2 KiB | dist/toolbar/chunk-chunk-PKTQCPHM.js |
| 484.6 KiB | dist/toolbar/chunk-chunk-PQM5GZBZ.js |
| 133.8 KiB | dist/toolbar/chunk-chunk-BOEVRRG3.js |
| 131.8 KiB | dist/toolbar/chunk-chunk-T5KY5WYR.js |
| 71.3 KiB | dist/toolbar/toolbar-app-746XUZSN.js |
| 69.0 KiB | dist/toolbar/chunk-chunk-27JL52RE.js |
| 35.6 KiB | dist/toolbar/chunk-chunk-E3UZ2534.js |
| 20.9 KiB | dist/toolbar/chunk-chunk-7PWPBHHX.js |
| 12.2 KiB | dist/toolbar/chunk-chunk-PIK3PADE.js |
Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile
✅ Dist folder size — 🔺 +2.6 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1443.04 MiB · 🔺 +2.6 KiB (+0.0%)
ℹ️ MCP UI apps size — 33 app(s), 17662.6 KB JS
Built size of each MCP UI app (main.js + styles.css).
| App | JS | CSS |
|---|---|---|
| debug | 600.0 KB | 195.2 KB |
| action | 458.2 KB | 195.2 KB |
| action-list | 564.9 KB | 195.2 KB |
| cohort | 457.2 KB | 195.2 KB |
| cohort-list | 563.8 KB | 195.2 KB |
| email-template | 457.0 KB | 195.2 KB |
| error-details | 472.9 KB | 195.2 KB |
| error-issue | 457.9 KB | 195.2 KB |
| error-issue-list | 564.7 KB | 195.2 KB |
| experiment | 562.0 KB | 195.2 KB |
| experiment-list | 565.6 KB | 195.2 KB |
| experiment-results | 567.1 KB | 195.2 KB |
| feature-flag | 567.6 KB | 195.2 KB |
| feature-flag-list | 571.4 KB | 195.2 KB |
| feature-flag-testing | 461.4 KB | 195.2 KB |
| inline-scan | 457.7 KB | 195.2 KB |
| insight-actors | 563.0 KB | 195.2 KB |
| invite-email-preview | 456.4 KB | 195.2 KB |
| llm-costs | 560.0 KB | 195.2 KB |
| session-recording | 459.0 KB | 195.2 KB |
| survey | 458.8 KB | 195.2 KB |
| survey-global-stats | 562.7 KB | 195.2 KB |
| survey-list | 565.5 KB | 195.2 KB |
| survey-stats | 562.7 KB | 195.2 KB |
| trace-span | 457.6 KB | 195.2 KB |
| trace-span-list | 564.7 KB | 195.2 KB |
| vision-observation-list | 563.9 KB | 195.2 KB |
| workflow | 457.5 KB | 195.2 KB |
| workflow-list | 564.2 KB | 195.2 KB |
| loops-review | 461.9 KB | 195.2 KB |
| query-results | 755.0 KB | 195.2 KB |
| render-ui | 838.1 KB | 195.2 KB |
| visual-review-snapshots | 462.0 KB | 195.2 KB |
⚠️ Backend snapshots — 1 updated (1 modified, 0 added, 0 deleted)
Query snapshots: Backend query snapshots updated
Changes: 1 snapshots (1 modified, 0 added, 0 deleted)
What this means:
- Query snapshots have been automatically updated to match current output
- These changes reflect modifications to database queries or schema
Next steps:
- Review the query changes to ensure they're intentional
- If unexpected, investigate what caused the query to change
⚠️ MCP snapshots — 1 updated (1 modified, 0 added, 0 deleted)
Snapshots: MCP unit test snapshots updated
Changes: 1 snapshots (1 modified, 0 added, 0 deleted)
What this means:
- Snapshots have been automatically updated to match current output
Next steps:
- Review the changes to ensure they're intentional
- If unexpected, investigate what caused the output to change
- coalesce NULL $ai_input_tokens to 0 so unknown-size generations land in the lowest bucket instead of inflating the >300k headline fallback - version the response cache namespace (v2) so a rolling deploy can't serve an old-shape payload against the new contract - drop the production-derived spend figure from a public code comment Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pass the internally-built input-size bucket expression as a parsed placeholder instead of interpolating it into the parse_select f-string, clearing the hogql-fstring-audit semgrep rule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Note 🤖 stamphog reviewed Codex's unresolved P2 comment ("Rank attributed spend before truncating tool rows") is confirmed still present in the diff: _fetch_by_tool truncates rows by the overcounted cost_usd before merging cost_attributed_usd, so a tool with the largest honest attributed cost can be dropped from a limited response — undermining the PR's own stated goal of honest attribution. This is a genuine unaddressed defect from an independent reviewer, not process noise.
Gate mechanics and policy version
Updated in place — this replaces 8 earlier stamphog review(s) on this PR. |
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
carlos-marchal-ph
left a comment
carlos-marchal-ph
left a comment
There was a problem hiding this comment.
@pauldambra sorry it took us so long to review this, the PR got kinda lost on the interwebs 🙏 Approving to unblock you, but I think some of the comments I added should be addressed.
| collapses those repeats before dividing, since a tool called twice in one | ||
| generation still only "caused" that generation's cost once. Generations with no | ||
| tools attribute their full cost to the same NULL bucket the unattributed | ||
| `_fetch_by_tool` rows use. Not truncated by `limit` -- callers merge by `tool` |
There was a problem hiding this comment.
This is the bit that isn't true. The query below picks up HogQL's default LIMIT 100, so it is truncated, just not by anything you passed it.
| AND {timestamp_filter} | ||
| ) | ||
| ) | ||
| GROUP BY tool |
There was a problem hiding this comment.
There's no LIMIT or ORDER BY here, so execute_hogql_query applies its default of 100 rows (DEFAULT_RETURNED_ROWS in posthog/hogql/constants.py). The new snapshot shows it landing as GROUP BY tool LIMIT 100. Any tool past the cap then falls through to the 0.0 default on line 784 and renders as cost_attributed_usd: 0.00, share_attributed: 0.0, which is exactly the number this PR is asking clients to trust in place of cost_usd. With no ORDER BY the surviving 100 are hash order, so which tools lose their attribution also shifts between refreshes.
I checked and there are already users well past 100 distinct tools in a 30 day window, so this is live rather than theoretical, and it gets worse as people add MCP servers.
Could the attribution be another aggregate on the existing _fetch_by_tool query rather than a second query? Wrap tools in arrayDistinct in an inner subquery, then select both sum(cost) and sum(cost / length(tools)) off the single arrayJoin, keeping the existing ORDER BY cost_usd DESC LIMIT {limit + 1}. That removes the row set mismatch entirely, saves a second scan of the same events on every uncached request, and fixes the dedupe asymmetry below in the same stroke.
| cost / length(if(empty(tools), [''], tools)) AS cost_share | ||
| FROM ( | ||
| SELECT | ||
| arrayDistinct(arrayFilter(x -> x != '', splitByChar(',', coalesce(properties.$ai_tools_called, '')))) AS tools, |
There was a problem hiding this comment.
This query dedupes, but _fetch_by_tool's on line 743 is still a plain splitByChar with no arrayDistinct. extractToolCallNames records names in call order and never dedupes, so "Bash,Read,Bash" is an ordinary value, as your docstring above says. That means one generation is counted twice on the Bash row: generation_count goes up by two and the full cost is added twice.
So generation_count's help text ("Number of $ai_generation events whose tool list includes this tool") isn't what the query returns, and the rewritten cost_usd text puts the whole overstatement down to multi-tool co-occurrence when a real chunk of it is a single generation charged repeatedly to the same tool. The tools that loop most within a turn come out worst, which is unfortunate given cost_usd is still the most prominent number on the row. Folding the two queries as suggested above fixes this too.
| "`summary.scoped_cost_usd`. Prefer `share_of_scoped` for headline percentages — it's computed " | ||
| "per row and doesn't require the totals to reconcile." | ||
| "`summary.scoped_cost_usd` and rows can't be added together. It measures co-occurrence, not " | ||
| "cost caused by the tool. Use `cost_attributed_usd` for a number that reconciles." |
There was a problem hiding this comment.
While you're rewriting these, scoped_cost_usd on line 418 still says it "matches the cost summed across by_tool / by_model", which is the claim this PR exists to retract. These strings ship into the MCP tool schema, so an agent reads both and has no way to reconcile them. Worth narrowing it to by_model and by_input_size, which do reconcile, and pointing at cost_attributed_usd for tools.
| # contract (this revision added `by_input_size`, `cost_attributed_usd`, `share_attributed` | ||
| # and removed `top_traces`). |
There was a problem hiding this comment.
This parenthetical records what this particular revision changed, so it's stale the next time anyone bumps the version. The two sentences above already carry the why.
| # multi-tool generation into both rows), `cost_attributed_usd` must split its $2.0 | ||
| # evenly across Bash and Read so the per-tool sum reconciles to scoped_cost_usd | ||
| # instead of overcounting -- this is the regression the whole PR exists to fix. | ||
| self._create_generation(tool="Bash,Read", cost=2.0, trace_id="multi") |
There was a problem hiding this comment.
This fixture is Bash,Read, where arrayDistinct is a no-op, so nothing in the suite exercises the repeat case the docstring was written for. Delete arrayDistinct from the query and everything still passes. One more generation with tool="Bash,Read,Bash", asserting Bash gets half rather than two thirds, would pin it down.
| assert sum(r["share_of_scoped"] for r in body["by_input_size"]["items"]) == 1.0 | ||
| assert by_bucket["<50k"]["avg_cost_per_generation"] == 0.75 | ||
|
|
||
| def test_top_traces_removed_from_response(self) -> None: |
There was a problem hiding this comment.
test_empty_result_when_no_events already asserts top_traces is absent, so this creates an event to re-assert the same thing. Is it earning its place?
|
This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| # written by an older process during a rolling deploy are never served against the new | ||
| # contract (this revision added `by_input_size`, `cost_attributed_usd`, `share_attributed` | ||
| # and removed `top_traces`). | ||
| CACHE_SCHEMA_VERSION = "v2" |
There was a problem hiding this comment.
Cache version was not bumped despite schema changes. The comment on lines 94-97 explicitly states this revision added by_input_size, cost_attributed_usd, share_attributed and removed top_traces, but CACHE_SCHEMA_VERSION remains "v2" (same as the old CACHE_VERSION on line 80). During a rolling deployment, an old process could write a cache entry with the old schema (missing by_input_size, cost_attributed_usd, share_attributed), and a new process could read it expecting the new schema, causing errors when trying to access the new fields.
# Should be bumped to v3 or higher
CACHE_SCHEMA_VERSION = "v3"| CACHE_SCHEMA_VERSION = "v2" | |
| CACHE_SCHEMA_VERSION = "v3" |
Spotted by Graphite
Is this helpful? React 👍 or 👎 to let us know.
| # written by an older process during a rolling deploy are never served against the new | ||
| # contract (this revision added `by_input_size`, `cost_attributed_usd`, `share_attributed` | ||
| # and removed `top_traces`). | ||
| CACHE_SCHEMA_VERSION = "v2" |
There was a problem hiding this comment.
Cache version was not bumped despite significant schema changes. The constant was renamed from CACHE_VERSION to CACHE_SCHEMA_VERSION but the value remains "v2". During a rolling deploy, this will cause:
- Old processes to serve cached responses missing
by_input_size,cost_attributed_usd, andshare_attributed - New processes to potentially serve cached responses containing the now-removed
top_tracesfield - Schema mismatch errors when clients expect the new shape but receive the old (or vice versa)
Fix:
CACHE_SCHEMA_VERSION = "v3"The comment on lines 94-97 explicitly states this constant should be bumped on any payload shape change, and this PR adds three new fields and removes one.
| CACHE_SCHEMA_VERSION = "v2" | |
| CACHE_SCHEMA_VERSION = "v3" |
Spotted by Graphite
Is this helpful? React 👍 or 👎 to let us know.
|
This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the |
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| # written by an older process during a rolling deploy are never served against the new | ||
| # contract (this revision added `by_input_size`, `cost_attributed_usd`, `share_attributed` | ||
| # and removed `top_traces`). | ||
| CACHE_SCHEMA_VERSION = "v2" |
There was a problem hiding this comment.
The cache version is still "v2" but the comment explicitly states it should be bumped when the payload shape changes. This PR adds by_input_size, cost_attributed_usd, and share_attributed fields and removes top_traces - all significant schema changes.
During a rolling deploy:
- An old process could write a cache entry without the new fields
- A new process could read that stale entry
- The new serializer will fail or return incomplete data to clients
Fix:
CACHE_SCHEMA_VERSION = "v3"| CACHE_SCHEMA_VERSION = "v2" | |
| CACHE_SCHEMA_VERSION = "v3" |
Spotted by Graphite
Is this helpful? React 👍 or 👎 to let us know.
|
This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the |
This branch was successfully deployed
2 participants
Problem
The personal LLM spend endpoint (
/api/llm_analytics/@me/spend/) attributes each generation's full cost to every tool in$ai_tools_called. Because that field is a comma-separated list and a single turn often calls several tools, the per-tool costs sum to well above the real total (around 122% in practice), and the most ubiquitous tool looks dominant purely by co-occurrence. Reading theby_toolbreakdown,Bashappeared to be roughly 60% of spend when in reality it was just present on the most turns. The genuine cost driver, how large each turn's input context is, was not exposed at all.Changes
Additive changes to
products/ai_observability/backend/api/personal_spend.pyand its response schema.by_toolrows gain two fields:cost_attributed_usd: each generation's cost split evenly across the distinct tools it called, so the column reconciles to the scoped generation cost instead of overcounting.share_attributed: that value normalized tosummary.scoped_cost_usd.cost_usdandshare_of_scopedstay, but their help text now explains they are co-occurrence measures that overstate ubiquitous tools and do not reconcile across rows, and points readers to the attributed fields for an honest number.by_input_sizebreakdown groups spend into input-token buckets (<50k,50k-100k,100k-150k,150k-200k,200k-300k,>300k). This is the primary lever: large-context turns dominate spend regardless of which tool they call.by_tool.toolhelp text now notes that every skill invocation collapses to the singleSkilltoken, because skill names are not recorded at ingestion, so that row is an upper bound across all skills combined rather than any one skill.top_tracesfield (and its serializers). It always returned empty and the PostHog Code client had already dropped it from its types.The change is backwards compatible. The only removal is the always-empty
top_traces. A companion PostHog Code PR (PostHog/code#3772) adoptscost_attributed_usd,share_attributed, andby_input_sizeas optional fields and rewrites the usage display and suggestion copy off the misleading "drives X%" line.How did you test this code?
Automated tests added to
test_personal_spend.py, each covering a regression the existing suite did not:test_by_tool_cost_attributed_splits_evenly_across_distinct_tools: a multi-tool generation splits its cost across distinct tools socost_attributed_usdreconciles, rather than assigning full cost to each tool.test_by_tool_cost_attributed_falls_back_to_null_tool_for_no_tools: a generation with no tools attributes its full cost to the null-tool bucket.test_by_input_size_buckets_generations_by_input_tokens: generations land in the correct input-size bucket and bucket costs sum toscoped_cost_usd.test_top_traces_removed_from_response: the response no longer carriestop_traces.What I (the agent) verified locally:
ruff checkandruff formatclean viahogli ci:preflight(zero failures), and repo-widemypypasses the way CI runs it (uv run mypy --cache-fine-grained ., "Success: no issues found in 16775 source files"). What I did not do: I have not run the backend test suite locally (it needs the Postgres/ClickHouse dev stack) and I have not exercised the endpoint by hand. CI runs the tests.Automatic notifications
Docs update
No user-facing docs cover this internal endpoint. The developer-facing OpenAPI help text is updated in this PR.
🤖 Agent context
Autonomy: Human-driven (agent-assisted) - Paul directed the work and is assigned as DRI.
The change grew out of an investigation into why one engineer's PostHog Code spend was high. Querying the production
$ai_generationdata (US, team 2) showed theby_toolbreakdown was misleading: it double-counts multi-tool turns, so the most-used tool looked like the cost driver. The real signal was input-context size, roughly half the spend came from turns carrying more than 300k input tokens. That motivated both the fractional-attribution fix and the new context-size breakdown.The backend change was produced by a Claude Sonnet subagent directed to follow the
/improving-drf-endpointsand/writing-testsconventions (schema annotations on every field, parameterized behavior-level tests). The contract (field names, bucket edges, additive-only, removetop_traces) was fixed up front so this PR and the companion client PR agree.top_tracesremoval was checked against the PostHog Code client first to confirm nothing rendered it.Note
The branch is a few commits behind master, and master has since touched the OpenAPI generation inputs. A rebase and
hogli build:openapiregeneration may be needed before this is marked ready, if CI flags generated-type drift.Created with PostHog Code