fix: preserve billing dimensions and journal usage for replay

[ramapitecusment]

Usage events lose billing dimensions and can disappear between the in-memory queue and the desktop collector. Several direct, streaming and media paths also bypass accounting, causing undercounting and making explicitly reported zero usage indistinguishable from absent measurements.

This change preserves the accounting evidence needed for per-event valuation and provides replayable delivery:

• Preserve raw usage, measurement/completeness state, cache creation lifetimes, response tiers and exact xAI USD ticks. Retain early hosted-tool/grounding metadata through native Gemini, Antigravity, AIStudio and plugin streams, including billing-only events without token counters. Explicitly measured zero remains distinct from unknown usage.
• Assign one event ID before synchronous/asynchronous plugin dispatch; propagate attempt/generation scope through normal HTTP handler contexts. Include endpoint, transport and event kind. Unmeasured coverage applies only to admitted requests, so unmatched routes, failed authentication and pre-authentication Home gates cannot fill the journal.
• Publish separate image-tool events in all Codex WebSocket execution modes and retain xAI video operation identity through creation and polling, even before usage arrives. Alpha Search records upstream latency and transport/read failures without duplicating Home 401 events. Management calls select provider-specific parsers, resolve Gemini/Vertex models from URLs, and merge SSE and JSON-array streams.
• Observe live/realtime terminal and transcription usage with per-content deduplication and the transcription model's identity. Oversized WebSocket frames are validated and projected incrementally into bounded accounting metadata while original bytes are forwarded unchanged; complete upstream usage survives downstream write failures. Malformed/incomplete frames are not presented as measured usage.
• Persist a private synchronous journal with authenticated GET/ACK replay. Opaque event IDs map to hashed filenames, with legacy file replay/ACK compatibility. Durable copies fingerprint credentials and exclude response headers and upstream failure bodies. The legacy queue retains its wire behavior and now queues an overflow event whenever any subscriber misses live delivery.

The change also fixes text-only normalization of translated image tool results while preserving multimodal behavior.

Validation

Regression tests were verified failing before the corresponding fixes and passing afterward, including source/protocol attribution, zero/unknown usage, billing-only streams, journal replay and privacy, mixed-subscriber overflow, video polling identity, and WebSocket image/terminal usage. Independent streaming-parser checks cover JSON grammar, reads split at every byte, exact large integers, duplicate keys, malformed/truncated input, reader errors and bounded allocation for large skipped values.

• go test ./... passes.
• go build -o test-output ./cmd/server && rm test-output passes.
• Race checks pass for the SDK usage manager, queue/journal, API/management handlers, usage helpers, live/realtime package and affected native executor scenarios.

Delivery and limits

The journal is a single-consumer local spool; unacknowledged files do not expire. Old collectors do not ACK it, so disk usage must be monitored until upgraded. Storage failures are surfaced, and neither unwritable storage nor usage never reported upstream can be reconstructed. Opaque WebRTC and arbitrary plugin traffic are not claimed to be fully measured. Realtime projection retains at most 256 KiB of accounting metadata with a nesting limit of 256; full text/audio content is skipped without a frame-size accounting cutoff. See docs/usage-accounting.md for the source matrix and contract.

Companion desktop PR #253 consumes this contract and implements per-event valuation, replay deduplication, provider-specific prices and explicit unknown-cost coverage. The desktop release must pin a core release containing this change.

[github-actions]

This pull request targeted main.

The base branch has been automatically changed to dev.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-12T15:23:43.931546Z	fa79464	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bd2e42c1c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 10cc498888

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 771f30123e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 84b18c0e4c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0a77686175

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 30c7821850

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1386ec7d23

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ramapitecusment]

@luispater @sususu98 I have extracted a first independently reviewable correction into #5817: three production lines to preserve DeepSeek cache-hit usage in the existing schema, with response/SSE regression tests. It reproduces on current dev and does not depend on this PR's journal or SDK additions. The earlier #3802 proposal is linked there to make the overlap explicit.

This PR combines several different decisions, so I am moving it to draft while separating them. Proposed review order:

1. Provider counter normalization fixes, each with a minimal upstream payload and expected canonical token buckets.
2. Request/attempt identity and HTTP/WebSocket source coverage.
3. Persistent usage delivery and replay as a separate design decision, with retention, disk/latency overhead and crash/ACK behavior made explicit.

Would you prefer the journal/delivery portion as an optional feature, or should that remain in the desktop collector? Guidance on that boundary would help avoid asking you to review an architecture you do not want. The current implementation and its tests remain here as a reference; #5817 can be merged independently.