fix(llm): decode Modal reasoning token usage - #691
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the OpenAI-compatible wire codec in fabro-llm to correctly account for Modal’s reasoning-token reporting shape, so downstream token accounting matches the provider’s semantics (reasoning tokens separated from visible output tokens).
Changes:
- Extend
ApiUsageto decode Modal’s top-levelusage.reasoning_tokens. - Normalize reasoning tokens by preferring
completion_tokens_details.reasoning_tokenswhen present, falling back to the top-level field. - Add focused unit tests covering both non-streaming and streaming usage payloads for Modal.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…oken-usage # Conflicts: # lib/components/fabro-llm/src/codec/openai_compatible/wire.rs
Replace the two envelope-level tests with focused `ApiUsage` tests that match the file's existing `token_counts_*` convention. The streaming and non-streaming tests were the same test paid for twice: `ApiResponse::usage` and `StreamChunk::usage` are both `Option<ApiUsage>`, so the envelope cannot change the result. Envelope-level usage decoding is already covered by `stream_chunk_usage_parsing`. Also pin the precedence rule this change introduces — nested detail wins over the flat spelling, and an empty `completion_tokens_details` still falls back — and document it on `token_counts`. Revert the unrelated `cost` doc edit that dropped the OpenRouter reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Merged Production change is unchanged — a review of reuse, quality, and efficiency found no issues with it. Notably: no helper extraction for the repeated Cleanups:
Verification: One out-of-scope finding for a possible follow-up: |
Summary
usage.reasoning_tokensfieldcompletion_tokens_details.reasoning_tokensfield when both spellings are presentModal reports
completion_tokensas an inclusive total and sends the reasoning subset beside it. Fabro previously discarded that sibling field, leaving all completion tokens in the visible-output bucket and reporting zero reasoning tokens.Verification
cargo nextest run -p fabro-llm— 671 passed, 24 skippedcargo +nightly-2026-04-14 fmt --check --allcargo +nightly-2026-04-14 clippy -p fabro-llm --all-targets -- -D warningsgit diff --check