Skip to content

fix(llm): decode Modal reasoning token usage - #691

Merged
brynary merged 3 commits into
mainfrom
fix/modal-reasoning-token-usage
Aug 1, 2026
Merged

fix(llm): decode Modal reasoning token usage#691
brynary merged 3 commits into
mainfrom
fix/modal-reasoning-token-usage

Conversation

@brynary

@brynary brynary commented Jul 30, 2026

Copy link
Copy Markdown
Member

Summary

  • decode Modal’s top-level usage.reasoning_tokens field
  • prefer the standard nested completion_tokens_details.reasoning_tokens field when both spellings are present
  • cover both streaming and non-streaming Chat Completions usage payloads

Modal reports completion_tokens as an inclusive total and sends the reasoning subset beside it. Fabro previously discarded that sibling field, leaving all completion tokens in the visible-output bucket and reporting zero reasoning tokens.

Verification

  • cargo nextest run -p fabro-llm — 671 passed, 24 skipped
  • cargo +nightly-2026-04-14 fmt --check --all
  • cargo +nightly-2026-04-14 clippy -p fabro-llm --all-targets -- -D warnings
  • git diff --check

Copilot AI review requested due to automatic review settings July 30, 2026 21:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the OpenAI-compatible wire codec in fabro-llm to correctly account for Modal’s reasoning-token reporting shape, so downstream token accounting matches the provider’s semantics (reasoning tokens separated from visible output tokens).

Changes:

  • Extend ApiUsage to decode Modal’s top-level usage.reasoning_tokens.
  • Normalize reasoning tokens by preferring completion_tokens_details.reasoning_tokens when present, falling back to the top-level field.
  • Add focused unit tests covering both non-streaming and streaming usage payloads for Modal.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

brynary and others added 2 commits August 1, 2026 09:34
…oken-usage

# Conflicts:
#	lib/components/fabro-llm/src/codec/openai_compatible/wire.rs
Replace the two envelope-level tests with focused `ApiUsage` tests that
match the file's existing `token_counts_*` convention.

The streaming and non-streaming tests were the same test paid for twice:
`ApiResponse::usage` and `StreamChunk::usage` are both `Option<ApiUsage>`,
so the envelope cannot change the result. Envelope-level usage decoding is
already covered by `stream_chunk_usage_parsing`.

Also pin the precedence rule this change introduces — nested detail wins
over the flat spelling, and an empty `completion_tokens_details` still
falls back — and document it on `token_counts`. Revert the unrelated
`cost` doc edit that dropped the OpenRouter reference.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 1, 2026 13:44
@brynary

brynary commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Merged main (resolved a conflict in wire.rs with the DeepSeek prompt_cache_hit_tokens work, which landed in the same function) and ran a simplification pass over the diff.

Production change is unchanged — a review of reuse, quality, and efficiency found no issues with it. Notably: no helper extraction for the repeated .as_ref().and_then(...).or(...) shape, since each of the three reads a different field, so a helper would need a closure per call site and end up longer. Field name and .or() also left as-is, matching the DeepSeek fallback next to it.

Cleanups:

  • Collapsed the two envelope tests into focused ApiUsage tests. They were one test paid for twice — ApiResponse::usage and StreamChunk::usage are both Option<ApiUsage>, so the envelope cannot change the result. Envelope-level usage decoding is already covered by stream_chunk_usage_parsing. The payloads also carried fields ApiUsage never deserializes (total_tokens, id, model, choices) plus a cached_tokens split unrelated to the fix.
  • Renamed to the token_counts_* convention used by the neighboring tests and regrouped them so the usage tests stay contiguous.
  • Added a test pinning the precedence rule this PR introduces: nested detail wins over the flat spelling, and an empty completion_tokens_details still falls back to it. That behavior was previously untested. Verified both new tests fail when .or(self.reasoning_tokens) is removed.
  • Documented the precedence on token_counts, which covered disjoint bucketing but not the alternate spellings.
  • Reverted the unrelated cost doc edit that dropped the concrete OpenRouter reference.

Verification: cargo nextest run -p fabro-llm (690 passed, 26 skipped), fmt --check --all, clippy -p fabro-llm --all-targets -- -D warnings, git diff --check.

One out-of-scope finding for a possible follow-up: codec/openai_compatible/response.rs:81 parses the whole response body a second time into a serde_json::Value for Response.raw, which appears to have no production reader (only two fabro-llm tests). Not touched here.

@brynary
brynary marked this pull request as ready for review August 1, 2026 13:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

@brynary
brynary merged commit 66b282d into main Aug 1, 2026
15 checks passed
@brynary
brynary deleted the fix/modal-reasoning-token-usage branch August 1, 2026 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants