Surface the model and thinking level agents actually use - #2659
Surface the model and thinking level agents actually use#2659colonelpanic8 wants to merge 9 commits into
Conversation
|
| Filename | Overview |
|---|---|
| packages/server/src/server/agent/providers/claude/agent.ts | Captures Claude runtime model and effort observations across live, coalesced, and restored timeline paths, including raw off-catalog identifiers. |
| packages/server/src/server/agent/providers/codex-app-server-agent.ts | Captures Codex runtime model and reasoning effort from thread lifecycle messages and clears stale observations after configuration changes. |
| packages/server/src/server/agent/providers/opencode-agent.ts | Retains OpenCode’s observed provider/model identity and resets it when model or thinking configuration changes. |
| packages/app/src/hooks/use-agent-model-display.ts | Centralizes runtime-preferred model and thinking display resolution for app surfaces. |
| packages/app/src/agent-stream/turn-attribution.ts | Resolves completed-turn attribution from the final attributed assistant message without crossing turn boundaries. |
| packages/app/src/runtime/replica-cache/index.ts | Preserves effective thinking attribution, including the distinction between an absent field and explicit null, in cached replicas. |
| packages/protocol/src/agent-types.ts | Extends shared agent and timeline types to carry runtime and per-turn attribution through existing protocol channels. |
Sequence Diagram
sequenceDiagram
participant Provider as Agent provider
participant Server as Paseo daemon
participant Store as Client session store
participant UI as App surfaces
Provider->>Server: Runtime model and thinking observations
Server->>Server: Normalize known values or preserve raw identifiers
Server->>Store: runtimeInfo and attributed timeline items
Store->>UI: Runtime-preferred model and thinking metadata
Reviews (9): Last reviewed commit: "fix: keep AgentControls under the comple..." | Re-trigger Greptile
Screenshots (web, desktop layout)Captured against a dev daemon running this branch's server code, with three real Codex agents (a parent plus two children created with a The setup happens to demonstrate the exact bug this PR fixes. All three agents were created with a configured thinking level, and Codex reports something different at runtime:
Before this PR the UI would have shown "Low" and "Medium" for those children. It now shows what Codex is actually doing. Subagents track — trailing
|
Added: per-turn model attributionScope grew by request — the thread now records which model produced each assistant turn, not just what the agent is running right now. Same feature, so it's in this PR rather than a follow-up. What it looks likeCompleted turn footers gain the model and thinking level that produced them: In a thread, every turn carries its own attribution: Together with the subagent rows from the earlier comment: How it works
History: Claude and OpenCode backfill from their persisted transcripts, so existing conversations gain attribution retroactively. Codex is forward-only — its transcript exposes no per-message model. Two things found by testing this against a live agent, not by reading the diff1. Codex turns were silently unattributed. Attribution was keyed on per-turn observations, but the real app-server reports model/effort on the thread response and usually omits them from turn notifications — so in practice nothing was ever stamped. Fixed by falling back to the thread-level observation (still something Codex reported), with a turn's own value preferred when present. The unit tests passed throughout, because they fed turn notifications that carried the fields; only a live agent showed the gap. 2. A rejected fix that rewrote history. I first tried also falling back to the model Paseo requested for the turn, so attribution would survive a model switch. It looked right and passed tests — then in the live app I watched an earlier Sol turn silently repaint itself as Luna after I switched models, because that fallback reflects current config and any replay reapplies it. Reverted. Recording a turn wrong is worse than recording nothing. Known limitationA turn taken immediately after a model switch may be unattributed until the runtime reports again — Codex clears its observation on Verification
Platform coverage is unchanged from the earlier comment: web only, no iOS/Android/Electron captures. |
When Claude reported a model that normalizeClaudeRuntimeModelId could not map to the catalog, the previously recognized model stayed in runtimeInfo.model and the real identifier was only kept in extra.runtimeModel — which no client reads. Every surface this PR adds would then assert the agent was running a model it had switched away from. An id we cannot map is still what Claude said it is running, so report it verbatim; clients already render an unknown id as-is rather than guessing. Reported by greptile on getpaseo#2659. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
d5e391f to
deee9c4
Compare
The previous fix made runtimeInfo report an unmappable model verbatim, but the timeline paths still stamped only normalized ids. An off-catalog model therefore showed up on current-agent surfaces while completed-turn footers and restored history silently lost it. All three sites now share one rule — normalize when we can, keep the raw identifier when we cannot — so live turns, coalesced messages, and backfilled history attribute the same way. Reported by greptile on getpaseo#2659. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
When Claude reported a model that normalizeClaudeRuntimeModelId could not map to the catalog, the previously recognized model stayed in runtimeInfo.model and the real identifier was only kept in extra.runtimeModel — which no client reads. Every surface this PR adds would then assert the agent was running a model it had switched away from. An id we cannot map is still what Claude said it is running, so report it verbatim; clients already render an unknown id as-is rather than guessing. Reported by greptile on getpaseo#2659. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous fix made runtimeInfo report an unmappable model verbatim, but the timeline paths still stamped only normalized ids. An off-catalog model therefore showed up on current-agent surfaces while completed-turn footers and restored history silently lost it. All three sites now share one rule — normalize when we can, keep the raw identifier when we cannot — so live turns, coalesced messages, and backfilled history attribute the same way. Reported by greptile on getpaseo#2659. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
9170eda to
425e5a5
Compare
|
Pushed 9915e10, a follow-up found by dogfooding the branch: the observed model was invisible for most of an agent's life.
Verified with new regression tests in |
Paseo showed only the *selected* model and thinking level. Providers can diverge from that selection — Codex collaboration modes inject their own model/effort, models fall back mid-session — and subagent rows showed no model at all, so there was no way to tell what an agent was really running. The observed-vs-configured channel already existed end to end (`runtimeInfo.model` / `runtimeInfo.thinkingOptionId` plus the daemon's runtime-preferring `effectiveThinkingOptionId`). It was unevenly populated and barely surfaced. No protocol changes were needed. Server — populate runtimeInfo from data the runtimes already send: - Codex: capture model and reasoningEffort from thread/start, thread/resume, and thread/turn notifications; normalize effort to catalog thinking ids. An omitted effort retains the last observation, an explicit null clears it. - Claude: capture per-turn assistant message models, not just the SDK init model, and stop dropping the raw `extra.runtimeModel` after run(). - OpenCode: keep the observed assistant providerID/modelID that was already being resolved for context-window lookups and prefer it in runtime info. - All three clear their observations on setModel/setThinkingOption, so a user's new selection is never shadowed by a stale observation. App — show it, runtime-preferring, and never guess: - Subagents track rows gain trailing "Model · Thinking" meta, and prefer the human task description over the provider's agent-type label. - Desktop tab tooltips show "Claude · Opus 4.5 · High". - The running-turn indicator and context window tooltip show model/thinking. - The composer thinking chip reads out the runtime value while the dropdown keeps marking the configured one, so a mutation stays relative to config. - The client stops discarding effectiveThinkingOptionId, preserving the load-bearing absent (daemon never sent it) vs explicit null distinction through the store and the offline replica cache. Display never substitutes a guess for an unknown: an off-catalog runtime model or thinking id renders verbatim rather than falling back to the configured model or the first available option. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Knowing what an agent is running right now doesn't tell you what produced the answer you're scrolling past. A turn from an hour ago may have run on a different model — after a fallback, a collaboration-mode override, or a deliberate switch — and nothing in the thread said so. Assistant timeline items now carry the model and thinking level that produced them, and completed turn footers read "Worked for 21s · GPT-5.6-Sol · High". Protocol: `model` and `thinkingOptionId` are optional additions to the assistant_message timeline item. Old clients ignore them, old daemons omit them, and absence renders nothing — so no capability gate is needed. Server: each provider stamps only what it actually observed for that turn, never the configured value, because a confident wrong attribution is worse than none. The stream coalescer and timeline projection preserve the fields while merging chunks. Claude and OpenCode also backfill history from their persisted transcripts; Codex is forward-only, since its transcript exposes no per-message model. App: attribution survives the three paths that rebuild assistant items (streaming append, block promotion, reducer replacement) and is add-only, so a later chunk arriving bare cannot erase what the first chunk reported. A turn that spans a model switch reports the model it ended on — the one the footer sits under. Labels resolve per turn from the providers snapshot rather than from the agent's current selection. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex reports its model and reasoning effort on the thread response and usually omits them from turn notifications, so keying attribution solely on per-turn observations left every real Codex turn unattributed. Fall back to the thread-level observation, which Codex also reported, while still preferring a turn's own value so a mid-session change lands on the right turn. Verified against a live Codex agent: turns now carry their model, and switching models mid-thread leaves already-recorded turns untouched rather than repainting history with the current selection. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
When Claude reported a model that normalizeClaudeRuntimeModelId could not map to the catalog, the previously recognized model stayed in runtimeInfo.model and the real identifier was only kept in extra.runtimeModel — which no client reads. Every surface this PR adds would then assert the agent was running a model it had switched away from. An id we cannot map is still what Claude said it is running, so report it verbatim; clients already render an unknown id as-is rather than guessing. Reported by greptile on getpaseo#2659. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous fix made runtimeInfo report an unmappable model verbatim, but the timeline paths still stamped only normalized ids. An off-catalog model therefore showed up on current-agent surfaces while completed-turn footers and restored history silently lost it. All three sites now share one rule — normalize when we can, keep the raw identifier when we cannot — so live turns, coalesced messages, and backfilled history attribute the same way. Reported by greptile on getpaseo#2659. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runtime model Claude reports was captured but invisible for most of an agent's life. The manager only refreshes runtimeInfo on thread_started and turn_completed, and Claude pre-assigns its session id, so thread_started never fires for it: the observation surfaced only when a turn *completed*. A running agent — exactly what the subagents track shows — reported no model at all, and an interrupted turn never reported one. - Claude now dispatches model_changed when it captures a runtime model, the same push path ACP uses, so the manager and clients see the observation the moment the first assistant chunk names it. - turn_failed and turn_canceled refresh runtimeInfo like turn_completed does; an interrupted turn still observed a model before it stopped. - The "<synthetic>" placeholder Claude Code stamps on frames with no real inference behind them is no longer treated as an observation; one had reached a real agent record as its display model. All four attribution sites now go through resolveObservedClaudeModelId, which the subagent sources already used. - Loading a persisted transcript re-seeds the observation from the last real assistant entry, so a daemon restart no longer blanks the observed model until the next turn completes. Also folds translateMessageToEvents' assistant and stream_event arms into helpers, dropping its complexity below the lint ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completed turn footers reported the wrong model. `resolveTurnAttribution` walked from the turn's start to the end of the *entire* stream and kept the last attribution it saw, so every footer showed whichever model the most recent message ran on. A session that started on Fable and later answered one question on Sonnet relabelled the whole history "Sonnet 5". It now walks the turn the way `collectAssistantTurnContent` does — from the turn's last item toward its first, in the strategy's traversal direction, stopping at the user message that opened the turn. Taking the traversal step from the strategy also fixes inverted lists, where the old hardcoded `+= 1` walked backwards through time. The thinking level on those footers was fabricated outright. Nothing observed one, so `resolveThinkingId` fell through to the model's default, which is `effortLevels[0]` — always "low" for every Claude model. Recorded turns now pass `thinkingFallback: "none"`: a turn reports what it ran at or nothing. Live agent surfaces keep the model-default fallback, where it answers the different question of what the configured model would think at. That leaves a real value to report: Claude Code records `effort` on assistant frames, as a sibling of `message` rather than a field of it — absent from `SDKAssistantMessage`, so it is read defensively and stamped only when it names a known level. Both the live path and transcript replay stamp it, so a turn now reads "Sonnet 5 · Low" only when it genuinely ran low. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude's runtimeInfo carried a model but never a thinking level, so every live surface — the subagents track, the composer chip, the tab tooltip — fell back to the configured level and would keep claiming "High" through a silent downgrade. That is the misreport this whole channel exists to prevent, left open on Claude's thinking dimension because the level looked unobservable. It is observable: Claude Code stamps `effort` on assistant frames. runtimeInfo now reports the observed level, and drops it on setModel and setThinkingOption so a stale observation cannot shadow a fresh selection or survive onto a model with different effort levels. Transcript replay re-seeds it, as it does the model. The key is omitted rather than null when nothing has been observed: its presence is what makes the runtime value outrank the configured one, so an unobserved level has to leave the configured level in charge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runtime-thinking wiring added two more optional chains to the resolveAgentModelSelection call, pushing AgentControls to a complexity of 21 against oxlint's ceiling of 20. Lift the slice-to-selection mapping to a module-level helper. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
31bc146 to
bb52162
Compare







Linked issue
Part of #2644
Type of change
What does this PR do
Paseo only ever showed the model and thinking level you had selected, never what the agent was actually running. Those can differ: Codex collaboration modes inject their own model and reasoning effort, models fall back mid-session, and a provider may refuse a configured model outright. Subagent rows showed no model at all, so with several subagents running there was no way to tell what any of them was using.
The observed-vs-configured channel already existed end to end —
runtimeInfo.model,runtimeInfo.thinkingOptionId, and the daemon's runtime-preferringeffectiveThinkingOptionId. It was unevenly populated and barely surfaced. No protocol changes were needed (git diff --stat -- packages/protocolis empty).Server — populate
runtimeInfofrom data the runtimes already send and Paseo was discarding:reasoningEffortfromthread/start,thread/resume, and thread/turn notifications, normalized to catalog thinking ids. An omitted effort retains the last observation; an explicitnullclears it and falls back to config.run()no longer drops the rawextra.runtimeModel.providerID/modelIDit was already resolving for context-window lookups.setModel/setThinkingOption, so a stale observation can never shadow a fresh user selection.App — surface it, runtime-preferring, and never guess:
Model · Thinkingmeta, and now prefer the human task description over the provider's agent-type label (Claude subagents were showing "general-purpose" instead of the Task description).Claude · Opus 4.5 · High.effectiveThinkingOptionId, preserving the load-bearing "absent (daemon never sent it)" vs "explicit null" distinction through the store and the offline replica cache.Display never substitutes a guess for something unknown: an off-catalog runtime model or thinking id renders verbatim rather than falling back to the configured model or to the first available option. Provider-subagent rows have no model on the wire yet and render no meta.
Partial provider coverage is intentional — ACP (Copilot) and Pi/OMP already reported runtime values correctly and are untouched.
How did you verify it
Automated, all run locally on this branch:
npm run typecheck— clean across all workspaces.npm run lint— 0 warnings, 0 errors.npm run format(Biome/oxfmt) — clean; also enforced by the pre-commit hook.codex-app-server-agent.test.ts(new cases for observed capture from start/resume/notifications, observed-over-configured precedence, absent-vs-null effort, and stale-observation clearing onsetModel/setThinkingOption),claude/agent.test.ts(62 passed),opencode-agent.test.ts(70 passed).composer/agent-controls/utils.test.ts(32 passed), plussubagents/select,subagents/track-presentation,utils/agent-snapshots,panels/agent-panel-descriptor,runtime/replica-cache— 91 passed across the six files.configures Codex app-server to use a custom provider base URL. I stashed only my changes and reproduced the identical failure on unmodifiedHEAD.Review: the full diff went through an adversarial review pass that specifically hunted stale-observation and absent-vs-null bugs. It found three real defects, all fixed in this branch and each now pinned by a regression test: the Codex and OpenCode
setModel/setThinkingOptionreset described above, and a display path that could read out a thinking level the agent was not using (the runtime id fell through tothinkingOptions[0]).Screenshots: posted in a follow-up comment — subagents track, desktop tab tooltip, context window tooltip, and the running-turn indicator, captured against a dev daemon running this branch with three real Codex agents. That setup incidentally reproduces the bug: two children configured
low/mediumare observed running athigh, so the old UI would have misreported them.Captured on web only (headless Chromium, desktop layout). iOS, Android, and Electron desktop are not captured — this is a headless machine. The compact layout is worth a visual check before merge (row meta truncates at
maxWidth: 40%, and track actions are always-visible on native/compact); happy to get a simulator/device capture if you want it.Checklist
npm run typecheckpassesnpm run lintpassesnpm run formatran (Biome)Disclosure: written with AI assistance (Claude Code). The design decisions, the review pass, and every verification claim above are things I ran and read myself; the one thing I could not stand up is called out rather than papered over.