Skip to content

Surface the model and thinking level agents actually use - #2659

Open
colonelpanic8 wants to merge 9 commits into
getpaseo:mainfrom
colonelpanic8:subagent-param-visibility
Open

Surface the model and thinking level agents actually use#2659
colonelpanic8 wants to merge 9 commits into
getpaseo:mainfrom
colonelpanic8:subagent-param-visibility

Conversation

@colonelpanic8

@colonelpanic8 colonelpanic8 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Linked issue

Part of #2644

Type of change

  • Bug fix
  • New feature (with prior issue + design alignment)
  • Refactor / code improvement
  • Docs

What does this PR do

Paseo only ever showed the model and thinking level you had selected, never what the agent was actually running. Those can differ: Codex collaboration modes inject their own model and reasoning effort, models fall back mid-session, and a provider may refuse a configured model outright. Subagent rows showed no model at all, so with several subagents running there was no way to tell what any of them was using.

The observed-vs-configured channel already existed end to end — runtimeInfo.model, runtimeInfo.thinkingOptionId, and the daemon's runtime-preferring effectiveThinkingOptionId. It was unevenly populated and barely surfaced. No protocol changes were needed (git diff --stat -- packages/protocol is empty).

Server — populate runtimeInfo from data the runtimes already send and Paseo was discarding:

  • Codex captures model and reasoningEffort from thread/start, thread/resume, and thread/turn notifications, normalized to catalog thinking ids. An omitted effort retains the last observation; an explicit null clears it and falls back to config.
  • Claude captures per-turn assistant-message models, not just the SDK init model, so a mid-session fallback is visible; and run() no longer drops the raw extra.runtimeModel.
  • OpenCode keeps the observed assistant providerID/modelID it was already resolving for context-window lookups.
  • All three clear their observations on setModel/setThinkingOption, so a stale observation can never shadow a fresh user selection.

App — surface it, runtime-preferring, and never guess:

  • Subagents track rows gain trailing Model · Thinking meta, and now prefer the human task description over the provider's agent-type label (Claude subagents were showing "general-purpose" instead of the Task description).
  • Desktop tab tooltips show Claude · Opus 4.5 · High.
  • The running-turn indicator and the context-window tooltip show model and thinking.
  • The composer thinking chip reads out the runtime value while the dropdown keeps marking the configured one, so a mutation stays relative to config.
  • The client stops discarding effectiveThinkingOptionId, preserving the load-bearing "absent (daemon never sent it)" vs "explicit null" distinction through the store and the offline replica cache.

Display never substitutes a guess for something unknown: an off-catalog runtime model or thinking id renders verbatim rather than falling back to the configured model or to the first available option. Provider-subagent rows have no model on the wire yet and render no meta.

Partial provider coverage is intentional — ACP (Copilot) and Pi/OMP already reported runtime values correctly and are untouched.

How did you verify it

Automated, all run locally on this branch:

  • npm run typecheck — clean across all workspaces.
  • npm run lint — 0 warnings, 0 errors.
  • npm run format (Biome/oxfmt) — clean; also enforced by the pre-commit hook.
  • Provider unit tests: codex-app-server-agent.test.ts (new cases for observed capture from start/resume/notifications, observed-over-configured precedence, absent-vs-null effort, and stale-observation clearing on setModel/setThinkingOption), claude/agent.test.ts (62 passed), opencode-agent.test.ts (70 passed).
  • App unit tests: composer/agent-controls/utils.test.ts (32 passed), plus subagents/select, subagents/track-presentation, utils/agent-snapshots, panels/agent-panel-descriptor, runtime/replica-cache — 91 passed across the six files.
  • Pre-existing unrelated failure, confirmed not mine: the full Codex test file stops at configures Codex app-server to use a custom provider base URL. I stashed only my changes and reproduced the identical failure on unmodified HEAD.

Review: the full diff went through an adversarial review pass that specifically hunted stale-observation and absent-vs-null bugs. It found three real defects, all fixed in this branch and each now pinned by a regression test: the Codex and OpenCode setModel/setThinkingOption reset described above, and a display path that could read out a thinking level the agent was not using (the runtime id fell through to thinkingOptions[0]).

Screenshots: posted in a follow-up comment — subagents track, desktop tab tooltip, context window tooltip, and the running-turn indicator, captured against a dev daemon running this branch with three real Codex agents. That setup incidentally reproduces the bug: two children configured low/medium are observed running at high, so the old UI would have misreported them.

Captured on web only (headless Chromium, desktop layout). iOS, Android, and Electron desktop are not captured — this is a headless machine. The compact layout is worth a visual check before merge (row meta truncates at maxWidth: 40%, and track actions are always-visible on native/compact); happy to get a simulator/device capture if you want it.

Checklist

  • One focused change. Unrelated cleanups split out.
  • npm run typecheck passes
  • npm run lint passes
  • npm run format ran (Biome)
  • UI changes include screenshots — web/desktop layout only; iOS, Android, and Electron not captured (headless machine), called out above
  • Tests added or updated where it made sense

Disclosure: written with AI assistance (Claude Code). The design decisions, the review pass, and every verification claim above are things I ran and read myself; the one thing I could not stand up is called out rather than papered over.

Comment thread packages/server/src/server/agent/providers/claude/agent.ts Outdated
@greptile-apps

greptile-apps Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR records the model and thinking level actually observed from supported agent runtimes and surfaces that attribution throughout the app.

  • Captures runtime model and reasoning observations from Claude, Codex, and OpenCode while clearing stale observations after configuration changes.
  • Preserves runtime attribution through protocol projections, session state, timeline history, and offline replica caching.
  • Displays observed model and thinking metadata in turn footers, composer controls, context-window details, desktop tabs, and subagent rows.
  • Adds regression coverage for runtime capture, off-catalog identifiers, timeline attribution, state propagation, and display resolution.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported off-catalog Claude model issues are fixed across current runtime state, live timeline attribution, coalesced messages, and restored history.

Important Files Changed

Filename Overview
packages/server/src/server/agent/providers/claude/agent.ts Captures Claude runtime model and effort observations across live, coalesced, and restored timeline paths, including raw off-catalog identifiers.
packages/server/src/server/agent/providers/codex-app-server-agent.ts Captures Codex runtime model and reasoning effort from thread lifecycle messages and clears stale observations after configuration changes.
packages/server/src/server/agent/providers/opencode-agent.ts Retains OpenCode’s observed provider/model identity and resets it when model or thinking configuration changes.
packages/app/src/hooks/use-agent-model-display.ts Centralizes runtime-preferred model and thinking display resolution for app surfaces.
packages/app/src/agent-stream/turn-attribution.ts Resolves completed-turn attribution from the final attributed assistant message without crossing turn boundaries.
packages/app/src/runtime/replica-cache/index.ts Preserves effective thinking attribution, including the distinction between an absent field and explicit null, in cached replicas.
packages/protocol/src/agent-types.ts Extends shared agent and timeline types to carry runtime and per-turn attribution through existing protocol channels.

Sequence Diagram

sequenceDiagram
  participant Provider as Agent provider
  participant Server as Paseo daemon
  participant Store as Client session store
  participant UI as App surfaces
  Provider->>Server: Runtime model and thinking observations
  Server->>Server: Normalize known values or preserve raw identifiers
  Server->>Store: runtimeInfo and attributed timeline items
  Store->>UI: Runtime-preferred model and thinking metadata
Loading

Reviews (9): Last reviewed commit: "fix: keep AgentControls under the comple..." | Re-trigger Greptile

@colonelpanic8

Copy link
Copy Markdown
Contributor Author

Screenshots (web, desktop layout)

Captured against a dev daemon running this branch's server code, with three real Codex agents (a parent plus two children created with a paseo.parent-agent-id label). Platform coverage is stated at the bottom.

The setup happens to demonstrate the exact bug this PR fixes. All three agents were created with a configured thinking level, and Codex reports something different at runtime:

Agent Configured Observed (runtimeInfo)
Investigate runtime model reporting gpt-5.6-sol / high gpt-5.6-sol / high
Map the Codex capture sites gpt-5.6-luna / low gpt-5.6-luna / high
Audit absent-vs-null semantics gpt-5.6-terra / medium gpt-5.6-terra / high

Before this PR the UI would have shown "Low" and "Medium" for those children. It now shows what Codex is actually doing.

Subagents track — trailing Model · Thinking, plus the composer chips

Subagents track showing model and thinking per row

Rows read GPT-5.6-Luna · High and GPT-5.6-Terra · High. Titles come from the description-first label change (these are Paseo subagents; provider subagents have no model on the wire yet and render no meta).

Desktop tab tooltip

Tab tooltip showing Codex, model and thinking

Codex · GPT-5.6-Sol · High — replaces the old Codex agent.

Context window tooltip

Context window tooltip with Model and Thinking rows

(Cropped to omit my provider plan usage, which the tooltip also renders.)

Running-turn indicator

Running turn indicator showing elapsed time, model and thinking

Captured mid-turn: 2s · GPT-5.6-Sol · High.

Also verified by hand

  • Expanding/collapsing the track, and the density rule: at a narrower pane the thinking chip drops its label to just the brain glyph (existing layout.ts behavior, unchanged).
  • agent update --thinking against the live daemon, confirming the stale-observation fix — after changing the configured level, getRuntimeInfo() follows config immediately instead of replaying the earlier observation.

Platform coverage — read this before trusting the above

Captured: web only (headless Chromium, 1600x1000, desktop layout). Not captured: iOS, Android, or Electron desktop. This machine is a headless server, so native and Electron captures need a display I do not have here; I tried a narrow-viewport web capture to stand in for the compact layout and the mobile navigation didn't cooperate, so I am not claiming it. The compact path does matter for this change — the subagent row meta has maxWidth: 40% and truncates before the title, and the track's action buttons are always visible on native/compact — so if you want that verified visually before merging, say so and I will get a real device or simulator capture rather than guess.

@colonelpanic8

Copy link
Copy Markdown
Contributor Author

Added: per-turn model attribution

Scope grew by request — the thread now records which model produced each assistant turn, not just what the agent is running right now. Same feature, so it's in this PR rather than a follow-up.

What it looks like

Completed turn footers gain the model and thinking level that produced them:

Turn footer showing Worked for 2s, GPT-5.6-Sol, High

In a thread, every turn carries its own attribution:

Thread with two attributed turns

Together with the subagent rows from the earlier comment:

Subagents track with model and thinking per row

How it works

assistant_message timeline items gained optional model / thinkingOptionId. Old clients ignore them, old daemons omit them, absence renders nothing — no capability gate needed. Providers stamp only what they actually observed, never the configured value.

History: Claude and OpenCode backfill from their persisted transcripts, so existing conversations gain attribution retroactively. Codex is forward-only — its transcript exposes no per-message model.

Two things found by testing this against a live agent, not by reading the diff

1. Codex turns were silently unattributed. Attribution was keyed on per-turn observations, but the real app-server reports model/effort on the thread response and usually omits them from turn notifications — so in practice nothing was ever stamped. Fixed by falling back to the thread-level observation (still something Codex reported), with a turn's own value preferred when present. The unit tests passed throughout, because they fed turn notifications that carried the fields; only a live agent showed the gap.

2. A rejected fix that rewrote history. I first tried also falling back to the model Paseo requested for the turn, so attribution would survive a model switch. It looked right and passed tests — then in the live app I watched an earlier Sol turn silently repaint itself as Luna after I switched models, because that fallback reflects current config and any replay reapplies it. Reverted. Recording a turn wrong is worse than recording nothing.

Known limitation

A turn taken immediately after a model switch may be unattributed until the runtime reports again — Codex clears its observation on setModel and doesn't necessarily re-report on the next turn. Verified behavior: past turns keep their recorded values; only the new turn is blank. I'd rather ship that than guess.

Verification

  • npm run typecheck, npm run lint, npm run format — all clean (also enforced by the pre-commit hook).
  • 341 tests across the touched app and server files, including new coverage for turn attribution, absent-vs-null, child-message isolation, and the mid-turn-change rule.
  • Screenshots above are a live Codex agent against a dev daemon running this branch.

Platform coverage is unchanged from the earlier comment: web only, no iOS/Android/Electron captures.

colonelpanic8 added a commit to colonelpanic8/paseo that referenced this pull request Jul 30, 2026
When Claude reported a model that normalizeClaudeRuntimeModelId could not
map to the catalog, the previously recognized model stayed in
runtimeInfo.model and the real identifier was only kept in
extra.runtimeModel — which no client reads. Every surface this PR adds would
then assert the agent was running a model it had switched away from.

An id we cannot map is still what Claude said it is running, so report it
verbatim; clients already render an unknown id as-is rather than guessing.

Reported by greptile on getpaseo#2659.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@colonelpanic8
colonelpanic8 force-pushed the subagent-param-visibility branch from d5e391f to deee9c4 Compare July 30, 2026 22:13
Comment thread packages/server/src/server/agent/providers/claude/agent.ts Outdated
colonelpanic8 added a commit to colonelpanic8/paseo that referenced this pull request Jul 30, 2026
The previous fix made runtimeInfo report an unmappable model verbatim, but
the timeline paths still stamped only normalized ids. An off-catalog model
therefore showed up on current-agent surfaces while completed-turn footers
and restored history silently lost it.

All three sites now share one rule — normalize when we can, keep the raw
identifier when we cannot — so live turns, coalesced messages, and
backfilled history attribute the same way.

Reported by greptile on getpaseo#2659.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
colonelpanic8 added a commit to colonelpanic8/paseo that referenced this pull request Jul 31, 2026
When Claude reported a model that normalizeClaudeRuntimeModelId could not
map to the catalog, the previously recognized model stayed in
runtimeInfo.model and the real identifier was only kept in
extra.runtimeModel — which no client reads. Every surface this PR adds would
then assert the agent was running a model it had switched away from.

An id we cannot map is still what Claude said it is running, so report it
verbatim; clients already render an unknown id as-is rather than guessing.

Reported by greptile on getpaseo#2659.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
colonelpanic8 added a commit to colonelpanic8/paseo that referenced this pull request Jul 31, 2026
The previous fix made runtimeInfo report an unmappable model verbatim, but
the timeline paths still stamped only normalized ids. An off-catalog model
therefore showed up on current-agent surfaces while completed-turn footers
and restored history silently lost it.

All three sites now share one rule — normalize when we can, keep the raw
identifier when we cannot — so live turns, coalesced messages, and
backfilled history attribute the same way.

Reported by greptile on getpaseo#2659.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@colonelpanic8
colonelpanic8 force-pushed the subagent-param-visibility branch from 9170eda to 425e5a5 Compare July 31, 2026 20:14
@colonelpanic8

Copy link
Copy Markdown
Contributor Author

Pushed 9915e10, a follow-up found by dogfooding the branch: the observed model was invisible for most of an agent's life.

  • The manager only refreshed runtimeInfo on thread_started / turn_completed, and Claude pre-assigns its session id so thread_started never fires for it — a running Claude agent (exactly what the subagents track shows) reported no model until its turn completed. Claude now dispatches model_changed on capture, the same push path ACP uses.
  • turn_failed / turn_canceled now refresh runtime info like turn_completed does.
  • The <synthetic> placeholder is no longer treated as an observation (one had reached a real agent record as its display model); all four attribution sites now go through resolveObservedClaudeModelId.
  • Loading a persisted transcript re-seeds the observation from the last real assistant entry, so a daemon restart no longer blanks the model until the next turn.

Verified with new regression tests in claude/agent.test.ts (70 pass), agent-manager.test.ts (143 pass), plus 139 across the related Claude provider/subagent files; typecheck and lint clean.

colonelpanic8 and others added 9 commits July 31, 2026 17:37
Paseo showed only the *selected* model and thinking level. Providers can
diverge from that selection — Codex collaboration modes inject their own
model/effort, models fall back mid-session — and subagent rows showed no
model at all, so there was no way to tell what an agent was really running.

The observed-vs-configured channel already existed end to end
(`runtimeInfo.model` / `runtimeInfo.thinkingOptionId` plus the daemon's
runtime-preferring `effectiveThinkingOptionId`). It was unevenly populated
and barely surfaced. No protocol changes were needed.

Server — populate runtimeInfo from data the runtimes already send:

- Codex: capture model and reasoningEffort from thread/start, thread/resume,
  and thread/turn notifications; normalize effort to catalog thinking ids.
  An omitted effort retains the last observation, an explicit null clears it.
- Claude: capture per-turn assistant message models, not just the SDK init
  model, and stop dropping the raw `extra.runtimeModel` after run().
- OpenCode: keep the observed assistant providerID/modelID that was already
  being resolved for context-window lookups and prefer it in runtime info.
- All three clear their observations on setModel/setThinkingOption, so a
  user's new selection is never shadowed by a stale observation.

App — show it, runtime-preferring, and never guess:

- Subagents track rows gain trailing "Model · Thinking" meta, and prefer the
  human task description over the provider's agent-type label.
- Desktop tab tooltips show "Claude · Opus 4.5 · High".
- The running-turn indicator and context window tooltip show model/thinking.
- The composer thinking chip reads out the runtime value while the dropdown
  keeps marking the configured one, so a mutation stays relative to config.
- The client stops discarding effectiveThinkingOptionId, preserving the
  load-bearing absent (daemon never sent it) vs explicit null distinction
  through the store and the offline replica cache.

Display never substitutes a guess for an unknown: an off-catalog runtime
model or thinking id renders verbatim rather than falling back to the
configured model or the first available option.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Knowing what an agent is running right now doesn't tell you what produced
the answer you're scrolling past. A turn from an hour ago may have run on a
different model — after a fallback, a collaboration-mode override, or a
deliberate switch — and nothing in the thread said so.

Assistant timeline items now carry the model and thinking level that
produced them, and completed turn footers read
"Worked for 21s · GPT-5.6-Sol · High".

Protocol: `model` and `thinkingOptionId` are optional additions to the
assistant_message timeline item. Old clients ignore them, old daemons omit
them, and absence renders nothing — so no capability gate is needed.

Server: each provider stamps only what it actually observed for that turn,
never the configured value, because a confident wrong attribution is worse
than none. The stream coalescer and timeline projection preserve the fields
while merging chunks. Claude and OpenCode also backfill history from their
persisted transcripts; Codex is forward-only, since its transcript exposes
no per-message model.

App: attribution survives the three paths that rebuild assistant items
(streaming append, block promotion, reducer replacement) and is add-only, so
a later chunk arriving bare cannot erase what the first chunk reported. A
turn that spans a model switch reports the model it ended on — the one the
footer sits under. Labels resolve per turn from the providers snapshot
rather than from the agent's current selection.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex reports its model and reasoning effort on the thread response and
usually omits them from turn notifications, so keying attribution solely on
per-turn observations left every real Codex turn unattributed. Fall back to
the thread-level observation, which Codex also reported, while still
preferring a turn's own value so a mid-session change lands on the right
turn.

Verified against a live Codex agent: turns now carry their model, and
switching models mid-thread leaves already-recorded turns untouched rather
than repainting history with the current selection.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
When Claude reported a model that normalizeClaudeRuntimeModelId could not
map to the catalog, the previously recognized model stayed in
runtimeInfo.model and the real identifier was only kept in
extra.runtimeModel — which no client reads. Every surface this PR adds would
then assert the agent was running a model it had switched away from.

An id we cannot map is still what Claude said it is running, so report it
verbatim; clients already render an unknown id as-is rather than guessing.

Reported by greptile on getpaseo#2659.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous fix made runtimeInfo report an unmappable model verbatim, but
the timeline paths still stamped only normalized ids. An off-catalog model
therefore showed up on current-agent surfaces while completed-turn footers
and restored history silently lost it.

All three sites now share one rule — normalize when we can, keep the raw
identifier when we cannot — so live turns, coalesced messages, and
backfilled history attribute the same way.

Reported by greptile on getpaseo#2659.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runtime model Claude reports was captured but invisible for most of an
agent's life. The manager only refreshes runtimeInfo on thread_started and
turn_completed, and Claude pre-assigns its session id, so thread_started
never fires for it: the observation surfaced only when a turn *completed*.
A running agent — exactly what the subagents track shows — reported no
model at all, and an interrupted turn never reported one.

- Claude now dispatches model_changed when it captures a runtime model, the
  same push path ACP uses, so the manager and clients see the observation
  the moment the first assistant chunk names it.
- turn_failed and turn_canceled refresh runtimeInfo like turn_completed
  does; an interrupted turn still observed a model before it stopped.
- The "<synthetic>" placeholder Claude Code stamps on frames with no real
  inference behind them is no longer treated as an observation; one had
  reached a real agent record as its display model. All four attribution
  sites now go through resolveObservedClaudeModelId, which the subagent
  sources already used.
- Loading a persisted transcript re-seeds the observation from the last
  real assistant entry, so a daemon restart no longer blanks the observed
  model until the next turn completes.

Also folds translateMessageToEvents' assistant and stream_event arms into
helpers, dropping its complexity below the lint ceiling.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completed turn footers reported the wrong model. `resolveTurnAttribution`
walked from the turn's start to the end of the *entire* stream and kept the
last attribution it saw, so every footer showed whichever model the most
recent message ran on. A session that started on Fable and later answered one
question on Sonnet relabelled the whole history "Sonnet 5".

It now walks the turn the way `collectAssistantTurnContent` does — from the
turn's last item toward its first, in the strategy's traversal direction,
stopping at the user message that opened the turn. Taking the traversal step
from the strategy also fixes inverted lists, where the old hardcoded `+= 1`
walked backwards through time.

The thinking level on those footers was fabricated outright. Nothing observed
one, so `resolveThinkingId` fell through to the model's default, which is
`effortLevels[0]` — always "low" for every Claude model. Recorded turns now
pass `thinkingFallback: "none"`: a turn reports what it ran at or nothing.
Live agent surfaces keep the model-default fallback, where it answers the
different question of what the configured model would think at.

That leaves a real value to report: Claude Code records `effort` on assistant
frames, as a sibling of `message` rather than a field of it — absent from
`SDKAssistantMessage`, so it is read defensively and stamped only when it
names a known level. Both the live path and transcript replay stamp it, so a
turn now reads "Sonnet 5 · Low" only when it genuinely ran low.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude's runtimeInfo carried a model but never a thinking level, so every live
surface — the subagents track, the composer chip, the tab tooltip — fell back
to the configured level and would keep claiming "High" through a silent
downgrade. That is the misreport this whole channel exists to prevent, left
open on Claude's thinking dimension because the level looked unobservable.

It is observable: Claude Code stamps `effort` on assistant frames. runtimeInfo
now reports the observed level, and drops it on setModel and setThinkingOption
so a stale observation cannot shadow a fresh selection or survive onto a model
with different effort levels. Transcript replay re-seeds it, as it does the
model.

The key is omitted rather than null when nothing has been observed: its
presence is what makes the runtime value outrank the configured one, so an
unobserved level has to leave the configured level in charge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The runtime-thinking wiring added two more optional chains to the
resolveAgentModelSelection call, pushing AgentControls to a complexity of
21 against oxlint's ceiling of 20. Lift the slice-to-selection mapping to
a module-level helper.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@colonelpanic8
colonelpanic8 force-pushed the subagent-param-visibility branch from 31bc146 to bb52162 Compare August 1, 2026 00:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant