Skip to content

feat(runtime): publish provider-neutral agent events - #666

Open
winterfx wants to merge 2 commits into
mainfrom
feat/agent-structured-events
Open

feat(runtime): publish provider-neutral agent events#666
winterfx wants to merge 2 commits into
mainfrom
feat/agent-structured-events

Conversation

@winterfx

@winterfx winterfx commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

What

Every runner collapsed its provider's structured event stream into a plain text transcript, so tool calls, tool results and token accounting never reached consumers. Only codex forwarded raw SDK events, pushing a six-way parse onto every client; claude, opencode and pi emitted a single {type:"output"} text frame; gemini and dsh produced no agent events at all. Nothing ever produced ProjectRunEventKindAgentActivity.

This adds a neutral AgentEvent union and an optional onEvent sink on RunnerOptions, and maps all six providers onto it. AttachAgentEvent needs no proto change — payload_json already carries anything; the frame name becomes the kind and the projector reads that instead of codex's item shapes.

Three rules that are load-bearing

Measured against recorded runs, not assumed:

  • inputTokens always excludes cached tokens. codex and gemini report an inclusive prompt count upstream and now subtract, so the field means the same thing everywhere.
  • dsh publishes byte-identical usage twice (assistant/chunk and assistant/message); only the latter is mapped, or every count doubles.
  • claude and pi close a step before its tool events arrive, so consumers must group by the step field, never by the step_start/step_end interval.

Fixed while mapping

  • claude never handled the user-role message carrying tool_result, so every tool's output was invisible; its completed tool_call used the content-block index as id and could not correlate with its result.
  • codex swallowed the top-level ThreadErrorEvent — the guard for item-bearing events returned before reaching it.
  • interactive sessions dropped effort and skills entirely.

dsh

Three fixes to reach parity with the other providers:

  • model was mandatory. It rejected any agent not spelling out <llm-provider-id>/<model-name> while codex and claude fall back to the daemon default. Incidental: SplitDshModel ran before the shared default resolution could be reached.
  • wire protocol was pinned to chat completions, so every turn against a Responses provider went through conversion — which also carries whatever the upstream sends that the bridge does not model. This gateway emits its own codex.response.metadata and responsesapi.websocket_timing events, and the chat-completions encoder rendered them into the assistant's answer. The constant was inherited rather than chosen: the profile configured dsh-base's llm-deepseek row, whose Config has no protocol field. dsh-base also mounts llm-pi-ai, dormant until a profile supplies routes and able to name its protocol per route, so the guest now follows the provider and stays on the passthrough path.
  • its own bash/fs sandboxes enforced nothing inside agent-compose's sandbox but added a sandbox_permissions escalation argument to every tool schema. Nothing tells the model which mode it holds, so it asks for one narrower than the danger-full-access it already has and the call is rejected as "not strictly wider" — a wasted turn each time (deepseek-harness#468). The unconfined executors drop ctx.shell.sandboxMode, removing the argument from the schema.

Testing

  • Fixtures under runtime/javascript/test/fixtures/providers/ are real recorded runs of one prompt against each provider, kept as regression cover for provider field drift — codex sends a cache_write_input_tokens its own SDK type does not declare.
  • provider-event-mapping.test.ts replays them through the mappers and asserts the invariants above (tool call/result ids correlate, usage is not double counted, inputTokens excludes cache, absent kinds stay absent).
  • Verified live against all six providers. codex, claude, pi and dsh complete cleanly; gemini is still refused by the prompt-attach whitelist (it persists no thread id and would lose context between turns); opencode fails on a pre-existing issue where AgentConfigFromDefinition discards agent.Model for that provider.
  • Go and runtime suites pass with no new failures against main's baseline.

Deliberately not included

Two guards were prototyped and then dropped, because measurement did not support them:

  • A proxy-side filter for stream parts the bridge could not model. With dsh following the provider's protocol, no provider in this configuration reaches the chat-completions encoder that renders them into text; claude does bridge, but into the Anthropic encoder, whose type:"raw" events the client ignores. Six runs without the filter stayed clean. The underlying defect is in ai-api-protocol-bridge, where every encoder renders an unmodelled event with fmt.Sprint into a user-visible field, and belongs upstream.
  • An idempotence guard in the dsh runner plugin. A session-id collision seen during development was attributed to apply() running twice, but eighteen runs without the guard produced none, so the attribution was wrong and the change is unjustified.

The WebSocket chat UI used to compare providers side by side is held on a separate branch; it is a harness rather than product surface.

🤖 Generated with Claude Code

@winterfx
winterfx force-pushed the feat/agent-structured-events branch from 02a51e4 to 639bcf5 Compare September 4, 2026 06:09
Every runner collapsed its provider's structured event stream into a plain
text transcript, so tool calls, tool results and token accounting never
reached consumers. Only codex forwarded raw SDK events, which pushed a
six-way parse onto every client; claude, opencode and pi emitted a single
`{type:"output"}` text frame, and gemini and dsh produced no agent events at
all. Nothing ever produced ProjectRunEventKindAgentActivity.

Introduce a neutral AgentEvent union (agent-event.ts) and an optional
`onEvent` sink on RunnerOptions, then map all six providers onto it. A
provider that structurally cannot produce a kind emits nothing of that kind
rather than an empty placeholder, so consumers can tell "did not happen"
from "never reported". The frame name becomes the kind and the daemon's
projector reads that instead of codex's item shapes; AttachAgentEvent needs
no proto change, since payload_json already carries anything.

Three behaviours are load-bearing and were measured against recorded runs:

- inputTokens always excludes cached tokens. codex and gemini report an
  inclusive prompt count upstream and now subtract, so the field means the
  same thing across providers.
- dsh publishes byte-identical usage twice (assistant/chunk and
  assistant/message); only the latter is mapped, or every count doubles.
- claude and pi close a step before its tool events arrive, so consumers
  must group by the `step` field, never by the step_start/step_end interval.

Fixed while mapping:

- claude never handled the user-role message that carries tool_result, so
  every tool's output was invisible; its completed tool_call used the
  content-block index as id and could not correlate with its result.
- codex swallowed the top-level ThreadErrorEvent, because the guard for
  item-bearing events returned before reaching it.
- interactive sessions dropped `effort` and `skills` entirely.

dsh needed three fixes to reach parity:

- it rejected any agent that did not spell out <llm-provider-id>/<model-name>
  while codex and claude fall back to the daemon default; the requirement was
  incidental, since SplitDshModel ran before the shared default resolution
  could be reached.
- its facade pinned chat completions, so every turn against a Responses
  provider went through protocol conversion — which also carries whatever the
  upstream sends that the bridge does not model, and this gateway's own
  codex.response.metadata and responsesapi.websocket_timing events reached
  the guest as assistant text. That constant was inherited, not chosen: the
  profile configured dsh-base's llm-deepseek row, whose Config has no protocol
  field. dsh-base also mounts llm-pi-ai, dormant until a profile supplies
  routes and able to name its protocol per route, so the guest now follows the
  provider and stays on the passthrough path.
- its profile kept dsh's own bash and fs sandboxes, which enforce nothing
  extra inside agent-compose's sandbox but do add a `sandbox_permissions`
  escalation argument to every tool schema. Nothing tells the model which
  mode it holds, so it asks for one narrower than the danger-full-access it
  already has and the call is rejected as "not strictly wider" — a wasted
  turn each time (deepseek-harness#468). The unconfined executors drop
  ctx.shell.sandboxMode, which removes the argument from the schema.

Fixtures are real recorded runs of one prompt against each provider, kept as
regression cover for provider field drift — codex sends a
cache_write_input_tokens its own SDK type does not declare.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R46kvDYupMBjXvJHaHoinr
@winterfx
winterfx force-pushed the feat/agent-structured-events branch from 639bcf5 to 29d0b86 Compare September 4, 2026 06:56
@winterfx

winterfx commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

@monkeyscan review这个pr

@winterfx
winterfx marked this pull request as ready for review September 4, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant