feat(runtime): publish provider-neutral agent events - #666
Open
winterfx wants to merge 2 commits into
Open
Conversation
winterfx
force-pushed
the
feat/agent-structured-events
branch
from
September 4, 2026 06:09
02a51e4 to
639bcf5
Compare
Every runner collapsed its provider's structured event stream into a plain
text transcript, so tool calls, tool results and token accounting never
reached consumers. Only codex forwarded raw SDK events, which pushed a
six-way parse onto every client; claude, opencode and pi emitted a single
`{type:"output"}` text frame, and gemini and dsh produced no agent events at
all. Nothing ever produced ProjectRunEventKindAgentActivity.
Introduce a neutral AgentEvent union (agent-event.ts) and an optional
`onEvent` sink on RunnerOptions, then map all six providers onto it. A
provider that structurally cannot produce a kind emits nothing of that kind
rather than an empty placeholder, so consumers can tell "did not happen"
from "never reported". The frame name becomes the kind and the daemon's
projector reads that instead of codex's item shapes; AttachAgentEvent needs
no proto change, since payload_json already carries anything.
Three behaviours are load-bearing and were measured against recorded runs:
- inputTokens always excludes cached tokens. codex and gemini report an
inclusive prompt count upstream and now subtract, so the field means the
same thing across providers.
- dsh publishes byte-identical usage twice (assistant/chunk and
assistant/message); only the latter is mapped, or every count doubles.
- claude and pi close a step before its tool events arrive, so consumers
must group by the `step` field, never by the step_start/step_end interval.
Fixed while mapping:
- claude never handled the user-role message that carries tool_result, so
every tool's output was invisible; its completed tool_call used the
content-block index as id and could not correlate with its result.
- codex swallowed the top-level ThreadErrorEvent, because the guard for
item-bearing events returned before reaching it.
- interactive sessions dropped `effort` and `skills` entirely.
dsh needed three fixes to reach parity:
- it rejected any agent that did not spell out <llm-provider-id>/<model-name>
while codex and claude fall back to the daemon default; the requirement was
incidental, since SplitDshModel ran before the shared default resolution
could be reached.
- its facade pinned chat completions, so every turn against a Responses
provider went through protocol conversion — which also carries whatever the
upstream sends that the bridge does not model, and this gateway's own
codex.response.metadata and responsesapi.websocket_timing events reached
the guest as assistant text. That constant was inherited, not chosen: the
profile configured dsh-base's llm-deepseek row, whose Config has no protocol
field. dsh-base also mounts llm-pi-ai, dormant until a profile supplies
routes and able to name its protocol per route, so the guest now follows the
provider and stays on the passthrough path.
- its profile kept dsh's own bash and fs sandboxes, which enforce nothing
extra inside agent-compose's sandbox but do add a `sandbox_permissions`
escalation argument to every tool schema. Nothing tells the model which
mode it holds, so it asks for one narrower than the danger-full-access it
already has and the call is rejected as "not strictly wider" — a wasted
turn each time (deepseek-harness#468). The unconfined executors drop
ctx.shell.sandboxMode, which removes the argument from the schema.
Fixtures are real recorded runs of one prompt against each provider, kept as
regression cover for provider field drift — codex sends a
cache_write_input_tokens its own SDK type does not declare.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R46kvDYupMBjXvJHaHoinr
winterfx
force-pushed
the
feat/agent-structured-events
branch
from
September 4, 2026 06:56
639bcf5 to
29d0b86
Compare
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R46kvDYupMBjXvJHaHoinr
Collaborator
Author
|
@monkeyscan review这个pr |
winterfx
marked this pull request as ready for review
September 4, 2026 10:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every runner collapsed its provider's structured event stream into a plain text transcript, so tool calls, tool results and token accounting never reached consumers. Only codex forwarded raw SDK events, pushing a six-way parse onto every client; claude, opencode and pi emitted a single
{type:"output"}text frame; gemini and dsh produced no agent events at all. Nothing ever producedProjectRunEventKindAgentActivity.This adds a neutral
AgentEventunion and an optionalonEventsink onRunnerOptions, and maps all six providers onto it.AttachAgentEventneeds no proto change —payload_jsonalready carries anything; the frame name becomes the kind and the projector reads that instead of codex's item shapes.Three rules that are load-bearing
Measured against recorded runs, not assumed:
inputTokensalways excludes cached tokens. codex and gemini report an inclusive prompt count upstream and now subtract, so the field means the same thing everywhere.assistant/chunkandassistant/message); only the latter is mapped, or every count doubles.stepfield, never by thestep_start/step_endinterval.Fixed while mapping
tool_result, so every tool's output was invisible; its completedtool_callused the content-block index as id and could not correlate with its result.ThreadErrorEvent— the guard for item-bearing events returned before reaching it.effortandskillsentirely.dsh
Three fixes to reach parity with the other providers:
<llm-provider-id>/<model-name>while codex and claude fall back to the daemon default. Incidental:SplitDshModelran before the shared default resolution could be reached.codex.response.metadataandresponsesapi.websocket_timingevents, and the chat-completions encoder rendered them into the assistant's answer. The constant was inherited rather than chosen: the profile configured dsh-base'sllm-deepseekrow, whose Config has no protocol field. dsh-base also mountsllm-pi-ai, dormant until a profile supplies routes and able to name its protocol per route, so the guest now follows the provider and stays on the passthrough path.sandbox_permissionsescalation argument to every tool schema. Nothing tells the model which mode it holds, so it asks for one narrower than thedanger-full-accessit already has and the call is rejected as "not strictly wider" — a wasted turn each time (deepseek-harness#468). The unconfined executors dropctx.shell.sandboxMode, removing the argument from the schema.Testing
runtime/javascript/test/fixtures/providers/are real recorded runs of one prompt against each provider, kept as regression cover for provider field drift — codex sends acache_write_input_tokensits own SDK type does not declare.provider-event-mapping.test.tsreplays them through the mappers and asserts the invariants above (tool call/result ids correlate, usage is not double counted,inputTokensexcludes cache, absent kinds stay absent).AgentConfigFromDefinitiondiscardsagent.Modelfor that provider.Deliberately not included
Two guards were prototyped and then dropped, because measurement did not support them:
type:"raw"events the client ignores. Six runs without the filter stayed clean. The underlying defect is inai-api-protocol-bridge, where every encoder renders an unmodelled event withfmt.Sprintinto a user-visible field, and belongs upstream.apply()running twice, but eighteen runs without the guard produced none, so the attribution was wrong and the change is unjustified.The WebSocket chat UI used to compare providers side by side is held on a separate branch; it is a harness rather than product surface.
🤖 Generated with Claude Code