Skip to content

fix: capture reasoning blocks from streaming responses - #815

Open
sjawhar wants to merge 1 commit into
LuthienResearch:mainfrom
trajectory-labs-pbc:capture-streaming-thinking
Open

sjawhar wants to merge 1 commit into
LuthienResearch:mainfrom
trajectory-labs-pbc:capture-streaming-thinking

Conversation

@sjawhar

@sjawhar sjawhar commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Capture thinking and redacted_thinking blocks when rebuilding a streamed response for conversation history.

_reconstruct_response_from_stream_events had branches for text and tool_use only; the line this replaces read # thinking blocks are intentionally excluded from history with no recorded rationale (its origin is not reachable — the line predates this clone's grafted root). A streamed call therefore recorded a complete-looking response holding no reasoning content, while the non-streaming path already stored the same blocks verbatim via an unfiltered model_dump(). History was inconsistent between the two transports depending only on whether the client streamed.

Change

  • thinking blocks accumulate thinking_delta text and record the signature_delta value. The signature arrives as a separate event from the text deltas, which is the failure mode a half-implementation would hit.
  • redacted_thinking blocks keep their opaque data payload — dropping it loses the only record that a reasoning block was present.
  • Block ordering is untouched, so reasoning blocks still precede visible text.

Why this is not a new data shape

Nothing under history/, debug/, ui/ or static/ references thinking at all. The event's consumers (passthrough_materialize/payloads.py, debug/service.py) handle the streaming and non-streaming events symmetrically, and already receive these blocks today from non-streaming calls. This makes streaming produce what non-streaming already produces.

Deliberately out of scope

The streaming path's record_inbound_response / record_outbound_response pass no body=, so request_logs.response_body stays NULL for streamed calls. Filling it would duplicate every streamed body into a second table under the 8 MB truncation rule, for no gain here. Separate concern.

Tests

Three cases added to TestReconstructResponseFromStreamEvents, written first and confirmed failing for the right reason (blocks dropped entirely — len([]) == 0):

  • test_captures_thinking_text_and_signature — multi-delta text accumulation plus the separately-arriving signature.
  • test_preserves_thinking_before_text_in_block_order — index ordering across mixed block types.
  • test_captures_redacted_thinking_payload — opaque payload preserved.

uv run pytest ...::TestReconstructResponseFromStreamEvents → 11/11 pass.

Coverage is unit-level only. The tests construct the SDK's own pydantic event models (RawContentBlockStartEvent, ThinkingDelta, SignatureDelta), so a wrong block shape fails construction rather than passing a hand-rolled dict. What they do not prove is the live streaming path reaching this function — that link is exercised by the existing text/tool_use cases. MockAnthropicServer emits only hardcoded text/tool_use blocks, so an end-to-end mock_e2e case needs a new mock response type; that is a follow-on rather than scope creep here.

Pre-existing failures, not from this change

scripts/dev_checks.sh reports three. They reproduce identically on unmodified main under xdist (uv run -m pytest -q --no-cov -n 4, which is what dev_checks.sh runs by default) and all pass single-process, so they are parallelism-sensitive and unrelated to this diff:

  • tests/luthien_cli/test_onboard.py::test_find_docker_ports_respects_env_vars
  • tests/luthien_proxy/unit_tests/test_config_registry.py::TestResolveDefaults::test_default_value_when_no_overrides
  • tests/luthien_proxy/unit_tests/test_config_registry.py::TestResolvePriority::test_db_ignored_for_non_db_settable

Changelog fragment: changelog.d/capture-streaming-thinking.md.

_reconstruct_response_from_stream_events rebuilt streamed responses for
conversation history with branches for text and tool_use only, so thinking
and redacted_thinking blocks were dropped. A streamed call therefore recorded
a complete-looking response holding no reasoning content, while the
non-streaming path stored the same blocks verbatim -- history was inconsistent
between the two transports.

Accumulate thinking_delta text and the signature_delta value into a thinking
block, and keep a redacted_thinking block's opaque data payload. Block
ordering is untouched, so reasoning blocks still precede visible text.

No consumer branches on block type, and the non-streaming path already emits
these blocks, so no downstream reader sees a new shape.

Omp-Session: 01a067a2-92a1-7000-85fe-5f7a3a650a38
@sjawhar
sjawhar marked this pull request as ready for review September 3, 2026 23:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant