fix(haystack): keep output message content on spans - #3760
Harsh23Kashyap wants to merge 1 commit into
Conversation
feiiiiii5
left a comment
There was a problem hiding this comment.
Checked the two claims this PR rests on from the Haystack side rather than from the diff. Both hold; one reviewer question that turns out to be a non-issue, recorded so nobody re-derives it.
The producer path is real (not a synthetic fixture). The PR description says Haystack's own streaming conversion yields a reply whose meta has no finish_reason. That is in haystack/components/generators/utils.py:172-174,189:
finish_reasons = [chunk.finish_reason for chunk in chunks if chunk.finish_reason]
finish_reason = finish_reasons[-1] if finish_reasons else None
...
"finish_reason": finish_reason,So a streamed reply from any provider that never sets finish_reason arrives with meta["finish_reason"] = None, and the removed guard (if ... (finish_reason := reply_meta.get("finish_reason")) is None: continue) discarded the whole reply — content and role — before the loop ever reached the tool_calls branch. That matches the issue, so the finish_reason gate removal is fixing a supported path rather than hardening a hypothetical one.
No empty content is emitted for tool-call-only replies. Checked against Haystack 3.2.0-rc0:
text+tool_calls .text='Let me check.' is_not_None=True tool_calls=1
tool_calls only .text=None is_not_None=False tool_calls=1
so if (text := reply.text) is not None yields content exactly for the #3759 case 1 reply (text plus tool calls, previously tool-calls-only) and skips the tool-call-only case. This also fixes a latent problem in the old else branch, which yielded MESSAGE_CONTENT, reply.text unconditionally and could hand None to the attribute for a finish_reason of e.g. stop with no text.
The asymmetry I suspected is not a bug. The new output path yields reply.role.value, while the input path at _wrappers.py:517-518 yields the raw enum. ChatRole is class ChatRole(str, Enum), OTel stores the instance, and it serializes as "user":
span attr type: ChatRole value: <ChatRole.USER: 'user'>
json.dumps({role: ChatRole.USER}) -> {"role": "user"}
So both spellings export the same string; no change needed here, just noting it because the inconsistency looks like a finding at a glance.
Two things I could not verify locally, so treat them as untested rather than as passing. I could not run this package's suite in my environment: collection of tests/openinference/haystack/test_instrumentor.py needs haystack_integrations (Cohere ranker import) and PyPI is timing out from here, so the PR's "65 passed, 5 skipped" is the author's number, not one I reproduced. And I did not check the n > 1 case against a live provider — only that the .0. → .{reply_index}. change makes multiple string replies addressable.
For the record, the remaining .0. literals in this file (_wrappers.py:521-523 prompt branch, :759-767 single-text embedder branch) are single-value elif arms, not loops, so they are correct as written.
Fixes #3759
_get_llm_output_message_attributesgated everything onfinish_reasonand indexed string replies with a hardcoded0, so reply content went missing from spans:finish_reasonwas skipped entirely (haystack's own streaming conversion produces exactly that when no chunk reports one)The extraction now records every reply under its own index: text and role always, tool calls whenever the reply carries them, regardless of
finish_reason.Regression test: a pipeline with a fake chat generator (text + tool call reply, reply without
finish_reason) and a fake text generator (two string replies), asserting each reply lands under its own index with its content. Fails before, passes after. Full haystack suite: 65 passed, 5 skipped (version/env gates), no changes needed to the existing cassette tests.