Skip to content

fix(haystack): keep output message content on spans - #3760

Open
Harsh23Kashyap wants to merge 1 commit into
Arize-ai:mainfrom
Harsh23Kashyap:fix/haystack-output-message-content
Open

Harsh23Kashyap wants to merge 1 commit into
Arize-ai:mainfrom
Harsh23Kashyap:fix/haystack-output-message-content

Conversation

@Harsh23Kashyap

Copy link
Copy Markdown

Fixes #3759

_get_llm_output_message_attributes gated everything on finish_reason and indexed string replies with a hardcoded 0, so reply content went missing from spans:

  • a reply with both text and tool calls kept only the tool calls
  • a reply whose meta has no finish_reason was skipped entirely (haystack's own streaming conversion produces exactly that when no chunk reports one)
  • multiple string replies overwrote each other at index 0

The extraction now records every reply under its own index: text and role always, tool calls whenever the reply carries them, regardless of finish_reason.

Regression test: a pipeline with a fake chat generator (text + tool call reply, reply without finish_reason) and a fake text generator (two string replies), asserting each reply lands under its own index with its content. Fails before, passes after. Full haystack suite: 65 passed, 5 skipped (version/env gates), no changes needed to the existing cassette tests.

@feiiiiii5 feiiiiii5 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the two claims this PR rests on from the Haystack side rather than from the diff. Both hold; one reviewer question that turns out to be a non-issue, recorded so nobody re-derives it.

The producer path is real (not a synthetic fixture). The PR description says Haystack's own streaming conversion yields a reply whose meta has no finish_reason. That is in haystack/components/generators/utils.py:172-174,189:

finish_reasons = [chunk.finish_reason for chunk in chunks if chunk.finish_reason]
finish_reason = finish_reasons[-1] if finish_reasons else None
...
"finish_reason": finish_reason,

So a streamed reply from any provider that never sets finish_reason arrives with meta["finish_reason"] = None, and the removed guard (if ... (finish_reason := reply_meta.get("finish_reason")) is None: continue) discarded the whole reply — content and role — before the loop ever reached the tool_calls branch. That matches the issue, so the finish_reason gate removal is fixing a supported path rather than hardening a hypothetical one.

No empty content is emitted for tool-call-only replies. Checked against Haystack 3.2.0-rc0:

text+tool_calls    .text='Let me check.'   is_not_None=True    tool_calls=1
tool_calls only    .text=None              is_not_None=False   tool_calls=1

so if (text := reply.text) is not None yields content exactly for the #3759 case 1 reply (text plus tool calls, previously tool-calls-only) and skips the tool-call-only case. This also fixes a latent problem in the old else branch, which yielded MESSAGE_CONTENT, reply.text unconditionally and could hand None to the attribute for a finish_reason of e.g. stop with no text.

The asymmetry I suspected is not a bug. The new output path yields reply.role.value, while the input path at _wrappers.py:517-518 yields the raw enum. ChatRole is class ChatRole(str, Enum), OTel stores the instance, and it serializes as "user":

span attr type: ChatRole  value: <ChatRole.USER: 'user'>
json.dumps({role: ChatRole.USER}) -> {"role": "user"}

So both spellings export the same string; no change needed here, just noting it because the inconsistency looks like a finding at a glance.

Two things I could not verify locally, so treat them as untested rather than as passing. I could not run this package's suite in my environment: collection of tests/openinference/haystack/test_instrumentor.py needs haystack_integrations (Cohere ranker import) and PyPI is timing out from here, so the PR's "65 passed, 5 skipped" is the author's number, not one I reproduced. And I did not check the n > 1 case against a live provider — only that the .0..{reply_index}. change makes multiple string replies addressable.

For the record, the remaining .0. literals in this file (_wrappers.py:521-523 prompt branch, :759-767 single-text embedder branch) are single-value elif arms, not loops, so they are correct as written.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

[bug] haystack instrumentor drops assistant reply content from spans

2 participants