Skip to content

fix: handle mixed-type lists in _flatten across instrumentors; isolate dspy extraction errors - #3728

Open
Harsh23Kashyap wants to merge 3 commits into
Arize-ai:mainfrom
Harsh23Kashyap:fix/dspy-extraction-isolation
Open

Harsh23Kashyap wants to merge 3 commits into
Arize-ai:mainfrom
Harsh23Kashyap:fix/dspy-extraction-isolation

Conversation

@Harsh23Kashyap

@Harsh23Kashyap Harsh23Kashyap commented Sep 13, 2026

Copy link
Copy Markdown

Description

Resolves #3727

_flatten has the same defect in nine instrumentors: for a list that mixes mappings with other types it takes the mapping branch and then recurses into every item, so a mixed-type content list (a normal shape for vision and tool messages) breaks attribute extraction. The visible impact differs by package:

  • instructor: AttributeError raised into user code from the patched call, before the wrapped function runs.
  • dspy: the same AttributeError into user code from LM spans.
  • llama-index: no crash, but the flattening failure is caught and logged, so the event loses every attribute.
  • autogen-agentchat, guardrails: the mixed-list path is not reachable from their current call sites; fixed for consistency so future call sites inherit the safe behavior.
  • langchain: also not currently reachable - message content is pre-flattened by _extract_message_kwargs before it reaches _flatten, and other structured payloads (document metadata, prompts) are JSON-serialized first. Fixed for consistency.
  • google-genai, portkey, groq: these copies of _flatten are unused today; fixed for consistency.

The dspy instrumentor had a second problem: no extractor had error isolation, so a circular reference in the inputs surfaced as ValueError: Circular reference detected from safe_json_dumps in user code. litellm got the same class of fix in #3578.

Changes:

  • _flatten recurses only into mapping items of a list; non-mapping parts pass through indexed. Output for all-mapping and all-scalar lists is unchanged. Same edit in all nine copies; every _flatten in the tree is now a safe copy.
  • dspy generator extractors get _suppress_extractor_errors, the same shape as litellm's: a failing extractor drops its attributes with a logged warning while the span and the wrapped call proceed. _get_input_value and _module_prediction_output_attributes fall back to a safe value on failure.

Verification:

  • One new regression test per package; each fails on main and passes with the fix.
  • instructor: end-to-end through _PatchWrapper with a fake create function. On main the mixed content list crashes with AttributeError: 'str' object has no attribute 'items'; with the fix the call completes and both content parts land on the span. Ran twice identical both ways.
  • llama-index: end-to-end through _finish_tracing with a fabricated event carrying the mixed list. On main the event keeps none of its attributes; with the fix the model name and both content parts survive. Ran twice identical both ways.
  • langchain: end-to-end through _update_span with a real HumanMessage carrying a mixed content list - no crash on either side, because langchain-core and _extract_message_kwargs normalize and pre-flatten content before _flatten runs. That run is what established the not-reachable status above.
  • dspy: the repro scripts from the issue run twice identical before (AttributeError / ValueError into user code) and after (the call reaches the LM).
  • Package suites: autogen-agentchat 7 passed, google-genai 122 passed, groq 12 passed, llama-index test_callback.py 14 passed and 4 xfailed, langchain 231 passed and 2 skipped. instructor, guardrails, portkey, and langchain each show a small set of pre-existing failures in this environment; the failure sets are identical on unmodified main.
  • ruff check and ruff format clean on the changed files.

Checklist:

  • Properly respects suppress tracing context (the suppression check at wrapper entry is untouched; the guards only affect attribute extraction)

@github-actions

github-actions Bot commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

Apply the same mixed-list handling added for dspy to the identical
_flatten copies in autogen-agentchat, google-genai, guardrails,
instructor, and llama-index: recurse only into Mapping items and pass
non-mapping items through with an indexed key.

- instructor: fixes an AttributeError raised into user calls when a
  message content list mixes dict parts and plain strings
- llama-index: fixes silent loss of every attribute on an event whose
  payload carries the same mixed-list shape
- autogen-agentchat, guardrails, google-genai: consistency fixes; the
  mixed-list path is not reachable from their current call sites (the
  google-genai copy is unused today), so these prevent the bug rather
  than fix a live one

Each package gains a regression test covering the mixed-type list.
@Harsh23Kashyap Harsh23Kashyap changed the title fix(dspy): isolate attribute extraction errors from user code fix: handle mixed-type lists in _flatten across instrumentors; isolate dspy extraction errors Sep 13, 2026
Extend the mixed-list handling to the remaining three defective
_flatten copies in the repo. With this commit every _flatten copy in
the tree recurses only into Mapping items and passes non-mapping items
through with an indexed key.

None of the three copies is reachable from current call sites:
langchain's message-content path is pre-flattened by
_extract_message_kwargs before it reaches _flatten, and the portkey
and groq copies have no call sites at all. These are consistency fixes
so future call sites inherit the safe behavior.

Each package gains a regression test covering the mixed-type list.
@Harsh23Kashyap

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Sep 13, 2026
@caroger
caroger self-requested a review September 17, 2026 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

[bug] DSPy instrumentor raises into user code from attribute extraction (_flatten mixed lists, unguarded extractors)

1 participant