feat: add schema-constrained structured outputs and recovery loop to Agent

[rautaditya2606]

Related Issues

• fixes Support schema-constrained structured outputs with automatic recovery in Agent #12989

Proposed Changes

Adds response_schema and max_schema_retries parameters to Agent, enabling schema-constrained output validation and automatic self-correction for terminal agent replies:

• Provider-agnostic post-hoc validation: Validates final assistant text completions against a Pydantic BaseModel subclass or JSON Schema dictionary without coupling to provider-specific client parameters.
• Automatic recovery loop: When the LLM generates invalid JSON or fails schema validation, the agent appends a targeted correction message (ChatMessage.from_user) containing the schema and error description, re-looping up to max_schema_retries times within the agent's overall max_agent_steps budget.
• Fail-fast initialization: Validates JSON schema dictionaries at construction time via Draft202012Validator.check_schema, raising ValueError on malformed schemas before running.
• Markdown fence handling: Pre-extracts JSON from markdown code blocks (json ... ) before validation to prevent unnecessary retries on formatted responses.
• Terminal gating: Gated strictly on model_exit_reason == _EXIT_REASON_TEXT, ensuring schema validation never fires on unexecuted tool-call messages when no tools are present.
• State & Pipeline typing: Registers "structured_output" in resolved_state_schema and output sockets (type: response_schema | None) while ensuring it is excluded from input sockets.
• Serialization & clone: Added serialization/deserialization support for Pydantic type paths and JSON schema dictionaries in to_dict, from_dict, and clone.

How did you test it?

Added TestAgentStructuredOutput in test/components/agents/test_agent.py covering:

• Init-time validation (type check, negative retries, reserved state key collision, and malformed schema check).
• Clean Pydantic model and JSON Schema dictionary extraction.
• Markdown-fenced JSON responses.
• Recovery loop on invalid JSON string followed by valid response.
• Recovery loop on schema mismatch followed by valid response.
• Exceeded retry budget returning structured_output=None without unhandled exceptions.
• max_agent_steps cutoff stopping recovery.
• Gating check: tool call with empty tools does not trigger spurious schema validation.
• run_async parity with sync run.
• Backward compatibility when response_schema=None.
• to_dict, from_dict, and clone serde roundtrips.

Quality checks run:

• hatch run fmt (clean)
• hatch run test:types haystack/components/agents/agent.py test/components/agents/test_agent.py (clean)
• hatch -e test run pytest test/components/agents/test_agent.py (138 passed)

Notes for the reviewer

The validation and retry logic is completely isolated to haystack/components/agents/agent.py. When response_schema is omitted, the execution path and output dictionary remain byte-identical to existing behavior.

Note: when response_schema is a Pydantic model, structured_output holds a live BaseModel instance in agent state. This hasn't been tested against Agent's breakpoint/snapshot resume path — if that machinery serializes arbitrary state values mid-run, a Pydantic instance may not round-trip the same way plain dicts do. Dict-schema mode is unaffected. Flagging as untested rather than claiming full compatibility.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: feat:
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@rautaditya2606 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/agents
agent.py					219, 227-231, 240-244, 1288
Project Total

This report was generated by python-coverage-comment-action

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[anakin87]

Thank you!

See #12989 (comment)

No changes are needed from your side.

We'll follow up in the issue/here if we decide to implement this feature.