Skip to content

Agent: exit_reason names a tool whose result is not last_message when tool calls run in parallel #12890

Description

@lets-order-some-fries

Describe the bug

Agent.run documents exit_reason as, verbatim:

the name of the tool that satisfied a tool exit condition (in which case last_message is that tool's result)

When the model requests several tools in one assistant message and the exit-condition tool is not the one it listed last, that promise does not hold: exit_reason names tool A while last_message carries tool B's result.

_check_exit_conditions is deliberately order-agnostic — its own docstring says "Every tool call in the message is checked, so the order of parallel tool calls does not matter" — but last_message is simply messages[-1], and tool result messages are appended in the order the model requested them. The two are only consistent by luck.

This matters because the docs advertise exit_reason for exactly this purpose: "useful for routing the output downstream (e.g. with a ConditionalRouter)". A router that branches on exit_reason == "get_weather" and then reads last_message silently consumes a different tool's output — no error, no warning.

Expected behavior

Either last_message is the exit tool's result as documented, or the documentation says how to actually find it (by origin.tool_name among the step's tool messages).

To Reproduce

from typing import Any, Optional
from haystack import component
from haystack.components.agents import Agent
from haystack.dataclasses import ChatMessage, ToolCall
from haystack.tools import Tool


@component
class FakeChatGenerator:
    """Returns one assistant message requesting two tools, then a plain reply."""

    def __init__(self) -> None:
        self.calls = 0

    @component.output_types(replies=list[ChatMessage])
    def run(self, messages: list[ChatMessage], tools: Optional[Any] = None, **kwargs: Any) -> dict[str, Any]:
        self.calls += 1
        if self.calls == 1:
            return {"replies": [ChatMessage.from_assistant(tool_calls=[
                ToolCall(id="call_1", tool_name="get_weather", arguments={"city": "Berlin"}),
                ToolCall(id="call_2", tool_name="add_to_calendar", arguments={"title": "trip"}),
            ])]}
        return {"replies": [ChatMessage.from_assistant("done")]}

    def to_dict(self) -> dict[str, Any]: return {"type": "FakeChatGenerator", "init_parameters": {}}
    @classmethod
    def from_dict(cls, data: dict[str, Any]) -> "FakeChatGenerator": return cls()


weather = Tool(name="get_weather", description="weather", function=lambda city: f"22C and sunny in {city}",
               parameters={"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]})
calendar = Tool(name="add_to_calendar", description="calendar", function=lambda title: "event created",
                parameters={"type": "object", "properties": {"title": {"type": "string"}}, "required": ["title"]})

agent = Agent(chat_generator=FakeChatGenerator(), tools=[weather, calendar], exit_conditions=["get_weather"])
agent.warm_up()
result = agent.run([ChatMessage.from_user("Weather in Berlin, and add a calendar entry")])

last = result["last_message"]
print("exit_reason               :", result["exit_reason"])
print("last_message produced by  :", last.tool_call_result.origin.tool_name)
print("last_message.result       :", last.tool_call_result.result)
print("last_message.text         :", repr(last.text))

Output on main (4505df3):

exit_reason               : get_weather
last_message produced by  : add_to_calendar
last_message.result       : event created
last_message.text         : None

Reproduces identically through Agent.run_async.

Additional context

The same claim appears in three more places, and one of them is independently inaccurate:

  • haystack/components/agents/agent.py:869 and :955 — the :returns: clause quoted above.
  • docs-website/docs/pipeline-components/agents-1/agent.mdx:69 — a stronger form: "In this case last_message is that tool's result — a tool-result ChatMessage whose text is empty — so exit_reason tells you how to consume it." text is None on a tool-result message, not empty, as the repro above shows.
  • test/components/agents/test_agent.py:1114 — a comment restating the same belief.

test_tool_exit_reports_the_first_matching_tool already builds exactly this parallel-call shape, but asserts only exit_reason and never last_message — which is why this went unnoticed.

Two ways to resolve it, and I'm happy to send either

  1. Documentation — correct the four places and say that the exit tool's result is found via tool_call_result.origin.tool_name among the step's tool messages. No behaviour change.
  2. Behaviour — have _check_exit_conditions report the exit-condition tool whose result is last among the step's tool messages, making the documented promise true. This changes which tool is reported when two exit-condition tools fire in one step, which test_tool_exit_reports_the_first_matching_tool currently pins.

I'd default to (1), because on #12621 @anakin87 raised the same preference for a neighbouring case — "just update the docstring to describe the actual behavior instead of introducing new complex logic". @sjrl, you added exit_reason in #12074, so you may have a view on whether the promise was meant to hold.

I'll open a PR for option (1) right after this; glad to switch it to (2) instead if you'd rather the behaviour changed.

FAQ Check

System:

  • OS: macOS 15 (arm64)
  • Haystack version: main @ 4505df3 (3.3.0-rc0)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Medium priority, add to the next sprint if no P1 available

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions