Skip to content

assertions.md

Latest commit

 

History

History
199 lines (141 loc) · 4.65 KB

assertions.md

File metadata and controls

199 lines (141 loc) · 4.65 KB
Raw

PlayAgent Assertions DSL Reference

1) Overview

Tool-call assertions let you test agent behavior from trace data, not just final text output. You can assert which tools were called, in what order, with which parameters, and how many times.

2) Entry point: assert_trace(trace_id)

from playagent import assert_trace

assert_trace("sess_a1b2c3d4").tool_called("search")

assert_trace(trace_id) returns an AssertionChain object.

3) Methods

tool_called(tool_name: str) -> AssertionChain

Asserts the tool was called at least once.

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search")

Failure message format:

Expected tool 'search' to be called, but no tool calls were made in session sess_a1b2c3d4.

no_tool_called(tool_name: str) -> AssertionChain

Asserts the tool was never called.

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).no_tool_called("delete_file")

Failure message format:

Expected tool 'delete_file' to never be called, but it was called at turn(s) 3 in session sess_a1b2c3d4.

before(other_tool: str) -> AssertionChain

Asserts the last referenced tool was called before other_tool.

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search").before("summarize")

Failure message format:

Expected tool 'search' to be called before 'summarize', but 'search' was called at turn 2 and 'summarize' was called at turn 1 in session sess_a1b2c3d4.

with_param(param_name: str, matcher) -> AssertionChain

Asserts the last referenced tool has a parameter that matches.

from playagent import contains

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search").with_param("query", contains("invoice"))

Failure message format:

Expected tool 'search' param 'query' to contain 'invoice', but got: 'weather report'.

exactly(n: int) -> AssertionChain

Asserts the last referenced tool was called exactly n times.

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search").exactly(2)

Failure message format:

Expected tool 'search' to be called exactly 2 time(s), but found 1 in session sess_a1b2c3d4.

4) String matchers

contains(substring: str)

Checks substring in str(value).

from playagent import contains
trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search").with_param("query", contains("invoice"))

starts_with(prefix: str)

Checks str(value).startswith(prefix).

from playagent import starts_with
trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search").with_param("query", starts_with("invoice"))

matches(pattern: str)

Checks re.search(pattern, str(value)).

from playagent import matches
trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search").with_param("query", matches(r"invoice\s+\d+"))

equals(value: Any)

Checks exact equality with ==.

from playagent import equals
trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search").with_param("query", equals("invoice 123"))

5) Chaining

Combine multiple checks in one expression:

from playagent import assert_trace, contains

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id) \
    .tool_called("search") \
    .before("summarize") \
    .with_param("query", contains("invoice")) \
    .exactly(1)

6) AssertionFailedError

AssertionFailedError is raised when an assertion fails.

  • It is a custom exception type for PlayAgent assertion failures.
  • It includes actionable context (expected vs found + session id).
  • It is not a subclass of AssertionError.
from playagent import AssertionFailedError, assert_trace

trace_id = "sess_a1b2c3d4"
try:
    assert_trace(trace_id).tool_called("search")
except AssertionFailedError as exc:
    print(exc)

7) Common patterns

Pattern 1: Required tool call exists

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search")

Pattern 2: Destructive tool is never called

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).no_tool_called("delete_file")

Pattern 3: Retrieval before synthesis

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search").before("summarize")

Pattern 4: Query must include ticket id

from playagent import contains

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search").with_param("query", contains("ticket-"))

Pattern 5: Retry count is bounded

trace_id = "sess_a1b2c3d4"
assert_trace(trace_id).tool_called("search").exactly(3)