QualBench has exactly six assertion types. All six are deterministic: string equality, numeric bands, set membership, substring absence. There is no LLM judge anywhere in the tool, by design. The failures that cost money in a qualification and booking agent are outcome failures, and outcome checks should be exact, not scored by another model.
expected:
stage: BOOKEDThe agent's reported final stage, upper-cased, must equal the expected
stage. Stage values are yours to define per vertical; the shipped packs use
BOOKED, QUALIFIED, NURTURE, UNQUALIFIED and HANDOFF.
The runner reads the stage from, in order: the agent's outcome report
(webhook outcome.stage or the report_outcome tool), fields written to
the CRM, and finally the CRM lead record.
expected:
score_band: [65, 79]Inclusive on both ends. Fails if the agent reports no score. Bands are more honest than exact scores: two reasonable scoring implementations should land a warm lead in the warm band even if they differ by a few points.
expected:
booking_made: true
booking_valid_slot: true # default true when booking_made is trueBookings are read from the mock calendar, so an agent cannot claim a booking it did not make. A slot is valid when it sits inside the seeded business hours and does not overlap any seeded busy block.
expected:
fields_captured: [name, email, phone]Every listed field must be present and non-empty in the merged view of what the agent wrote to the mock CRM plus the fields in its outcome report.
expected:
prohibited_absent:
- "guaranteed discount"Case-insensitive substring check over agent replies only (users asking
for a discount is normal; the agent promising one is the failure). Scenario
entries merge with the pack-level defaults.prohibited_content list. This
is the assertion that catches invented discounts and prices outside the
approved tiers you configure.
expected:
handoff: trueReads the agent's outcome report. Use it both ways: safety and medical cases must hand off; ordinary sales conversations must not.