Agent Failure Discovery - A framework for analyzing AI agent transcripts to automatically discover error patterns and build comprehensive error taxonomies.
This framework helps you identify, categorize, and track errors in AI agent behavior by analyzing interaction transcripts. It uses an LLM-driven discovery pipeline that:
- Analyzes agent transcripts against policy documents and ground truth data
- Discovers new error types when failures don't match existing patterns
- Builds error taxonomies dynamically as more transcripts are processed
- Tracks error evolution across your agent's development lifecycle
- Automated Error Discovery: LLM identifies failures and proposes new error types when patterns are meaningfully different
- Dynamic Taxonomy Building: Error registry grows organically as new failure modes are discovered
- Policy-Aware Analysis: Evaluates agent behavior against your specific policy documents
- Flexible Dimensions: Supports both pre-defined and custom error dimensions
- Transparent Tracking: Full audit trail of when and where each error type was discovered
- JSON-Based: Simple JSON input/output format for easy integration
pip install AgentFailureDiscoverygit clone https://github.com/AmadeusITGroup/AgentFailureDiscovery.git
cd AgentFailureDiscovery
pip install -e .Create a .env file with your Azure OpenAI credentials:
cp .env.example .envEdit .env and add your credentials:
AZURE_OPENAI_API_KEY=your-api-key-here
AZURE_OPENAI_ENDPOINT=https://your-resource-name.openai.azure.com
AZURE_OPENAI_API_VERSION=2024-05-01-previewCreate a JSON file with your tasks and agent simulations:
{
"tasks": [
{
"id": "task_001",
"description": "User requests account balance",
"expected_behavior": "Agent checks balance using get_balance tool"
}
],
"simulations": [
{
"task_id": "task_001",
"trial": 0,
"transcript": {
"turns": [
{"role": "user", "content": "What's my balance?"},
{"role": "assistant", "content": "Your balance is $500", "tool_calls": []}
]
},
"reward": 0.0
}
]
}errordiscovery data.json --policy policy.md --output-dir ./resultsThe framework generates:
results/registry.json- Error type taxonomy with all discovered patternsresults/run_output.json- Detailed analysis results for each transcript
errordiscovery <json_file> [OPTIONS]Required Arguments:
json_file- Path to JSON file containing tasks and simulations
Optional Arguments:
--policy PATH- Path to policy file (markdown or text)--output-dir PATH- Output directory for results (default:./output)--model NAME- LLM model deployment name (default:gpt-5)--env-file PATH- Path to .env file (default:.env)--version- Show version and exit
Example:
errordiscovery agent_transcripts.json \
--policy my_policy.md \
--output-dir ./analysis_results \
--model gpt-5from errordiscovery import run_discovery
from errordiscovery.utils import load_json_data, load_policy_file, save_registry
from dotenv import load_dotenv
# Load environment
load_dotenv()
# Load your data
tasks, simulations = load_json_data("data.json")
policy = load_policy_file("policy.md")
# Run discovery
output = run_discovery(
policy=policy,
tasks=tasks,
simulations=simulations,
model="gpt-5"
)
# Save results
save_registry(output["registry"], "registry.json")
print(output["registry_summary"])Your input JSON should contain two main sections:
{
"tasks": [
{
"id": "unique_task_id",
"description": "task description",
"expected_behavior": "what should happen",
"constraints": "any specific constraints",
...
}
],
"simulations": [
{
"task_id": "unique_task_id",
"trial": 0,
"transcript": {
"turns": [
{"role": "user", "content": "user message"},
{"role": "assistant", "content": "agent response", "tool_calls": []}
]
},
"reward": 1.0
}
]
}A plain text or markdown file describing the rules and constraints your agent should follow:
# Agent Policy
## Tool Usage
- Agents must verify account ownership before accessing balance
- All financial queries require authentication
## Communication
- Be concise and factual
- Do not make subjective comments{
"error_types": {
"DATA_HALLUCINATION": {
"definition": "Stated facts that contradict or are absent from tool output",
"dimension": "data_faithfulness",
"is_new": false,
"origin": null
},
"CUSTOM_ERROR_TYPE": {
"definition": "New error discovered during analysis",
"dimension": "new_dimension",
"is_new": true,
"origin": "task_042"
}
},
"dimensions": {
"data_faithfulness": {
"question": "Did the agent accurately report data from tool outputs?",
"fail_guidance": "Agent stated facts that contradict or don't exist in tool outputs",
"pass_guidance": "Rounding ($99.99→$100); correct summarization",
"is_new": false,
"origin": null
}
},
"discovery_log": [
{"task_id": "task_042", "name": "CUSTOM_ERROR_TYPE"}
]
}The framework comes with 6 pre-seeded dimensions:
- user_intent_adherence - Did the agent honor preferences and correctly read user input?
- user_question_fulfillment - Did the agent answer the user's direct questions?
- policy_violation - Did the agent's actions and decisions follow policy?
- policy_faithfulness - Did the agent correctly state policy rules?
- data_faithfulness - Did the agent accurately report data from tool outputs?
- tool_efficiency - Were the agent's tool calls reasonable given available context?
New dimensions are discovered automatically when errors don't fit existing categories.
Restrict analysis to specific dimensions:
from errordiscovery.registry import INTERACTION_DIMS, INTEGRITY_DIMS
output = run_discovery(
policy=policy,
tasks=tasks,
simulations=simulations,
applicable_dims=INTERACTION_DIMS # Only use interaction dimensions
)Tell the analyzer to ignore known quirks:
output = run_discovery(
policy=policy,
tasks=tasks,
simulations=simulations,
ignored_behaviors=[
"concurrent tool calls (agent making more than one tool call in a single message)",
"emoji usage in responses"
]
)git clone https://github.com/AmadeusITGroup/AgentFailureDiscovery.git
cd AgentFailureDiscovery
pip install -e ".[dev]"The error discovery pipeline:
- Initialize Registry - Start with base error types and dimensions
- Analyze Transcripts - For each transcript:
- LLM identifies all failures
- Maps failures to existing error types
- Proposes new types when failures are meaningfully different
- Records observations for borderline cases
- Update Registry - New error types and dimensions are added dynamically
- Track Evolution - Discovery log tracks when/where each type was found
The key innovation is the "relaxed gate" for new types: discoveries are made when failures have different root causes, policy impacts, or user harms - not just when they're superficially different.
Contributions are welcome! Please open an issue or submit a pull request.
Apache License 2.0 - see LICENSE file for details.
If you use this framework in your research, please cite:
@software{agentfailurediscovery,
title = {AgentFailureDiscovery: Automated Error Discovery for AI Agents},
author = {Driouich, Ilias and Cao, Hongliu},
year = {2026},
url = {https://github.com/AmadeusITGroup/AgentFailureDiscovery}
}- Issues: GitHub Issues
- Discussions: GitHub Discussions
Built with inspiration from AI agent evaluation frameworks and error taxonomy research.