This repository contains a lightweight, decoupled Java-based framework for conducting Trajectory-Based Evaluations (Evals) on autonomous AI agents.
Unlike traditional deterministic software testing, Agentic AI testing requires evaluating an agent’s behavioral footprint across its entire multi-step lifecycle, monitoring its system boundaries, and auditing its internal reasoning paths.
This test framework architecture implements advanced boundary validation patterns inspired by investigative psychology and criminology frameworks:
- Means, Motive, and Opportunity (Boundary Vulnerability Testing): Programmatic evaluations that isolate an agent's technical capabilities (
Means) from its environment (Opportunity) to ensure it cannot execute structural mutations outside its explicit context layer. - Trajectory Alignment (Cognitive Deconstruction): Auditing the hidden internal reasoning chain (
internalThoughts) for early signs of logical drift, shortcuts, or rule rationalization before the system calls an external tool or executes an unrecoverable payload. - Entropy Validation (The Broken Windows Theory): Turn-by-turn verification designed to track data context degradation. This prevents small, unhandled execution anomalies from compounding over long history tracks and causing total model failure.
The suite is structured into clear, decoupled layers using clean software engineering paradigms:
src/ai/evals/agent/
├── AgentExecutionTrace.java # Immutable runtime payload capturing the agent state
├── EvalResult.java # Structured container for evaluation criteria outputs
├── Evaluator.java # Baseline operational contract interface for all testers
├── SecurityBoundaryEvaluator.java# Deterministic evaluator checking structural bounds
├── ReasoningAuditorEvaluator.java # Behavioral judge analyzing reasoning chains (LLM Mock)
└── AgentTestPipeline.java # Orchestration pipeline executing the test runners