Agent evaluation and capability measurement are advancing through several complementary efforts that emerged alongside this project and continue to develop. HAL provides a shared harness and cost-aware leaderboard for running agent evaluations. Epoch AI, METR, and Arena AI are leading independent evaluation efforts. Epoch AI measures capability across benchmarks, METR studies how reliably agents complete tasks of different lengths, and Arena AI compares model outputs through large-scale blind evaluations. At a time when AI systems are advancing rapidly and entering increasingly consequential settings, efforts such as these provide an essential empirical basis for understanding their capabilities, limitations, safety, and broader implications.
-These evaluation records can also support the development of new methods. Recent efforts such as Jev and CLM-8B develop learned decision models that can act as judges or verifiers. CLM-8B is post-trained on approximately one million agent trajectories from the Agent Data Protocol dataset, together with terminal traces from Endless-Terminals and LiteCoder-Terminal-SFT. Each trajectory step pairs the agent's current context with the action it took. By releasing trajectories together with task, verifier, and result information, Messier can facilitate similar work while potentially providing additional context for training and evaluating learned judges.
+These evaluation records can also support the development of new methods. Recent efforts such as Jev, CLM-8B, and Laya develop learned decision models that can act as judges or verifiers. CLM-8B is post-trained on approximately one million agent trajectories from the Agent Data Protocol dataset, together with terminal traces from Endless-Terminals and LiteCoder-Terminal-SFT. Each trajectory step pairs the agent's current context with the action it took. By releasing trajectories together with task, verifier, and result information, Messier can support training similar decision models while potentially providing additional context for evaluating learned judges.
BRIDGE and Agent Psychometrics use results for individual tasks and Item Response Theory to study task difficulty and agent capability. We provide an initial analysis in this paper and explore this direction in greater depth in a follow-up paper: Predicting Task Difficulty Without Rollouts.
Looking ahead, it will be important to examine when a shared scale provides a good fit, whether tasks grouped by occupation or industry require separate but linked scales, and whether multidimensional or hierarchical IRT can better represent the factors that contribute to performance. Such extensions are well established in psychometrics, but remain underexplored here. The present fit estimates one ability per model after selecting its best observed scaffold within each benchmark. A complementary extension could retain model–scaffold pairs as distinct agents and model relationships among agents that share a model family, developer, or scaffold. Hierarchical IRT could borrow statistical strength across these relationships while preserving distinct agent estimates, improving estimates when observations are sparse and helping study the contributions of models, scaffolds, and their interaction.