How does tool access change an agent's ability to track hidden state? Statecraft provides synthetic tasks with controlled dependencies and success conditions, exposed through native tools or a Python workspace. Recorded runs let you inspect dependency errors, bookkeeping failures, and decisions based on stale observations.
Task environments · Recorded experiments · Writeup
With Python 3.12 or later and uv:
git clone https://github.com/latentpaths/statecraft.git
cd statecraft
uv sync --locked --group dev
uv run statecraft generate --family hidden_dag --seed 1 \
--output /tmp/statecraft-task.json
uv run statecraft run --task /tmp/statecraft-task.json --mode oracle \
--output /tmp/statecraft-results.jsonl
uv run statecraft summarize --input /tmp/statecraft-results.jsonlThis generates a task, solves it with the reference oracle, and summarizes the record. Use a fresh output path for each independent run: records are append-only. Oracle success checks task mechanics; it does not measure LLM performance.
| Task family | Required actions |
|---|---|
hidden_dag |
Discover intermediate values and compute a target in dependency order |
repair_chain |
Inspect and repair machines while tracking locations, tools, and observation versions |
The core package implements task generation, hidden-state transitions, oracle runs, datasets, and metrics. Two Verifiers packages expose those tasks: standard-statecraft uses native tools; ptc-statecraft uses a Python workspace.
The experiment guide explains matched grids, retained records, aggregation, and plots. These synthetic comparisons measure performance under the recorded task and execution settings; they do not establish general tool-use reliability. Some hard-grid PTC runs timed out and have no raw results, so inspect missing-run coverage before comparing summaries.
For model calls, dataset export, and grid previews, see the evaluation guide.
uv run --locked pytest tests
uv build --package statecraft
uv build --package standard-statecraft
uv build --package ptc-statecraftThe existing license reserves all rights. Use, reproduction, modification, distribution, benchmarking, evaluation, dataset generation, model training, and derivative works require prior written permission. Contact Ayush Bodade.
@misc{bodadearticlestatecraft,
author = {Bodade, Ayush},
title = {{Statecraft: A Synthetic Benchmark Experiment for Programmatic Tool Calling}},
year = {2026},
month = {05},
day = {20},
date = {2026-05-20},
howpublished = {Latent Paths},
url = {https://www.latentpaths.com/writings/statecraft},
note = {Blog post}
}