Skip to content

About

Hidden, stateful environments for comparing native and programmatic tool calling.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Statecraft

How does tool access change an agent's ability to track hidden state? Statecraft provides synthetic tasks with controlled dependencies and success conditions, exposed through native tools or a Python workspace. Recorded runs let you inspect dependency errors, bookkeeping failures, and decisions based on stale observations.

Task environments · Recorded experiments · Writeup

Try a task offline

With Python 3.12 or later and uv:

git clone https://github.com/latentpaths/statecraft.git
cd statecraft
uv sync --locked --group dev
uv run statecraft generate --family hidden_dag --seed 1 \
  --output /tmp/statecraft-task.json
uv run statecraft run --task /tmp/statecraft-task.json --mode oracle \
  --output /tmp/statecraft-results.jsonl
uv run statecraft summarize --input /tmp/statecraft-results.jsonl

This generates a task, solves it with the reference oracle, and summarizes the record. Use a fresh output path for each independent run: records are append-only. Oracle success checks task mechanics; it does not measure LLM performance.

What the agent does

Task family Required actions
hidden_dag Discover intermediate values and compute a target in dependency order
repair_chain Inspect and repair machines while tracking locations, tools, and observation versions

The core package implements task generation, hidden-state transitions, oracle runs, datasets, and metrics. Two Verifiers packages expose those tasks: standard-statecraft uses native tools; ptc-statecraft uses a Python workspace.

Evaluate and inspect

The experiment guide explains matched grids, retained records, aggregation, and plots. These synthetic comparisons measure performance under the recorded task and execution settings; they do not establish general tool-use reliability. Some hard-grid PTC runs timed out and have no raw results, so inspect missing-run coverage before comparing summaries.

For model calls, dataset export, and grid previews, see the evaluation guide.

Development

uv run --locked pytest tests
uv build --package statecraft
uv build --package standard-statecraft
uv build --package ptc-statecraft

License

The existing license reserves all rights. Use, reproduction, modification, distribution, benchmarking, evaluation, dataset generation, model training, and derivative works require prior written permission. Contact Ayush Bodade.

Citation

@misc{bodadearticlestatecraft,
  author = {Bodade, Ayush},
  title = {{Statecraft: A Synthetic Benchmark Experiment for Programmatic Tool Calling}},
  year = {2026},
  month = {05},
  day = {20},
  date = {2026-05-20},
  howpublished = {Latent Paths},
  url = {https://www.latentpaths.com/writings/statecraft},
  note = {Blog post}
}

About

Hidden, stateful environments for comparing native and programmatic tool calling.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages