Reproducible evaluations of AI models, agents, and inference systems.
This repository publishes protocols, manifests, summaries, and verification tools. Large or generated evidence bundles are attached to immutable GitHub Releases instead of being committed to Git history.
The goal is not to collect favorable demonstrations. Every publication should retain failed runs, state what the executable contract proves, and distinguish capability from repeatability.
| Release | Experiment | Result |
|---|---|---|
2026.07.29.1 |
Local coding agents: filesystem containment | One reviewed capability pass; subsequent repeatability batch 0/3 |
See coding-agents/filesystem-containment for the protocol, limitations, and release manifest.
| Study | Status | Next gate |
|---|---|---|
| Local-first routing for repository-level coding agents | Published calibration pilot; confirmatory no-go | Develop the separately named successor without pooling v1 outcomes |
Draft and calibration artifacts are not published benchmark claims. They are retained so protocol changes, invalidated runs, and treatment-selection decisions remain auditable before confirmatory execution.
Each experiment must provide:
- a stable experiment identifier;
- an explicit schema version;
- source and harness revisions;
- model and runtime identity;
- hardware and execution constraints;
- exact verification commands;
- success, failure, and timeout results;
- known limitations;
- checksums for every released asset.
Releases use CalVer:
YYYY.MM.DD.N
The final component is a sequence number for multiple publications on the same day. Published release tags and assets are never replaced. Corrections receive a new release.
Download a release archive and its checksum file into the same directory:
./scripts/verify-release \
SHA256SUMS \
ai-experiment-coding-agents-filesystem-containment-2026.07.29.1.tar.gzThe script verifies the checksum and inspects the archive before extraction.
The repository is intentionally independent from any single model or harness. An experiment may use GPTCode, Codex, OpenCode, Gemini, a custom evaluator, or another system, provided the manifest records the dependency precisely.
Repository-authored code, metadata, and documentation are available under the MIT License. Evidence bundles may contain third-party or generated material; each experiment must record the applicable source licenses.