Skip to content

docs: rewrite README for clearer positioning and onboarding #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
instax-dutta merged 1 commit into from
Jul 19, 2026
Merged

docs: rewrite README for clearer positioning and onboarding #11

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
250 changes: 83 additions & 167 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,138 +1,102 @@
# AgentLoop

**A harness-agnostic, self-verifying autonomy wrapper. The verification oracle is the product.**
Run coding agents in a verification loop until the task actually passes your checks.

AgentLoop is *not* a coding agent and *not* a BYOK harness. It drives the coding
agent you already use — **OpenCode by default** (free models, no key from us),
also Kilo Code / Claude Code / Aider / Codex / Goose — in a loop until the goal is
*actually met and proven correct*:
AgentLoop re-invokes your chosen agent, feeds failures back into the next iteration, and stops only when a human-authored verifier says the work is correct.

- Works with multiple agent harnesses
- Retries until verification passes
- Resumes after crashes
- Supports held-out verification
- Supports cost caps and notifications

```text
goal -> agent edits -> verify fails -> failure fed back -> agent retries -> verify passes
```
goal + feedback ──► your agent edits the sandbox ──► verification oracle
▲ │
└────────────── (fail) ◄───────────────────────────┘
(pass) ──► DONE (and a sealed oracle agrees)
```

The harness supplies the model and auth. AgentLoop only adds three things
people keep rebuilding by hand:
## Why this exists

Coding agents often stop halfway or pass only the obvious cases. A naive loop that checks whether the agent said "DONE" fails in two common ways:

* **continuity** — it loops until the goal is really done (no "please clarify"),
* **the verification oracle** — a correctness gate, not "it runs and prints",
* **safety** — your key is never exposed to the wrapped agent; work is
git-checkpointed and, if the process dies, **resumed** from where it stopped.
1. **Partial completion.** The agent quits mid-task; the loop thinks it won.
2. **Overfitting.** If the checks are visible to the agent, it can pass without being generally correct.

## Why this exists (read this before you write a 20-line bash loop)
AgentLoop fixes both by separating the agent from the verifier:

Coding agents are famous for **stopping halfway** and claiming they're done
(OpenCode issue #24685 and many duplicates; the "stopped halfway" essays). A
naive loop — *"run the agent, check if it said DONE"* — fails two ways:
- the agent makes changes
- the verifier decides whether the work is actually correct
- AgentLoop keeps retrying until the verifier passes

1. **Partial completion.** The agent quits mid-task; your loop thinks it won.
2. **Overfitting.** If you bake the exact test cases into the check, the agent
can pass without being *generally* correct (the SWE-bench false-green bug).
## Why people use it

AgentLoop fixes both: it gates on a **human-authored oracle** the agent can't
edit, and the oracle can run against **held-out inputs the agent never saw**
(see below). It also survives a crash: rerun it and it resumes the same goal.
AgentLoop is useful when "looks right" is not enough and you need proven correctness.

## Install / quick start
- Works with OpenCode, Kilo Code, Claude Code, Aider, Codex, and Goose
- Keeps a run going even if the process crashes
- Lets you use human-authored or held-out checks
- Tracks cost and can stop before runaway bills
- Sends terminal-state notifications

## Quick start

```bash
git clone <this> agentloop && cd agentloop
python3 -m pip install -e .
# optional: direct-mode OpenAI-compatible API client
# python3 -m pip install -e '.[direct]'
# optional: developer tooling (ruff, pre-commit)
# python3 -m pip install -e '.[dev]'
cp .env.example .env
chmod +x launch.sh stop.sh verify.sh mock_agent.sh oracle.py
```

You need **OpenCode installed** (`npm i -g opencode` or your package manager) and
a free model configured in it. No API key is required by AgentLoop itself.
You need **OpenCode installed** (`npm i -g opencode` or your package manager) and a free model configured in it. No API key is required by AgentLoop itself.

### One-liner

```bash
agentloop "build a JSON linter" --verify "bash verify.sh"
agentloop --dry-run --harness opencode --verify "bash verify.sh" # show config, do nothing
agentloop --init # scaffold goal.txt + verify.sh + .env
agentloop --dry-run --harness opencode --verify "bash verify.sh"
agentloop --init
agentloop --version
agentloop --status # show current run status
agentloop --serve # start web monitoring UI
agentloop --status
agentloop --serve
```

Or the file-based flow:

1. Put the task in `goal.txt`.
2. Write a **verification oracle** — a script that exits 0 only when the work is
correct. The agent never edits this, so it can't fake it.
3. `./launch.sh` (or `agentloop`), `tail -f agentloop.log`, `./stop.sh`.

### Windows (PowerShell)

```powershell
.\launch.ps1 # start the loop (background job)
.\stop.ps1 # stop gracefully
```

### Multi-agent: run a plan

```bash
agentloop --run plan.md
```

Parses a markdown file with `- [ ] task` items and spawns one agent loop per task.
2. Write a verification oracle — a script that exits 0 only when the work is correct.
3. Run `./launch.sh`, watch `tail -f agentloop.log`, stop with `./stop.sh`.

## The verification oracle (the important part)
## The verification oracle

The agent's *own* tests are not trusted: it can validate "it runs" without
noticing wrong numbers. Instead, a separate, human-authored check is run by the
harness on each iteration:
The agent's own tests are not trusted: it can validate "it runs" without noticing wrong numbers. Instead, a separate human-authored check is run by the harness on each iteration:

* Exit 0 → accepted, loop ends, `agentloop.summary.txt` written, `NOTIFY_CMD` fired.
* Non-zero → the failure output is injected back into the next prompt and the
agent **must keep working**.
- Exit 0 → accepted, loop ends, `agentloop.summary.txt` is written, notifications fire.
- Non-zero → the failure output is injected back into the next prompt and the agent must keep working.

### Sealed / held-out oracle (defeats overfitting)
### Sealed / held-out oracle

A plain check can still be gamed if the author leaks the exact cases to the
agent. To stop that, keep a **case file outside the sandbox** and split it into
*visible* (may be shown to the agent) and *held-out* (the agent never sees them):
A plain check can still be gamed if the author leaks the exact cases to the agent. To stop that, keep a case file outside the sandbox and split it into visible and held-out inputs:

```bash
# 1) auto-generate fresh test inputs from a reference program
python oracle.py gen --reference "python ref.py" --n 200 --out cases.txt --seed 42

# 2) record a trusted reference's behaviour (keep oracle.json OUTSIDE the sandbox)
python oracle.py record \
--reference "python ref.py" --inputs cases.txt --visible 3 \
--out .agentloop/oracle_sealed/oracle.json --seal "$ORACLE_SEAL"

# 3) the verifier grades the candidate against ALL inputs (visible + held-out)
python oracle.py grade \
--candidate "python sandbox/solution.py" \
--oracle .agentloop/oracle_sealed/oracle.json --seal "$ORACLE_SEAL"
```

The candidate only **PASSES if it is correct on the inputs it has never seen**.
A wrong `--seal` makes grading report `TAMPERED`. See `verify_template.sh` for a
drop-in template. This is the feature that makes AgentLoop worth using over a
hand-rolled loop — it's the same idea as dedicated "verification oracle"
projects, packaged for the autonomous-coding loop.
The candidate only passes if it is correct on inputs it has never seen. A wrong `--seal` makes grading report `TAMPERED`.

## Crash-safe resume + summary

AgentLoop writes `agentloop.state.json` every iteration. If the process (or your
machine) dies, just run it again with the **same goal** and it resumes from the
last iteration instead of re-planning:
AgentLoop writes `agentloop.state.json` every iteration. If the process or machine dies, run it again with the same goal and it resumes from the last iteration instead of re-planning.

```
```text
resuming from iter 2 (cost so far: $0.10)
```

When the run ends it writes `agentloop.summary.txt` (`status=... iters=... elapsed=... goal=...`) and fires notifications.
When the run ends it writes `agentloop.summary.txt` and fires notifications.

## Hard per-run cost cap

Expand All @@ -142,118 +106,75 @@ Prevent runaway API bills with `MAX_COST_USD`:
MAX_COST_USD=5 agentloop "my task" --verify "bash verify.sh"
```

The loop tracks running cost in `agentloop.state.json` and stops with
`status=over-budget` when the cap is exceeded. In CLI mode, set
`ESTIMATED_COST_PER_ITER` (default $0.10) to approximate cost. In direct mode,
actual token counts from the API response are used when available.
The loop tracks running cost in `agentloop.state.json` and stops with `status=over-budget` when the cap is exceeded. In CLI mode, set `ESTIMATED_COST_PER_ITER` to approximate cost. In direct mode, actual token counts are used when available.

## Hands-off notifications

Send terminal-state summaries via generic command, Telegram, Discord, or Slack:

```bash
# Generic: any shell command
NOTIFY_CMD='curl -s -X POST https://hooks.example.com -d "{kind}: {msg}"'

# Telegram (set both)
NOTIFY_TELEGRAM_BOT_TOKEN=123456:ABC-DEF
NOTIFY_TELEGRAM_CHAT_ID=-1001234567890

# Discord (webhook URL)
NOTIFY_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...

# Slack (webhook URL)
NOTIFY_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
```

`{kind}` (completed/blocked/stopped/timeout/over-budget) and `{msg}` are
substituted into the generic command. Native adapters use the REST API directly
with no extra dependencies.
`{kind}` and `{msg}` are substituted into the generic command. Native adapters use the REST API directly.

## Run monitoring

```bash
agentloop --status # terminal status display
agentloop --serve # web UI at http://localhost:8080 (auto-refresh 5s)
agentloop --serve --port 9090 # custom port
agentloop --status
agentloop --serve
agentloop --serve --port 9090
```

## Harnesses (`.env`)

```bash
AGENT_MODE=cli
AGENT_PRESET=opencode # primary target; ships free models, no key needed
AGENT_PRESET=opencode
VERIFY_CMD="bash verify.sh"
# NOTIFY_CMD='...' # optional
# TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=... # optional
# MAX_COST_USD=5 # optional cost cap
# NOTIFY_CMD='...'
# TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=...
# MAX_COST_USD=5
```

If `AGENT_PRESET` is empty, AgentLoop auto-detects an installed CLI (with
version checks). Or set `AGENT_CMD` to any command; the prompt is injected via
`$AGENTLOOP_PROMPT`:
If `AGENT_PRESET` is empty, AgentLoop auto-detects an installed CLI with version checks. Or set `AGENT_CMD` to any command; the prompt is injected via `$AGENTLOOP_PROMPT`:

```bash
AGENT_CMD='opencode run "$AGENTLOOP_PROMPT" --auto'
```

| Preset | Command |
|-----------|------------------------------------------------------|
| opencode | `opencode run "$AGENTLOOP_PROMPT" --auto` |
| kilocode | `kilocode run "$AGENTLOOP_PROMPT"` |
| Preset | Command |
|-----------|---------|
| opencode | `opencode run "$AGENTLOOP_PROMPT" --auto` |
| kilocode | `kilocode run "$AGENTLOOP_PROMPT"` |
| claude | `claude -p "$AGENTLOOP_PROMPT" --dangerously-skip-permissions` |
| aider | `aider --message "$AGENTLOOP_PROMPT" --yes` |
| codex | `codex exec "$AGENTLOOP_PROMPT"` |
| goose | `goose run "$AGENTLOOP_PROMPT"` |

**Hidden strength:** because AgentLoop re-invokes the harness *every iteration*,
a harness that "stops halfway" (the OpenCode bug above) just becomes one failed
iteration — the loop absorbs it and keeps going.
| aider | `aider --message "$AGENTLOOP_PROMPT" --yes` |
| codex | `codex exec "$AGENTLOOP_PROMPT"` |
| goose | `goose run "$AGENTLOOP_PROMPT"` |

## Direct mode (legacy / offline)

If you have no harness installed, `AGENT_MODE=direct` calls an OpenAI-compatible
API directly (one continuous reasoning thread, same tools + oracle). Set
`KILO_API_KEY` / `KILO_BASE_URL` / `KILO_MODEL` in `.env`.

## Log rotation

Logs are automatically rotated at `LOG_MAX_MB` (default 10 MB) with 3 backup
files. Set in `.env`:

```
LOG_MAX_MB=25
```

## Input auto-generation

`oracle.py gen` automatically produces fresh test inputs from a reference
program using multiple generative strategies (random integers, floats, strings,
structured data):

```bash
oracle.py gen --reference "python ref.py" --n 200 --out cases.txt --seed 42
oracle.py record --reference "python ref.py" --inputs cases.txt --visible 10 --out oracle.json
oracle.py grade --candidate "python cand.py" --oracle oracle.json
```
If you have no harness installed, `AGENT_MODE=direct` calls an OpenAI-compatible API directly. Set `KILO_API_KEY`, `KILO_BASE_URL`, and `KILO_MODEL` in `.env`.

## Examples gallery

The `examples/` directory contains three working verifier samples:

| Example | Description | Verifier Type |
|---------|-------------|---------------|
| `tax-demo/` | US tax calculator | Fixed cases in bash |
| `json-linter/` | JSON syntax validator | Temporary test files |
| `refactor-regression/` | Refactor with identical output | Held-out oracle (gen + record + grade) |

Each example has a `goal.txt` (the task) and `verify.sh` (the oracle).
Each example has a `goal.txt` and `verify.sh`.

## Tests

```bash
python3 test_oracle.py # the verification gate + held-out oracle (no LLM)
python3 test_loop.py # full loop using mock_agent.sh + cost cap + status + more
python3 test_oracle.py
python3 test_loop.py
```

Both run deterministically without any model.
Expand All @@ -265,17 +186,25 @@ pip install -e '.[dev]'
pre-commit install
```

## Limitations

- The wrapped agent runs with `cwd=sandbox`; AgentLoop relies on the harness's own permission model. For hard isolation, run inside a container.
- Without `VERIFY_CMD`, the loop falls back to a full-line `DONE` / `BLOCKED` signal from the agent; a verifier is strongly recommended.
- Each CLI iteration is a fresh agent invocation; continuity is maintained by feeding goal + last failure back, not by an in-agent session.
- The held-out case file should live outside the sandbox; the seal is a tamper signal, not absolute security.
- API keys and `ORACLE_SEAL` are stripped from the agent environment; the seal is passed only to `VERIFY_CMD`.

## Files

* `agentloop.py` — orchestrator (cli + direct modes, resume, notify, CLI)
* `oracle.py` — verification oracle + sealed/held-out record/grade + input gen
* `verify.sh` / `verify_template.sh` — example oracle + scaffold template
* `mock_agent.sh`— deterministic stand-in for a real agent CLI (for tests)
* `launch.sh` / `stop.sh` — Linux/Mac launcher scripts
* `launch.ps1` / `stop.ps1` — Windows launcher scripts
* `goal.txt` / `.env.example` / `CHANGELOG.md` / `CONTRIBUTING.md`
* `.pre-commit-config.yaml` — ruff linting hooks
* `examples/` — 3 working verifier samples
- `agentloop.py` — orchestrator (CLI + direct modes, resume, notify)
- `oracle.py` — verification oracle + sealed / held-out record / grade + input generation
- `verify.sh` / `verify_template.sh` — example oracle + scaffold template
- `mock_agent.sh` — deterministic stand-in for a real agent CLI
- `launch.sh` / `stop.sh` — Linux / Mac launcher scripts
- `launch.ps1` / `stop.ps1` — Windows launcher scripts
- `goal.txt` / `.env.example` / `CHANGELOG.md` / `CONTRIBUTING.md`
- `.pre-commit-config.yaml` — ruff linting hooks
- `examples/` — working verifier samples

## Exit codes

Expand All @@ -288,16 +217,3 @@ pre-commit install
| exhausted (max iters) | 4 |
| over-budget (cost cap) | 5 |
| stopped (STOP / signal) | 130 |

## Limitations

* The wrapped agent runs with `cwd=sandbox`; AgentLoop relies on the harness's
own permission model. For hard isolation, run inside a container (see ISSUES.md).
* Without a `VERIFY_CMD`, the loop falls back to a full-line `DONE`/`BLOCKED`
signal from the agent — a verifier is strongly recommended.
* Each CLI iteration is a fresh agent invocation; "continuity" is maintained by
AgentLoop feeding goal + last failure back, not by an in-agent session.
* The held-out case file should live outside the sandbox; the seal is a
tamper *signal*, not absolute security — keep secrets out of the agent's reach.
* API keys and `ORACLE_SEAL` are stripped from the agent environment; the seal
is passed only to `VERIFY_CMD`. Never put secrets in the sandbox.
Loading