Trellis

Standalone optimization lab for Smithers' Trellis component. The first target is ARC-AGI-3 using only Claude Fable 5 and GPT-5.6 Sol.

ARC-AGI-3

The harness pins the official evaluator as a submodule and exposes Trellis as an OpenAI-compatible model endpoint. The official Python agent remains in charge of frames, action validation, budgets, scorecards, and RHAE scoring.

git submodule update --init --recursive
bun install --frozen-lockfile
uv sync --project vendor/arc-agi-3-benchmarking
cp .env.example .env
bun run arc:gateway
# another terminal
uv run python arc3/run.py --sample-size 17 --seed smithers-trellis-v1

The official SDK automatically issues an anonymous ARC key when ARC_API_KEY is omitted. Seventeen games are sampled without replacement from the 25-game public suite. Under the conservative p=0.5 finite-population normal approximation, that estimates a suite-level percentage within about 15 percentage points at 95% confidence. The report also emits the observed t-based interval for the bounded per-game RHAE scores.

Raw manifests, official scorecard IDs, recordings, and reports go under results/; secrets and local databases are ignored.

Hindsight experiment

Every generated Trellis author/worker task is wrapped by the reusable local MemoryTrellis composite. Recall is bounded to 2,048 tokens and fail-open; successful-task retention is asynchronous. Memory is split between a shared ARC strategy bank and an isolated per-game world-model bank. The prompts state that recalled material is advisory and current-game evidence wins.

Run the same frozen manifest in three arms. Restart the gateway with the shown mode before each arm so its health response records the active treatment.

TRELLIS_MEMORY_MODE=baseline bun run arc:gateway
uv run python arc3/run.py --variant baseline --sample-size 17 --seed smithers-trellis-v1

TRELLIS_MEMORY_MODE=recall bun run arc:gateway
uv run python arc3/run.py --variant recall --sample-size 17 --seed smithers-trellis-v1

TRELLIS_MEMORY_MODE=learn bun run arc:gateway
uv run python arc3/run.py --variant learn --sample-size 17 --seed smithers-trellis-v1

python arc3/compare.py results/baseline/report.json results/learn/report.json

baseline cannot read or write memory; recall reads but cannot contaminate the bank; learn reads and retains completed task digests. Promotion requires at least 17 identical paired games, a positive lower 95% bound on RHAE delta, and regressions on no more than 25% of games. Point estimates never promote a prompt, primer, skill, or harness refinement.

Prompts live in prompts/*.mdx. Deterministic unit and Smithers graph-extraction tests cover policy isolation, prompt contracts, sampling, reports, and the promotion gate. The official evaluator remains the end-to-end authority.

results/report.json reports the sample mean RHAE as the expected full-suite score plus an observed 95% t interval with finite-population correction. It also retains every per-game score, completed-level count, and action count. Trellis agent harnesses and optimization experiments for external benchmarks