Launch Evidence-Based AI Engineering

README.md

@@ -15,7 +15,8 @@ code.
 [Website](https://gptcode.dev) ·
 [Architecture](https://gptcode.dev/#architecture) ·
 [Documentation](https://gptcode.dev/guides/getting-started) ·
-[Engineering essay](https://gptcode.dev/blog/the-workflow-is-the-source-of-truth/)
+[Engineering thesis](https://gptcode.dev/blog/the-workflow-is-the-source-of-truth/) ·
+[Evaluation essay](https://gptcode.dev/blog/2026-07-29-one-successful-agent-run-proves-almost-nothing)
 
 <p align="center">
   <a href="https://gptcode.dev">

_roadmap.md

@@ -38,4 +38,18 @@ engineering portfolio.
 - [x] Propagate cancellation through agent-run command process trees and report local-model telemetry without false precision.
 - [x] Remove the disconnected legacy `monitor` and imperative `release` command implementations while preserving public-surface compatibility tests.
 - [ ] Isolate Live as an optional local observability protocol and retire unused training and experimental command surfaces.
+- [x] Build a content-free Codex history scanner and validate turn-level Git, patch, verification, and completion evidence against the local corpus.
+- [x] Build and container-validate a path-safe historical patch replayer, and establish that legacy Codex sessions lack the dirty-worktree baseline needed for deterministic replay.
+- [x] Capture complete initial and final snapshots for new agent experiments and prove deterministic bundle restoration end to end.
+- [x] Run a real local-model evaluation against a concurrent Go fixture and use its failures to fix local routing, repository-grounded planning, and retry context.
+- [x] Expand the evaluation corpus across concurrency, temporal semantics, and filesystem containment, with failure-inclusive suite aggregation and per-run time budgets.
+- [x] Compare GPT-OSS and Qwen3-Coder on the same smoke corpus, review the apparent pass, and strengthen the contract after identifying a false positive.
+- [x] Identify a local configuration that produces a human-reviewed fixture pass and measure three-run repeatability on the strengthened cache contract.
+- [x] Find a fully GPU-backed local configuration that passes the strengthened safe-store contract, while retaining Qwen's extended-budget failure as negative evidence.
+- [x] Stream verbose agent stages from the evidence suite without sacrificing the replayable output bundle.
+- [x] Measure Devstral safe-store repeatability and retain the 0/3 timeout result alongside the earlier reviewed capability pass.
+- [x] Launch Evidence-Based AI Engineering with a full paper, technical brief, reusable result figure, and evidence-linked distribution package.
+- [ ] Improve local safe-store convergence; neither Qwen nor Devstral currently supports a reliability claim on the strengthened contract.
+- [ ] Extend reviewed repeatability beyond one fixture before publishing a general coding-agent success-rate claim.
+- [ ] Add an OpenCode importer to challenge the vendor-neutral evidence model.
 - [ ] Raise coverage in legacy workflow packages without presenting the public fixture as repository-wide coverage.

benchmarks/README.md

@@ -0,0 +1,106 @@
+# GPTCode agent evaluation
+
+The evaluation harness runs coding agents against the same clean Git fixture
+and records an inspectable evidence bundle. It is designed to answer a narrow
+question: did the agent produce a change that satisfies executable,
+task-specific checks without losing the initial repository state?
+
+An experiment configuration names the agent process explicitly. Commands are
+executed directly, without an implicit shell. The evidence directory must live
+outside the evaluated repository so the run cannot accidentally measure its
+own artifacts.
+
+```json
+{
+  "id": "go-ledger-deadlock",
+  "repository": "/tmp/go-ledger",
+  "output": "/tmp/go-ledger-evidence",
+  "agent": {
+    "name": "gptcode-local",
+    "args": [
+      "/Users/jadercorrea/bin/gt",
+      "do",
+      "Fix the opposing-transfer deadlock without changing the public API"
+    ]
+  },
+  "verifications": [
+    {
+      "name": "tests-and-race-detector",
+      "args": ["go", "test", "-race", "./..."]
+    },
+    {
+      "name": "static-analysis",
+      "args": ["go", "vet", "./..."]
+    }
+  ]
+}
+```
+
+Run it with:
+
+```bash
+go run ./scripts/evidence-run -config /tmp/experiment.json
+```
+
+Every result retains failures as evidence. A successful agent process does not
+make a run pass: every verification command must also exit successfully.
+
+## Repeatability suites
+
+The suite runner creates a fresh Git repository for every fixture repetition
+and aggregates all outcomes. It runs sequentially so local CPU and memory
+contention do not bias model comparisons.
+
+```bash
+go run ./scripts/evidence-suite \
+  -verbose \
+  -config benchmarks/suites/local-gpt-oss.json \
+  -output /tmp/gptcode-go-core-quality
+```
+
+`-verbose` streams the agent's inspectable stages while retaining identical
+stdout and stderr in the evidence bundle. It exposes model selection, planning,
+tool execution, retries, and deterministic checks; it does not print private
+model chain-of-thought.
+
+Use `benchmarks/suites/local-qwen3-coder.json` for the equivalent Qwen suite.
+Each committed local suite records and executes its required GPTCode profile
+selection in `setup.json`; the agent name is not merely a user-supplied label.
+`benchmarks/suites/local-qwen3-coder-cache.json` preserves the first
+human-reviewed three-run result on the strengthened cache contract.
+`benchmarks/suites/local-devstral-small-2-safe-store.json` records the
+hardware-tuned 16k Devstral configuration that produced the first reviewed
+pass on strengthened filesystem containment.
+
+The initial corpus covers three distinct failure classes:
+
+- lock ordering and deterministic concurrency;
+- TTL boundary semantics under concurrent cache access;
+- path traversal and symlink containment at a filesystem boundary.
+
+The committed suite uses three repetitions. During harness development,
+`-repetitions 1` provides a smoke test without presenting it as a consistency
+measurement. `run_timeout_seconds` is a hard agent budget; verification still
+runs after a timeout so the resulting repository state remains evidence.
+Fixtures with `require_failing_baseline` also write `baseline.json` and abort
+the suite if every check already passes before the agent runs.
+
+Development results are recorded under `benchmarks/results/`. They distinguish
+system defects from model failures and never promote a one-run smoke test to a
+repeatability claim.
+
+## Bundle contract
+
+The generated directory contains:
+
+- `manifest.json`: schema, base commit, timestamps, and snapshot hashes.
+- `initial.tar` and `final.tar`: deterministic repository snapshots excluding
+  Git metadata.
+- `initial.patch`, `agent.patch`, and `final.patch`: binary-capable Git diffs.
+- `events.jsonl`: lifecycle events.
+- `commands.jsonl`: exact commands, outputs, exit codes, and durations.
+- `verification.json`: the machine-readable pass/fail decision.
+- `report.md`: a compact human-readable summary.
+
+Snapshots are intentionally complete. Only use controlled public fixtures:
+real repositories may contain credentials or proprietary untracked files.

benchmarks/fixtures/go-expiring-cache/cache.go

@@ -0,0 +1,53 @@
+package expiringcache
+
+import (
+	"sync"
+	"time"
+)
+
+type entry struct {
+	value     string
+	expiresAt time.Time
+}
+
+type Cache struct {
+	mu      sync.RWMutex
+	entries map[string]entry
+	now     func() time.Time
+}
+
+func New() *Cache {
+	return newWithClock(time.Now)
+}
+
+func newWithClock(now func() time.Time) *Cache {
+	return &Cache{
+		entries: make(map[string]entry),
+		now:     now,
+	}
+}
+
+func (c *Cache) Set(key, value string, ttl time.Duration) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.entries[key] = entry{
+		value:     value,
+		expiresAt: c.now().Add(ttl),
+	}
+}
+
+func (c *Cache) Get(key string) (string, bool) {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	item, found := c.entries[key]
+	if !found {
+		return "", false
+	}
+	return item.value, true
+}
+
+func (c *Cache) Len() int {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	return len(c.entries)
+}

benchmarks/fixtures/go-expiring-cache/cache_test.go

@@ -0,0 +1,67 @@
+package expiringcache
+
+import (
+	"sync"
+	"testing"
+	"time"
+)
+
+func TestGetReturnsLiveEntry(t *testing.T) {
+	now := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC)
+	cache := newWithClock(func() time.Time { return now })
+	cache.Set("session", "active", time.Minute)
+
+	value, found := cache.Get("session")
+	if !found || value != "active" {
+		t.Fatalf("Get() = %q, %v, want active, true", value, found)
+	}
+}
+
+func TestGetEvictsExpiredEntry(t *testing.T) {
+	now := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC)
+	cache := newWithClock(func() time.Time { return now })
+	cache.Set("session", "expired", time.Minute)
+	now = now.Add(time.Minute)
+
+	if value, found := cache.Get("session"); found || value != "" {
+		t.Fatalf("Get() = %q, %v, want empty, false", value, found)
+	}
+	if length := cache.Len(); length != 0 {
+		t.Fatalf("Len() = %d, want expired entry removed", length)
+	}
+}
+
+func TestLenPurgesExpiredEntriesWithoutLookup(t *testing.T) {
+	now := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC)
+	cache := newWithClock(func() time.Time { return now })
+	cache.Set("expired", "stale", time.Minute)
+	cache.Set("live", "current", 2*time.Minute)
+	now = now.Add(time.Minute)
+
+	if length := cache.Len(); length != 1 {
+		t.Fatalf("Len() = %d, want only the live entry", length)
+	}
+	if value, found := cache.Get("live"); !found || value != "current" {
+		t.Fatalf("Get(live) = %q, %v, want current, true", value, found)
+	}
+}
+
+func TestCacheSupportsConcurrentReadersAndWriters(t *testing.T) {
+	cache := New()
+	var workers sync.WaitGroup
+	for worker := range 8 {
+		workers.Add(1)
+		go func() {
+			defer workers.Done()
+			for iteration := range 250 {
+				key := string(rune('a' + worker))
+				cache.Set(key, "value", time.Minute)
+				cache.Get(key)
+				if iteration%10 == 0 {
+					cache.Len()
+				}
+			}
+		}()
+	}
+	workers.Wait()
+}

benchmarks/fixtures/go-expiring-cache/go.mod

@@ -0,0 +1,3 @@
+module example.com/expiringcache
+
+go 1.22

benchmarks/fixtures/go-expiring-cache/quality_test.go

@@ -0,0 +1,22 @@
+package expiringcache
+
+import (
+	"bytes"
+	"go/format"
+	"os"
+	"testing"
+)
+
+func TestCacheImplementationIsFormatted(t *testing.T) {
+	source, err := os.ReadFile("cache.go")
+	if err != nil {
+		t.Fatal(err)
+	}
+	formatted, err := format.Source(source)
+	if err != nil {
+		t.Fatalf("format.Source() error = %v", err)
+	}
+	if !bytes.Equal(source, formatted) {
+		t.Fatal("cache.go is not gofmt-formatted")
+	}
+}

benchmarks/fixtures/go-expiring-cache/task.md

@@ -0,0 +1,16 @@
+# Enforce cache expiration
+
+`Cache.Get` currently returns entries after their TTL has elapsed, and `Len`
+continues to count those stale entries.
+
+Make expiration authoritative while preserving the exported API. A lookup at
+the exact expiration instant is expired and must remove the stale entry.
+Maintain race-free concurrent reads and writes; do not introduce background
+goroutines or wall-clock sleeps.
+
+The implementation must pass:
+
+```text
+go test -race ./...
+go vet ./...
+```

benchmarks/fixtures/go-ledger/concurrency_test.go

@@ -0,0 +1,68 @@
+package ledger
+
+import (
+	"sync"
+	"testing"
+	"time"
+)
+
+func TestSelfTransferCompletes(t *testing.T) {
+	account := NewAccount(1_000)
+	completed := make(chan error, 1)
+	go func() {
+		completed <- Transfer(account, account, 10)
+	}()
+
+	select {
+	case err := <-completed:
+		if err != nil {
+			t.Fatalf("Transfer() error = %v", err)
+		}
+	case <-time.After(250 * time.Millisecond):
+		t.Fatal("self-transfer deadlocked")
+	}
+
+	if balance := account.Balance(); balance != 1_000 {
+		t.Errorf("balance = %d, want 1000", balance)
+	}
+}
+
+func TestOpposingTransfersComplete(t *testing.T) {
+	left := NewAccount(1_000)
+	right := NewAccount(1_000)
+	start := make(chan struct{})
+	var workers sync.WaitGroup
+	workers.Add(2)
+
+	go func() {
+		defer workers.Done()
+		<-start
+		for range 1_000 {
+			_ = Transfer(left, right, 1)
+		}
+	}()
+	go func() {
+		defer workers.Done()
+		<-start
+		for range 1_000 {
+			_ = Transfer(right, left, 1)
+		}
+	}()
+
+	close(start)
+	completed := make(chan struct{})
+	go func() {
+		workers.Wait()
+		close(completed)
+	}()
+
+	select {
+	case <-completed:
+	case <-time.After(2 * time.Second):
+		t.Fatal("opposing transfers deadlocked")
+	}
+
+	if total := left.Balance() + right.Balance(); total != 2_000 {
+		t.Errorf("total balance = %d, want 2000", total)
+	}
+}

benchmarks/fixtures/go-ledger/go.mod

@@ -0,0 +1,3 @@
+module example.com/ledger
+
+go 1.22