diff --git a/README.md b/README.md index c9586fd..49d506a 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,8 @@ code. [Website](https://gptcode.dev) · [Architecture](https://gptcode.dev/#architecture) · [Documentation](https://gptcode.dev/guides/getting-started) · -[Engineering essay](https://gptcode.dev/blog/the-workflow-is-the-source-of-truth/) +[Engineering thesis](https://gptcode.dev/blog/the-workflow-is-the-source-of-truth/) · +[Evaluation essay](https://gptcode.dev/blog/2026-07-29-one-successful-agent-run-proves-almost-nothing)

diff --git a/_roadmap.md b/_roadmap.md index 47c9edc..e97fd32 100644 --- a/_roadmap.md +++ b/_roadmap.md @@ -38,4 +38,18 @@ engineering portfolio. - [x] Propagate cancellation through agent-run command process trees and report local-model telemetry without false precision. - [x] Remove the disconnected legacy `monitor` and imperative `release` command implementations while preserving public-surface compatibility tests. - [ ] Isolate Live as an optional local observability protocol and retire unused training and experimental command surfaces. +- [x] Build a content-free Codex history scanner and validate turn-level Git, patch, verification, and completion evidence against the local corpus. +- [x] Build and container-validate a path-safe historical patch replayer, and establish that legacy Codex sessions lack the dirty-worktree baseline needed for deterministic replay. +- [x] Capture complete initial and final snapshots for new agent experiments and prove deterministic bundle restoration end to end. +- [x] Run a real local-model evaluation against a concurrent Go fixture and use its failures to fix local routing, repository-grounded planning, and retry context. +- [x] Expand the evaluation corpus across concurrency, temporal semantics, and filesystem containment, with failure-inclusive suite aggregation and per-run time budgets. +- [x] Compare GPT-OSS and Qwen3-Coder on the same smoke corpus, review the apparent pass, and strengthen the contract after identifying a false positive. +- [x] Identify a local configuration that produces a human-reviewed fixture pass and measure three-run repeatability on the strengthened cache contract. +- [x] Find a fully GPU-backed local configuration that passes the strengthened safe-store contract, while retaining Qwen's extended-budget failure as negative evidence. +- [x] Stream verbose agent stages from the evidence suite without sacrificing the replayable output bundle. +- [x] Measure Devstral safe-store repeatability and retain the 0/3 timeout result alongside the earlier reviewed capability pass. +- [x] Launch Evidence-Based AI Engineering with a full paper, technical brief, reusable result figure, and evidence-linked distribution package. +- [ ] Improve local safe-store convergence; neither Qwen nor Devstral currently supports a reliability claim on the strengthened contract. +- [ ] Extend reviewed repeatability beyond one fixture before publishing a general coding-agent success-rate claim. +- [ ] Add an OpenCode importer to challenge the vendor-neutral evidence model. - [ ] Raise coverage in legacy workflow packages without presenting the public fixture as repository-wide coverage. diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..a3da318 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,106 @@ +# GPTCode agent evaluation + +The evaluation harness runs coding agents against the same clean Git fixture +and records an inspectable evidence bundle. It is designed to answer a narrow +question: did the agent produce a change that satisfies executable, +task-specific checks without losing the initial repository state? + +An experiment configuration names the agent process explicitly. Commands are +executed directly, without an implicit shell. The evidence directory must live +outside the evaluated repository so the run cannot accidentally measure its +own artifacts. + +```json +{ + "id": "go-ledger-deadlock", + "repository": "/tmp/go-ledger", + "output": "/tmp/go-ledger-evidence", + "agent": { + "name": "gptcode-local", + "args": [ + "/Users/jadercorrea/bin/gt", + "do", + "Fix the opposing-transfer deadlock without changing the public API" + ] + }, + "verifications": [ + { + "name": "tests-and-race-detector", + "args": ["go", "test", "-race", "./..."] + }, + { + "name": "static-analysis", + "args": ["go", "vet", "./..."] + } + ] +} +``` + +Run it with: + +```bash +go run ./scripts/evidence-run -config /tmp/experiment.json +``` + +Every result retains failures as evidence. A successful agent process does not +make a run pass: every verification command must also exit successfully. + +## Repeatability suites + +The suite runner creates a fresh Git repository for every fixture repetition +and aggregates all outcomes. It runs sequentially so local CPU and memory +contention do not bias model comparisons. + +```bash +go run ./scripts/evidence-suite \ + -verbose \ + -config benchmarks/suites/local-gpt-oss.json \ + -output /tmp/gptcode-go-core-quality +``` + +`-verbose` streams the agent's inspectable stages while retaining identical +stdout and stderr in the evidence bundle. It exposes model selection, planning, +tool execution, retries, and deterministic checks; it does not print private +model chain-of-thought. + +Use `benchmarks/suites/local-qwen3-coder.json` for the equivalent Qwen suite. +Each committed local suite records and executes its required GPTCode profile +selection in `setup.json`; the agent name is not merely a user-supplied label. +`benchmarks/suites/local-qwen3-coder-cache.json` preserves the first +human-reviewed three-run result on the strengthened cache contract. +`benchmarks/suites/local-devstral-small-2-safe-store.json` records the +hardware-tuned 16k Devstral configuration that produced the first reviewed +pass on strengthened filesystem containment. + +The initial corpus covers three distinct failure classes: + +- lock ordering and deterministic concurrency; +- TTL boundary semantics under concurrent cache access; +- path traversal and symlink containment at a filesystem boundary. + +The committed suite uses three repetitions. During harness development, +`-repetitions 1` provides a smoke test without presenting it as a consistency +measurement. `run_timeout_seconds` is a hard agent budget; verification still +runs after a timeout so the resulting repository state remains evidence. +Fixtures with `require_failing_baseline` also write `baseline.json` and abort +the suite if every check already passes before the agent runs. + +Development results are recorded under `benchmarks/results/`. They distinguish +system defects from model failures and never promote a one-run smoke test to a +repeatability claim. + +## Bundle contract + +The generated directory contains: + +- `manifest.json`: schema, base commit, timestamps, and snapshot hashes. +- `initial.tar` and `final.tar`: deterministic repository snapshots excluding + Git metadata. +- `initial.patch`, `agent.patch`, and `final.patch`: binary-capable Git diffs. +- `events.jsonl`: lifecycle events. +- `commands.jsonl`: exact commands, outputs, exit codes, and durations. +- `verification.json`: the machine-readable pass/fail decision. +- `report.md`: a compact human-readable summary. + +Snapshots are intentionally complete. Only use controlled public fixtures: +real repositories may contain credentials or proprietary untracked files. diff --git a/benchmarks/fixtures/go-expiring-cache/cache.go b/benchmarks/fixtures/go-expiring-cache/cache.go new file mode 100644 index 0000000..71564b0 --- /dev/null +++ b/benchmarks/fixtures/go-expiring-cache/cache.go @@ -0,0 +1,53 @@ +package expiringcache + +import ( + "sync" + "time" +) + +type entry struct { + value string + expiresAt time.Time +} + +type Cache struct { + mu sync.RWMutex + entries map[string]entry + now func() time.Time +} + +func New() *Cache { + return newWithClock(time.Now) +} + +func newWithClock(now func() time.Time) *Cache { + return &Cache{ + entries: make(map[string]entry), + now: now, + } +} + +func (c *Cache) Set(key, value string, ttl time.Duration) { + c.mu.Lock() + defer c.mu.Unlock() + c.entries[key] = entry{ + value: value, + expiresAt: c.now().Add(ttl), + } +} + +func (c *Cache) Get(key string) (string, bool) { + c.mu.RLock() + defer c.mu.RUnlock() + item, found := c.entries[key] + if !found { + return "", false + } + return item.value, true +} + +func (c *Cache) Len() int { + c.mu.RLock() + defer c.mu.RUnlock() + return len(c.entries) +} diff --git a/benchmarks/fixtures/go-expiring-cache/cache_test.go b/benchmarks/fixtures/go-expiring-cache/cache_test.go new file mode 100644 index 0000000..b65ef28 --- /dev/null +++ b/benchmarks/fixtures/go-expiring-cache/cache_test.go @@ -0,0 +1,67 @@ +package expiringcache + +import ( + "sync" + "testing" + "time" +) + +func TestGetReturnsLiveEntry(t *testing.T) { + now := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC) + cache := newWithClock(func() time.Time { return now }) + cache.Set("session", "active", time.Minute) + + value, found := cache.Get("session") + if !found || value != "active" { + t.Fatalf("Get() = %q, %v, want active, true", value, found) + } +} + +func TestGetEvictsExpiredEntry(t *testing.T) { + now := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC) + cache := newWithClock(func() time.Time { return now }) + cache.Set("session", "expired", time.Minute) + now = now.Add(time.Minute) + + if value, found := cache.Get("session"); found || value != "" { + t.Fatalf("Get() = %q, %v, want empty, false", value, found) + } + if length := cache.Len(); length != 0 { + t.Fatalf("Len() = %d, want expired entry removed", length) + } +} + +func TestLenPurgesExpiredEntriesWithoutLookup(t *testing.T) { + now := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC) + cache := newWithClock(func() time.Time { return now }) + cache.Set("expired", "stale", time.Minute) + cache.Set("live", "current", 2*time.Minute) + now = now.Add(time.Minute) + + if length := cache.Len(); length != 1 { + t.Fatalf("Len() = %d, want only the live entry", length) + } + if value, found := cache.Get("live"); !found || value != "current" { + t.Fatalf("Get(live) = %q, %v, want current, true", value, found) + } +} + +func TestCacheSupportsConcurrentReadersAndWriters(t *testing.T) { + cache := New() + var workers sync.WaitGroup + for worker := range 8 { + workers.Add(1) + go func() { + defer workers.Done() + for iteration := range 250 { + key := string(rune('a' + worker)) + cache.Set(key, "value", time.Minute) + cache.Get(key) + if iteration%10 == 0 { + cache.Len() + } + } + }() + } + workers.Wait() +} diff --git a/benchmarks/fixtures/go-expiring-cache/go.mod b/benchmarks/fixtures/go-expiring-cache/go.mod new file mode 100644 index 0000000..6926ffb --- /dev/null +++ b/benchmarks/fixtures/go-expiring-cache/go.mod @@ -0,0 +1,3 @@ +module example.com/expiringcache + +go 1.22 diff --git a/benchmarks/fixtures/go-expiring-cache/quality_test.go b/benchmarks/fixtures/go-expiring-cache/quality_test.go new file mode 100644 index 0000000..39f8ea9 --- /dev/null +++ b/benchmarks/fixtures/go-expiring-cache/quality_test.go @@ -0,0 +1,22 @@ +package expiringcache + +import ( + "bytes" + "go/format" + "os" + "testing" +) + +func TestCacheImplementationIsFormatted(t *testing.T) { + source, err := os.ReadFile("cache.go") + if err != nil { + t.Fatal(err) + } + formatted, err := format.Source(source) + if err != nil { + t.Fatalf("format.Source() error = %v", err) + } + if !bytes.Equal(source, formatted) { + t.Fatal("cache.go is not gofmt-formatted") + } +} diff --git a/benchmarks/fixtures/go-expiring-cache/task.md b/benchmarks/fixtures/go-expiring-cache/task.md new file mode 100644 index 0000000..b2ae0ef --- /dev/null +++ b/benchmarks/fixtures/go-expiring-cache/task.md @@ -0,0 +1,16 @@ +# Enforce cache expiration + +`Cache.Get` currently returns entries after their TTL has elapsed, and `Len` +continues to count those stale entries. + +Make expiration authoritative while preserving the exported API. A lookup at +the exact expiration instant is expired and must remove the stale entry. +Maintain race-free concurrent reads and writes; do not introduce background +goroutines or wall-clock sleeps. + +The implementation must pass: + +```text +go test -race ./... +go vet ./... +``` diff --git a/benchmarks/fixtures/go-ledger/concurrency_test.go b/benchmarks/fixtures/go-ledger/concurrency_test.go new file mode 100644 index 0000000..b87f599 --- /dev/null +++ b/benchmarks/fixtures/go-ledger/concurrency_test.go @@ -0,0 +1,68 @@ +package ledger + +import ( + "sync" + "testing" + "time" +) + +func TestSelfTransferCompletes(t *testing.T) { + account := NewAccount(1_000) + completed := make(chan error, 1) + go func() { + completed <- Transfer(account, account, 10) + }() + + select { + case err := <-completed: + if err != nil { + t.Fatalf("Transfer() error = %v", err) + } + case <-time.After(250 * time.Millisecond): + t.Fatal("self-transfer deadlocked") + } + + if balance := account.Balance(); balance != 1_000 { + t.Errorf("balance = %d, want 1000", balance) + } +} + +func TestOpposingTransfersComplete(t *testing.T) { + left := NewAccount(1_000) + right := NewAccount(1_000) + start := make(chan struct{}) + var workers sync.WaitGroup + workers.Add(2) + + go func() { + defer workers.Done() + <-start + for range 1_000 { + _ = Transfer(left, right, 1) + } + }() + go func() { + defer workers.Done() + <-start + for range 1_000 { + _ = Transfer(right, left, 1) + } + }() + + close(start) + completed := make(chan struct{}) + go func() { + workers.Wait() + close(completed) + }() + + select { + case <-completed: + case <-time.After(2 * time.Second): + t.Fatal("opposing transfers deadlocked") + } + + if total := left.Balance() + right.Balance(); total != 2_000 { + t.Errorf("total balance = %d, want 2000", total) + } +} diff --git a/benchmarks/fixtures/go-ledger/go.mod b/benchmarks/fixtures/go-ledger/go.mod new file mode 100644 index 0000000..4bb9136 --- /dev/null +++ b/benchmarks/fixtures/go-ledger/go.mod @@ -0,0 +1,3 @@ +module example.com/ledger + +go 1.22 diff --git a/benchmarks/fixtures/go-ledger/ledger.go b/benchmarks/fixtures/go-ledger/ledger.go new file mode 100644 index 0000000..3598890 --- /dev/null +++ b/benchmarks/fixtures/go-ledger/ledger.go @@ -0,0 +1,38 @@ +package ledger + +import ( + "errors" + "sync" +) + +var ErrInsufficientFunds = errors.New("insufficient funds") + +type Account struct { + mu sync.Mutex + balance int +} + +func NewAccount(balance int) *Account { + return &Account{balance: balance} +} + +func (a *Account) Balance() int { + a.mu.Lock() + defer a.mu.Unlock() + return a.balance +} + +// Transfer moves funds while preserving the total held by both accounts. +func Transfer(from, to *Account, amount int) error { + from.mu.Lock() + defer from.mu.Unlock() + to.mu.Lock() + defer to.mu.Unlock() + + if from.balance < amount { + return ErrInsufficientFunds + } + from.balance -= amount + to.balance += amount + return nil +} diff --git a/benchmarks/fixtures/go-ledger/ledger_test.go b/benchmarks/fixtures/go-ledger/ledger_test.go new file mode 100644 index 0000000..251e47d --- /dev/null +++ b/benchmarks/fixtures/go-ledger/ledger_test.go @@ -0,0 +1,33 @@ +package ledger + +import ( + "errors" + "testing" +) + +func TestTransfer(t *testing.T) { + from := NewAccount(100) + to := NewAccount(25) + + if err := Transfer(from, to, 40); err != nil { + t.Fatalf("Transfer() error = %v", err) + } + if got := from.Balance(); got != 60 { + t.Errorf("source balance = %d, want 60", got) + } + if got := to.Balance(); got != 65 { + t.Errorf("destination balance = %d, want 65", got) + } +} + +func TestTransferRejectsInsufficientFunds(t *testing.T) { + from := NewAccount(10) + to := NewAccount(20) + + if err := Transfer(from, to, 11); !errors.Is(err, ErrInsufficientFunds) { + t.Fatalf("Transfer() error = %v, want ErrInsufficientFunds", err) + } + if got := from.Balance() + to.Balance(); got != 30 { + t.Errorf("total balance = %d, want 30", got) + } +} diff --git a/benchmarks/fixtures/go-ledger/quality_test.go b/benchmarks/fixtures/go-ledger/quality_test.go new file mode 100644 index 0000000..998c6cd --- /dev/null +++ b/benchmarks/fixtures/go-ledger/quality_test.go @@ -0,0 +1,41 @@ +package ledger + +import ( + "bytes" + "go/format" + "go/parser" + "go/token" + "os" + "strconv" + "testing" +) + +func TestLedgerImplementationIsFormatted(t *testing.T) { + source, err := os.ReadFile("ledger.go") + if err != nil { + t.Fatal(err) + } + formatted, err := format.Source(source) + if err != nil { + t.Fatalf("format.Source() error = %v", err) + } + if !bytes.Equal(source, formatted) { + t.Fatal("ledger.go is not gofmt-formatted") + } +} + +func TestLedgerImplementationAvoidsUnsafe(t *testing.T) { + file, err := parser.ParseFile(token.NewFileSet(), "ledger.go", nil, parser.ImportsOnly) + if err != nil { + t.Fatal(err) + } + for _, imported := range file.Imports { + path, err := strconv.Unquote(imported.Path.Value) + if err != nil { + t.Fatal(err) + } + if path == "unsafe" { + t.Fatal("ledger.go must not depend on package unsafe for lock ordering") + } + } +} diff --git a/benchmarks/fixtures/go-ledger/task.md b/benchmarks/fixtures/go-ledger/task.md new file mode 100644 index 0000000..8d39fd4 --- /dev/null +++ b/benchmarks/fixtures/go-ledger/task.md @@ -0,0 +1,17 @@ +# Eliminate opposing-transfer deadlocks + +Two goroutines transferring funds in opposite directions can each hold one +account lock while waiting forever for the other. A transfer to the same +account deterministically attempts to lock the same mutex twice. + +Fix both deadlocks while preserving the existing exported API, error semantics, +and total balance. Keep the implementation idiomatic, `gofmt`-formatted, and +do not use package `unsafe`. The provided regression and quality tests must +pass: + +```text +go test -race ./... +go vet ./... +``` + +Do not weaken or remove synchronization. diff --git a/benchmarks/fixtures/go-safe-store/go.mod b/benchmarks/fixtures/go-safe-store/go.mod new file mode 100644 index 0000000..a05a534 --- /dev/null +++ b/benchmarks/fixtures/go-safe-store/go.mod @@ -0,0 +1,3 @@ +module example.com/safestore + +go 1.22 diff --git a/benchmarks/fixtures/go-safe-store/quality_test.go b/benchmarks/fixtures/go-safe-store/quality_test.go new file mode 100644 index 0000000..c368717 --- /dev/null +++ b/benchmarks/fixtures/go-safe-store/quality_test.go @@ -0,0 +1,22 @@ +package safestore + +import ( + "bytes" + "go/format" + "os" + "testing" +) + +func TestStoreImplementationIsFormatted(t *testing.T) { + source, err := os.ReadFile("store.go") + if err != nil { + t.Fatal(err) + } + formatted, err := format.Source(source) + if err != nil { + t.Fatalf("format.Source() error = %v", err) + } + if !bytes.Equal(source, formatted) { + t.Fatal("store.go is not gofmt-formatted") + } +} diff --git a/benchmarks/fixtures/go-safe-store/store.go b/benchmarks/fixtures/go-safe-store/store.go new file mode 100644 index 0000000..5c02463 --- /dev/null +++ b/benchmarks/fixtures/go-safe-store/store.go @@ -0,0 +1,22 @@ +package safestore + +import ( + "os" + "path/filepath" +) + +type Store struct { + root string +} + +func New(root string) *Store { + return &Store{root: root} +} + +func (s *Store) Write(name string, content []byte) error { + path := filepath.Join(s.root, name) + if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil { + return err + } + return os.WriteFile(path, content, 0o600) +} diff --git a/benchmarks/fixtures/go-safe-store/store_test.go b/benchmarks/fixtures/go-safe-store/store_test.go new file mode 100644 index 0000000..5564b7a --- /dev/null +++ b/benchmarks/fixtures/go-safe-store/store_test.go @@ -0,0 +1,109 @@ +package safestore + +import ( + "errors" + "os" + "path/filepath" + "testing" +) + +func TestWriteStoresNestedFileInsideRoot(t *testing.T) { + root := t.TempDir() + store := New(root) + + if err := store.Write("reports/summary.txt", []byte("verified\n")); err != nil { + t.Fatalf("Write() error = %v", err) + } + content, err := os.ReadFile(filepath.Join(root, "reports", "summary.txt")) + if err != nil { + t.Fatal(err) + } + if string(content) != "verified\n" { + t.Fatalf("content = %q", content) + } +} + +func TestWriteAllowsDotsInsidePathSegment(t *testing.T) { + root := t.TempDir() + store := New(root) + + name := "reports/v1..v2.txt" + if err := store.Write(name, []byte("valid\n")); err != nil { + t.Fatalf("Write() error = %v for legitimate name", err) + } + content, err := os.ReadFile(filepath.Join(root, filepath.FromSlash(name))) + if err != nil { + t.Fatal(err) + } + if string(content) != "valid\n" { + t.Fatalf("content = %q", content) + } +} + +func TestWriteRejectsTraversal(t *testing.T) { + parent := t.TempDir() + parent, err := filepath.EvalSymlinks(parent) + if err != nil { + t.Fatal(err) + } + root := filepath.Join(parent, "store") + if err := os.Mkdir(root, 0o700); err != nil { + t.Fatal(err) + } + store := New(root) + + err = store.Write("../escaped.txt", []byte("escape")) + if err == nil { + t.Fatal("Write() error = nil, want traversal rejection") + } + if _, statErr := os.Stat(filepath.Join(parent, "escaped.txt")); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("file escaped root: %v", statErr) + } +} + +func TestWriteRejectsTraversalAfterNormalizingSegments(t *testing.T) { + parent := t.TempDir() + parent, err := filepath.EvalSymlinks(parent) + if err != nil { + t.Fatal(err) + } + root := filepath.Join(parent, "store") + if err := os.Mkdir(root, 0o700); err != nil { + t.Fatal(err) + } + store := New(root) + + err = store.Write("reports/../../escaped.txt", []byte("escape")) + if err == nil { + t.Fatal("Write() error = nil, want normalized traversal rejection") + } + if _, statErr := os.Stat(filepath.Join(parent, "escaped.txt")); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("file escaped root: %v", statErr) + } +} + +func TestWriteRejectsAbsolutePath(t *testing.T) { + root := t.TempDir() + store := New(root) + + if err := store.Write(filepath.Join(t.TempDir(), "escaped.txt"), []byte("escape")); err == nil { + t.Fatal("Write() error = nil, want absolute path rejection") + } +} + +func TestWriteRejectsSymlinkedParent(t *testing.T) { + root := t.TempDir() + outside := t.TempDir() + if err := os.Symlink(outside, filepath.Join(root, "linked")); err != nil { + t.Fatal(err) + } + store := New(root) + + err := store.Write("linked/escaped.txt", []byte("escape")) + if err == nil { + t.Fatal("Write() error = nil, want symlink rejection") + } + if _, statErr := os.Stat(filepath.Join(outside, "escaped.txt")); !errors.Is(statErr, os.ErrNotExist) { + t.Fatalf("file escaped through symlink: %v", statErr) + } +} diff --git a/benchmarks/fixtures/go-safe-store/task.md b/benchmarks/fixtures/go-safe-store/task.md new file mode 100644 index 0000000..60ee6ca --- /dev/null +++ b/benchmarks/fixtures/go-safe-store/task.md @@ -0,0 +1,16 @@ +# Contain writes inside the store root + +`Store.Write` accepts attacker-controlled names. Traversal, absolute paths, and +symlinked parent directories can currently write outside the configured root. + +Contain every write beneath the root while preserving the exported API and +support for legitimate nested paths. Reject unsafe input before creating +directories or files. Do not resolve the problem by deleting symlinks, changing +the process working directory, or weakening file permissions. + +The implementation must pass: + +```text +go test -race ./... +go vet ./... +``` diff --git a/benchmarks/results/2026-07-28-local-smoke.md b/benchmarks/results/2026-07-28-local-smoke.md new file mode 100644 index 0000000..cd13283 --- /dev/null +++ b/benchmarks/results/2026-07-28-local-smoke.md @@ -0,0 +1,116 @@ +# Local GPT-OSS smoke evaluation — 2026-07-28 + +This is a development smoke test, not a benchmark claim. Each fixture ran once; +repeatability requires the three repetitions declared by the suite. + +## Corpus + +| Fixture | Failure class | Deterministic contract | +| --- | --- | --- | +| `ledger-deadlock` | lock ordering and self-deadlock | tests, race detector, formatting, no `unsafe`, vet | +| `cache-expiration` | TTL boundary and stale eviction | tests, race detector, formatting, vet | +| `safe-store` | traversal and symlink containment | tests, race detector, formatting, vet | + +## First smoke + +The first run exposed harness and CLI defects before it could measure the model +fairly: + +- a planner stage escaped the local profile and attempted OpenRouter; +- structured `required_files` output could not be decoded; +- Go formatting remained probabilistic. + +Result: **0/3 passed**, median duration **1m57s**. These runs remain preserved +locally, but are not treated as model-performance evidence. + +## Corrected smoke + +The CLI was rebuilt with: + +- a hard prohibition on cloud backends in local mode; +- support for string and structured movement file references; +- deterministic `gofmt` on modified Go files before verification. + +| Fixture | Result | Duration | Final failure | +| --- | ---: | ---: | --- | +| `ledger-deadlock` | failed | 7m38s | invalid imports and unresolved `unsafe` reference | +| `cache-expiration` | failed | 5m50s | `Len` still counted the expired entry | +| `safe-store` | failed | 5m31s | malformed final source after retries | + +Corrected result: **0/3 passed**, median duration **5m50s**. + +No corrected run contacted OpenRouter. The model identified each root cause and +made relevant edits, but did not converge to a verified implementation within +three attempts. The current evidence therefore supports local exercisability, +not acceptable coding-agent reliability. + +## Next measurement + +Do not publish a comparative success-rate claim yet. First: + +1. enforce the ten-minute per-run budget in the suite; +2. test a faster local model on the same corpus; +3. run all three repetitions only for configurations that pass at least one + smoke fixture; +4. publish every run, including failures and timeouts. + +## Qwen3-Coder comparison + +The same corrected 1×3 smoke was run with `qwen3-coder:latest`. + +| Fixture | Initial result | Duration | +| --- | ---: | ---: | +| `ledger-deadlock` | failed | 1m38s | +| `cache-expiration` | failed | 1m41s | +| `safe-store` | passed contract | 1m51s | + +Initial aggregate: **1/3 passed**, median **1m41s**. This was substantially +faster than GPT-OSS, but human review found that the safe-store implementation +rejected every name containing `..`, including the legitimate +`reports/v1..v2.txt`. + +The contract was strengthened to preserve dots inside ordinary path segments. +A new Qwen safe-store run then failed in **1m35s** after three attempts. +Therefore the earlier pass is classified as a contract false positive, not a +quality-confirmed success. + +That conclusion changed after closing the editor feedback loop. + +## Grounded closed-loop Qwen result + +The CLI previously stopped the editor as soon as every planned file had been +touched. It also omitted tests and `task.md` from planning and retry context. +The corrected loop now: + +- grounds the initial plan and retries in implementation, tests, and task + constraints; +- lets the editor run verification and repair the same file within one turn; +- permits up to twenty tool calls for inspect → edit → verify → repair; +- rejects empty local-model answers and propagates Ollama HTTP errors. + +Human review found two more fixture weaknesses before accepting a result: + +- safe-store traversal appeared to pass only because `/var` is a symlink on + macOS; the strengthened test resolves the temporary root first; +- cache `Len` was not required to purge expired entries unless `Get` had + already been called; the strengthened test now checks independent cleanup. + +Qwen did not solve the strengthened safe-store contract within the ten-minute +budget. It did solve the strengthened cache contract, and three clean +repetitions produced: + +| Repetition | Result | Duration | +| --- | ---: | ---: | +| 1 | passed | 4m07s | +| 2 | passed | 3m56s | +| 3 | passed | 4m46s | + +Reviewed aggregate: **3/3 passed**, median **4m07s**, with `go test -race ./...` +and `go vet ./...` passing after every agent run. The three patches used +different locking strategies but all preserved the exported API, treated the +exact TTL boundary as expired, and purged stale entries through both `Get` and +`Len`. + +This is evidence of repeatability for one bounded temporal/concurrency task, +not a general coding-agent success-rate claim. Filesystem containment remains +an observed failure class. diff --git a/benchmarks/results/2026-07-29-local-filesystem.md b/benchmarks/results/2026-07-29-local-filesystem.md new file mode 100644 index 0000000..5d116e9 --- /dev/null +++ b/benchmarks/results/2026-07-29-local-filesystem.md @@ -0,0 +1,93 @@ +# Local filesystem-containment evaluation — 2026-07-29 + +This development evaluation compares local configurations on the strengthened +`safe-store` fixture. It is a single-run capability check, not a repeatability +or general success-rate claim. + +## Hardware and execution + +- Apple M1 Pro, 16 GPU cores, 32 GB unified memory. +- Ollama served every measured inference locally. +- `ollama ps` reported Qwen at 24 GB and **100% GPU** with a 65,536-token + context. +- Devstral Small 2 used 25 GB at 65,536 tokens and offloaded 8% to CPU. +- Setting `GPTCODE_OLLAMA_CONTEXT_LENGTH=16384` reduced Devstral to 17 GB and + kept it at **100% GPU**. + +Kimi was not evaluated locally. Current Kimi releases do not fit this hardware +as a comparable fully local coding-agent configuration. + +## Adaptive retry budget + +The CLI now stops after three equivalent deterministic-verification failures. +Timing noise is normalized before comparison, while a changed failure resets +the counter. This permits larger local-model budgets without spending every +attempt on a demonstrated plateau. + +Qwen received 30 minutes and eight allowed attempts. It stopped after six +attempts in **16m27s** when the symlink-parent failure repeated three times. +More time did not resolve the strengthened contract. + +## Devstral calibration + +The first Devstral run exposed a fixed 120-second Ollama client timeout before +the model could edit. The provider now defaults to ten minutes per request and +supports: + +- `GPTCODE_OLLAMA_TIMEOUT`, parsed as a Go duration; +- `GPTCODE_OLLAMA_CONTEXT_LENGTH`, sent to Ollama as `num_ctx`. + +At 32k, Devstral remained fully GPU-backed but timed out at 30 minutes after +repeating an incorrect `EvalSymlinks` strategy and entering a no-diff tool +loop. + +At 16k, Devstral completed in **26m08s**: + +| Gate | Result | +| --- | ---: | +| Agent exit | passed | +| `go test -count=1 -race ./...` | passed | +| `go vet ./...` | passed | +| Files modified | 1 | +| API cost | $0 | + +The first edit failed only the symlinked-parent test. The second edit passed +the deterministic contract. The evidence suite streamed the verbose execution +while preserving the same stdout and stderr in `commands.jsonl`. + +## Human review + +The Devstral patch correctly rejects the fixture's traversal, absolute-path, +and pre-existing symlink cases while permitting nested paths and ordinary dots +inside path segments. + +It is not presented as a general production filesystem primitive. Its +symlink check and subsequent write are separate operations, leaving a TOCTOU +window under a concurrently hostile filesystem. The result proves the bounded +fixture contract, not race-free containment against an active attacker. + +One reviewed pass justifies further repetition. It does not justify a general +model reliability claim. + +## Three-run repeatability result + +The same committed 16k configuration was then run three times from clean +fixtures with a 30-minute budget per run. + +| Repetition | Result | Duration | Final observed failure | +| --- | ---: | ---: | --- | +| 1 | timed out | 30m01s | legitimate nested paths rejected | +| 2 | timed out | 30m01s | legitimate nested paths rejected | +| 3 | timed out | 30m01s | absolute path and symlink escape accepted | + +Repeatability result: **0/3 passed**, median **30m01s**. + +All three runs remained fully local and GPU-backed. They produced relevant +edits, but did not converge before the task budget. The earlier reviewed pass +remains evidence that this configuration can solve the bounded contract; the +repeatability suite is stronger evidence that it cannot yet do so reliably. + +The runs also exposed nondeterministic Go temporary-directory IDs in otherwise +equivalent test failures. The plateau detector now normalizes those paths so +future local runs do not mistake ephemeral filesystem names for engineering +progress. diff --git a/benchmarks/suites/local-devstral-small-2-safe-store.json b/benchmarks/suites/local-devstral-small-2-safe-store.json new file mode 100644 index 0000000..7f3a08c --- /dev/null +++ b/benchmarks/suites/local-devstral-small-2-safe-store.json @@ -0,0 +1,43 @@ +{ + "id": "go-safe-store-devstral-small-2", + "output": "/tmp/gptcode-devstral-small-2-safe-store", + "repetitions": 1, + "run_timeout_seconds": 1800, + "setup": [ + { + "name": "select-local-devstral-profile", + "args": ["gt", "profiles", "use", "local.devstral"] + } + ], + "agent": { + "name": "gptcode-local-devstral-small-2-16k", + "args": [ + "/usr/bin/env", + "GPTCODE_OLLAMA_CONTEXT_LENGTH=16384", + "GPTCODE_OLLAMA_TIMEOUT=6m", + "gt", + "do", + "Read task.md, implement the requested fix, and meet every regression and quality constraint in the repository.", + "--verbose", + "--max-attempts", + "8" + ] + }, + "fixtures": [ + { + "id": "safe-store", + "source": "benchmarks/fixtures/go-safe-store", + "require_failing_baseline": true, + "verifications": [ + { + "name": "tests-race-and-quality-contract", + "args": ["go", "test", "-count=1", "-race", "./..."] + }, + { + "name": "static-analysis", + "args": ["go", "vet", "./..."] + } + ] + } + ] +} diff --git a/benchmarks/suites/local-gpt-oss.json b/benchmarks/suites/local-gpt-oss.json new file mode 100644 index 0000000..1251e37 --- /dev/null +++ b/benchmarks/suites/local-gpt-oss.json @@ -0,0 +1,69 @@ +{ + "id": "go-core-quality", + "output": "/tmp/gptcode-go-core-quality", + "repetitions": 3, + "run_timeout_seconds": 600, + "setup": [ + { + "name": "select-local-gpt-oss-profile", + "args": ["gt", "profiles", "use", "local.gptoss"] + } + ], + "agent": { + "name": "gptcode-local-gpt-oss", + "args": [ + "gt", + "do", + "Read task.md, implement the requested fix, and meet every regression and quality constraint in the repository.", + "--max-attempts", + "3" + ] + }, + "fixtures": [ + { + "id": "ledger-deadlock", + "source": "benchmarks/fixtures/go-ledger", + "require_failing_baseline": true, + "verifications": [ + { + "name": "tests-race-and-quality-contract", + "args": ["go", "test", "-count=1", "-race", "./..."] + }, + { + "name": "static-analysis", + "args": ["go", "vet", "./..."] + } + ] + }, + { + "id": "cache-expiration", + "source": "benchmarks/fixtures/go-expiring-cache", + "require_failing_baseline": true, + "verifications": [ + { + "name": "tests-race-and-quality-contract", + "args": ["go", "test", "-count=1", "-race", "./..."] + }, + { + "name": "static-analysis", + "args": ["go", "vet", "./..."] + } + ] + }, + { + "id": "safe-store", + "source": "benchmarks/fixtures/go-safe-store", + "require_failing_baseline": true, + "verifications": [ + { + "name": "tests-race-and-quality-contract", + "args": ["go", "test", "-count=1", "-race", "./..."] + }, + { + "name": "static-analysis", + "args": ["go", "vet", "./..."] + } + ] + } + ] +} diff --git a/benchmarks/suites/local-qwen3-coder-cache.json b/benchmarks/suites/local-qwen3-coder-cache.json new file mode 100644 index 0000000..f84d261 --- /dev/null +++ b/benchmarks/suites/local-qwen3-coder-cache.json @@ -0,0 +1,39 @@ +{ + "id": "go-cache-qwen3-coder-reviewed", + "output": "/tmp/gptcode-qwen-cache-reviewed", + "repetitions": 3, + "run_timeout_seconds": 600, + "setup": [ + { + "name": "select-local-qwen3-coder-profile", + "args": ["gt", "profiles", "use", "local.evaluation"] + } + ], + "agent": { + "name": "gptcode-local-qwen3-coder", + "args": [ + "gt", + "do", + "Read task.md, implement the requested fix, and meet every regression and quality constraint in the repository.", + "--max-attempts", + "3" + ] + }, + "fixtures": [ + { + "id": "cache-expiration", + "source": "benchmarks/fixtures/go-expiring-cache", + "require_failing_baseline": true, + "verifications": [ + { + "name": "tests-race-and-quality-contract", + "args": ["go", "test", "-count=1", "-race", "./..."] + }, + { + "name": "static-analysis", + "args": ["go", "vet", "./..."] + } + ] + } + ] +} diff --git a/benchmarks/suites/local-qwen3-coder.json b/benchmarks/suites/local-qwen3-coder.json new file mode 100644 index 0000000..17a460d --- /dev/null +++ b/benchmarks/suites/local-qwen3-coder.json @@ -0,0 +1,69 @@ +{ + "id": "go-core-quality-qwen3-coder", + "output": "/tmp/gptcode-go-core-quality-qwen3-coder", + "repetitions": 3, + "run_timeout_seconds": 600, + "setup": [ + { + "name": "select-local-qwen3-coder-profile", + "args": ["gt", "profiles", "use", "local.evaluation"] + } + ], + "agent": { + "name": "gptcode-local-qwen3-coder", + "args": [ + "gt", + "do", + "Read task.md, implement the requested fix, and meet every regression and quality constraint in the repository.", + "--max-attempts", + "3" + ] + }, + "fixtures": [ + { + "id": "ledger-deadlock", + "source": "benchmarks/fixtures/go-ledger", + "require_failing_baseline": true, + "verifications": [ + { + "name": "tests-race-and-quality-contract", + "args": ["go", "test", "-count=1", "-race", "./..."] + }, + { + "name": "static-analysis", + "args": ["go", "vet", "./..."] + } + ] + }, + { + "id": "cache-expiration", + "source": "benchmarks/fixtures/go-expiring-cache", + "require_failing_baseline": true, + "verifications": [ + { + "name": "tests-race-and-quality-contract", + "args": ["go", "test", "-count=1", "-race", "./..."] + }, + { + "name": "static-analysis", + "args": ["go", "vet", "./..."] + } + ] + }, + { + "id": "safe-store", + "source": "benchmarks/fixtures/go-safe-store", + "require_failing_baseline": true, + "verifications": [ + { + "name": "tests-race-and-quality-contract", + "args": ["go", "test", "-count=1", "-race", "./..."] + }, + { + "name": "static-analysis", + "args": ["go", "vet", "./..."] + } + ] + } + ] +} diff --git a/docs/_distribution/2026-07-29-capability-is-not-reliability.md b/docs/_distribution/2026-07-29-capability-is-not-reliability.md new file mode 100644 index 0000000..e9dfc92 --- /dev/null +++ b/docs/_distribution/2026-07-29-capability-is-not-reliability.md @@ -0,0 +1,266 @@ +# Capability Is Not Reliability + +Canonical paper: +https://gptcode.dev/blog/2026-07-29-one-successful-agent-run-proves-almost-nothing + +Technical brief: +https://gptcode.dev/blog/2026-07-29-capability-is-not-reliability + +Evidence release: +https://github.com/jadercorrea/ai-experiments/releases/tag/2026.07.29.1 + +## LinkedIn + +I watched a local coding agent solve a security-sensitive task. + +It changed one Go file, preserved the public API, passed the tests under the +race detector, and passed static analysis. No hosted model was used. + +Then I ran the same configuration three more times. + +It failed every run. + +That experiment reinforced a distinction that is easy to miss when evaluating +AI systems: + +Capability is not reliability. + +One successful run proves that a system can produce a result. It tells us very +little about how often it will produce that result under controlled repetition. + +The first campaign demonstrated bounded capability: 1/1 passed in 26 minutes. +The subsequent repeatability campaign was 0/3, with every run reaching the +30-minute timeout. + +The failed runs were not noise to discard. They exposed different failures at +the security boundary and defects in the surrounding agent harness: hidden +progress, mismatched timeouts, wasteful retries, and incorrect final status. + +I published the full protocol, patches, command logs, snapshots, verification +results, limitations, and checksum as an immutable release. + +Engineering decisions should depend on repeated, inspectable evidence, not the +most memorable demonstration. + +One successful run demonstrates possibility. Engineering depends on +repeatability. + +Full paper: +https://gptcode.dev/blog/2026-07-29-one-successful-agent-run-proves-almost-nothing + +Raw evidence: +https://github.com/jadercorrea/ai-experiments/releases/tag/2026.07.29.1 + +## X thread + +1/10 + +I watched a local coding agent solve a security-sensitive Go task. It preserved +the public API, passed `go test -race`, passed `go vet`, and used no paid API. + +Then I ran the same configuration three more times. + +It failed every run. + +2/10 + +The lesson was not that local models are useless. + +It was that capability is not reliability. + +One successful run proves a system can produce a result. It does not tell us +how often it will. + +3/10 + +The first campaign was a capability check: + +1/1 passed in 26m08s. + +The subsequent repeatability campaign: + +0/3 passed. Every run timed out at roughly 30 minutes. + +These campaigns answer different questions and should not be combined into an +artificial 1/4 score. + +4/10 + +The task enforced filesystem containment: + +- preserve legitimate nested paths; +- reject traversal and absolute paths; +- reject a pre-existing symlink escape; +- preserve the public API; +- pass the race detector and static analysis. + +5/10 + +An earlier model “passed” by rejecting every filename containing `..`. + +That also rejected the legitimate name `v1..v2.txt`. + +The agent had not solved traversal. It had prohibited a substring. + +Better evaluation made the model look worse and the evidence more truthful. + +6/10 + +The failed runs mattered. + +Two rejected legitimate nested paths. Another accepted an absolute path and a +symlink escape. Some retries regressed cases that had already passed. + +Those artifacts explain more than an aggregate score. + +7/10 + +The experiment also found defects in the agent system: + +- a provider timeout shorter than the suite budget; +- progress hidden until completion; +- retries repeating equivalent failures; +- final status contradicting successful verification. + +8/10 + +Fixing those defects did not make the model smarter. + +It made the system more observable, economical, and honest when the model was +not capable enough. + +9/10 + +A useful agent evaluation retains the initial state, task, configuration, +commands, patch, failed runs, verification output, final snapshot, duration, +and human-review boundary. + +The unit of evidence is the restorable run, not the selected transcript. + +10/10 + +I published the complete paper and immutable 2026.07.29.1 evidence release. + +One successful run demonstrates possibility. + +Engineering depends on repeatability. + +https://gptcode.dev/blog/2026-07-29-one-successful-agent-run-proves-almost-nothing + +## Hacker News + +Suggested title: + +One Successful Agent Run Proves Almost Nothing + +Submission URL: + +https://gptcode.dev/blog/2026-07-29-one-successful-agent-run-proves-almost-nothing + +First comment: + +Author here. I ran a local coding model against a small but security-sensitive +filesystem-containment fixture. It passed one capability check, then timed out +in all three runs of a subsequent repeatability campaign. + +The article is about the evaluation method rather than the model: clean +workspaces, required failing baselines, retained failed patches, deterministic +verification, final snapshots, human-review boundaries, and why capability and +reliability are different claims. + +The raw evidence is published in the immutable `2026.07.29.1` release: +https://github.com/jadercorrea/ai-experiments/releases/tag/2026.07.29.1 + +The main limitation is also documented: the historical runner came from a +precursor working tree whose exact transient build commit was not retained. +Future campaigns require a clean immutable harness revision before execution. + +I would be especially interested in how others predeclare agent evaluations, +retain negative runs, and decide when a local model should escalate to a hosted +one. + +## Five-minute video + +### 0:00–0:35 — Hook + +I watched a local coding agent solve a security-sensitive task. It changed one +Go file, preserved the public API, passed the Go race detector, and passed +static analysis. + +Then I ran the same configuration three more times. It failed every run. + +That is the difference between capability and reliability. + +### 0:35–1:15 — The task + +The fixture was a small Go file store. Legitimate nested files had to work, but +absolute paths, parent traversal, and escape through a pre-existing symlink had +to fail. + +It also contained a false-positive guard: `v1..v2.txt` is a legitimate +filename. An earlier model looked successful only because it rejected every +name containing two adjacent dots. + +The contract had to distinguish a real solution from a plausible shortcut. + +### 1:15–2:05 — The result + +With Devstral Small 2 running through Ollama at a 16,384-token context, the +capability check passed in 26 minutes and 8 seconds. + +The same configuration then entered a separate repeatability campaign. Three +fresh workspaces, the same task, the same model profile, the same verification +commands, and the same 30-minute budget. + +All three runs timed out. + +Show the result figure here. + +The first campaign proved possibility. The second did not support a reliability +claim. + +### 2:05–3:00 — What failures revealed + +The failed runs were valuable. Two rejected legitimate nested paths. Another +accepted an absolute path and a symlink escape. Some retries regressed +previously passing behavior. + +The experiment also exposed defects in the agent harness: an internal timeout +shorter than the suite budget, no visible progress during long runs, repeated +equivalent failures, and a final status that could contradict successful +verification. + +Fixing these problems did not improve the model. It improved the truthfulness +of the system around it. + +### 3:00–4:00 — What counts as evidence + +For agent evaluation, I now want the initial repository, exact task and +configuration, clean workspace, command stream, patch, deterministic +verification, final snapshot, duration, failed runs, and human-review boundary. + +Tests establish truth only within the encoded contract. The successful patch +still retained a possible check-to-write race against an actively hostile +concurrent process. That limitation belongs in the result. + +### 4:00–4:40 — The local inference boundary + +Local AI may follow the path of recording tools. Much of the work that once +required a professional studio can now be done well at home. Studios remain +important for specialized work, but they are no longer mandatory for every +recording. + +Likewise, frontier models and managed inference will remain essential, but not +every bounded engineering task should necessarily require them. + +The system should try locally, verify deterministically, and escalate structured +evidence when progress plateaus. + +### 4:40–5:00 — Close + +The complete experiment, including failed patches and snapshots, is published +under release `2026.07.29.1`. + +One successful run demonstrates possibility. + +Engineering depends on repeatability. diff --git a/docs/_layouts/post.html b/docs/_layouts/post.html index f400c22..ff8b453 100644 --- a/docs/_layouts/post.html +++ b/docs/_layouts/post.html @@ -4,6 +4,11 @@

+ {% if page.series %} +

+ {{ page.series }}{% if page.format %} · {{ page.format }}{% endif %} +

+ {% endif %}

{{ page.title }}