Launch Evidence-Based AI Engineering - #25
Merged
Merged
Conversation
Add a content-free Codex evidence scanner and a TDD-built, path-safe ordered patch replayer. Validate repository and turn-level evidence across the local corpus, exercise verification inside a locked-down Docker container, and document the critical finding that legacy sessions omit the initial dirty-worktree baseline required for deterministic replay.
Local agent runs were not credible evidence: historical sessions lacked reproducible baselines, fresh runs hid progress until completion, Ollama used a fixed timeout, and retry budgets could be exhausted without measurable improvement. The same gaps made local execution slower, opaque, and easy to overstate. Build deterministic, replayable evaluation bundles around clean fixtures; ground planning and retries in repository contracts; stream verbose stages while preserving captured output; tune Ollama timeout and context for available hardware; and stop equivalent verification plateaus early. Strengthened fixtures retain false positives and failures, while reviewed Qwen cache and Devstral filesystem results demonstrate bounded, zero-API-cost capability. General reliability still requires repeatability across more fixtures. The Devstral filesystem result is one reviewed contract pass, and its generated patch is not claimed as a race-free filesystem primitive against an actively hostile concurrent process.
The local-agent study had the depth of an engineering paper but made its central capability-versus-reliability result expensive to discover and difficult to distribute outside the site. Launch Evidence-Based AI Engineering with an executive summary, accessible experiment figure, stronger conclusion, 1,071-word technical brief, and measured copy for LinkedIn, X, Hacker News, and a five-minute video. Tie every format back to the immutable evidence release and protect the editorial contract with site tests. The paper remains the canonical technical account and the brief is an entry point, not a replacement. Browser-based page inspection was unavailable in this session, so the generated HTML, responsive markup, full-resolution graphic, Jekyll build, and repository quality gate were verified independently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
ai-experimentsrelease2026.07.29.1Why
A selected successful run demonstrated bounded capability but could not support a reliability claim. The following three-run campaign timed out every time. Publishing both campaigns, their negative artifacts, and their limitations makes the engineering claim inspectable instead of promotional.
Validation
make verifyLimitations
The historical runner predates a retained immutable build revision, and the successful filesystem patch retains a possible symlink check-to-write race. Both limitations are documented. The capability check and repeatability campaign remain separate because they served different experimental purposes.