Skip to content

Add the 44%-vs-85% cross-agent harness audit to §3, with deep note - #88

Open
uipreliga wants to merge 1 commit into
benchflow-ai:mainfrom
uipreliga:add-coder-eval-cross-agent-note
Open

uipreliga wants to merge 1 commit into
benchflow-ai:mainfrom
uipreliga:add-coder-eval-cross-agent-note

Conversation

@uipreliga

Copy link
Copy Markdown
Contributor

Adds one §3 entry plus its deep note under notes/articles/. This is the follow-up @xdotli invited on #61, where he called this case "exactly the kind of war story §3 and §6 are short on." Thanks for merging that one.

Affiliation, per CONTRIBUTING: I work on Coder Eval and co-authored the linked post. Disclosing up front — the tool entry in §5a was self-submitted too.

What it is

We ran Codex and Claude Code over the same 417 tasks and got 44.1% vs 85.3%. Before publishing that, we hand-audited 227 of the 228 failures — no sampling, every label required to cite a mechanism from the run record — and sorted them into four buckets:

Bucket Count Share
Harness-unfair 144 63%
Ambiguous 49 22%
Genuine 25 11%
Strict criteria 9 4%

11% of the failures were the model. The ambiguous bucket turned out to be one cluster of 30 tasks downstream of a single broken tool, which puts ~89% outside the model's control. The largest single cause was one flaky file-edit backend: 106 of 228 failures. Two others were our own eval bugs — an activation check that only recognized Claude Code's Skill call (30 tasks lost regardless of what Codex did), and a command check that graded command text rather than effect.

Then we fixed the harness and re-ran: 44% → 76%, against 82%. The gap collapsed.

Why it clears the bar

  • Show your work — a per-failure census with counts and named mechanisms, not a "harnesses matter" take. The post publishes the 22% we could not adjudicate rather than allocating it, states its own bias ("the incentive runs toward charity"), and says plainly that a 227-failure hand-audit does not scale and is a one-time instrument.
  • Every number is quotable — all figures appear verbatim in the linked post; the note's Verified quotes section pins ten of them.
  • It adds something §3 lacks. The section has the thesis from Lee, Brand's AlgoTune ranking flip, Hodgson's teardown, and Anthropic's VirBench spread. What it hasn't had is a practitioner reporting the number they nearly published and then itemizing why it was wrong — including a re-run that closes the loop. The reusable output is the inverse law of gap size: rank per-skill gaps, and the biggest ones are the least likely to be real, because infrastructure failures are binary and total while capability differences are graded. It comes with a decisive test (did that agent's tooling actually install for this skill?) and it points readers at their own most impressive-looking results.
  • ⚠️ Caveat, stated in the note: one suite, two agents, one org. The taxonomy and the heuristic travel; the specific percentages should not be read as a general Codex-vs-Claude-Code result.

Housekeeping

  • Note follows the house shape — Summary / Key points / Verified quotes (verbatim, with source URL) / What it adds / Themes.
  • Bumped 143 deep reading notes144 in both places it appears (header line and § Deep notes).
  • I did not cross-list into §6 even though you mentioned it, since CONTRIBUTING says cross-list only when genuinely warranted. Happy to add it there instead of, or as well as, §3 — your call.
  • Did not add a (notes: ...) pointer on the entry; only 2 of 443 entries carry one and both point at a research/notes/ path that no longer exists. Say the word if you want it, and whether that path should be fixed.

Trim the annotation if it runs long for the section — it's above the §3 median and I'd rather you cut it than have it sit oversized.

Signed-off-by: uipreliga <tomasz.religa@uipath.com>

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant