Research Readiness Audit: Paper/Blog/Code/Meaningfulness Assessment

[nv78]

Summary

Audit of the CodeBench project against DESIGN_DOC.md on main, performed on branch claude/relaxed-albattani-FA8RR. Scores are 1-10.

Scores

Dimension	Score	Justification
Paper readiness	2/10	No draft existed before this audit; the only "results" anywhere in the repo (and in DESIGN_DOC.md) are explicitly hypothesized "Expected results" tables for experiments whose code (security scanners, semantic oracle, repo-integration harness, 1,800-task dataset) does not exist yet.
Blog post readiness	2/10	No accessible non-academic writeup existed. README.md described a benchmark with empty placeholder leaderboard tables, and DESIGN_DOC.md is written for a technical/research audience only.
Research code quality/completeness	4/10	src/codebench/ (core.py, data.py, evaluate.py) is clean, type-annotated (pydantic models), and has solid unit test coverage (tests/test_core.py, test_data.py, test_evaluate.py, test_suite_and_scoring.py all pass conceptually well-designed cases). However: it implements only a generic harness + scoring layer. data.py's make_benchmark/make_rollout_benchmark/make_test_suite generate seeded-random synthetic results, not real model outputs. There is no LLM integration, no Semgrep/Bandit/CodeQL integration (only a 7-pattern regex heuristic in evaluate.py::security_score), no semantic oracle, no repo-context harness, and no data/tasks/ (the 1,800-task dataset from the design doc doesn't exist — only 10 hand-written SAMPLE_TASKS). Nothing in the repo has ever produced a number describing a real coding assistant.
Research meaningfulness	7/10	The core idea — that pass@1 alone overstates production-readiness, and that security-safety, specification-gaming, and codebase-integration gaps deserve first-class metrics — is well-motivated, timely, and plausible given known issues with LLM-generated code security (cf. CyberSecEval-style findings) and benchmark contamination concerns. It's not deeply novel (similar concerns have been raised piecemeal elsewhere) but the combination into one benchmark with three concrete metrics (SSR/SGR/CIR) is a reasonable, citable contribution if executed at the claimed scale (1,800 tasks, 8 systems, human CWE labeling). The risk is scope: the design doc commits to a large empirical program (multi-scanner security analysis, semantic oracles, repo-context evaluation, contamination analysis) that is currently 0% implemented.

Improvement Plan

1. [Implemented] Add BLOG.md — plain-language summary of the vision, the three new metrics (SSR/SGR/CIR), and an explicit, honest "what's built vs. what's still synthetic" status section.
2. [Implemented] Add PAPER_DRAFT.md — paper skeleton mirroring DESIGN_DOC.md's structure (Abstract, Related Work placeholder, Dataset, Metrics, Experiments 0-5, Discussion, Limitations, Target Venue). Every numeric claim from DESIGN_DOC.md's "Expected Results" tables is explicitly re-tagged as (projected, pending full experiment run) rather than presented as a result.
3. [Implemented] Add experiments/exp0_baseline.py — the first script in the repo that performs real code execution (not RNG): it exec()s 5 of the 10 SAMPLE_TASKS reference solutions and runs hand-written smoke-test assertions against them, then computes real pass@1/pass@5 (via the existing leaderboard() function) and real security_score values. Output is written to results/exp0_baseline.json. This deliberately does not call any external LLM API or fabricate model-quality numbers — it validates that the harness's execution/scoring pipeline works end-to-end on real code, which is a prerequisite for ever running a real model through it.
4. [Implemented] Add results/README.md — explains what is/isn't a real measured result, to prevent results/ from silently accumulating synthetic numbers that get mistaken for findings later.
5. [Implemented] Add tests/test_exp0_baseline.py and link all new docs from README.md, including replacing the empty Agent Comparison Table's implicit claim with an explicit "intentionally empty, no real agent has been run yet" note.
6. [Left as future work] Wire up one real LLM API end-to-end (prompt → generated code → real pytest execution) for at least one of the 10 sample tasks, to get the first genuinely model-derived pass@1 number.
7. [Left as future work] Replace the security_score regex heuristic with a real Bandit integration (pure-Python AST scanner, no external service needed) and compare its findings against the regex heuristic on the same code samples.
8. [Left as future work] Build out a ~50-100 task dataset (versus the 10 prototypes / 1,800 target) with real security labels and a basic semantic oracle, to make Experiment 1 (SSR) and Experiment 2 (SGR) executable even at small scale.
9. [Left as future work] Related-work audit section in PAPER_DRAFT.md (currently a placeholder) — compare explicitly against CyberSecEval, SecurityEval, and SWE-bench framing to sharpen the novelty claim.

What was added (this PR/branch: claude/relaxed-albattani-FA8RR)

• BLOG.md
• PAPER_DRAFT.md
• results/README.md
• experiments/__init__.py
• experiments/exp0_baseline.py
• tests/test_exp0_baseline.py
• README.md (updated to link the above and clarify what's synthetic vs. real)

No fabricated experimental numbers were added. Every quantitative claim from DESIGN_DOC.md's expected-results tables is explicitly labeled "(projected, pending full experiment run)" in PAPER_DRAFT.md; the only numbers computed by actually running code are in results/exp0_baseline.json (produced by experiments/exp0_baseline.py), and those are pipeline-validation numbers on reference solutions, not model-quality results.