Why This Matters
Code generation benchmarks are the most crowded subfield in NLP. Reviewers will challenge "how is this different from HumanEval, MBPP, SWE-bench, or BigCodeBench?" A systematic audit is essential to articulate the exact differentiators — especially the security dimension, which is the primary novelty.
Audit Tasks
1. Comprehensive Code Benchmark Survey
Catalog all existing code generation and evaluation benchmarks. For each, document:
- Name, venue, year
- Task types (generation, completion, repair, security)
- Languages covered
- Whether it evaluates security/vulnerability properties
- Contamination risk level
- Key gaps relative to our contribution
Key works to cover: HumanEval, MBPP, APPS, CodeContests, DS-1000, SWE-bench, SWE-bench Verified, BigCodeBench, LiveCodeBench, EvalPlus, CRUXEval, CyberSecEval, CWE-bench, SecurityEval, LLMSecEval
2. Security Benchmark Novelty
The security evaluation angle is our primary differentiator. Document exactly which security task types are NOT covered by CyberSecEval, SecurityEval, and CWE-bench. Our novelty must be in this gap.
3. Specification Gaming Novelty
Specification gaming and intent specification compliance evaluation is underexplored. Search specifically for papers evaluating whether code models follow specifications vs. exploit them.
4. Anticipate the "Already Done" Review
Strongest objections: "BigCodeBench already covers diverse coding tasks" and "CyberSecEval already covers security." Write rebuttals focusing on our integrated security + specification compliance evaluation and contamination-resistant design.
5. Citation Completeness Check
- Search Semantic Scholar for code generation evaluation papers from 2022–2025
- Ensure all evaluated models (GPT-4o, Claude, Gemini, Codestral, etc.) are cited
- Check that SWE-bench, EvalPlus, and LiveCodeBench (which updated HumanEval) are cited and distinguished
Deliverable
A related_work_audit.md file with comparison table, novelty matrix, strongest objections + rebuttals, and refined novelty statement.
Why This Matters
Code generation benchmarks are the most crowded subfield in NLP. Reviewers will challenge "how is this different from HumanEval, MBPP, SWE-bench, or BigCodeBench?" A systematic audit is essential to articulate the exact differentiators — especially the security dimension, which is the primary novelty.
Audit Tasks
1. Comprehensive Code Benchmark Survey
Catalog all existing code generation and evaluation benchmarks. For each, document:
Key works to cover: HumanEval, MBPP, APPS, CodeContests, DS-1000, SWE-bench, SWE-bench Verified, BigCodeBench, LiveCodeBench, EvalPlus, CRUXEval, CyberSecEval, CWE-bench, SecurityEval, LLMSecEval
2. Security Benchmark Novelty
The security evaluation angle is our primary differentiator. Document exactly which security task types are NOT covered by CyberSecEval, SecurityEval, and CWE-bench. Our novelty must be in this gap.
3. Specification Gaming Novelty
Specification gaming and intent specification compliance evaluation is underexplored. Search specifically for papers evaluating whether code models follow specifications vs. exploit them.
4. Anticipate the "Already Done" Review
Strongest objections: "BigCodeBench already covers diverse coding tasks" and "CyberSecEval already covers security." Write rebuttals focusing on our integrated security + specification compliance evaluation and contamination-resistant design.
5. Citation Completeness Check
Deliverable
A
related_work_audit.mdfile with comparison table, novelty matrix, strongest objections + rebuttals, and refined novelty statement.