Write ethics & broader impact statement for paper submission

[nv78]

Why This Matters

Code generation benchmark papers at ICSE, NeurIPS, and security venues face scrutiny on dual-use risks — especially security-related tasks. A thorough ethics statement is required by most venues and preempts reviewer objections about responsible disclosure and potential for misuse.

Statement Outline

Intended Use and Benefits

• CodeBench enables rigorous evaluation of AI coding assistants, especially for security and specification correctness — dimensions that existing benchmarks underweight
• Benefits: helps development teams select AI tools that produce safer, more correct code; raises the bar for what "state of the art" means in code generation

Security Research and Responsible Disclosure

• The benchmark includes security-related coding tasks (vulnerability detection, secure coding pattern compliance)
• Responsible disclosure: any real exploits or vulnerabilities discovered during benchmark construction should be reported to affected maintainers before public release
• The benchmark does NOT include tasks that require generating novel attack code; all security tasks are defensive (detecting vulnerabilities, not creating them)
• Document the responsible disclosure process used for any CVEs discovered

Contamination and Benchmark Gaming

• Code generation models may have been trained on problems similar to the benchmark
• Releasing a full benchmark publicly enables targeted fine-tuning that inflates scores without improving real-world capability
• Mitigation: maintain a private holdout set for leaderboard evaluation; periodically refresh public problems; document contamination analysis in the paper

Bias in Code Ecosystem Coverage

• The benchmark likely over-represents Python and JavaScript; performance on other languages (Rust, Go, C++) may differ significantly
• Document language coverage; do not claim generalizability beyond covered languages
• Task distributions may reflect typical US/EU software engineering patterns; international coding conventions may differ

Environmental Impact

• Report CO2 emissions for all model evaluations
• Note: evaluating large code models at scale can be compute-intensive; document hardware and runtime

Checklist

• Draft ethics section with responsible disclosure and dual-use discussion
• Confirm no real exploits in the public release; document security review process
• Run contamination analysis; document in paper
• Compute and report CO2 emissions
• Have a security-focused co-author review the ethics statement