Add statistical rigor: confidence intervals, significance testing, and variance analysis

[nv78]

Why This Matters

Code benchmark papers are high-profile and highly scrutinized. HumanEval contamination debates have made the community hypersensitive to result reliability. Reviewers at ICSE, NeurIPS, and security venues will ask: "How much variance is there from sampling temperature?" and "Are security score differences significant?" Without rigorous statistics, claims will be dismissed.

Required Statistical Additions

1. Bootstrap Confidence Intervals on All Main Metrics

• Compute 95% bootstrap CIs (1000 resamples) for every metric: pass@k, security_score, functional correctness, specification compliance rate
• Report as mean ± CI in all tables
• For pass@k: use the unbiased Chen et al. (2021) estimator; report variance across at least 20 samples per problem

2. Temperature Sensitivity Analysis

• Run key models at temperature ∈ {0.0, 0.2, 0.4, 0.8} and report performance variance
• This directly addresses reviewer concern about cherry-picked temperature settings
• Report which temperature setting was used for reported numbers and whether it was tuned on the test set (it should not be)

3. Significance Testing for Model Comparisons

• Run paired bootstrap test for every model comparison (both functional and security dimensions)
• Threshold: p < 0.05 after Bonferroni correction for multiple comparisons
• Mark significant differences with †; provide full p-values in appendix
• Use McNemar's test for pass/fail comparisons on the same problem instances

4. Contamination Sensitivity Analysis

• Report performance with and without suspected contaminated problems (e.g., LeetCode problems that appear in common training sets)
• Verify that performance ordering across models is stable after removing contaminated problems
• This is the single most important analysis for benchmark paper credibility in 2025+

5. Inter-Rater Reliability for Security Scoring

• If security scores involve human judgment, report inter-rater agreement (Cohen's κ or Krippendorff's α)
• For automated security scoring, report false positive/negative rates against expert-labeled ground truth

Implementation Checklist

• Add bootstrap CI computation with pass@k unbiased estimator to src/codebench/evaluate.py
• Run temperature sensitivity sweep for all major models
• Run contamination sensitivity analysis; identify and flag contaminated problems
• Add significance tests for all model comparisons
• Update tables with CIs and significance markers
• Add statistical appendix

Reference

• Chen et al. (2021) "Evaluating Large Language Models Trained on Code" — unbiased pass@k estimator
• Dror et al. (2018) "The Hitchhiker's Guide to Testing Statistical Significance in NLP"