Key research questions: does security-adjusted pass@k reveal a different model ranking than functional pass@k?

[nv78]

The Questions Reviewers Will Ask

Q1: Does security-adjusted pass@k produce a meaningfully different model ranking than functional pass@k?

• Why it matters: If rankings are identical, security adjustment adds no information; if they diverge, we've found something important
• Experiment: Compute Kendall τ between functional pass@k and security-adjusted pass@k rankings across 15+ models
• Bar: τ < 0.6 would show significant divergence, validating the security dimension as independent signal

Q2: Are AI-generated security vulnerabilities correlated with model capability (bigger = more vulnerable or less)?

• Experiment: Plot security-adjusted pass@k vs. functional pass@k for models of varying scale; test for correlation
• Bar: Non-trivial finding either way (bigger models are more secure, or surprisingly less secure) — both are publishable

Q3: Does the contamination detection pipeline identify models that are gaming the benchmark?

• Experiment: Test contamination detection on models known to have memorized benchmark data (o3, GPT-4o) vs. those less likely to have seen the data
• Bar: Statistically significant difference in contamination scores between high-suspicion and low-suspicion models

Q4: Does the regression-detection metric (issue #7) identify failures that pass@k misses?

• Experiment: Find cases where an agent achieves pass@k on the target function but introduces a regression in a related function
• Bar: ≥10% of apparent "fixes" introduce a regression detectable by the differential coverage analysis

Tracking

• Security-adjusted vs. functional ranking correlation computed for 15+ models
• Model scale vs. security vulnerability analysis run
• Contamination detection pipeline validated on known cases
• Regression rate measurement implemented