Improve the work: multi-language support, real-world task collection, and security test suite hardening

[nv78]

Critical Improvements Before Submission

Ablations and Analysis Gaps

• Language breakdown: current benchmark is predominantly Python; add JavaScript, TypeScript, Go, Rust — the languages most used in production systems
• Task category breakdown: report pass@k separately for bug fix, feature addition, refactor, API migration, test generation — reviewers will ask for this
• Model context window: test whether providing more file context (full repo vs. single file vs. just the function) changes security vulnerability rates
• Prompt engineering sensitivity: test whether simple prompt additions ("Ensure the code is secure", "Check for SQL injection") reduce vulnerability rates — and by how much

Security Test Suite Hardening

• CWE coverage audit: verify that test cases cover all OWASP Top 10; document which CWEs have <5 test cases and fill gaps
• Security test false positive rate: have a security engineer manually audit 100 security test failures to measure false positive rate — critical for credibility
• Severity weighting: not all CWEs are equal; implement CVSS score weighting in security-adjusted pass@k
• Novel CWE injection: current security tests are known patterns; add AI-specific vulnerabilities (prompt injection, insecure LLM output handling)

Contamination Detection Hardening

• Cross-check with training data timestamps: for models with known training cutoffs, verify that contamination detection aligns with the cutoff boundary
• Semantic similarity contamination: current detection uses exact/near-exact match; add semantic similarity detection for paraphrased contamination
• Contamination rate audit: publish a per-model contamination probability score as part of the leaderboard

Real-World Task Collection Pipeline

• Automated GitHub issue ingestion: build a pipeline to continuously collect real, resolved GitHub issues (with test suites) from high-quality repos
• Manual quality filter: each task needs a human expert to verify it's well-defined, has deterministic tests, and doesn't contain leaked solutions
• Deduplication: implement semantic deduplication to prevent near-duplicate tasks inflating apparent coverage

Tracking

• JavaScript/TypeScript task collection initiated
• CWE coverage audit complete
• Security test false positive audit (100 samples) scheduled
• GitHub issue ingestion pipeline MVP built