P0: Make accuracy benchmarks fail closed and measure real incremental latency

[shaggitza]

Priority

P0 — trustworthy accuracy and performance claims.

Problem

Current benchmark ingestion and reporting can hide invalid input, overstate coverage, and label cold no-cache execution as incremental latency.

Evidence:

• benchmarks/real_world/evaluate.py:437-463,620-648,703,771-777
• benchmarks/real_world/run_current.py:433-670
• src/fastapi_endpoint_detector/analyzer/scip_analyzer.py:133-161
• benchmarks/results/verification-v1/mypy/evaluation.json
• benchmarks/results/verification-v1/scip/evaluation.json
• benchmarks/results/verification-v1/route-census/findings.md

Checked-in verification evidence currently reports:

• mypy normalized TP=3, FP=0, FN=68 (4.23% recall);
• SCIP normalized TP=0, FP=0, FN=71;
• 48 normalized false negatives attributed to discovery and 20/19 to propagation;
• one stress PR contributes 106 of 177 normalized FastAPI atoms.
  These artifacts may represent an older code revision and should not be advertised as current performance without provenance.

Integrity defects:

• Duplicate (repository, PR) truth or prediction rows silently overwrite in dictionaries.
• Runner always uses --no-cache, reports the wall clock as incremental_seconds, hardcodes index time to zero, and SCIP force-reindexes.
• Prediction coverage only checks record presence, even for unresolved/empty records.
• Empty truth/empty prediction PRs receive zero macro scores rather than contributing negative-control specificity.

Required design

• Validate schemas, unique keys/candidate IDs, confidence values, finite timing, and exact selected-key coverage before scoring.
• Record source revision, tool/version, configuration, source inventory, and cache/index fingerprints.
• Separate cold build, warm no-change query, one-file incremental update, baseline+target preparation, peak RSS, and cache size.
• Report p50/p95/max, record/completed/inventory/changed-symbol coverage, unresolved hunk rate, discovery vs propagation FNs, confidence-tier precision, and negative-control specificity.
• Keep raw exact and normalized metrics separate; never treat missing/not-evaluable labels as negatives.

Accuracy gates

• Supported generated typed DAG subset: 100% precision and recall.
• Dead-code/unrelated-change controls: zero HIGH/MEDIUM candidates.
• First milestone: normalized recall >=20%, precision >=90%, no new negative-control FP.
• Propagation milestone: recall >=50%, precision >=90%, every repository non-zero recall.
• Fresh blind release corpus: recall >=80%, precision >=90%, per-repo recall >=70%, resolved coverage >=95%.
• HIGH direct evidence precision remains 100%; LOW remains report-only.
• Paired 10,000-sample PR bootstrap guards against >2-point precision regression.
• Incremental p95 <=30s for the documented medium-project class; cold build and memory reported separately.

The repeatedly inspected 60-PR corpus should be treated as regression/training data; publication claims require a fresh prediction-blind corpus.

Related: #100, #103, #144, #148.

[shaggitza]

Progress merged in #289 (d06ca41).

Completed in this tranche:

• strict fail-closed truth/prediction/manifest readers; duplicate JSON members/rows, malformed candidates, non-finite timing, anonymous IDs, unknown schema-v3 fields, and missing selected predictions are rejected;
• schema-v3 runner output binds exact prediction bytes, selected keys, candidate/config/source metadata, secure execution flags, and per-PR records;
• cold no-cache analyzer wall time is no longer mislabeled incremental and fabricated zero index time is removed;
• record/completion/inventory coverage, positive-PR macro metrics, negative-control specificity, and protocol-specific p50/p95/max are separated;
• unknown and not_evaluable labels remain outside scoring/negative controls;
• unsafe/dry-run output is explicitly ineligible; full attestation and incremental-gate eligibility remain false;
• all nine checked-in legacy prediction artifacts remain readable without modifying frozen artifacts.

CI passed Ruff, mypy, FastAPI minimum compatibility, Python 3.10/3.11/3.12 tests, coverage, and package build. Independent review returned PASS.

Issue remains open for real warm/no-change and one-file incremental measurements, backend reuse/invalidation telemetry, source-inventory and truth-selection attestation, changed-symbol/unresolved-hunk coverage, confidence-tier/release gates, and fresh prediction-blind validation.

[shaggitza]

Merged the independently reviewed truthful telemetry foundation in PR #304 as f812c36. Full CI passed across Python 3.10–3.12, lint/type checks, minimum FastAPI compatibility, and package build. The merged schema-v4 contract records measured cold/no-cache preparation and comparison timing while explicitly marking unsupported warm/incremental/RSS/cache-size fields not_measured; timing keys and prediction context are authenticated. Broader benchmark publication/calibration scope remains open.