Skip to content

Full-EIP property generation + bug-DB cross-check #90

Closed as not planned
Closed as not planned

Description

@grandchildrice

M2 · track:2-C · track:2-D · pipeline — target 7/31 (step 4) · depends on the Lean provider and dataset issues · blocks the cross-client audit issue · part of #86

Background

Run the property stage (01a → 01b → 01e, with the Lean provider where it applies) across the full in-scope EIP set, then measure coverage against the refined bug DB: for each historical bug, is there a generated property that would catch it? Gaps feed back into generation. The paper states it plainly — "as models improve, property generation becomes the binding constraint on recall" — so closing coverage gaps here is the highest-leverage work for 2-C. Keep the feedback loop bounded: full generation + a coverage report is the must-have; the iteration is the stretch and must not slip 8/31.

Tasks

  1. Assemble the EIP set (start from the Fusaka corpus referenced in outputs/01a_STATE.json; confirm current scope), and run 01a → 01b → 01e over all of it.
  2. Build the coverage check: join generated 01e properties to bug-DB rows on eip_or_spec + spec_clause / category; mark each historical bug as covered / not covered by ≥1 property.
  3. Emit a coverage report (outputs/coverage/ + a chart under benchmarks/): counts plus the uncovered list with reasons.
  4. (Stretch) For uncovered clauses, feed the gap back into generation (prompt or Lean) and re-measure; record before / after.
  5. Tests: the coverage join is correct on a small fixture with known covered and known uncovered rows.

Interfaces & contracts

  • Coverage record: { "bug_id": "...", "covered": true, "matched_property_ids": [], "reason_if_uncovered": "" }.

Done when

  • 01e properties exist for the full set; the run is resumable.
  • The coverage report lists covered / uncovered historical bugs with reasons.
  • (Stretch) at least one iteration shows a measurable coverage increase.

Review checklist

  • The join key matches the dataset schema exactly — no silent misses from key drift.
  • The uncovered list carries reasons, not just a count.
  • The stretch loop is genuinely optional and cannot block the deliverable.
  • Token / cost stays inside the per-phase budget (the harness enforces it; confirm no override).

Out of scope

Running the audit on clients. Reproduction.

Activity

  1. added
    M2Milestone 2 grant work
    track:2-DDeliverable 2-D: SPECA x Lean integration
    track:2-CDeliverable 2-C: cumulative bug & fix report
    pipelineFull pipeline runs
    on Jun 30, 2026
  2. grandchildrice commented on Jul 19, 2026

    @grandchildrice
    CollaboratorAuthor

    歴史的脆弱性データセット(#89)がほぼ完成したので、こちらの cross-check 作業に着手可能です。

    • 正規アーティファクト: NyxFoundation/ethereum-vuln-dataset の data/ethereum_vulns.parquet(2,225行)。スキーマは data/manifest.json、閲覧用は data/ethereum_vulns.preview.csv。
    • マッチングキー: 当初案の spec_clause / category ではなく、layer / label(プロトコル領域、設計は docs/label_design.md)+ stride / cwe_top25 を使ってください。
    • 残項目は「既知新規バグ4件(c-kzg-4844 / Lodestar / Nimbus / Prysm)の行追加」のみ(Refine the historical vulnerability dataset #89 参照)。cross-check の実装はこれを待たずに進められます。
  3. grandchildrice commented on Jul 22, 2026

    @grandchildrice
    CollaboratorAuthor

    2026-07-22 M2 再構成のためクローズ。 3トラック(a=gasper / b=EL / c=vuln-dataset)× 4フェーズの新構成に作り直しました → トラッカー #86。

    この issue の引き継ぎ先: トラック b/c(#144-#147)に分割。EL spec と vuln-dataset 起点の Lean→01e として再編。

    作業内容・議論は新 issue とトラッカーに移行済みです。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

M2Milestone 2 grant workpipelineFull pipeline runstrack:2-CDeliverable 2-C: cumulative bug & fix reporttrack:2-DDeliverable 2-D: SPECA x Lean integration

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions