Skip to content

Refine the historical vulnerability dataset #89

Closed

Description

@grandchildrice

M2 · track:2-C · data — target 7/7 (step 3) · blocks the full-EIP cross-check issue (#90) · part of #86

Background

Property generation leans on known historical bugs, and the paper shows property coverage is the binding constraint on recall — so this dataset's quality directly caps detection. The data is currently spread across labeled findings under benchmarks/results/…, tables under csv/, and the public NyxFoundation/vulnerability-reports dataset on Hugging Face. Consolidate into one clean, queryable dataset with a fixed schema that the cross-check issue (#90) can match against.

Where it lives

Collection and storage happens in NyxFoundation/ethereum-vuln-dataset, not in this repo. That repo is the canonical home for the refined dataset (schema, loader, validator, provenance). speca consumes it as an external input — the 01e pluggable provider work (#87) will be able to ingest it via a GitHub link / artifact.

No Hugging Face upload for now. Keep the existing HF dataset as a source to migrate from, but do not publish the refined dataset to HF in this issue. (The pluggable ingestion in #87 still supports an HF link as one of the source types for the future; that's separate from publishing.)

Tasks

  1. In NyxFoundation/ethereum-vuln-dataset: locate every source — speca's csv/, benchmarks/results/rq1/.../ labels, the HF NyxFoundation/vulnerability-reports dataset, any loader under scripts/ — and list what each holds in the PR.
  2. Fix one record schema: id, client, eip_or_spec, spec_clause, severity (CRITICAL/HIGH/MEDIUM/LOW), category (CWE / STRIDE tag), root_cause, fix_commit, source (e.g. Sherlock #, contest), property_id (if linked).
  3. Migrate and deduplicate existing entries into that schema; verify each fix_commit resolves.
  4. Backfill spec_clause and category where missing — these are the keys the cross-check (Full-EIP property generation + bug-DB cross-check #90) matches on.
  5. Add a loader + validator in the dataset repo and a test asserting schema conformance and unique ids.
  6. Document provenance (where each row came from). HF publishing is explicitly deferred — note the migration-from-HF path but do not push back to HF.

Interfaces & contracts

Done when

  • One file, one schema in NyxFoundation/ethereum-vuln-dataset, every field populated where data exists, fix-commit links checked.
  • Loader + validator pass under test.
  • The four known novel bugs are present and correctly tagged as a sanity check: c-kzg-4844 f18ba082, Lodestar 3b98c59c, Nimbus b3a3f3f9, Prysm b5bdd65f.
  • No HF upload performed (deferred).

Review checklist

  • The dataset lives in NyxFoundation/ethereum-vuln-dataset, not duplicated back into speca.
  • The schema is the single source of truth — no parallel ad-hoc formats left behind.
  • Dedup is real (spot-check merged rows).
  • spec_clause / category populated; without them the cross-check can't run.
  • Provenance documented; HF deferral noted explicitly.

Out of scope

Using the dataset to improve generation (#90). New bug discovery (#91). Publishing to Hugging Face (deferred).

Activity

  1. added
    M2Milestone 2 grant work
    track:2-CDeliverable 2-C: cumulative bug & fix report
    dataDatasets / data wrangling
    on Jun 30, 2026
  2. grandchildrice commented on Jul 19, 2026

    @grandchildrice
    CollaboratorAuthor

    検証結果(close前チェック)

    NyxFoundation/ethereum-vuln-dataset を Done-when に沿って検証した。

    完了確認 ✅

    • 単一の正規アーティファクト: data/ethereum_vulns.parquet(+ ethereum_vulns.csv / preview.csv)、2,225行・33列。スキーマは data/manifest.json に固定。id は全行ユニーク。
    • ローダー/バリデータ + テスト: pipeline/build_security_dataset.py + tests/test_security_dataset.py。CI(build.yml)が「決定的リビルド(row count / id set drift検出)」+ pytest を実行。
    • 来歴(provenance): authority_tier / n_signals / evidence / severity_source 列 + docs/collection.md / BUILD_REPORT.md で文書化。
    • HFアップロード未実施: README に "mirror is planned" とあるのみ。延期方針どおり。
    • fix_commit は 1,959 / 2,225 行で populated。

    スキーマについて(当初案からの意図的変更)

    当初案の eip_or_spec / spec_clause / category / property_id は、layer / label(プロトコル領域)+ stride / cwe_top25 ベースに再設計(設計文書: docs/label_design.md、#88 で共有済み)。#90 のクロスチェックはこのキーでマッチングする前提に読み替える。

    未達 ❌(closeブロッカー)

    Done-when の sanity check「4件の既知新規バグの収録」が未達。リポジトリ全体を grep しても以下のコミットはどのファイルにもヒットしない:

    • c-kzg-4844 f18ba082
    • Lodestar 3b98c59c
    • Nimbus b3a3f3f9
    • Prysm b5bdd65f

    source_platform の一覧にも c-kzg-4844 が存在しないため、ライブラリ系リポジトリはクロール対象外だった可能性が高い。この4行を追加(または収録済みなら該当行のidを提示)したら close する。

  3. grandchildrice commented on Jul 19, 2026

    @grandchildrice
    CollaboratorAuthor

    上記の未達項目(既知新規バグ4件の収録)は不要と判断し、要件から落とします。この4件はSPECA自身のM1新規発見バグであり、historical fixのクロール対象(クライアントリポジトリ)外のライブラリを含むため、このデータセットの守備範囲外とする。#90 の cross-check もこの4件を前提にしない。

    それ以外の Done-when はすべて検証済み(上記コメント参照):

    • 正規アーティファクト data/ethereum_vulns.parquet(2,225行)+ manifest.json、id全行ユニーク
    • ローダー/バリデータ+テスト、CIでの決定的リビルド検証
    • 来歴の文書化、label設計(docs/label_design.md)
    • HFアップロード未実施(延期方針どおり)

    完了として close します。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

M2Milestone 2 grant workdataDatasets / data wranglingtrack:2-CDeliverable 2-C: cumulative bug & fix report

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions