Repository navigation
Description
M2 · track:2-C · data — target 7/7 (step 3) · blocks the full-EIP cross-check issue (#90) · part of #86
Background
Property generation leans on known historical bugs, and the paper shows property coverage is the binding constraint on recall — so this dataset's quality directly caps detection. The data is currently spread across labeled findings under benchmarks/results/…, tables under csv/, and the public NyxFoundation/vulnerability-reports dataset on Hugging Face. Consolidate into one clean, queryable dataset with a fixed schema that the cross-check issue (#90) can match against.
Where it lives
Collection and storage happens in NyxFoundation/ethereum-vuln-dataset, not in this repo. That repo is the canonical home for the refined dataset (schema, loader, validator, provenance). speca consumes it as an external input — the 01e pluggable provider work (#87) will be able to ingest it via a GitHub link / artifact.
No Hugging Face upload for now. Keep the existing HF dataset as a source to migrate from, but do not publish the refined dataset to HF in this issue. (The pluggable ingestion in #87 still supports an HF link as one of the source types for the future; that's separate from publishing.)
Tasks
- In
NyxFoundation/ethereum-vuln-dataset: locate every source —speca'scsv/,benchmarks/results/rq1/.../labels, the HFNyxFoundation/vulnerability-reportsdataset, any loader underscripts/— and list what each holds in the PR. - Fix one record schema:
id,client,eip_or_spec,spec_clause,severity(CRITICAL/HIGH/MEDIUM/LOW),category(CWE / STRIDE tag),root_cause,fix_commit,source(e.g. Sherlock #, contest),property_id(if linked). - Migrate and deduplicate existing entries into that schema; verify each
fix_commitresolves. - Backfill
spec_clauseandcategorywhere missing — these are the keys the cross-check (Full-EIP property generation + bug-DB cross-check #90) matches on. - Add a loader + validator in the dataset repo and a test asserting schema conformance and unique
ids. - Document provenance (where each row came from). HF publishing is explicitly deferred — note the migration-from-HF path but do not push back to HF.
Interfaces & contracts
- Canonical artifact: a single file in
NyxFoundation/ethereum-vuln-dataset(e.g.historical_bugs.csvor a JSONL) — pick one and document it. The loader returns typed records matching Full-EIP property generation + bug-DB cross-check #90's expectations. specaconsumes this dataset as an external input (GitHub link / release artifact); Pluggable property providers and reproduction backends #87 wires the ingestion path.
Done when
- One file, one schema in
NyxFoundation/ethereum-vuln-dataset, every field populated where data exists, fix-commit links checked. - Loader + validator pass under test.
- The four known novel bugs are present and correctly tagged as a sanity check: c-kzg-4844
f18ba082, Lodestar3b98c59c, Nimbusb3a3f3f9, Prysmb5bdd65f. - No HF upload performed (deferred).
Review checklist
- The dataset lives in
NyxFoundation/ethereum-vuln-dataset, not duplicated back intospeca. - The schema is the single source of truth — no parallel ad-hoc formats left behind.
- Dedup is real (spot-check merged rows).
spec_clause/categorypopulated; without them the cross-check can't run.- Provenance documented; HF deferral noted explicitly.
Out of scope
Using the dataset to improve generation (#90). New bug discovery (#91). Publishing to Hugging Face (deferred).
Activity
grandchildrice commentedon Jul 19, 2026
検証結果(close前チェック)
NyxFoundation/ethereum-vuln-datasetを Done-when に沿って検証した。完了確認 ✅
data/ethereum_vulns.parquet(+ethereum_vulns.csv/preview.csv)、2,225行・33列。スキーマはdata/manifest.jsonに固定。idは全行ユニーク。pipeline/build_security_dataset.py+tests/test_security_dataset.py。CI(build.yml)が「決定的リビルド(row count / id set drift検出)」+ pytest を実行。authority_tier/n_signals/evidence/severity_source列 +docs/collection.md/BUILD_REPORT.mdで文書化。fix_commitは 1,959 / 2,225 行で populated。スキーマについて(当初案からの意図的変更)
当初案の
eip_or_spec/spec_clause/category/property_idは、layer/label(プロトコル領域)+stride/cwe_top25ベースに再設計(設計文書:docs/label_design.md、#88 で共有済み)。#90 のクロスチェックはこのキーでマッチングする前提に読み替える。未達 ❌(closeブロッカー)
Done-when の sanity check「4件の既知新規バグの収録」が未達。リポジトリ全体を grep しても以下のコミットはどのファイルにもヒットしない:
f18ba0823b98c59cb3a3f3f9b5bdd65fsource_platformの一覧にもc-kzg-4844が存在しないため、ライブラリ系リポジトリはクロール対象外だった可能性が高い。この4行を追加(または収録済みなら該当行のidを提示)したら close する。grandchildrice commentedon Jul 19, 2026
上記の未達項目(既知新規バグ4件の収録)は不要と判断し、要件から落とします。この4件はSPECA自身のM1新規発見バグであり、historical fixのクロール対象(クライアントリポジトリ)外のライブラリを含むため、このデータセットの守備範囲外とする。#90 の cross-check もこの4件を前提にしない。
それ以外の Done-when はすべて検証済み(上記コメント参照):
data/ethereum_vulns.parquet(2,225行)+manifest.json、id全行ユニークdocs/label_design.md)完了として close します。