Repository navigation
Description
M2 · track:2-D · orchestrator — target 7/7 (step 2) · blocks the Lean provider (#88) and Kurtosis reproduction (#92) issues · part of #86
PR #94 status (merged 2026-07-05)
PR #94 implemented this issue over 3 review rounds (see PR history/comments for the full back-and-forth, including two bugs found and fixed post-merge-candidate: a PHASE_CONFIGS singleton mutation and a verdict-string filter typo that silently zeroed out confirmed_findings). The Tasks / Done-when / Review-checklist items below are checked off against what was independently verified in code, not against the PR description's claims. This issue is intentionally kept open — the merge does not close it — because of the unchecked items (version-pin is decorative-only; no fixture-identity regression test for the default path). Close once those are folded into a follow-up or into #88/#92.
Background
Property generation is currently a single LLM phase: 01e runs prompts/01e_prop_worker.md and writes outputs/01e_PARTIAL_*.json. There's no way to produce that same property schema from anything other than the prompt, and there's no E2E verification / reproduction step after the 04 review gates — PoCs are generated manually through prompts/05_poc.md. 2-D needs both sides — property generation and post-04 verification/reproduction — to be first-class, swappable backends so the Lean provider (#88) and the Kurtosis E2E harness (#92) drop in the way any phase does. This issue lands the seams only.
Design philosophy — make 01e generation AND 04 verification aggressively pluggable
The goal is not just "add a Lean option" or "add a Kurtosis option." It's to turn both 01e (property generation) and the post-04 verification step into plugin points where the source, refinement, and verification backend are all swappable, taking the existing CLI as the reference for how options/flags are surfaced. Concretely, the seam should be able to grow into all of these (this issue builds the seams; later issues fill implementations):
Property generation (01e) plugins:
- Selectable generation method (default vs. optional):
- default — spec-based: the current prompt path (
prompts/01e_prop_worker.md). - optional — Lean4-based: properties derived from Lean 4 verification of the spec (implemented in Lean4 FV property provider: EIP spec →
01eproperties + Kurtosis tests #88 via thespeca-lean4-pluginexternal plugin).
- default — spec-based: the current prompt path (
- Dataset ingestion as a source: pull properties / historical-bug context from an external dataset via a HuggingFace link or a GitHub artifact/release link (e.g. the refined dataset from Refine the historical vulnerability dataset #89 in
NyxFoundation/ethereum-vuln-dataset). The ingestion accepts either link type. - Ingest an existing
01edataset directly: load a previously-generated01eproperty set instead of regenerating, so runs can be replayed / reused. - Recursive self-improvement pass: after
01egeneration, optionally run a refinement loop that takes the generated01eJSON and improves it (dedup, tighten assertions, raise coverage) before it flows downstream.
Verification / reproduction (post-04) plugins:
- The
03audit reports candidate findings; the04review gates filter them (Dead Code → Trust Boundary → Scope Check). After04filtering, optionally run a selectable verification backend that takes the surviving confirmed findings and executes an E2E check. - Whether to run a verification backend, and which one, is selectable (default: none/off). Selecting
kurtosisplugs inNyxFoundation/kurtosis-harness— an E2E reproduction/verification base for Ethereum clients — to run E2E tests against a running client. Other backends can be added the same way.
Treat each of these as a provider / post-processor / verification-backend that satisfies one interface, selected by config/flag with safe defaults. The point of this issue is that adding any future source, refinement step, or verification backend should be a plug-in, not a harness change.
Design note recorded for the implementers (not part of this issue's scope, carried over from PR #94 review):
- Kurtosis (Kurtosis reproduction of confirmed findings #92): prefer a flexible design over a rigid fixture-replay model — have the LLM read the target repo's README to understand its environment, then have it generate the PoC-execution code into
outputs/itself, and use the harness to verify the generated PoC actually reproduces the finding, rather than only replaying a pre-baked Kurtosis fixture. - Lean (Lean4 FV property provider: EIP spec →
01eproperties + Kurtosis tests #88): the ideal is the same flexible approach, but the realistic near-term path is closer to importing an already-existing01eproperty set (theExistingPropertyProviderpath landed here) rather than live Lean invocation — matches the plan already sketched in Lean4 FV property provider: EIP spec →01eproperties + Kurtosis tests #88's comment thread (gasper-lean4→01e-propertiesgenerated externally, then run through SPECA).
External-plugin boundary — officially certified plugins
Heavy or domain-specific execution does not live in this repo. It belongs in dedicated plugin repositories that speca references and officially certifies (pinned version, documented invocation interface):
NyxFoundation/speca-lean4-plugin— Lean 4 execution for theleangeneration method (Lean4 FV property provider: EIP spec →01eproperties + Kurtosis tests #88). Keeps thelean/laketoolchain out of corespeca.NyxFoundation/kurtosis-harness— the E2E verification/reproduction base for Ethereum clients, plugged in when thekurtosisverification backend is selected (used by Kurtosis reproduction of confirmed findings #92). Keeps the Kurtosis devnet machinery out of corespeca.
This issue makes both plugin boundaries explicit (resolution point + version pin + stub) so #88 and #92 plug into them rather than vendoring Lean / Kurtosis into speca.
Tasks
- Read
scripts/orchestrator/config.py(PhaseConfig),factory.py(create_orchestrator),base.py, andrunner.py— and review how the existing CLI surfaces options — and write the current extension seam into the PR description so the next issues don't have to rediscover it. (seam documented across PR feat(orchestrator): pluggable property providers and verification backends #94's description + the 3 review threads) - Add a
provider(generation-method) field to the property phase config — defaultprompt(spec-based), withleanand dataset/existing-01esources as recognized values. No behavior change when the field is absent. (property_providerfield,config.py) - Define a
PropertyProviderinterface. Input:01bsubgraphs +outputs/BUG_BOUNTY_SCOPE.json(and, for ingestion providers, a HF/GitHub source link). Output: the existing01eproperty schema fromschemas.py. Wrap the current prompt path as the first implementation so every future provider (Lean, dataset-ingest, existing-01e-ingest) satisfies one interface. (providers.py:PropertyProviderprotocol +PromptPropertyProvider/LeanPropertyProvider/DatasetPropertyProvider/ExistingPropertyProvider) - Define a post-generation refinement hook (the recursive-self-improvement seam): an optional pass that takes generated
01eJSON and returns refined01eJSON. Off by default, opt-in by config/flag, no-op stub for now. (run_refinement_pass,refinement_pass_enabledconfig field) - Define a
VerificationBackendinterface for the post-04step: input is the confirmed findings surviving the04gates (04_PARTIAL_*.json) +outputs/TARGET_INFO.json; output is a reproduction/verification record. Selectable by config/flag (default off /none);kurtosisresolves to the externalNyxFoundation/kurtosis-harnessplugin. Register it as a post-04step with a no-op stub for now. (NullVerificationBackend/KurtosisVerificationBackend, wired intoPhase04Orchestrator.run()) - Make the plugin boundaries explicit: the
leanprovider resolves tospeca-lean4-pluginand thekurtosisverification backend resolves tokurtosis-harness— both external, version-pinned, invoked across the seam. Add the registration/resolution points and stubs, but no Lean/Kurtosis logic here. Partially done:plugin_ref+plugin_versionfields exist on both provider/backend classes and are surfaced in theNotImplementedErrormessage, butresolve_provider()/resolve_verification_backend()never read or enforceplugin_version— it's a decorative pin today, not an operative one. Needs to become enforced (e.g. checked against the installed plugin's actual commit) once Lean4 FV property provider: EIP spec →01eproperties + Kurtosis tests #88/Kurtosis reproduction of confirmed findings #92 wire in real resolution. - Add the reproduction/verification-record Pydantic schema to
schemas.py. (VerificationRecord, matches the proposed shape below) - Tests: provider selection resolves to the right implementation; the refinement hook and the verification backend are skipped unless enabled; selecting
kurtosisdispatches to the (stubbed) external-plugin resolution point; on a fixture, the default prompt provider still emits byte-identical01eoutput. Partially done: the first three are tested and passing (30 provider tests, incl. a regression test for the verdict-filter bug found in review). The last one is not: there is no fixture-based test asserting the defaultpromptprovider's01eoutput is byte-identical pre/post this PR — only config-default assertions exist.
Interfaces & contracts
- Property output stays exactly the
01eschema —property_id,text,type,assertion,severity,covers,reachability{classification, entry_points, attacker_controlled, bug_bounty_scope},bug_bounty_eligible,exploitability. Every provider and the refinement hook must preserve it (backend-specific data, e.g.lean_status, is additive only). - Generation method and verification backend are config/flag-selected; the defaults reproduce today's behavior exactly (prompt generation, no post-
04verification). - External plugins, referenced + version-pinned, not vendored:
lean→speca-lean4-plugin;kurtosis→kurtosis-harness. - Proposed verification/reproduction record (finalize names in
schemas.py):{ "property_id": "PROP-...", "verdict": "reproduced | not-reproduced | error", "harness": "kurtosis", "artifact_path": "...", "logs_path": "...", "notes": "" }
Done when
-
01eruns unchanged withproviderunset / default (fixture output identical). Not verified — no fixture-identity regression test exists (see Task 8). -
provider: leanis accepted and dispatches to thespeca-lean4-pluginresolution point as a stub that raises "not implemented" (filled by Lean4 FV property provider: EIP spec →01eproperties + Kurtosis tests #88). (CLI-verified end to end: zero Claude CLI/network calls, raises with the version-pin-aware message) - Dataset-ingest / existing-
01e-ingest providers and the recursive-refinement hook exist as recognized, opt-in seams (no-op stubs), each satisfying the one interface. - The post-
04VerificationBackendis selectable; default is off;kurtosisdispatches to thekurtosis-harnessresolution point as a no-op/not-implementedstub (filled by Kurtosis reproduction of confirmed findings #92). (verdict filter bug found in review — was silently dispatching 0 findings — fixed and regression-tested) -
pytest tests/ -vgreen, new schema covered. (691/691 passing)
Review checklist
- The harness (
base.py/runner.py) is untouched; the seams are added at config/factory level only, mirroring how the existing CLI exposes options. Not satisfied, by necessity:Phase01Orchestrator.run()/Phase04Orchestrator.run()inbase.pywere modified to actually dispatch to providers/backends. Without this, the seam had zero runtime effect (the exact bug the first review round caught —--property-provider leansilently ran the normal prompt worker instead of raising). Recording this as a deliberate, reviewed deviation from the original constraint, not an oversight. - Diff shows no behavior change for existing
01e/04runs. Plausible but unproven — defaults are preserved and the prompt/CLI code path is logically untouched, but there's no fixture-identity test proving byte-for-byte equivalence (same gap as the Done-when item above). - The provider / refinement / verification-backend interfaces are minimal, named consistently with existing schemas, and genuinely pluggable (a new source or verification base = a new plug-in, not a harness edit).
- Both external plugin boundaries are explicit and external — no Lean/
lakeor Kurtosis dependency pulled into corespeca. - The verification backend runs strictly after the
04gates, on the filtered confirmed findings. (note:DOWNGRADED— a real confirmed-but-severity-capped verdict — is currently excluded from the dispatch filter; worth revisiting when Kurtosis reproduction of confirmed findings #92 lands) - Schema additions land in
schemas.pywith matching tests.
Out of scope
Lean proof logic / speca-lean4-plugin implementation (#88), Kurtosis E2E execution / kurtosis-harness implementation (#92), and the actual recursive-self-improvement / ingestion logic. This issue ships seams + stubs + schema + the pluggable design only.
Activity
hirorogo commentedon Jul 2, 2026
Implementation complete — PR #94 opened.
Branch: feat/issue-87-pluggable-providers
What was implemented:
New file: scripts/orchestrator/providers.py
Updated: scripts/orchestrator/schemas.py
Updated: scripts/orchestrator/config.py
Updated: scripts/orchestrator/factory.py
Updated: scripts/run_phase.py
New file: tests/test_pluggable_providers.py
fix(orchestrator): address PR #94 follow-up review
grandchildrice commentedon Jul 19, 2026
残項目ステータス更新(PR #115 マージ後)
このIssueをopenのままにしている残項目の現状を整理:
完了 ✅
LeanPropertyProvider.plugin_version = "v0.1.0"(providers.py:245)がテストで固定され、auto-clone パスでは--branch v0.1.0として実際に効いている(PR 01e: wire the lean property provider to speca-lean4-plugin v0.1.0 (+ provider:lean CI smoke) #115)properties-leanジョブ)未完了 ⬜(このIssueをcloseする条件)
SPECA_LEAN4_PLUGIN_DIR指定時(= CI が使うパス)は shape チェックのみで、checkout が pin と一致するかの照合(git describe等)がない。CI YAML も'v0.1.0'を3箇所ハードコードしており、コード側の pin と cross-check されない。base.py:753-754)のみ。collector.py/factory.py/config.py追加上記⬜を follow-up PR(または #88/#92 のスコープ内)で消化したら close。
grandchildrice commentedon Jul 19, 2026
残項目ステータス更新(PR #119 マージ後)
_verify_plugin_version()が全解決パスでgit tag --points-at HEAD照合、CI の'v0.1.0'ハードコードも撤去されコード側 pin から導出。scripts/run_phase.py(workflowの実エントリポイント)が依然漏れている(01e: lean provider follow-ups — enforce version pin, CI injection/path-filter fixes, clone hardening #119 レビュー参照)。このIssueのcloseに残るのは上記⬜の2点。あわせて #119 レビューの新規指摘(plain-dir がネストした外側リポジトリのタグを拾う
git -C問題、並行clone の rename 衝突)もここで追跡する。Merge pull request #127 from NyxFoundation/feat/lean-provider-followu…
sururu-k commentedon Jul 20, 2026
PR #137 (CI 全緑・11/11 checks pass) で未チェック項目 2 件の根拠が揃いました。チェック操作は issue オーナーに委ねます。
Task 6(pin が decorative)→ 根拠あり
plugin_commitを両 plugin 境界に追加し、git rev-parse HEADとの commit 照合を強制。tag は動かせるため operative な pin は commit(v0.1.0 =aa123f34、kurtosis-harness =f92be45c)。動かされた/偽造されたv0.1.0tag を持つ wrong-commit checkout は実 git リポジトリでの敵対的デモで reject を確認(PR 本文に生ログ)。resolve_provider()/resolve_verification_backend()が resolve 時に pin を強制(ローカル設定済み checkout:SPECA_LEAN4_PLUGIN_DIR/SPECA_KURTOSIS_PLUGIN_DIR)。未設定時は subprocess ゼロ(テストで保証)なのでデフォルト経路のコストは不変。auto-clone/cache 経路はgenerate()内の解決点で照合。01eproperties + Kurtosis tests #88 Task 1 の残項目(「pin の強制(checkout 照合)は未了」)もこれで解消。SPECA_LEAN4_ALLOW_VERSION_MISMATCH=1/SPECA_KURTOSIS_ALLOW_VERSION_MISMATCH=1(loud warning に降格)。CI の properties-lean は実 clone HEAD = pin commit で強制の match 経路を通過して緑。Task 8 / Done-when 1 項目め / Review checklist「no behavior change」→ 根拠あり
01e_prompt_partial_golden.json)が pre-seam コード(68ead915= PR feat(orchestrator): pluggable property providers and verification backends #94 マージの親)の collector 出力と byte-identical(1228 bytes、3 者一致)であることを機械検証 — 自己一致ではなく pre-seam 挙動へのアンカー。生出力は PR feat(providers): operative plugin version pins + pre-seam-anchored default-path identity guard (#87 Tasks 6/8) #137 本文。tests/fixtures/phase_config_preseam_golden.json+test_default_configs_additive_over_preseam_golden: pre-seam の 01e/04 PhaseConfig 全キーが等値で現存(seam は additive-only: 追加 6 フィールドのみ、削除/変更ゼロ)を恒久テスト化。68ead915..4e5d4955の diff は prompts/ と collector.py に非接触 → routing テスト(既存)+ config golden + byte golden で seam の全接触面をカバー。feat(providers): make plugin version pins operative and anchor the de…
grandchildrice commentedon Jul 22, 2026
2026-07-22 M2 再構成のためクローズ。 3トラック(a=gasper / b=EL / c=vuln-dataset)× 4フェーズの新構成に作り直しました → トラッカー #86。
この issue の引き継ぎ先: 基盤として完了(provider seam / judge harness)。新トラック a-2 #143 ほかが引き継ぎ。
作業内容・議論は新 issue とトラッカーに移行済みです。
sururu-k commentedon Jul 22, 2026
みすってました!すみません…