Skip to content

Pluggable property providers and reproduction backends #87

Closed as not planned

Description

@grandchildrice

M2 · track:2-D · orchestrator — target 7/7 (step 2) · blocks the Lean provider (#88) and Kurtosis reproduction (#92) issues · part of #86

PR #94 status (merged 2026-07-05)

PR #94 implemented this issue over 3 review rounds (see PR history/comments for the full back-and-forth, including two bugs found and fixed post-merge-candidate: a PHASE_CONFIGS singleton mutation and a verdict-string filter typo that silently zeroed out confirmed_findings). The Tasks / Done-when / Review-checklist items below are checked off against what was independently verified in code, not against the PR description's claims. This issue is intentionally kept open — the merge does not close it — because of the unchecked items (version-pin is decorative-only; no fixture-identity regression test for the default path). Close once those are folded into a follow-up or into #88/#92.

Background

Property generation is currently a single LLM phase: 01e runs prompts/01e_prop_worker.md and writes outputs/01e_PARTIAL_*.json. There's no way to produce that same property schema from anything other than the prompt, and there's no E2E verification / reproduction step after the 04 review gates — PoCs are generated manually through prompts/05_poc.md. 2-D needs both sides — property generation and post-04 verification/reproduction — to be first-class, swappable backends so the Lean provider (#88) and the Kurtosis E2E harness (#92) drop in the way any phase does. This issue lands the seams only.

Design philosophy — make 01e generation AND 04 verification aggressively pluggable

The goal is not just "add a Lean option" or "add a Kurtosis option." It's to turn both 01e (property generation) and the post-04 verification step into plugin points where the source, refinement, and verification backend are all swappable, taking the existing CLI as the reference for how options/flags are surfaced. Concretely, the seam should be able to grow into all of these (this issue builds the seams; later issues fill implementations):

Property generation (01e) plugins:

  • Selectable generation method (default vs. optional):
  • Dataset ingestion as a source: pull properties / historical-bug context from an external dataset via a HuggingFace link or a GitHub artifact/release link (e.g. the refined dataset from Refine the historical vulnerability dataset #89 in NyxFoundation/ethereum-vuln-dataset). The ingestion accepts either link type.
  • Ingest an existing 01e dataset directly: load a previously-generated 01e property set instead of regenerating, so runs can be replayed / reused.
  • Recursive self-improvement pass: after 01e generation, optionally run a refinement loop that takes the generated 01e JSON and improves it (dedup, tighten assertions, raise coverage) before it flows downstream.

Verification / reproduction (post-04) plugins:

  • The 03 audit reports candidate findings; the 04 review gates filter them (Dead Code → Trust Boundary → Scope Check). After 04 filtering, optionally run a selectable verification backend that takes the surviving confirmed findings and executes an E2E check.
  • Whether to run a verification backend, and which one, is selectable (default: none/off). Selecting kurtosis plugs in NyxFoundation/kurtosis-harness — an E2E reproduction/verification base for Ethereum clients — to run E2E tests against a running client. Other backends can be added the same way.

Treat each of these as a provider / post-processor / verification-backend that satisfies one interface, selected by config/flag with safe defaults. The point of this issue is that adding any future source, refinement step, or verification backend should be a plug-in, not a harness change.

Design note recorded for the implementers (not part of this issue's scope, carried over from PR #94 review):

External-plugin boundary — officially certified plugins

Heavy or domain-specific execution does not live in this repo. It belongs in dedicated plugin repositories that speca references and officially certifies (pinned version, documented invocation interface):

This issue makes both plugin boundaries explicit (resolution point + version pin + stub) so #88 and #92 plug into them rather than vendoring Lean / Kurtosis into speca.

Tasks

  1. Read scripts/orchestrator/config.py (PhaseConfig), factory.py (create_orchestrator), base.py, and runner.py — and review how the existing CLI surfaces options — and write the current extension seam into the PR description so the next issues don't have to rediscover it. (seam documented across PR feat(orchestrator): pluggable property providers and verification backends #94's description + the 3 review threads)
  2. Add a provider (generation-method) field to the property phase config — default prompt (spec-based), with lean and dataset/existing-01e sources as recognized values. No behavior change when the field is absent. (property_provider field, config.py)
  3. Define a PropertyProvider interface. Input: 01b subgraphs + outputs/BUG_BOUNTY_SCOPE.json (and, for ingestion providers, a HF/GitHub source link). Output: the existing 01e property schema from schemas.py. Wrap the current prompt path as the first implementation so every future provider (Lean, dataset-ingest, existing-01e-ingest) satisfies one interface. (providers.py: PropertyProvider protocol + PromptPropertyProvider/LeanPropertyProvider/DatasetPropertyProvider/ExistingPropertyProvider)
  4. Define a post-generation refinement hook (the recursive-self-improvement seam): an optional pass that takes generated 01e JSON and returns refined 01e JSON. Off by default, opt-in by config/flag, no-op stub for now. (run_refinement_pass, refinement_pass_enabled config field)
  5. Define a VerificationBackend interface for the post-04 step: input is the confirmed findings surviving the 04 gates (04_PARTIAL_*.json) + outputs/TARGET_INFO.json; output is a reproduction/verification record. Selectable by config/flag (default off / none); kurtosis resolves to the external NyxFoundation/kurtosis-harness plugin. Register it as a post-04 step with a no-op stub for now. (NullVerificationBackend/KurtosisVerificationBackend, wired into Phase04Orchestrator.run())
  6. Make the plugin boundaries explicit: the lean provider resolves to speca-lean4-plugin and the kurtosis verification backend resolves to kurtosis-harness — both external, version-pinned, invoked across the seam. Add the registration/resolution points and stubs, but no Lean/Kurtosis logic here. Partially done: plugin_ref + plugin_version fields exist on both provider/backend classes and are surfaced in the NotImplementedError message, but resolve_provider()/resolve_verification_backend() never read or enforce plugin_version — it's a decorative pin today, not an operative one. Needs to become enforced (e.g. checked against the installed plugin's actual commit) once Lean4 FV property provider: EIP spec → 01e properties + Kurtosis tests #88/Kurtosis reproduction of confirmed findings #92 wire in real resolution.
  7. Add the reproduction/verification-record Pydantic schema to schemas.py. (VerificationRecord, matches the proposed shape below)
  8. Tests: provider selection resolves to the right implementation; the refinement hook and the verification backend are skipped unless enabled; selecting kurtosis dispatches to the (stubbed) external-plugin resolution point; on a fixture, the default prompt provider still emits byte-identical 01e output. Partially done: the first three are tested and passing (30 provider tests, incl. a regression test for the verdict-filter bug found in review). The last one is not: there is no fixture-based test asserting the default prompt provider's 01e output is byte-identical pre/post this PR — only config-default assertions exist.

Interfaces & contracts

  • Property output stays exactly the 01e schema — property_id, text, type, assertion, severity, covers, reachability{classification, entry_points, attacker_controlled, bug_bounty_scope}, bug_bounty_eligible, exploitability. Every provider and the refinement hook must preserve it (backend-specific data, e.g. lean_status, is additive only).
  • Generation method and verification backend are config/flag-selected; the defaults reproduce today's behavior exactly (prompt generation, no post-04 verification).
  • External plugins, referenced + version-pinned, not vendored: lean → speca-lean4-plugin; kurtosis → kurtosis-harness.
  • Proposed verification/reproduction record (finalize names in schemas.py):
    { "property_id": "PROP-...", "verdict": "reproduced | not-reproduced | error",
      "harness": "kurtosis", "artifact_path": "...", "logs_path": "...", "notes": "" }

Done when

  • 01e runs unchanged with provider unset / default (fixture output identical). Not verified — no fixture-identity regression test exists (see Task 8).
  • provider: lean is accepted and dispatches to the speca-lean4-plugin resolution point as a stub that raises "not implemented" (filled by Lean4 FV property provider: EIP spec → 01e properties + Kurtosis tests #88). (CLI-verified end to end: zero Claude CLI/network calls, raises with the version-pin-aware message)
  • Dataset-ingest / existing-01e-ingest providers and the recursive-refinement hook exist as recognized, opt-in seams (no-op stubs), each satisfying the one interface.
  • The post-04 VerificationBackend is selectable; default is off; kurtosis dispatches to the kurtosis-harness resolution point as a no-op/not-implemented stub (filled by Kurtosis reproduction of confirmed findings #92). (verdict filter bug found in review — was silently dispatching 0 findings — fixed and regression-tested)
  • pytest tests/ -v green, new schema covered. (691/691 passing)

Review checklist

  • The harness (base.py / runner.py) is untouched; the seams are added at config/factory level only, mirroring how the existing CLI exposes options. Not satisfied, by necessity: Phase01Orchestrator.run() / Phase04Orchestrator.run() in base.py were modified to actually dispatch to providers/backends. Without this, the seam had zero runtime effect (the exact bug the first review round caught — --property-provider lean silently ran the normal prompt worker instead of raising). Recording this as a deliberate, reviewed deviation from the original constraint, not an oversight.
  • Diff shows no behavior change for existing 01e / 04 runs. Plausible but unproven — defaults are preserved and the prompt/CLI code path is logically untouched, but there's no fixture-identity test proving byte-for-byte equivalence (same gap as the Done-when item above).
  • The provider / refinement / verification-backend interfaces are minimal, named consistently with existing schemas, and genuinely pluggable (a new source or verification base = a new plug-in, not a harness edit).
  • Both external plugin boundaries are explicit and external — no Lean/lake or Kurtosis dependency pulled into core speca.
  • The verification backend runs strictly after the 04 gates, on the filtered confirmed findings. (note: DOWNGRADED — a real confirmed-but-severity-capped verdict — is currently excluded from the dispatch filter; worth revisiting when Kurtosis reproduction of confirmed findings #92 lands)
  • Schema additions land in schemas.py with matching tests.

Out of scope

Lean proof logic / speca-lean4-plugin implementation (#88), Kurtosis E2E execution / kurtosis-harness implementation (#92), and the actual recursive-self-improvement / ingestion logic. This issue ships seams + stubs + schema + the pluggable design only.

Activity

  1. added
    M2Milestone 2 grant work
    track:2-DDeliverable 2-D: SPECA x Lean integration
    orchestratorAsync orchestrator / harness
    on Jun 30, 2026
  2. hirorogo commented on Jul 2, 2026

    @hirorogo
    Collaborator

    Implementation complete — PR #94 opened.

    Branch: feat/issue-87-pluggable-providers

    What was implemented:

    New file: scripts/orchestrator/providers.py

    • PropertyProvider protocol (generate method)
    • VerificationBackend protocol (verify method)
    • Four provider impls: PromptPropertyProvider (delegates to Claude CLI runner, unchanged), LeanPropertyProvider (stub referencing NyxFoundation/speca-lean4-plugin), DatasetPropertyProvider (stub for HuggingFace/GitHub dataset ingestion), ExistingPropertyProvider (loads properties from existing JSON file on disk)
    • Two backend impls: NullVerificationBackend (returns []), KurtosisVerificationBackend (stub referencing NyxFoundation/kurtosis-harness)
    • run_refinement_pass hook (identity pass for now)
    • resolve_provider() and resolve_verification_backend() factory helpers

    Updated: scripts/orchestrator/schemas.py

    • PropertyProviderName and VerificationBackendName enums
    • VerificationRecord Pydantic model

    Updated: scripts/orchestrator/config.py

    • Four new PhaseConfig fields: property_provider, dataset_source_url, refinement_pass_enabled, verification_backend

    Updated: scripts/orchestrator/factory.py

    • create_orchestrator now accepts property_provider and verification_backend kwargs

    Updated: scripts/run_phase.py

    • --property-provider, --dataset-source, --enable-refinement, --verification-backend flags

    New file: tests/test_pluggable_providers.py

    • 26 tests, all passing (184 total)
  3. added a commit that references this issue on Jul 5, 2026
  4. grandchildrice commented on Jul 19, 2026

    @grandchildrice
    CollaboratorAuthor

    残項目ステータス更新(PR #115 マージ後)

    このIssueをopenのままにしている残項目の現状を整理:

    完了 ✅

    未完了 ⬜(このIssueをcloseする条件)

    • ⬜ Version-pin の「強制」: SPECA_LEAN4_PLUGIN_DIR 指定時(= CI が使うパス)は shape チェックのみで、checkout が pin と一致するかの照合(git describe 等)がない。CI YAML も 'v0.1.0' を3箇所ハードコードしており、コード側の pin と cross-check されない。
    • ⬜ デフォルト(LLM)パスの fixture-identity 回帰テスト: 依然として存在しない。PR 01e: wire the lean property provider to speca-lean4-plugin v0.1.0 (+ provider:lean CI smoke) #115 の保護は構造的なゲート(base.py:753-754)のみ。
    • ⬜ (#115レビューで追加)plugin が空の properties を返しても exit 0 で「completed」になる件のランタイム警告、smoke path filter への collector.py/factory.py/config.py 追加

    上記⬜を follow-up PR(または #88/#92 のスコープ内)で消化したら close。

  5. grandchildrice commented on Jul 19, 2026

    @grandchildrice
    CollaboratorAuthor

    残項目ステータス更新(PR #119 マージ後)

    このIssueのcloseに残るのは上記⬜の2点。あわせて #119 レビューの新規指摘(plain-dir がネストした外側リポジトリのタグを拾う git -C 問題、並行clone の rename 衝突)もここで追跡する。

  6. added a commit that references this issue on Jul 20, 2026
  7. sururu-k commented on Jul 20, 2026

    @sururu-k
    Collaborator

    PR #137 (CI 全緑・11/11 checks pass) で未チェック項目 2 件の根拠が揃いました。チェック操作は issue オーナーに委ねます。

    Task 6(pin が decorative)→ 根拠あり

    • plugin_commit を両 plugin 境界に追加し、git rev-parse HEAD との commit 照合を強制。tag は動かせるため operative な pin は commit(v0.1.0 = aa123f34、kurtosis-harness = f92be45c)。動かされた/偽造された v0.1.0 tag を持つ wrong-commit checkout は実 git リポジトリでの敵対的デモで reject を確認(PR 本文に生ログ)。
    • resolve_provider() / resolve_verification_backend() が resolve 時に pin を強制(ローカル設定済み checkout: SPECA_LEAN4_PLUGIN_DIR / SPECA_KURTOSIS_PLUGIN_DIR)。未設定時は subprocess ゼロ(テストで保証)なのでデフォルト経路のコストは不変。auto-clone/cache 経路は generate() 内の解決点で照合。
    • kurtosis backend はスタブのまま境界だけ operative 化 — Kurtosis reproduction of confirmed findings #92 は強制を継承するだけ。Lean4 FV property provider: EIP spec → 01e properties + Kurtosis tests #88 Task 1 の残項目(「pin の強制(checkout 照合)は未了」)もこれで解消。
    • 開発用逃げ道: SPECA_LEAN4_ALLOW_VERSION_MISMATCH=1 / SPECA_KURTOSIS_ALLOW_VERSION_MISMATCH=1(loud warning に降格)。CI の properties-lean は実 clone HEAD = pin commit で強制の match 経路を通過して緑。

    Task 8 / Done-when 1 項目め / Review checklist「no behavior change」→ 根拠あり

  8. grandchildrice commented on Jul 22, 2026

    @grandchildrice
    CollaboratorAuthor

    2026-07-22 M2 再構成のためクローズ。 3トラック(a=gasper / b=EL / c=vuln-dataset)× 4フェーズの新構成に作り直しました → トラッカー #86。

    この issue の引き継ぎ先: 基盤として完了(provider seam / judge harness)。新トラック a-2 #143 ほかが引き継ぎ。

    作業内容・議論は新 issue とトラッカーに移行済みです。

  9. sururu-k commented on Jul 22, 2026

    @sururu-k
    Collaborator

    2026-07-22 M2 再構成のためクローズ。 3トラック(a=gasper / b=EL / c=vuln-dataset)× 4フェーズの新構成に作り直しました → トラッカー #86。

    この issue の引き継ぎ先: 基盤として完了(provider seam / judge harness)。新トラック a-2 #143 ほかが引き継ぎ。

    作業内容・議論は新 issue とトラッカーに移行済みです。

    みすってました!すみません…

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

M2Milestone 2 grant workorchestratorAsync orchestrator / harnesstrack:2-DDeliverable 2-D: SPECA x Lean integration

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions