Skip to content

feat: add miner-bench harness for local miner self-test - #17

Open
dataCenter430 wants to merge 1 commit into
0xsigurd:mainfrom
dataCenter430:feat/miner-bench-harness
Open

dataCenter430 wants to merge 1 commit into
0xsigurd:mainfrom
dataCenter430:feat/miner-bench-harness

Conversation

@dataCenter430

@dataCenter430 dataCenter430 commented May 18, 2026

Copy link
Copy Markdown

Adds miner_bench/: a standalone harness that lets miners evaluate attack strategies against the same scoring rules the mainnet validator applies (faithful port of verify_and_score), without consuming validator resources or polluting on-chain history.

  • scoring.py: dependency-light port of PerturbValidator.verify_and_score with the same SSIM/PSNR/Linf/RMSE gates and score combination.
  • challenge.py: Pexels fetch + Challenge dataclass; mirrors the validator's challenge generation (prompt rotation, photo sampling, label-from-model).
  • strategy.py: MinerStrategy ABC + PGDStrategy reference baseline (textbook untargeted PGD, respects validator's effective max-delta cap).
  • runner.py: run_bench() + CLI with per-challenge + aggregate reporting, optional JSON dump for downstream dashboards.
  • scripts/bench_miner.py: thin CLI entrypoint.
  • README in package: install, CLI flags, library API, plug-in interface, calibration notes vs mainnet.

Smoke-tested end-to-end against live Pexels with EfficientNetV2-M on GPU: 5/5 PGD challenges scored cleanly through the validator-parity scorer.

Test logs:

$ python scripts/bench_miner.py --n-challenges 10 --seed 42 --pgd-steps 25

[bench] loading EfficientNetV2-M on cuda...
[bench] challenge 1/10 ...
  prompt=dog          true=german shepherd           eps=0.0805 attack_ms=  350 score=0.7543 reason=success
[bench] challenge 2/10 ...
  prompt=fish         true=eel                       eps=0.1446 attack_ms=  106 score=0.9151 reason=success
[bench] challenge 3/10 ...
  prompt=cat          true=persian cat               eps=0.1961 attack_ms=  177 score=0.7604 reason=success
[bench] challenge 4/10 ...
  prompt=caprine      true=ibex                      eps=0.1811 attack_ms=  116 score=0.9202 reason=success
[bench] challenge 5/10 ...
  prompt=porcine      true=wild boar                 eps=0.1318 attack_ms=  262 score=0.4492 reason=success
[bench] challenge 6/10 ...
  prompt=amphibian    true=tailed frog               eps=0.1491 attack_ms=  183 score=0.7593 reason=success
[bench] challenge 7/10 ...
  prompt=insect       true=long-horned beetle        eps=0.1225 attack_ms=  131 score=0.9162 reason=success
[bench] challenge 8/10 ...
  prompt=fish         true=goldfish                  eps=0.1441 attack_ms=  128 score=0.9161 reason=success
[bench] challenge 9/10 ...
  prompt=cat          true=egyptian cat              eps=0.1139 attack_ms=  131 score=0.7605 reason=success
[bench] challenge 10/10 ...
  prompt=bird         true=jacamar                   eps=0.0601 attack_ms=   92 score=0.9128 reason=success

strategy        : pgd
challenges      : 10 / requested 10
success rate    : 100.0% (10/10)
mean score      : 0.8064
mean attack_ms  : 168
reasons         :
  success                        10

Sample JSON dump entry (--out results.json)

{
  "task_id": "bench-1779210580533-1",
  "prompt": "dog",
  "true_label": "german shepherd",
  "epsilon": 0.0805,
  "attack_ms": 350,
  "evaluation": {
    "score": 0.7543058425454864,
    "reason": "success",
    "model_prediction": "kelpie",
    "response_time_ms": 350,
    "norm": 0.007843166589736938,
    "rmse": 0.004740638192743063,
    "epsilon": 0.0805,
    "ssim": 0.9930430054664612,
    "psnr_db": 46.4832641328272
  }
}

Adds `miner_bench/`: a standalone harness that lets miners evaluate
attack strategies against the same scoring rules the mainnet validator
applies (faithful port of `verify_and_score`), without consuming validator
resources or polluting on-chain history.

- scoring.py: dependency-light port of `PerturbValidator.verify_and_score`
  with the same SSIM/PSNR/Linf/RMSE gates and score combination.
- challenge.py: Pexels fetch + Challenge dataclass; mirrors the validator's
  challenge generation (prompt rotation, photo sampling, label-from-model).
- strategy.py: `MinerStrategy` ABC + `PGDStrategy` reference baseline
  (textbook untargeted PGD, respects validator's effective max-delta cap).
- runner.py: `run_bench()` + CLI with per-challenge + aggregate reporting,
  optional JSON dump for downstream dashboards.
- scripts/bench_miner.py: thin CLI entrypoint.
- README in package: install, CLI flags, library API, plug-in interface,
  calibration notes vs mainnet.

Smoke-tested end-to-end against live Pexels with EfficientNetV2-M on GPU:
5/5 PGD challenges scored cleanly through the validator-parity scorer.
@dataCenter430
dataCenter430 force-pushed the feat/miner-bench-harness branch from c9e4cab to bc9940a Compare May 18, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant