Skip to content

docs: add Exp. 8 benchmark results - #12

Merged
leofmarciano merged 7 commits into
mainfrom
devin/1793683164-exp8-benchmarks
Nov 3, 2026
Merged

leofmarciano merged 7 commits into
mainfrom
devin/1793683164-exp8-benchmarks

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What and why

Adds the Exp. 8 replication results to the README and the eval tooling behind them.

  • README: test-set results on Qwen3.5-0.8B for the base model, the paper's checkpoint and our Opus 5.5 edits, with thinking on (paper protocol, 32k tokens) and off (model card sampling, 8k tokens). One sample per problem.
  • Takeaway: Opus's 4-neuron edit (4,096 weights, 7 forward passes) takes the base from 8.4% / 12.4% to 40.0% / 53.2% (MATH-500 / GSM8K), below the paper's edit at 52.4% / 61.6%. The base with thinking off already gets 46.4% / 56.6%, and no edit beats it in non-thinking mode.
  • experiments/exp8: evaluate.py --no-thinking, reuse of complete generations, table.py, math-verify in the requirements.

Still missing: Astra's edit with thinking on, plus random-patch controls for the Opus edit. The GPU pod was removed before those finished.

How it was tested

  • ruff check . && ruff format --check .
  • pytest
  • Added a CHANGELOG.md entry (if user-facing)

Evals ran on Lium GPUs (1×H100, then 4×L40S) with vLLM 0.33.

devin-ai-integration Bot and others added 7 commits November 3, 2026 11:59
…_edit

Co-Authored-By: Leonardo Marciano <leonardo.marciano@toolzz.me>
Co-Authored-By: Leonardo Marciano <leonardo.marciano@toolzz.me>
Co-Authored-By: Leonardo Marciano <leonardo.marciano@toolzz.me>
Co-Authored-By: Leonardo Marciano <leonardo.marciano@toolzz.me>
Co-Authored-By: Leonardo Marciano <leonardo.marciano@toolzz.me>
Co-Authored-By: Leonardo Marciano <leonardo.marciano@toolzz.me>
Co-Authored-By: Leonardo Marciano <leonardo.marciano@toolzz.me>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@leofmarciano
leofmarciano merged commit 164da4e into main Nov 3, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant