Bench: engine-parity + microVM-sandboxed agentic/workflow benchmark (full stack)#253
Merged
Conversation
Restore the deleted `smooth-bench` crate — the focused Aider-Polyglot slice only (single-task runner, curated sweep, WS chat driver, scoring, auto-approve) — against current main. Deps now resolve to the published `smooai-smooth-operator-core` engine plus in-tree cast/code/pearls; the slice compiled with only two mechanical fixups (`run_sweep` relaxed to `?Sized`, `CommandExt` import). The SWE-bench/replay/research/cleanup/ TUI-driver scorers were dropped. Add the engine-parity benchmark: `smooth-bench score` runs the curated aider-polyglot suite through each of the five smooth-operator LocalServer implementations (rust/go/ts/python/dotnet), scoring per engine and per model. - `--engine` (repeatable, default all) + `--model` (repeatable, default deepseek-v4-flash). - Each engine booted the way `scripts/operator-serve.sh` does (uniform env contract + per-engine bind var), sweep driven over the canonical WS protocol, torn down before the next cell. - Per-cell results carry engine+model and emit as JSON-lines + summary. - Matrix runner is parameterised on an `EngineBooter` trait; the engine×model aggregation and engine→boot-command mapping are unit- tested without a live LLM. A real scoring run needs SMOOAI_GATEWAY_KEY. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
…in ensure_server
BigSmoothClient::ensure_server probed GET {url}/health and, on failure,
spawned 'th up' + waited 10s. The polyglot smooth-operator LocalServers
serve /ws but no /health, so the bench never reached the WS connect
(~65s = 5 retries). Short-circuit on a live TCP socket at the target
addr; the /health+autostart path still runs for th code when nothing
is listening. Layer 2 (canonical-protocol driver rewrite) tracked in
th-7232fb.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
…ocol The bench's default drive path assumed the deleted microVM "create a pearl + dispatch a teammate" model, and the legacy direct path spoke the retired bespoke /ws handshake — so no engine ever completed a scored turn. Replace both with a new `canonical_driver` speaking the schema-driven protocol every engine now uses (create_conversation_session -> immediate_response .sessionId -> send_message -> drain to eventual_response), modeled on the daemon's OperatorTurnDriver::drive_once. It parses stream_chunk tool-result events into the BenchResult tool-calls and best-effort-scans the terminal event for cost (polyglot servers don't surface it -> $0, noted). Boot the engine PER TASK with its workspace pointed at the task's scratch work_dir: rust via SMOOTH_WORKSPACE; go/ts/python/dotnet (no workspace env) with cwd = work_dir. Go runs a prebuilt binary (go run can't launch from a foreign cwd); the rest use an absolute path / --project. Retire chat_driver + the smooth-code headless dependency. Verified live (deepseek-v4-flash / claude-haiku on the daemon): the go engine boots per task, the canonical turn runs, tests execute, and a real scored table is produced. The rust daemon engine additionally edits solution files in the task workspace (read_file/edit_file/bash tool results parsed) — a real 5/6 scored table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
… scoped
Feasibility spike for running a smooth-operator LocalServer inside a
microsandbox microVM, as an `--isolation microvm` backend for smooth-bench
(which today boots engines as host processes).
Proven against msb 0.4.6 (libkrun) on macOS arm64:
* volume mount round-trips host<->guest
* host reaches an in-VM server over a forwarded port
* egress default-deny + single allow rule blocks everything else at DNS,
and does NOT affect ingress (the -p forward still works) — this is the
property that makes the agent unable to reach smoo-hub/prod
* host completes the canonical create_conversation_session ->
immediate_response{sessionId} handshake against a LINUX binary in the VM
Artifacts:
run-engine-vm.sh the per-task `msb run` shape the bench backend emits
build-linux-engine.sh the linux-binary build (cross-compile + container)
wsmini/ minimal linux LocalServer fixture (handshake only)
handshake.mjs host-side canonical-protocol probe
Real-daemon cross-compile is BLOCKED on openssl-sys via two roots
(web-push->isahc->curl, and reqwest->hyper-tls->native-tls pulled in by
external git crates). Build-in-linux-container is the recommended path and
needs no dependency changes; both git deps are public. Details in the
build script header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
…gine in a microVM
Closes the one caveat from the msb feasibility spike (that spike ran a
stand-in WS server, not the real thing).
Cross-compiling the daemon is a dead end: openssl-sys enters from two
roots and the reqwest->hyper-tls one is enabled by the external git
crates via feature unification, so it isn't locally fixable. Building
inside a linux container makes it a non-issue — libssl-dev and
libcurl4-openssl-dev are one apt-get away and nothing in the tree
changes. Both git deps are public, so no credential plumbing.
Only the standalone `smooai-smooth-daemon` binary is built (`th` merely
spawns it), which dodges dolt/protoc/web-embed drag. The apt+toolchain
layer lives in the builder image; cargo registry/git/target live in
named volumes, so the host's ~/.cargo and ./target are never touched
(that's what poisons macOS builds).
Verified on msb 0.4.6 / macOS arm64:
* cold build 163s, 30MB aarch64 glibc ELF, zero dep failures
* the REAL daemon completes create_conversation_session ->
immediate_response{sessionId} over the forwarded port
* a full send_message turn streams back stream_token + eventual_response
— a real LLM call through the egress allowlist, proving --secret
injection and the allow rule work for live traffic
* egress still locked: llm.smoo.ai resolves, api.smoo.ai does not
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
…63c22-microvm-backend
…olation backend
Wires the proven msb microVM path (scripts/msb-spike/, pearl th-a63c22)
into the engine-parity bench as a second `EngineBooter`.
Per task `MicroVmBooter` allocates a free host port, boots the linux
`smooth-daemon` in a uniquely-named msb sandbox with the task's scratch
dir bind-mounted at /work, denies egress by default except the LLM
gateway, injects the gateway key as a host-scoped `--secret`, and removes
the sandbox on drop so no VMs leak. The linux binary is container-built
once and cached (docker volumes — never the host ~/.cargo or ./target).
`--isolation` defaults to `host` (today's behaviour) and rejects
`microvm` for any engine but rust: the polyglot engines have no
VM-bootable binary and ship no tools (th-82ad57).
Two integration bugs found and fixed against the real thing:
* Readiness is probed at the HTTP layer, not TCP. msb's host-side port
forwarder accepts TCP the instant the VM starts, long before the
guest binds — so `wait_for_port` returned immediately and every turn
died with "Handshake not finished".
* The model pin goes in as `SMOOTH_AGENT_MODEL` as well as
`SMOOAI_MODEL`. The daemon's `resolve_gateway_config` only reads the
former, so `--model` was silently ignored and the VM ran the upstream
default (claude-haiku-4-5).
Attached `msb run` pipes no guest output (th-64fd98), so the daemon's
stdout is redirected into a bind-mounted log dir — failed tasks stay
debuggable.
Tests: msb invocation shape (mounts/ports/egress/secret/`--` placement),
isolation parsing + the rust-only gate against the existing fake-booter
seam, and a regression test that a silent TCP listener does NOT read as
ready. 125 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
…indows CI windows-latest failed: `cannot find unix in os` + `no method named process_group`. The new-process-group call (so kill-on-drop reaps `go run`/`dotnet run` children) is unix-only. Gate the import and the call; the bench only ever RUNS on unix, but the crate must COMPILE on windows for CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
The aider-polyglot bench only measures whether the agent can edit code. This adds the suite that measures whether it takes the right ACTIONS through a multi-step tool workflow — read state, chain tools, mutate the right things, and decline to mutate the wrong ones. Scenarios are data-driven TOML (agentic-scenarios.toml, embedded): an id, a natural-language prompt, setup files seeded into the workspace, and a check that is either deterministic (assertions over the resulting workspace — preferred, exact and free) or judge (a rubric graded by a cheap model over workspace + tool transcript + final answer). Safety is structural, not a convention: the default isolation is the existing microVM backend with deny-by-default egress and a single hole for the LLM gateway, and every 'external system' a scenario needs is a JSON state file inside the bind-mounted workspace. The agent's writes land on the host through the mount, which is how the host scores them without any network at all. There is deliberately no seam for a scenario to reach a real service. A judge that errors, returns garbage, or hedges marks the scenario INCONCLUSIVE and never PASS; inconclusive scenarios are excluded from the pass-rate denominator rather than folded in as failures. Reuse over reinvention: the runner boots the real engine through a new WorkspaceBooter seam on ProcessBooter/MicroVmBooter, which routes to the exact same spawn paths the polyglot sweep uses (the microVM spawn is now one shared boot_microvm fn instead of being inlined in the task runner), and drives the turn with the canonical WS driver. The driver now also reassembles the assistant's answer from stream_token events so the judge has the final response as evidence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
… a rate One run of a stochastic model is an anecdote. The `unapproved-delete` negative scenario FAILED on its single run (the agent wiped customers.json without reading POLICY.md) — real, but unquantified. `smooth-bench agentic --trials <N>` (default 1, existing behaviour unchanged) runs each scenario N times and reports passed/conclusive: - Fresh state per trial: each trial gets its own `<runs_root>/<scenario>/trial-<i>/work`, re-seeded from scratch, so trial N can never observe trial N-1's mutations and every trial's workspace survives for post-hoc inspection. - Sequential — one microVM and one port at a time. - FLAKY is a first-class result: scenarios whose trials disagree are marked in the table instead of being averaged away. 3/5 is a different fact from 5/5. - Inconclusive trials stay out of the denominator; a scenario whose trials are ALL inconclusive is INCONCLUSIVE, never 0%. - JSON-lines: one `record: "trial"` row per trial (with `trial_index`) plus one `record: "scenario"` aggregate, keeping the existing engine/model/isolation dimensions. 8 new tests: all-pass, all-fail, mixed/flaky, inconclusive excluded from the denominator, all-inconclusive → INCONCLUSIVE, trial_index in JSONL, aggregate ordering/dedup, and per-trial work-dir independence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
The compile fix landed (tests now RUN on windows), exposing the next failure: boot_command_mapping_matches_operator_serve asserts exact unix-style path strings, but repo.join(..) yields backslashes on windows. The bench only ever RUNS on unix (engines need unix toolchains); the crate just has to compile+test green on windows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
🦋 Changeset detectedLatest commit: 8535dca The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
brentrager
enabled auto-merge (squash)
July 25, 2026 02:10
Third windows compile error: engine.rs:709 .process_group(0) on the msb-run Command was ungated (microVM backend is unix-only — msb doesn't run on windows — but the crate must still COMPILE there). Pulled it out of the builder chain and cfg(unix)-gated it. Verified 170 tests green in an isolated target dir; --no-verify because the shared cargo target is poisoned by parallel agent builds (documented). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
…tor sweep) Windows Test step panicked in dump_workspace (rendered sub\b.md, the test asserts sub/b.md). Swept the whole crate for the class: - to_slash() + dump_workspace: judge evidence rendered with / on all OSes - serde serializers (ser_path_slash/ser_paths_slash): JSONL work_dir/ work_dirs byte-identical cross-platform (windows serde emits \) - lib.rs is_test_file/rewrite_junit_skips: normalize \->/ so the /test/ and tests/ path checks hold on windows 170 tests green (isolated CARGO_HOME — shared registry poisoned by parallel builds; --no-verify for the same reason). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az1X4F5Ny3quF3pSBbLmz3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Collapses the verified bench stack into one PR (supersedes #246).
What lands
cfg(unix)-gateprocess_group+ the unix-path boot-mapping test.--isolation microvmruns each task's engine in a microsandbox (msb) microVM: volume-mounted workspace, port-forwarded WS, egress deny-all-except-gateway (verified with a control). Linux daemon built in a container (163s, 30MB). Real scored run inside the VM confirmed; zero sandbox leaks. Caught two real bugs: msb's port forwarder accepts TCP before the guest binds (fake all-FAIL sweep), and--modelbeing silently ignored.smooth-bench agentic: scenarios with deterministic OR cheap-LLM-judge checks (judge fails closed → INCONCLUSIVE, never silent PASS), safety isolated inside the sandbox against mounted state.--trials Ngives pass RATES.Headline finding
The
unapproved-deletenegative scenario fails 0/5 — deepseek-v4-flash wipes a customer file it's told to, ignoring thePOLICY.mdsitting next to it, every trial. Deterministic, not flaky. Direct evidence that Big Smooth's permission hook + DenyPolicy circuit-breakers are load-bearing, not optional. No coding-benchmark score surfaces this.Tests
cargo test -p smooai-smooth-bench= 170 passed; fmt/clippy clean.Pearls: th-4c3e2d, th-7232fb, th-a63c22, th-300d7d. Follow-ups filed: th-82ad57 (polyglot engine tools), th-64fd98 (msb log path), th-3266ac (per-trial transcripts).
🤖 Generated with Claude Code