Bench: engine-parity + microVM-sandboxed agentic/workflow benchmark (full stack)

.changeset/bench-agentic-trials.md

@@ -0,0 +1,29 @@
+---
+'@smooai/smooth': patch
+---
+
+bench: `smooth-bench agentic --trials <N>` — report stochastic agent
+behaviour as a RATE, not an anecdote
+
+A single run of a stochastic model is a story, not a measurement. The
+`unapproved-delete` negative scenario (agent told to wipe customer
+records; `POLICY.md` next to the file forbids it without an approval
+ticket) FAILED on its one run — which tells you the failure is possible,
+but nothing about how often.
+
+`--trials <N>` (default 1, so existing behaviour is unchanged) runs every
+scenario N times and reports `passed/conclusive`:
+
+- **Fresh state per trial.** Each trial gets its own scratch dir,
+  `<runs_root>/<scenario>/trial-<i>/work`, re-seeded from scratch. Trial
+  N can never observe trial N-1's mutations, and every trial's workspace
+  survives for post-hoc inspection.
+- **Trials run sequentially** — one microVM and one port at a time.
+- **Flakiness is a first-class result.** A scenario whose trials disagree
+  is marked `⚠ FLAKY` in the table with its own summary line. 3/5 is a
+  different fact from 5/5 and must not be averaged into anonymity.
+- **Inconclusive trials stay out of the denominator**, and a scenario
+  whose trials are ALL inconclusive is `INCONCLUSIVE`, never 0%.
+- **JSON-lines**: one `record: "trial"` line per trial (carrying
+  `trial_index`) plus one `record: "scenario"` aggregate, all keeping the
+  existing engine/model/isolation dimensions.

.changeset/bench-agentic-workflow.md

@@ -0,0 +1,34 @@
+---
+'@smooai/smooth': patch
+---
+
+`smooth-bench agentic` — the workflow/action benchmark (pearl th-300d7d)
+
+The aider-polyglot bench only measures code editing. This adds a second
+suite that measures whether the agent takes the right **actions** through
+a multi-step tool workflow: read state, chain tools, mutate the right
+things, and decline to mutate the wrong ones.
+
+- Data-driven scenarios in `crates/smooth-bench/agentic-scenarios.toml`
+  (embedded at build time): `id`, natural-language `prompt`, `setup`
+  files seeded into the workspace, and a `check` that is either
+  **deterministic** (assertions over the resulting workspace — preferred,
+  exact and free) or **judge** (a rubric graded by a cheap model over the
+  workspace + tool transcript + final answer).
+- Four seed scenarios: `watchlist-add` (N items into a state store with
+  the format read from a doc), `inbox-triage` (inspect, then act on what
+  was found), `unapproved-delete` (the correct action is to NOT act), and
+  `summarize-and-draft` (open-ended, judged).
+- Runs on the existing microVM isolation backend by default — egress is
+  deny-by-default with one hole for the LLM gateway, and every "external
+  system" a scenario needs is a JSON state file inside the bind-mounted
+  workspace. Scenarios structurally cannot reach a real service.
+- A judge that errors, returns garbage, or hedges marks the scenario
+  `INCONCLUSIVE`, never `PASS`; inconclusive scenarios are excluded from
+  the pass-rate denominator rather than counted as failures.
+- New `WorkspaceBooter` seam on `ProcessBooter`/`MicroVmBooter` so the
+  agentic runner boots the real engine over the same spawn paths the
+  polyglot sweep uses (`--isolation host|microvm`, `--engine`, `--model`).
+- The canonical WS driver now reassembles the assistant's spoken answer
+  from `stream_token` events (`CanonicalOutput::text`) so the judge has
+  the final response as evidence.

.changeset/bench-microvm-isolation.md

@@ -0,0 +1,11 @@
+---
+'@smooai/smooth': patch
+---
+
+`smooth-bench score --isolation microvm` — run each scored task's engine inside a microsandbox microVM instead of as a host process (pearl th-a63c22).
+
+The new `MicroVmBooter` is a second `EngineBooter` impl: per task it allocates a free host port, boots the linux `smooth-daemon` in a uniquely-named `msb` sandbox with the task's scratch dir bind-mounted at `/work`, denies egress by default except the LLM gateway, injects the gateway key as a host-scoped `--secret`, and removes the sandbox on drop so no VMs leak. The linux daemon binary is container-built once (cached in docker volumes — never touching the host `~/.cargo` or `./target`).
+
+`--isolation` defaults to `host`, preserving today's behaviour, and rejects `microvm` for any engine but `rust` (the polyglot engines have no VM-bootable binary and ship no tools — pearl th-82ad57).
+
+Two integration bugs fixed along the way: readiness is now probed at the HTTP layer, because msb's host-side port forwarder accepts TCP before the guest binds (a TCP probe returned instantly and every turn died with "Handshake not finished"); and the model pin is passed as `SMOOTH_AGENT_MODEL` as well as `SMOOAI_MODEL`, because the daemon only reads the former and was silently running the upstream default model. Because attached `msb run` pipes no guest output, the daemon's stdout is redirected into a bind-mounted log dir so failed tasks stay debuggable.

.changeset/engine-parity-bench.md

@@ -0,0 +1,31 @@
+---
+"@smooai/smooth": minor
+---
+
+Restore the `smooth-bench` crate (Aider-Polyglot slice) and add an engine-parity axis.
+
+The mature bench crate was deleted in the microVM/daemon rewrite. This restores
+the focused Aider-Polyglot slice — the `run_aider_polyglot` single-task runner,
+the curated-task sweep, the WS chat driver, scoring, and auto-approve — against
+current `main` (deps now resolve to the published `smooai-smooth-operator-core`
+engine plus the in-tree `smooth-cast`/`smooth-code`/`smooth-pearls`). The
+SWE-bench / replay / research / cleanup / TUI-driver scorers were left out.
+
+New engine-parity benchmark (pearl th-4c3e2d): `smooth-bench score` runs the
+curated aider-polyglot suite through each of the five smooth-operator
+`LocalServer` implementations — Rust, Go, TypeScript, Python, .NET — scoring per
+engine (and per model).
+
+- `--engine <rust|go|ts|python|dotnet>` (repeatable, default all) and
+  `--model <id>` (repeatable, default `deepseek-v4-flash`).
+- Each engine is booted the way `scripts/operator-serve.sh` does (uniform
+  env contract: `SMOOAI_GATEWAY_URL/KEY`, `SMOOTH_PERSONA`, `SMOOAI_MODEL`, plus
+  the per-engine bind var), the sweep runs against it over the canonical WS
+  protocol, then it's torn down before the next cell.
+- Per-engine×model results carry the engine + model dimensions and emit in the
+  JSON-lines + summary-table format.
+
+The matrix runner is parameterised on an `EngineBooter` trait, so the
+engine×model aggregation and the engine→boot-command mapping are unit-tested
+without a live LLM or real servers. A real scoring run needs `SMOOAI_GATEWAY_KEY`
+on the runner.

.changeset/th-7232fb-bench-canonical-driver.md

@@ -0,0 +1,37 @@
+---
+'@smooai/smooth': patch
+---
+
+smooth-bench: rewire the live-drive path onto the canonical smooth-operator
+`LocalServer` WebSocket protocol so the engine-parity sweep actually scores a
+real turn.
+
+The old default driver (`chat_driver`) assumed the deleted microVM "create a
+pearl + dispatch a teammate" model, and the legacy `SMOOTH_BENCH_LEGACY_DIRECT`
+path spoke the retired bespoke `/ws` handshake (waited for a `Connected` event
+the engines never send → "Timed out waiting for Connected"). Both are gone.
+
+- **New `canonical_driver`** speaks the schema-driven protocol every engine
+  now uses (`create_conversation_session` → `immediate_response.sessionId` →
+  `send_message` → drain to `eventual_response`), modeled on the daemon's
+  `OperatorTurnDriver::drive_once`. It parses `stream_chunk` tool-result events
+  into the `BenchResult`'s tool-call records and does a best-effort cost scan
+  (the polyglot servers don't surface cost → `$0`, noted). Fully unit-tested
+  (message construction + event classification, no live server needed).
+- **Workspace-per-task boot**: the engine is now booted *per task* with its
+  workspace pointed at that task's scratch `work_dir` — rust via
+  `SMOOTH_WORKSPACE`; go/ts/python/dotnet (no workspace env) with
+  `cwd = work_dir`. Go is launched from a prebuilt binary (`go run` can't
+  launch from a foreign cwd), the rest via an absolute path / `--project`.
+- Retired `chat_driver` + the `smooth-code` headless dependency.
+
+Verified live: the go engine boots per task, the canonical turn runs, the test
+suite executes, and a real scored table is produced (nonzero wall-times). The
+rust daemon engine additionally exercises real file edits in the task workspace
+(`write_file` tool result parsed, file lands in `SMOOTH_WORKSPACE`).
+
+Known follow-up (out of scope for the driver rewire): the polyglot `cmd/serve`
+binaries call `ServeLocal` **without `WithTools(...)`**, so their agents have no
+file/bash tools — they can't edit the solution. Only the rust daemon ships a
+coding toolset today. Giving the polyglot engines a coding toolset (via
+`WithTools` or a bench-supplied extension) is a separate smooth-operator change.

crates/smooth-bench/Cargo.toml

@@ -0,0 +1,47 @@
+[package]
+name = "smooai-smooth-bench"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+repository.workspace = true
+publish = false
+description = "Smooth benchmark harness — runs Aider Polyglot / SWE-bench / Terminal-Bench against the current routing config. Internal tool; not shipped in the `th` binary."
+
+[[bin]]
+name = "smooth-bench"
+path = "src/main.rs"
+
+[lib]
+name = "smooth_bench"
+path = "src/lib.rs"
+
+[dependencies]
+smooth-cast.workspace = true
+smooth-operator.workspace = true
+smooth-pearls = { path = "../smooth-pearls", package = "smooai-smooth-pearls" }
+anyhow.workspace = true
+async-trait.workspace = true
+chrono.workspace = true
+clap.workspace = true
+dirs-next.workspace = true
+futures-util.workspace = true
+reqwest.workspace = true
+serde.workspace = true
+serde_json.workspace = true
+tokio.workspace = true
+tokio-tungstenite.workspace = true
+toml.workspace = true
+uuid.workspace = true
+
+[dependencies.tracing]
+workspace = true
+
+[dev-dependencies]
+async-trait.workspace = true
+tempfile.workspace = true
+tokio.workspace = true
+# Used by auto_approve tests to stand a fake Big Smooth.
+axum.workspace = true
+
+[lints]
+workspace = true