Skip to content

Browser runner fixes and honest benchmarks - #457

Open
ealmloff wants to merge 5 commits into
backprop-basefrom
bench-accuracy
Open

ealmloff wants to merge 5 commits into
backprop-basefrom
bench-accuracy

Conversation

@ealmloff

@ealmloff ealmloff commented Sep 8, 2026

Copy link
Copy Markdown
Member

Five commits on top of e42d26a89, split out of the backprop branch so they can be read on their own. The base branch backprop-base is pinned at that commit; nothing here changes backprop or PR #430.

The preview page never finished loading

https://floneum.github.io/kalosm/pr-preview/pr-430 rendered and then froze within a few seconds. The tab was not crashing, it was deadlocked: the first replay-hit resolve arms the tuning clock, Chrome reports TIMESTAMP_QUERY, and read_timestamps then blocks the main thread on a map_async callback that only the event loop can deliver. wasm now never requests the feature, the same shape as the existing subgroup and cooperative-matrix gates, so every consumer takes its "not timed" path.

Two more faults surfaced once the page ran:

  • detach() did a host round trip, which is unavailable on wasm. It is now a device-side copy through a new Target::copy (pool allocation plus a queued buffer copy on GPU, an AlignedBuf memcpy on CPU). detach_async is an alias.
  • Tint rejects 3.4028235e38f, which is how naga prints f32::MAX, so any module containing that literal was invalid and the kernel silently never ran. Sampling read zeros. The emitter now spells the value as bitcast<f32>(bits) and refuses to const-fold to it.

The full conformance suite passes in Chrome, on the deployed preview as well as locally.

The benchmark page was reporting fabricated wins

It claimed roughly 10x over burn on nearly everything. Three separate inflations:

  1. fusor measured nothing. Each iteration rebuilt the same expression; the graph hash-conses, so it was the same node, and a node whose class still holds a device buffer resolves to zero launches. A 23-iteration case did one resolve. Iterations now drop the cached buffer first.
  2. Unequal fences. fusor stopped at a queue fence while burn's only awaitable is a full download. Both sides now retrieve a result.
  3. Mean over median. burn's first round carries its warmup: elementwise cases ran a 0.45 ms median under a 5.35 ms mean, and the mean ratio printed "11.33x faster" where the median said 1.07x slower.

Then two accuracy problems that survived those fixes:

  1. Cases were measuring each other. Every case shared one session, whose e-graph only grows, so a case ran against forty others' leftovers. A batched matmul reported 8.35 ms that reports 0.15 ms alone. Device::isolated gives each case its own session and graph over the same backend, and the registry and the sweep both use it.
  2. Rounds were shorter than the clock. A browser clamps performance.now() to 100 microseconds, so cheap cases sat on a handful of ticks and moved up to 3.1x between runs. The iteration count is now calibrated from an unrecorded probe round to land near 20 ms, capped at 256 so a round does not pin hundreds of live outputs.

Measured over three consecutive full-page runs, the median run-to-run spread is 1.03x, and the widest is burn's own variance rather than the harness.

The resulting picture is mixed, which is the point. fusor is ahead on quantized paired SiLU (7.9x), Q4K gemv (6.8x), RMS norm (5.1x), rope (3.2x), Q8 gemv (2.9x) and layer norm (3.1x); behind on conv1d (15x), elementwise (1.7 to 2.7x), dense matmul (2.0x) and attention (1.7x). conv1d and dense matmul are the real gaps.

cargo run -p fusor-conformance --features burn-bench --example bench_compare and --example bench_sweep <case> run the same comparison natively. Note that native numbers do not predict the page: burn's readback carries a fixed ~35 ms cost natively that it does not have in a browser.

Every route but the root 404'd

https://floneum.github.io/kalosm/benchmarks/dense_batched_matmul returned GitHub's 404 page, and so did /benchmarks, on the live site and the preview alike. GitHub Pages has no single-page fallback: it answers any path the build did not write a file for with a 404, so the benchmark tabs were reachable by clicking through from the root and by nothing else. Not a link, not a reload, not the back button after a reload.

The build and publish steps are both reusable workflows this repo only calls into, so the fix is in the app: the router now keeps the route in the URL fragment. Every URL asks the host for the one page that exists. Verified against a plain static file server, which is the same thing Pages does: /#/benchmarks/dense_batched_matmul loads the detail page directly and runs its sweep, and /#/tests reloads into a passing suite.

Links change shape, from /benchmarks/<case> to /#/benchmarks/<case>.

Conformance ran for 17 minutes

The binary force-set FUSOR_VERIFY_MEMBERS=1, which races every e-class member of every launch to value-check it, on every fuzz run of every case. One small sampling case executed about 470 candidates, and sampling alone was 72% of the suite.

The sweep is now a runtime flag rather than an environment read, and fuzz_case turns it on for one run per case, chosen per case so the swept shape varies across the suite. 17m39s to 1m43s, same 746 results, 737 passed, 9 skipped, 0 failed. The trade is per case: kernel variants are swept at one shape instead of three.

Verification

  • Full conformance green natively (746 results) and in Chrome (373 cases), on the deployed preview too.
  • Whisper transcribes correctly; the Qwen vision tower produces a coherent caption.
  • Both CI clippy invocations, wasm clippy, cargo fmt --check, and the fusor unit tests are clean.
  • All five workflows passed on cb89fd95f.

https://claude.ai/code/session_01AUTdnWQ8FGUP5FKR7dJsKs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant