Skip to content

macbox bench: a scaling and regression suite for concurrent simulators - #24

Open
0xdaniell wants to merge 23 commits into
mainfrom
bench-suite
Open

0xdaniell wants to merge 23 commits into
mainfrom
bench-suite

Conversation

@0xdaniell

@0xdaniell 0xdaniell commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

macbox bench: one suite that answers "how many simulators does this host run at once, each one smooth and responsive?" and guards a change against that answer. It builds on simfps, GlassBench and the frame strip from #18. Nothing outside src/bench/ changes except a bench subcommand in macbox and one .gitignore line.

What it does

  • macbox bench scale --n 1,2,3,...: boots one more simulator per step and measures every one of them in the same window: distinct app frames/s and the time between them (p50/p95/p99/max, longest gap), an input-latency proxy, guest CPU and memory per simulator, QEMU CPU by thread group, GPU busy time, host and guest swap. It reports the largest N that still meets each quality target (smooth60, smooth30, responsive; your own with --targets) and Jain's fairness index. Mixes: all animating, one animating beside still ones, an agent-like launch/screenshot/terminate cycle, idle.
  • idle, lifecycle (clone, first boot, launch, first frame, shutdown, resume, delete), soak, ab (A B B A on one live switch, restored byte for byte), compare (two result files, with a noise band) and chart.
  • Bench simulators are made the way production ones are (a clone of Simlock's source device, the Slim profile applied and checked, the idle guard told to keep them) in a device set of their own. One difference: Simlock deletes a clone's LaunchServices database before its first boot, the bench keeps it, so the lifecycle's first boot below is not a lease's (17-22 s here). A run deletes what it made; the next run refuses to start if one could not.
  • Gates, each PASS / FAIL / SKIP (an unreadable gate is SKIP, never PASS): frame order from the strip, torn screenshots, pixel-exact static screen, Coherent's census and process counters, kernel GPU errors, crash reports of the bench and of the simulator's graphics path, swap on host and guest. Settle fails closed: a guest that does not get quiet is not measured.
  • DRY=1 runs the whole control flow against stubs with injectable faults.

Tested

  • 159 unit tests pass (python3 -m unittest discover -s tests -t . in src/bench), including the dry runs with injected faults.
  • On a VM (macOS 26.7.1 guest, iPhone 17 Pro, iOS 26.5, Arrow Lake iGPU, 2026-11-17): tools, check, seed, cleanup, scale at 3x for N = 1, 2, 3 in the all-busy and one-busy mixes, and lifecycle (5 cycles, no boot storm). No gate failed in the final runs; three were SKIP (no golden picture stored yet, QEMU's log and the hang flag not readable from where it ran). All animating:
animating simulators distinct frames/s, each p95 between them input proxy p50 hand-made measurement, same night
1 57.6 (60.0 and 59.2 in the two runs before it) 23.8 ms 283 ms 60
2 39.6, 39.4 45 ms 310-313 ms 37-41
3 23.1, 23.6, 23.0 75 ms 350-363 ms 22-25

One animating beside still ones: it keeps 60 distinct frames/s at 1, 2 and 3 booted (59.9, 59.8, 60.0), and the still ones answer the input proxy as fast as it does (p95 305-311 ms against 285-304 ms, 0 of 75 probes missed). Lifecycle medians: clone 2.9 s, first boot 5.6 s, launch call 0.5 s, first app frame 1.0 s, shutdown 3.2 s, resume boot 5.5 s, delete 0.2 s; no steady memory growth over the 5 cycles.

The first VM runs did not agree with the hand-made numbers. Each cause was found by bisecting on one simulator and is fixed in the last four commits:

  • simfps locked every surface twice (frame strip, then the input marker): 54-58 screen frames/s against 104-110 with one lock (A B B A, 12 s windows). Strip and marker are now read under one lock.
  • simctl openurl as the input probe raises an "Open in GlassBench?" alert on iOS 26 that stays over the app, through a relaunch too: 121 → 80-83 screen frames/s. No longer tried.
  • simctl ui appearance as the probe redraws the whole screen: 43 distinct frames/s against 59 in the same window without it. The probe is now a Darwin notification GlassBench listens for (simctl spawn <udid> notifyutil -p ...): 60.0 distinct with it, 60.0 without.
  • The tear check flagged a card gap sliding under the navigation bar (3 of 3 runs). Of 50 saved screenshots it can judge it now flags 3; two of those were looked at and are torn.
  • On a still screen a picture read the moment its surface changes showed the frame before it in 14-15 of 15 probes (30 ms later the same surface showed the new one), so probes were counted as missed. On a still screen simfps now looks at an unchanged surface again (never where it reads the strip: that would be a second lock of most pictures at 30 a second); the result says how many probes were answered that way.
  • The lifecycle's leak gate failed on QEMU's RSS rising once and then staying flat (likely the host bringing swapped pages back; not counted in that run). It judges RSS + swap and only a steady growth now.
  • A routine crash report of an unrelated daemon failed the run. Reports of the bench's programs and of the graphics path fail it; the others are listed.

Not tested on a VM (stubs and unit tests only): idle, soak, ab, compare on real results, the agent mix, the lifecycle's boot storm, the 1x device type, N above 3. src/bench/README.md says so too.

Known limits

  • The input number is a proxy: it includes starting simctl spawn (about 280 ms here with one simulator alone), not a finger's path or the Device panel's stream. Read it as a trend over N.
  • What reading the strip costs is not stable: over seven runs the measured window's screen rate was +2, -2.5, -5.6, -12.7, -20, -23 and -31% against the unviewed window of the same run. In the one run with the three largest gaps (79-82 measured, 102-115 unviewed, interleaved over ten minutes) the distinct frames were still right (59.3 without a probe). The cause was not found; the result reports the gap as observer every time.
  • Screenshots tear now and then here (a known defect of the stack, not of the bench): the tear gate fails the step and the run, that N does not count as sustained, and the sweep goes on (covered by a dry-run test; no VM run met a tear after the fix).
  • The Simlock source folder defaults to ~/.simlock/devices/ios-sources; an install that names its state directory differently gives it with --source-set or BENCH_SIMLOCK_SOURCES (the VM runs did).
  • The memory gate stops a sweep when the host swaps the VM's memory out during a window (it did once: 690 MB in 40 s). That is the host's state, and the right reason not to trust the window.

…frame stamps; GlassBench -static and an input marker (not compiled yet: no Swift on the authoring host)
…detection, targets, A/B rule, result schema, compare, chart) with unit tests
…r-simulator CPU/memory, input probes, screenshots, lifecycle) with a stub for dry runs; host probes (QEMU thread CPU split, GPU fdinfo, cgroup, PSI), gates, run safety
…olden and A/B runner behind src/bench/run, with a DRY=1 stub guest
…acbox bench entry point, dry-run fault tests
…es, clone timing, light appearance, foreign launchd_sim check
…s, settle recorded, census dir from QEMU, cgroup-only swap gate
…host, appearance reset, README lifecycle timings, more tests
…rified cleanup that exits non-zero, idle-guard keep per simulator

The bench's set is now 'bench' (not per run) so a seed can be kept; a run takes the set with a lock and refuses to start
while an earlier run left devices, a lock or overrides. Teardown spares seeds, removes each device's overrides folder and
~/.sim-keep file, looks again and reports what is left (exit 20). macbox bench cleanup. sim-idle-guard keep <udid> <min>
(max 240) for every simulator; absent tool recorded, refusing tool fails. capture_env no longer drops runtime_used and
device_type. Stub: the files outside the device live under its state dir.
…d Slim seed made from Simlock's source device

ONE seed per device type in the bench's set: simctl --set <Simlock sources> clone <source> <seed> <bench set> (the only
command that names Simlock's set), idle-guard keep, boot, simslim --set <bench set> on <seed> --profile (the sources are
widget-trimmed, not Slim), >= 100 services off checked, the app, a real settle (two quiet samples, widget icons present),
shutdown. Kept between runs, reused, --rebuild-seed; a half-made seed is deleted, a stale app refuses reuse. Every
simulator = simctl clone of the seed + the seed's disabled.plist copied to its UDID folder (checked) before its first
boot. Fresh stock devices only behind --stock-devices (one at a time, guard in front); --slim/simslim-per-device removed.
No Simlock source: a clear message offering the stock path. Lifecycle cycles clone the seed. Stub models a source set,
simslim, plutil, apps and chronod icons.
…nt before every window, a dead frame counter, foreign simulators

core.settle raises with the top processes by CPU (two quiet samples in a row, --settle-max 300): the calibration settle
too. core.window counts the bench set's own Booted devices first (the guest's count is what a step records as 'booted')
and fails on a mismatch with the expected number; a simulator outside the bench booted before or during a window yields /
makes the step unstable. A new 'counter' gate fails any window in which simfps answered nothing or an error for a
simulator or an animating one read no counter (scale steps, soak, observer window). The soak gates are sections now (they
set the exit code). Stub: quiet background model (the clean DRY run used to pass with an unsettled N=3), faults unsettled,
lostsim, counterfail, foreign. No result file is found by pattern: a test pins it.
…he VM unchanged and answering, a monitor that aborts while waiting

--disk-start-gib 18 / --disk-boot-gib 14 / --disk-abort-gib 11 on --disk-path (the filesystem the VM's storage guard watches);
a disk that cannot be read refuses. Guard before EVERY boot (simulators boot one at a time; the boot storm is the one
exception), guest swap-outs since the run began and the compressor (--max-compressor-mb 2048) refuse. A monitor thread
(Run.watch) checks the host disk and QEMU's pid AND start time always, and the Mac (ssh answers twice, someone else's
simulator, swap, compressor) outside measurement windows; a breach aborts the run from that thread: partial result,
the one cleanup (the second caller waits for the first), closing sections, exit 8/9/7/10. Stub: faults diskstart, disklow,
diskabort, diskunreadable, qemugone, swapgrow, compressor.
…nds; check lists Simlock, simslim and the idle guard; compare warns on mixed device sources

The seed's simslim profile is the only .json in the simslim config folder unless --slim-profile / BENCH_SLIM_PROFILE says
otherwise (no machine-specific name in the repo).
…ores the bench's own transitions, icons recorded not gated, abort race, deadline verdict, lifecycle guard
A Mac that runs simulators files crash reports for other daemons all day
(widget extensions, searchd, UsageTrackingAgent). One of those failed the
first run on a VM. New reports of the bench's own programs and of the
processes the measured frames pass through still fail a step and the run;
the others are recorded (stability.other_ips, the gate's text).
--strict-crashes judges every report, --crash-names adds names.
…ads nothing

simfps locked every surface twice when it read both the frame strip and the
input marker. A lock of a surface the simulator just drew makes it wait for
the pixels: with two per surface one 3x simulator composed 54-58 frames/s
instead of 104-110 (measured, A B B A, 12 s windows). Strip and marker are
now read under one lock (108-110).

The observer check's second window never got --no-strip, so it compared two
identical windows. It now reads neither strip nor marker and injects nothing:
simfps locks no surface there.
…elow the bar

What the first runs on a VM showed, each measured on one 3x simulator:
- simctl openurl is answered by an "Open in GlassBench?" alert on iOS 26.
  Nobody taps it, it stays over the app through a relaunch, and the screen
  rate fell from 121 to 80-83 frames/s for the rest of the run. auto no
  longer tries it.
- an appearance switch as the probe redraws the whole screen: 0.5 s to
  answer, and the window lost a quarter of its distinct frames (43 against
  59 a second). The probe is now a Darwin notification GlassBench -probe
  listens for (simctl spawn notifyutil -p); appearance is the fallback.
  It is answered 280 ms after simctl spawn is started with one simulator
  alone, so the responsive bar is 500 ms, not 150.
- the tear check flagged a card gap sliding under the navigation bar's glass
  (3 of 3 runs). Its band starts below the bar; of 50 saved screenshots it
  can judge it now flags 3 (two looked at: torn).
- a torn screenshot fails the step and the run and that N is not sustained,
  but the sweep goes on (a dry-run test; no VM run met one since).
- the observer's cost is INFO, not FAIL: a lock of every picture costs
  frames (between nothing and 31% over seven runs; why it varies is not
  known); only a window that did not answer fails.
- Simlock's source folder is named by its install: the default is
  ~/.simlock/devices/ios-sources, --source-set or BENCH_SIMLOCK_SOURCES
  says otherwise (check and every guest call pass it on).

README: what has run on a real VM so far (tools, check, seed, cleanup and
the all-busy 3x sweep for N = 1, 2, 3, with its numbers) and what has only
run against the stubs.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T08:32:28.171966Z 32f96d6 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dcbf36fc99

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +45 to +46
span = times_s[-1] - times_s[0]
out["fps"] = r1((n - 1) / span if span > 0 else 0.0)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include the full measurement window in frame and stall metrics

When a busy simulator freezes before the window ends, calculating the span from only its first and last distinct frames hides the freeze. A synthetic 40-second window that animates at 60 fps for one second and then repeats its last counter is reported by sim_metrics as 60 fps with a 17 ms longest gap, and passes smooth60; the counter and strip gates also pass. This can certify a frozen simulator as sustainable or hide an A/B regression. Pass the scheduled window boundaries into the metric calculation, use the full duration for frame rate, and include boundary stalls.

Useful? React with 👍 / 👎.

Comment on lines +399 to +403
a_ = analysis.strip.analyze_counters([(t, c) for t, c, _ in raw])
sm.update(analysis.motion_metrics([t for t, _ in a_["unique"]], a.nominal_fps))
sm["strip"] = {k: a_[k] for k in ("repeats", "lost", "backwards", "jumps", "unreadable", "span_ticks")}
ok, why = analysis.strip.strip_gate(a_)
sm["strip_gate"] = {"ok": ok, "why": why}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Analyze agent counters separately for each app launch

For the agent mix, agent_loop intentionally terminates and relaunches GlassBench every period, and each new process resets FrameStrip.count to zero. Analyzing the entire window as one continuous counter sequence therefore treats normal relaunches as frames shown backwards: two ordinary 180-frame launches produce 179 backwards-frame errors in this analyzer. Intentional app-off stretches also contribute unreadable samples. Segment strip validation by successful launches and exclude intentional app-off periods so a healthy multi-cycle agent workload does not fail its correctness gate.

Useful? React with 👍 / 👎.

ok = f0 > 0 and f1 >= 0.95 * f0
ctx.section("soak_fps", "PASS" if ok else "FAIL", "SOAK %s: slowest simulator %.1f fps at the start, %.1f after %g min (bar: not below 95%%)" % ("PASS" if ok else "FAIL", f0, f1, a.soak_minutes))
ctx.doc["gates"].update(ctx.coh.check(ctx.marks["coh"]))
ctx.doc["gates"]["strip"] = gates.strip_gate_of(res["end"]["sims"])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Roll up strip failures from both soak windows

If frames are out of order in the initial soak window but the final window is clean, this assignment discards the initial strip failure and emits a PASS strip section. Unlike the counter gate immediately below, the strip gate only checks res["end"]; neither the FPS comparison nor the run-wide gates checks the initial strip result. A soak can consequently exit successfully despite recording corrupted frame ordering. Aggregate both windows and preserve a failure from either one.

Useful? React with 👍 / 👎.

Comment on lines +99 to +102
risk = max((d.get("overwrite_risk", 0) or 0) for d in surf)
errs = max((d.get("sync_errors", 0) or 0) for d in surf)
res["coherent_census"] = gate("FAIL" if risk or errs else "PASS",
"%d report(s): overwrite_risk max %d, sync_errors max %d" % (len(surf), risk, errs))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Compare census counters against their per-surface baseline

If a surface experienced an overwrite risk or synchronization error before the run and remains alive, its next periodic census report incorrectly fails the new run even when neither counter increased. The renderer's shadow_coherent.rs::Coherent::line emits cumulative counters by loading the underlying atomics, so selecting newly appended reports does not make their values window-local. Appending an identical surface report with overwrite_risk: 1 after mark() reproduces this false failure, which can stop an otherwise healthy sweep. Snapshot the counters by surface identity in mark() and judge their increments, rather than testing the new reports' absolute values.

Useful? React with 👍 / 👎.

for s_, bs in zip(fresh, b["sims"]):
ctx.sims.append({"idx": s_["idx"], "udid": s_["udid"]})
usable.append(bs["usable_s"])
boot = {"new": new, "all_usable_s": round(time.time() - t0, 2) if not parallel else groups[0][1]["all_s"], "each_usable_s": usable}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Report the complete duration of a stock-device storm

When lifecycle runs with --stock-devices and a storm of more than one simulator, grow(..., parallel=True) takes the serial create-and-boot path, but this expression still selects only the first boot group's duration. Four sequential 10-second boots therefore produce all_usable_s: 10 instead of 40, which run_lifecycle stores as boot_all_s and describes as all simulators booting at once. This makes the stock-device storm results misleading. Aggregate the actual serial duration and label the execution correctly, or implement concurrent booting before using the parallel timing branch.

Useful? React with 👍 / 👎.

…and only steady ones

The one-busy mix and the lifecycle ran on a VM for the first time:
- on a still screen a picture read the moment its surface changes showed the
  frame before it in 14-15 of 15 probes; 30 ms later the same surface showed
  the new one. The probes counted as missed (3, 3 and 1 of 15) or were
  answered by the flip before. simfps now looks at an unchanged surface
  again after 30, 150 and 600 ms (a line with counter -3, not a frame), and
  the result counts probes answered that way (input_second_looks).
- the lifecycle's leak gate failed on +215 MB of QEMU RSS per cycle from 3
  samples: the host bringing swapped pages back, one step then flat. It
  judges RSS + swap now, FAILs only a steady growth (4 samples, R2 >= 0.8)
  and says SKIP for a slope above the bar without that; 5 cycles by default.

README: both scenarios' numbers in what has run on a VM.
simfps looked again at an unchanged surface 30 ms after it appeared, also
while reading the strip. With 2 or 3 animating simulators a picture stays
longer than that, so it would have been a second lock of most pictures (two
locks halved the frame rate). It now does so only where it does not read
the strip, which is the still screen it was made for and the only place it
ran on a VM.

README: a bench clone keeps the LaunchServices database a Simlock lease
loses, so its first boot is not a lease's; the proxy on a moving screen may
carry one picture's interval; the RSS step in the first lifecycle run is
likely swap-in, it was not counted.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 32f96d6541

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +142 to +143
f0 = min([s["fps"] for s in res["start"]["sims"] if s.get("fps")] or [0])
f1 = min([s["fps"] for s in res["end"]["sims"] if s.get("fps")] or [0])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include zero-FPS simulators in the soak comparison

When one simulator stops producing distinct app frames during the soak but another keeps animating, the truthiness filter removes the stalled simulator's fps: 0.0 from the minimum. For example, an initial fleet of [60, 60] fps and a final fleet of [60, 0] fps reports 60 fps in both windows and passes. The counter and strip gates do not necessarily catch this: repeated readable counters yield zero distinct FPS while passing both gates. Filter only missing values, not zero, so a completely stalled simulator fails the soak's FPS-retention check.

Useful? React with 👍 / 👎.

Comment thread src/bench/macbench/ab.py
raise Stop(reason)
ctx.host.n = n
core.grow(ctx, n, core.roles_for(mix, n))
ctx.probe_method = scale.calibrate(ctx, ctx.sims[0])["chosen"] if mix in ("all-busy", "one-busy") and metric == "input_p95" else "none"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enable input probes when latency is a requested A/B tail

When running ab --metric total_fps --tails input_p95, this condition disables input probes because latency is not the primary metric, even though it is an explicitly requested regression guard. Consequently, every arm's input_p95 is None, and abrule.decide treats that missing tail as not worse, allowing an ADOPT decision without measuring the requested latency constraint. Enable and calibrate probes when either the primary metric or a requested tail requires them, and prevent an unavailable required tail from silently permitting adoption.

Useful? React with 👍 / 👎.

Comment thread src/bench/macbench/ab.py
raise Stop(reason)
ctx.host.n = n
core.grow(ctx, n, core.roles_for(mix, n))
ctx.probe_method = scale.calibrate(ctx, ctx.sims[0])["chosen"] if mix in ("all-busy", "one-busy") and metric == "input_p95" else "none"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enable input probes when latency is a requested tail

When ab --metric total_fps --tails input_p95 is used, this condition disables input probes because latency is not the primary metric. Every arm consequently has input_p95: None, and abrule.decide() treats the unmeasured tail as not worse, allowing ADOPT without checking the explicitly requested latency constraint. Calibrate and enable probes when either the primary metric or a requested tail requires input measurements.

Useful? React with 👍 / 👎.

Comment on lines +78 to +80
med = {k: _median([c[k] for c in cyc if c.get(k) is not None]) for k in keys}
data["median_s"] = med
ok = all(v is not None for v in med.values()) and bool(cyc)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fail lifecycle runs when any launch produces no app frame

If one cold or resumed launch never produces a readable app frame but subsequent cycles succeed, cmd_cycle() records that launch's first-frame time as None. Removing those values before checking only the medians hides the failed launch: first-frame results [None, 0.2, 0.2, 0.2] produce LIFECYCLE PASS with a 0.2-second median. Without a crash, the run-wide gates do not detect this failure either. Validate first-frame availability for every cycle before aggregating successful timings, and fail the lifecycle section if any launch never yielded a frame.

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant