Repository navigation
Conversation
…frame stamps; GlassBench -static and an input marker (not compiled yet: no Swift on the authoring host)
…detection, targets, A/B rule, result schema, compare, chart) with unit tests
…r-simulator CPU/memory, input probes, screenshots, lifecycle) with a stub for dry runs; host probes (QEMU thread CPU split, GPU fdinfo, cgroup, PSI), gates, run safety
…olden and A/B runner behind src/bench/run, with a DRY=1 stub guest
…acbox bench entry point, dry-run fault tests
…Mesa is flagged; meminfo parse fixed
…es, clone timing, light appearance, foreign launchd_sim check
…s, settle recorded, census dir from QEMU, cgroup-only swap gate
…host, appearance reset, README lifecycle timings, more tests
…rified cleanup that exits non-zero, idle-guard keep per simulator The bench's set is now 'bench' (not per run) so a seed can be kept; a run takes the set with a lock and refuses to start while an earlier run left devices, a lock or overrides. Teardown spares seeds, removes each device's overrides folder and ~/.sim-keep file, looks again and reports what is left (exit 20). macbox bench cleanup. sim-idle-guard keep <udid> <min> (max 240) for every simulator; absent tool recorded, refusing tool fails. capture_env no longer drops runtime_used and device_type. Stub: the files outside the device live under its state dir.
…d Slim seed made from Simlock's source device ONE seed per device type in the bench's set: simctl --set <Simlock sources> clone <source> <seed> <bench set> (the only command that names Simlock's set), idle-guard keep, boot, simslim --set <bench set> on <seed> --profile (the sources are widget-trimmed, not Slim), >= 100 services off checked, the app, a real settle (two quiet samples, widget icons present), shutdown. Kept between runs, reused, --rebuild-seed; a half-made seed is deleted, a stale app refuses reuse. Every simulator = simctl clone of the seed + the seed's disabled.plist copied to its UDID folder (checked) before its first boot. Fresh stock devices only behind --stock-devices (one at a time, guard in front); --slim/simslim-per-device removed. No Simlock source: a clear message offering the stock path. Lifecycle cycles clone the seed. Stub models a source set, simslim, plutil, apps and chronod icons.
…nt before every window, a dead frame counter, foreign simulators core.settle raises with the top processes by CPU (two quiet samples in a row, --settle-max 300): the calibration settle too. core.window counts the bench set's own Booted devices first (the guest's count is what a step records as 'booted') and fails on a mismatch with the expected number; a simulator outside the bench booted before or during a window yields / makes the step unstable. A new 'counter' gate fails any window in which simfps answered nothing or an error for a simulator or an animating one read no counter (scale steps, soak, observer window). The soak gates are sections now (they set the exit code). Stub: quiet background model (the clean DRY run used to pass with an unsettled N=3), faults unsettled, lostsim, counterfail, foreign. No result file is found by pattern: a test pins it.
…he VM unchanged and answering, a monitor that aborts while waiting --disk-start-gib 18 / --disk-boot-gib 14 / --disk-abort-gib 11 on --disk-path (the filesystem the VM's storage guard watches); a disk that cannot be read refuses. Guard before EVERY boot (simulators boot one at a time; the boot storm is the one exception), guest swap-outs since the run began and the compressor (--max-compressor-mb 2048) refuse. A monitor thread (Run.watch) checks the host disk and QEMU's pid AND start time always, and the Mac (ssh answers twice, someone else's simulator, swap, compressor) outside measurement windows; a breach aborts the run from that thread: partial result, the one cleanup (the second caller waits for the first), closing sections, exit 8/9/7/10. Stub: faults diskstart, disklow, diskabort, diskunreadable, qemugone, swapgrow, compressor.
…nds; check lists Simlock, simslim and the idle guard; compare warns on mixed device sources The seed's simslim profile is the only .json in the simslim config folder unless --slim-profile / BENCH_SLIM_PROFILE says otherwise (no machine-specific name in the repo).
…ores the bench's own transitions, icons recorded not gated, abort race, deadline verdict, lifecycle guard
A Mac that runs simulators files crash reports for other daemons all day (widget extensions, searchd, UsageTrackingAgent). One of those failed the first run on a VM. New reports of the bench's own programs and of the processes the measured frames pass through still fail a step and the run; the others are recorded (stability.other_ips, the gate's text). --strict-crashes judges every report, --crash-names adds names.
…ads nothing simfps locked every surface twice when it read both the frame strip and the input marker. A lock of a surface the simulator just drew makes it wait for the pixels: with two per surface one 3x simulator composed 54-58 frames/s instead of 104-110 (measured, A B B A, 12 s windows). Strip and marker are now read under one lock (108-110). The observer check's second window never got --no-strip, so it compared two identical windows. It now reads neither strip nor marker and injects nothing: simfps locks no surface there.
…elow the bar What the first runs on a VM showed, each measured on one 3x simulator: - simctl openurl is answered by an "Open in GlassBench?" alert on iOS 26. Nobody taps it, it stays over the app through a relaunch, and the screen rate fell from 121 to 80-83 frames/s for the rest of the run. auto no longer tries it. - an appearance switch as the probe redraws the whole screen: 0.5 s to answer, and the window lost a quarter of its distinct frames (43 against 59 a second). The probe is now a Darwin notification GlassBench -probe listens for (simctl spawn notifyutil -p); appearance is the fallback. It is answered 280 ms after simctl spawn is started with one simulator alone, so the responsive bar is 500 ms, not 150. - the tear check flagged a card gap sliding under the navigation bar's glass (3 of 3 runs). Its band starts below the bar; of 50 saved screenshots it can judge it now flags 3 (two looked at: torn). - a torn screenshot fails the step and the run and that N is not sustained, but the sweep goes on (a dry-run test; no VM run met one since). - the observer's cost is INFO, not FAIL: a lock of every picture costs frames (between nothing and 31% over seven runs; why it varies is not known); only a window that did not answer fails. - Simlock's source folder is named by its install: the default is ~/.simlock/devices/ios-sources, --source-set or BENCH_SIMLOCK_SOURCES says otherwise (check and every guest call pass it on). README: what has run on a real VM so far (tools, check, seed, cleanup and the all-busy 3x sweep for N = 1, 2, 3, with its numbers) and what has only run against the stubs.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dcbf36fc99
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| span = times_s[-1] - times_s[0] | ||
| out["fps"] = r1((n - 1) / span if span > 0 else 0.0) |
There was a problem hiding this comment.
Include the full measurement window in frame and stall metrics
When a busy simulator freezes before the window ends, calculating the span from only its first and last distinct frames hides the freeze. A synthetic 40-second window that animates at 60 fps for one second and then repeats its last counter is reported by sim_metrics as 60 fps with a 17 ms longest gap, and passes smooth60; the counter and strip gates also pass. This can certify a frozen simulator as sustainable or hide an A/B regression. Pass the scheduled window boundaries into the metric calculation, use the full duration for frame rate, and include boundary stalls.
Useful? React with 👍 / 👎.
| a_ = analysis.strip.analyze_counters([(t, c) for t, c, _ in raw]) | ||
| sm.update(analysis.motion_metrics([t for t, _ in a_["unique"]], a.nominal_fps)) | ||
| sm["strip"] = {k: a_[k] for k in ("repeats", "lost", "backwards", "jumps", "unreadable", "span_ticks")} | ||
| ok, why = analysis.strip.strip_gate(a_) | ||
| sm["strip_gate"] = {"ok": ok, "why": why} |
There was a problem hiding this comment.
Analyze agent counters separately for each app launch
For the agent mix, agent_loop intentionally terminates and relaunches GlassBench every period, and each new process resets FrameStrip.count to zero. Analyzing the entire window as one continuous counter sequence therefore treats normal relaunches as frames shown backwards: two ordinary 180-frame launches produce 179 backwards-frame errors in this analyzer. Intentional app-off stretches also contribute unreadable samples. Segment strip validation by successful launches and exclude intentional app-off periods so a healthy multi-cycle agent workload does not fail its correctness gate.
Useful? React with 👍 / 👎.
| ok = f0 > 0 and f1 >= 0.95 * f0 | ||
| ctx.section("soak_fps", "PASS" if ok else "FAIL", "SOAK %s: slowest simulator %.1f fps at the start, %.1f after %g min (bar: not below 95%%)" % ("PASS" if ok else "FAIL", f0, f1, a.soak_minutes)) | ||
| ctx.doc["gates"].update(ctx.coh.check(ctx.marks["coh"])) | ||
| ctx.doc["gates"]["strip"] = gates.strip_gate_of(res["end"]["sims"]) |
There was a problem hiding this comment.
Roll up strip failures from both soak windows
If frames are out of order in the initial soak window but the final window is clean, this assignment discards the initial strip failure and emits a PASS strip section. Unlike the counter gate immediately below, the strip gate only checks res["end"]; neither the FPS comparison nor the run-wide gates checks the initial strip result. A soak can consequently exit successfully despite recording corrupted frame ordering. Aggregate both windows and preserve a failure from either one.
Useful? React with 👍 / 👎.
| risk = max((d.get("overwrite_risk", 0) or 0) for d in surf) | ||
| errs = max((d.get("sync_errors", 0) or 0) for d in surf) | ||
| res["coherent_census"] = gate("FAIL" if risk or errs else "PASS", | ||
| "%d report(s): overwrite_risk max %d, sync_errors max %d" % (len(surf), risk, errs)) |
There was a problem hiding this comment.
Compare census counters against their per-surface baseline
If a surface experienced an overwrite risk or synchronization error before the run and remains alive, its next periodic census report incorrectly fails the new run even when neither counter increased. The renderer's shadow_coherent.rs::Coherent::line emits cumulative counters by loading the underlying atomics, so selecting newly appended reports does not make their values window-local. Appending an identical surface report with overwrite_risk: 1 after mark() reproduces this false failure, which can stop an otherwise healthy sweep. Snapshot the counters by surface identity in mark() and judge their increments, rather than testing the new reports' absolute values.
Useful? React with 👍 / 👎.
| for s_, bs in zip(fresh, b["sims"]): | ||
| ctx.sims.append({"idx": s_["idx"], "udid": s_["udid"]}) | ||
| usable.append(bs["usable_s"]) | ||
| boot = {"new": new, "all_usable_s": round(time.time() - t0, 2) if not parallel else groups[0][1]["all_s"], "each_usable_s": usable} |
There was a problem hiding this comment.
Report the complete duration of a stock-device storm
When lifecycle runs with --stock-devices and a storm of more than one simulator, grow(..., parallel=True) takes the serial create-and-boot path, but this expression still selects only the first boot group's duration. Four sequential 10-second boots therefore produce all_usable_s: 10 instead of 40, which run_lifecycle stores as boot_all_s and describes as all simulators booting at once. This makes the stock-device storm results misleading. Aggregate the actual serial duration and label the execution correctly, or implement concurrent booting before using the parallel timing branch.
Useful? React with 👍 / 👎.
…and only steady ones The one-busy mix and the lifecycle ran on a VM for the first time: - on a still screen a picture read the moment its surface changes showed the frame before it in 14-15 of 15 probes; 30 ms later the same surface showed the new one. The probes counted as missed (3, 3 and 1 of 15) or were answered by the flip before. simfps now looks at an unchanged surface again after 30, 150 and 600 ms (a line with counter -3, not a frame), and the result counts probes answered that way (input_second_looks). - the lifecycle's leak gate failed on +215 MB of QEMU RSS per cycle from 3 samples: the host bringing swapped pages back, one step then flat. It judges RSS + swap now, FAILs only a steady growth (4 samples, R2 >= 0.8) and says SKIP for a slope above the bar without that; 5 cycles by default. README: both scenarios' numbers in what has run on a VM.
simfps looked again at an unchanged surface 30 ms after it appeared, also while reading the strip. With 2 or 3 animating simulators a picture stays longer than that, so it would have been a second lock of most pictures (two locks halved the frame rate). It now does so only where it does not read the strip, which is the still screen it was made for and the only place it ran on a VM. README: a bench clone keeps the LaunchServices database a Simlock lease loses, so its first boot is not a lease's; the proxy on a moving screen may carry one picture's interval; the RSS step in the first lifecycle run is likely swap-in, it was not counted.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 32f96d6541
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| f0 = min([s["fps"] for s in res["start"]["sims"] if s.get("fps")] or [0]) | ||
| f1 = min([s["fps"] for s in res["end"]["sims"] if s.get("fps")] or [0]) |
There was a problem hiding this comment.
Include zero-FPS simulators in the soak comparison
When one simulator stops producing distinct app frames during the soak but another keeps animating, the truthiness filter removes the stalled simulator's fps: 0.0 from the minimum. For example, an initial fleet of [60, 60] fps and a final fleet of [60, 0] fps reports 60 fps in both windows and passes. The counter and strip gates do not necessarily catch this: repeated readable counters yield zero distinct FPS while passing both gates. Filter only missing values, not zero, so a completely stalled simulator fails the soak's FPS-retention check.
Useful? React with 👍 / 👎.
| raise Stop(reason) | ||
| ctx.host.n = n | ||
| core.grow(ctx, n, core.roles_for(mix, n)) | ||
| ctx.probe_method = scale.calibrate(ctx, ctx.sims[0])["chosen"] if mix in ("all-busy", "one-busy") and metric == "input_p95" else "none" |
There was a problem hiding this comment.
Enable input probes when latency is a requested A/B tail
When running ab --metric total_fps --tails input_p95, this condition disables input probes because latency is not the primary metric, even though it is an explicitly requested regression guard. Consequently, every arm's input_p95 is None, and abrule.decide treats that missing tail as not worse, allowing an ADOPT decision without measuring the requested latency constraint. Enable and calibrate probes when either the primary metric or a requested tail requires them, and prevent an unavailable required tail from silently permitting adoption.
Useful? React with 👍 / 👎.
| raise Stop(reason) | ||
| ctx.host.n = n | ||
| core.grow(ctx, n, core.roles_for(mix, n)) | ||
| ctx.probe_method = scale.calibrate(ctx, ctx.sims[0])["chosen"] if mix in ("all-busy", "one-busy") and metric == "input_p95" else "none" |
There was a problem hiding this comment.
Enable input probes when latency is a requested tail
When ab --metric total_fps --tails input_p95 is used, this condition disables input probes because latency is not the primary metric. Every arm consequently has input_p95: None, and abrule.decide() treats the unmeasured tail as not worse, allowing ADOPT without checking the explicitly requested latency constraint. Calibrate and enable probes when either the primary metric or a requested tail requires input measurements.
Useful? React with 👍 / 👎.
| med = {k: _median([c[k] for c in cyc if c.get(k) is not None]) for k in keys} | ||
| data["median_s"] = med | ||
| ok = all(v is not None for v in med.values()) and bool(cyc) |
There was a problem hiding this comment.
Fail lifecycle runs when any launch produces no app frame
If one cold or resumed launch never produces a readable app frame but subsequent cycles succeed, cmd_cycle() records that launch's first-frame time as None. Removing those values before checking only the medians hides the failed launch: first-frame results [None, 0.2, 0.2, 0.2] produce LIFECYCLE PASS with a 0.2-second median. Without a crash, the run-wide gates do not detect this failure either. Validate first-frame availability for every cycle before aggregating successful timings, and fail the lifecycle section if any launch never yielded a frame.
Useful? React with 👍 / 👎.
macbox bench: one suite that answers "how many simulators does this host run at once, each one smooth and responsive?" and guards a change against that answer. It builds onsimfps, GlassBench and the frame strip from #18. Nothing outsidesrc/bench/changes except abenchsubcommand inmacboxand one.gitignoreline.What it does
macbox bench scale --n 1,2,3,...: boots one more simulator per step and measures every one of them in the same window: distinct app frames/s and the time between them (p50/p95/p99/max, longest gap), an input-latency proxy, guest CPU and memory per simulator, QEMU CPU by thread group, GPU busy time, host and guest swap. It reports the largest N that still meets each quality target (smooth60,smooth30,responsive; your own with--targets) and Jain's fairness index. Mixes: all animating, one animating beside still ones, an agent-like launch/screenshot/terminate cycle, idle.idle,lifecycle(clone, first boot, launch, first frame, shutdown, resume, delete),soak,ab(A B B A on one live switch, restored byte for byte),compare(two result files, with a noise band) andchart.DRY=1runs the whole control flow against stubs with injectable faults.Tested
python3 -m unittest discover -s tests -t .insrc/bench), including the dry runs with injected faults.tools,check,seed,cleanup,scaleat 3x for N = 1, 2, 3 in theall-busyandone-busymixes, andlifecycle(5 cycles, no boot storm). No gate failed in the final runs; three were SKIP (no golden picture stored yet, QEMU's log and the hang flag not readable from where it ran). All animating:One animating beside still ones: it keeps 60 distinct frames/s at 1, 2 and 3 booted (59.9, 59.8, 60.0), and the still ones answer the input proxy as fast as it does (p95 305-311 ms against 285-304 ms, 0 of 75 probes missed). Lifecycle medians: clone 2.9 s, first boot 5.6 s, launch call 0.5 s, first app frame 1.0 s, shutdown 3.2 s, resume boot 5.5 s, delete 0.2 s; no steady memory growth over the 5 cycles.
The first VM runs did not agree with the hand-made numbers. Each cause was found by bisecting on one simulator and is fixed in the last four commits:
simfpslocked every surface twice (frame strip, then the input marker): 54-58 screen frames/s against 104-110 with one lock (A B B A, 12 s windows). Strip and marker are now read under one lock.simctl openurlas the input probe raises an "Open in GlassBench?" alert on iOS 26 that stays over the app, through a relaunch too: 121 → 80-83 screen frames/s. No longer tried.simctl ui appearanceas the probe redraws the whole screen: 43 distinct frames/s against 59 in the same window without it. The probe is now a Darwin notification GlassBench listens for (simctl spawn <udid> notifyutil -p ...): 60.0 distinct with it, 60.0 without.simfpsnow looks at an unchanged surface again (never where it reads the strip: that would be a second lock of most pictures at 30 a second); the result says how many probes were answered that way.Not tested on a VM (stubs and unit tests only):
idle,soak,ab,compareon real results, theagentmix, the lifecycle's boot storm, the 1x device type, N above 3.src/bench/README.mdsays so too.Known limits
simctl spawn(about 280 ms here with one simulator alone), not a finger's path or the Device panel's stream. Read it as a trend over N.observerevery time.~/.simlock/devices/ios-sources; an install that names its state directory differently gives it with--source-setorBENCH_SIMLOCK_SOURCES(the VM runs did).