From 1fb9dc8fe20de03ecbcc90be2770d0267285418a Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Fri, 25 Sep 2026 17:11:07 +0000 Subject: [PATCH 01/27] docs: observability design for error repro and perf timing Specify local events and timings for the silent failures already hit (touch no-op, HID death, Connecting stall, false MJPEG stall, camera-helper shutdown). Remote export stays opt-in and off by default. Co-authored-by: Evan Bacon --- docs/observability.md | 209 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 209 insertions(+) create mode 100644 docs/observability.md diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 000000000..d150bc2d3 --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,209 @@ +# Observability + +Design only. No SDK, exporter, or runtime change is implied by this document. + +serve-sim is a local Mac process that streams a simulator and injects input. The failures that have cost time are silent: a command exits 0, `--list` says `running: true`, or the preview says Connecting while the capture side is still producing frames. The useful record is the fields that distinguish those cases, plus timings on the paths we have already had to measure by hand. + +## Goals + +1. **Error reproduction.** Each event below is a field that was missing from a real report. With it, a log line is enough to tell injector-sent from guest-unchanged, signal from exit 0, and a dead consumer from a dead producer. +2. **Performance timing.** Spans for the hot paths where a fix already depended on a number: first frame, inject send, camera frame interval, helper startup/shutdown, reconnect. + +Both stay on the machine unless someone opts into export. + +## Non-goals + +- Dashboards, error budgets, release scorecards, or a feedback program. +- Shipping an OpenTelemetry SDK in the first slice. +- Guest-effect checks that keep pixels, OCR, or accessibility text. +- A default backend, a prompt, or a notice on startup. + +## Privacy and opt-out + +Default is **off**. Nothing is written for telemetry and nothing is sent. + +| Control | Effect | +|---|---| +| unset | off | +| `SERVE_SIM_TELEMETRY=0` \| `off` \| `false` | off | +| `--no-telemetry` | off (wins over every other switch) | +| `SERVE_SIM_TELEMETRY=local` or `--telemetry=local` | JSON-lines on the machine only | +| `SERVE_SIM_TELEMETRY=1` \| `on` \| `remote` or `--telemetry=remote` | local events, plus OTLP **only** when `OTEL_EXPORTER_OTLP_ENDPOINT` is set to a URL the operator chose | + +There is no config file today. A future `telemetry` key uses the same enum (`off` \| `local` \| `remote`). Precedence is CLI, then env, then file. `off` wins. + +CI and agent runs stay silent. Opt-in is an explicit env or flag in that job, not something inferred from `CI=`. + +`SERVE_SIM_DEBUG` and `SERVE_SIM_DEBUG_HID` stay local stderr for the person who set them. Those lines are debug logs. They are not attached to exported spans. + +`--list` health (HID state, last successful inject, stream frame age) is process status for the operator sitting at the terminal. It is not an export path. + +Events carry enums, timings, versions, and coarse runtime strings. They leave out: + +- MJPEG, JPEG, screenshots, camera frames, and any other pixels +- App content, accessibility-tree text, clipboard +- Credentials, tokens, cookies, `Authorization` headers +- Raw UDIDs and device names on any export (see `device.id_hash`) +- Home-directory paths (basenames only) +- Raw WebSocket payloads + +`device.id_hash` is 8 hex chars of SHA-256 of the UDID. Local debug may also print an 8-character UDID prefix on its own line. Remote export gets the hash only. `session.id` is a new UUID per server process. + +## Common attributes + +Present on every event and span: + +| Field | Type | Example / notes | +|---|---|---| +| `serve_sim.version` | string | `0.1.47` | +| `serve_sim.arch` | enum | `arm64` \| `x64` | +| `os.version` | string | macOS `major.minor` only | +| `xcode.version` | string | `27.0` (marketing version, not the build serial) | +| `ios.runtime` | string | `iOS 27.0` | +| `session.id` | string | ephemeral UUID | +| `device.id_hash` | string | 8 hex chars | +| `sim.ui_host` | enum | `none` \| `simulator_app` \| `device_hub` \| `unknown` | + +`sim.ui_host=none` is the headless `simctl boot` case with no Simulator.app or Device Hub. + +## Error reproduction + +| Issue | What the report could not show | Event | +|---|---|---| +| [#153](https://github.com/EvanBacon/serve-sim/issues/153) | Touch exits 0 and HID debug prints, screenshots identical, lock works. Missing injector health vs guest effect, runtime, UI host, client connected, send ack, coordinate space, version. | `input.inject` | +| [#136](https://github.com/EvanBacon/serve-sim/issues/136) | HID died; button, tap, and gesture still exit 0; `--list` stays `running: true`. | `hid.health` + `input.inject` | +| [#143](https://github.com/EvanBacon/serve-sim/issues/143) | Helper exit was null (signal) under `--max-concurrency=1`. Missing signal vs code, last log, shutdown phase, placeholder-timer vs socket-close timing. | `camera.helper.exit` | +| [#103](https://github.com/EvanBacon/serve-sim/issues/103) | Preview live for 1–2s, then Connecting forever on 0.1.40+. Missing state transitions, last frame time, WS close reason, first-frame vs later stall, whether it recovered. | `stream.state` | +| [#128](https://github.com/EvanBacon/serve-sim/issues/128) | MJPEG `` watchdog showed a frame-stall after rotation while `/stream.mjpeg` was still sending. | `stream.stall` | +| [#102](https://github.com/EvanBacon/serve-sim/issues/102) | Helper stayed up for days after the sim was Shutdown. Closed by [#152](https://github.com/EvanBacon/serve-sim/pull/152); the booted-vs-alive pair is the canary. | `helper.lifecycle` | + +### `input.inject` + +One event per touch phase, button, or gesture step. This is the record [#153](https://github.com/EvanBacon/serve-sim/issues/153) and [#136](https://github.com/EvanBacon/serve-sim/issues/136) needed: `button lock` goes through `IndigoHIDMessageForButton` (target hardware, `0x33`) and can succeed while digitizer touch (`0x32`, `IndigoHIDMessageForMouseNSEvent`) is built and sent into a session with no UI host. + +| Field | Type | Notes | +|---|---|---| +| `hid.client` | enum | `created` \| `missing` \| `send_unloaded` | +| `hid.state` | enum | `ok` \| `stale` \| `gone` (same enum as `hid.health`) | +| `inject.kind` | enum | `touch` \| `button` \| `gesture` \| `key` \| `rotate` | +| `inject.phase` | enum | `begin` \| `move` \| `end` \| `press` | +| `inject.result` | enum | `sent` \| `nil_message` \| `no_client` \| `throw` | +| `inject.ack` | enum | `none` until a reply exists. Later `ok` \| `timeout` | +| `inject.ack_ms` | int, ms | omitted while `inject.ack=none` | +| `inject.guest_effect` | enum | `not_checked` by default. Opt-in local `--verify` may set `changed` \| `unchanged` and then discard the frames | +| `input.coord_space` | enum | `normalized_0_1` | +| `input.screen_w` | int, px | width passed into the injector | +| `input.screen_h` | int, px | height passed into the injector | +| `input.target` | enum | `digitizer` \| `hardware_button` | +| `input.x`, `input.y` | number | local events only. Remote export omits them | +| `error.code` | string | stable token on `throw` (`hid_client_missing`, `mouse_symbol_missing`, …) | +| `cli.exit_code` | int | `0` only when `inject.result=sent` and `hid.state=ok`. `no_client`, `nil_message`, `gone`, and `throw` exit non-zero | + +Today's `tap` is fire-and-forget: the socket open succeeds, `begin`/`end` are written, and the process exits 0 with no reply handler. The event records that as `inject.result=sent`, `inject.ack=none`, `inject.guest_effect=not_checked`. A sent digitizer event and a delivered one stop looking the same once `hid.client`, `sim.ui_host`, and `input.target` are on the line. + +### `hid.health` + +Emitted on transition and included in `--list`. [#136](https://github.com/EvanBacon/serve-sim/issues/136) stayed invisible because `running` is "pidfile exists", which stays true after the HID client is dead and after a `simctl shutdown` / `boot`. + +| Field | Type | Notes | +|---|---|---| +| `hid.state` | enum | `ok` last send landed. `stale` no successful send for `hid.stale_after_ms` while commands are still arriving. `gone` client missing or sim not booted | +| `hid.last_ok_at_ms` | int | unix ms of the last `inject.result=sent`. `0` if none | +| `hid.stale_after_ms` | int, ms | threshold in force (start at 30000) | +| `hid.helper_age_ms` | int, ms | age of this helper process, not of the simulator | +| `sim.booted` | bool | `simctl` still lists the UDID as Booted | +| `list.running` | bool | today's flag, kept so `running:true` + `hid.state:gone` is one object | + +### `camera.helper.exit` + +[#143](https://github.com/EvanBacon/serve-sim/issues/143) failed in the test's `exitCode !== 0` branch: Node reports `null` when the child dies on a signal, and the shm name was already unlinked. The race was the placeholder timer's in-flight `PublishFrame` versus surface release (`SIGSEGV` / use-after-free, or `SIGPIPE` if a client vanished mid-write). + +[#161](https://github.com/EvanBacon/serve-sim/pull/161) is on main (`516cdbf`). `StopPlaceholderSource` now cancels the timer and waits for its cancel handler before `ReleaseSurfaces`, and the helper returns 0. Issue #143 is still open; treat that as the bug being mitigated, and keep these fields as the regression canary. A future red CI run should show signal number and phase instead of `exited with null`. + +| Field | Type | Notes | +|---|---|---| +| `camera.helper_exit_code` | int or null | `null` when the process is signaled | +| `camera.helper_signal` | int or null | signal number (`11` SIGSEGV, `13` SIGPIPE, `6` SIGABRT). `null` on a normal exit | +| `camera.shutdown_phase` | enum | `running` \| `unlink_shm` \| `stop_placeholder` \| `stop_video` \| `stop_webcam` \| `close_socket` \| `release_surfaces` \| `exited` | +| `camera.placeholder_joined` | bool | timer cancel handler finished before unmap | +| `camera.placeholder_join_ms` | int, ms | wait on that semaphore | +| `camera.socket_close_ms` | int, ms | shutdown request → accept source cancelled | +| `camera.source` | enum | `placeholder` \| `webcam` \| `image` \| `video` \| `none` | +| `camera.last_log` | string | last ~600 chars of helper stderr. Basenames only, no frame bytes | +| `error.code` | string | `helper_signaled` \| `helper_nonzero` \| `helper_ok` | + +### `stream.state` + +[#103](https://github.com/EvanBacon/serve-sim/issues/103) went live, then stuck on Connecting from 0.1.40 onward. A later note on 0.1.44: after the preview had already rendered, a browser/HID interaction flipped it back to Connecting and it never recovered, while `/stream.mjpeg` was still returning 200 and transferring megabytes. That is a consumer state-machine bug, not a dead simulator. The transition log is the repro. + +| Field | Type | Notes | +|---|---|---| +| `stream.state` | enum | `starting` \| `connecting` \| `live` \| `stalled` \| `reconnecting` \| `stopped` | +| `stream.prev_state` | enum | previous value | +| `stream.saw_first_frame` | bool | false = never left Connecting; true = stall after a good frame | +| `stream.last_frame_age_ms` | int, ms | age on **this consumer**. `-1` if none yet | +| `stream.consumer` | enum | `mjpeg_img` \| `mjpeg_fetch` \| `avcc` \| `relay` | +| `stream.producer_ok` | bool | capture side has a frame newer than the stall threshold (counter only) | +| `stream.ws_close_code` | int | RFC 6455 code, `0` if the socket is open | +| `stream.ws_close_reason` | string | short token (`normal`, `abnormal`, `error`). Not the payload | +| `stream.recovered` | bool | reached `live` again after `stalled` or a second `connecting` | + +### `stream.stall` + +[#128](https://github.com/EvanBacon/serve-sim/issues/128) fired the overlay "Stream is not producing frames" after rotation. The native `` and a parallel `fetch()` watchdog are separate consumers; either can time out while the other, and the helper, are healthy. The startup watchdog in `SimulatorView` is 6000 ms and keys off MJPEG `--frame` boundaries. Relay mode treats 2000 ms without a frame as stale. The event records which detector fired and whether the producer was still moving. + +| Field | Type | Notes | +|---|---|---| +| `stream.stall_reason` | enum | `startup_watchdog` \| `relay_stale` \| `consumer_stalled_producer_live` \| `producer_silent` | +| `stream.stall_threshold_ms` | int, ms | timeout that fired | +| `stream.last_frame_age_ms` | int, ms | consumer frame age | +| `stream.fps` | number | frames in the last 1s on that consumer | +| `stream.producer_fps` | number | frames published in the same 1s | +| `stream.false_positive` | bool | `producer_fps > 0` while the consumer is past threshold | +| `stream.orientation` | enum | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` | +| `stream.restart_ms` | int, ms | time until the next `live`, omitted if it never returns | + +### `helper.lifecycle` + +[#102](https://github.com/EvanBacon/serve-sim/issues/102) was an orphaned `serve-sim-bin` at ~100% CPU with the simulator already Shutdown (parent `launchd`, uptime measured in days). [#152](https://github.com/EvanBacon/serve-sim/pull/152) exits detached helpers after simulator shutdown. One event on the way out, and one if the process is still alive while `sim.booted=false`, is enough to catch a regression. + +| Field | Type | Notes | +|---|---|---| +| `sim.booted` | bool | | +| `helper.uptime_ms` | int, ms | | +| `helper.parent` | enum | `launchd` \| `serve-sim` \| `other` | +| `helper.action` | enum | `exit_sim_shutdown` \| `still_alive_sim_down` | + +## Performance timing + +Durations are milliseconds. Histograms are for a later exporter; local events just store the number. Suggested buckets are upper bounds in ms. + +| Span | Start → end | Fields | Buckets | +|---|---|---|---| +| `stream.startup` (`stream.ttff_ms`) | stream request accepted → first JPEG or AVCC frame delivered to that subscriber | `boot_ms` (sim already booted, else time until Booted), `ws_ms` (socket open), `indigo_ms` (first framebuffer/Indigo snapshot), `first_jpeg_ms` (encode + first byte to the subscriber). Sum is `stream.ttff_ms` | 50, 100, 250, 500, 1000, 2000, 5000 | +| `input.send` (`input.send_ms`) | handler entered → `rawSend` returned | `inject.kind`, `inject.result` | 1, 5, 10, 25, 50, 100, 250 | +| `input.roundtrip` (`input.roundtrip_ms`) | `rawSend` → ack | emitted only once `inject.ack` is `ok` or `timeout` | 5, 10, 25, 50, 100, 250, 500, 1000 | +| `camera.frame` (`camera.frame_interval_ms`) | `PublishFrame` n → n+1 | `camera.source` | 16, 33, 50, 100, 250, 1000 | +| `camera.shm_map` (`camera.shm_map_ms`) | `shm_open` → mapped header readable | `camera.shm_map_result` enum `ok` \| `enoent` \| `error` | 1, 5, 10, 50, 100, 500 | +| `camera.helper_startup` (`camera.helper_startup_ms`) | process start → first frame seq published | `camera.source` | 10, 50, 100, 250, 500, 1000, 5000 | +| `camera.helper_shutdown` (`camera.helper_shutdown_ms`) | shutdown action or signal → process exit | `camera.placeholder_join_ms`, `camera.socket_close_ms`, `camera.helper_exit_code`, `camera.helper_signal` | 10, 50, 100, 500, 1000, 5000 | +| `stream.reconnect` (`stream.reconnect_ms`) | leave `live` → return to `live` | `stream.reconnect_attempt` (int, starts at 1), `stream.stall_reason` | 100, 250, 500, 1000, 2000, 5000 | + +`stream.ttff_ms` is the number the idle-floor test already budgets by hand (first JPEG after `stream.mjpeg` is opened). `camera.helper_shutdown_ms` split into placeholder-join and socket-close is the number [#143](https://github.com/EvanBacon/serve-sim/issues/143) was missing when the helper died between "running" and exit 0. `input.send_ms` without `input.roundtrip_ms` is an honest measurement of today's fire-and-forget path; round-trip appears only when an ack exists. + +Crashes in our own process add `error.code` plus a stack trimmed to serve-sim frames, with home directories stripped. No media, no request bodies. + +## Phased rollout + +1. **Local events.** Structured JSON-lines when telemetry is `local`. Same schema as the tables above. `--list` grows `hid.state`, `hid.last_ok_at_ms`, and `stream.last_frame_age_ms` so an agent can see a dead injector without enabling telemetry. No new dependency. +2. **Optional OTLP.** When telemetry is `remote` and `OTEL_EXPORTER_OTLP_ENDPOINT` is set, export the same spans and events. Map `serve_sim.version` → `service.version`, `os.version` → `os.version`, `os.type=darwin`. Histograms use the buckets above. No endpoint is built in. Unset endpoint means stay on phase 1 even if the flag says remote. +3. **Default remains off.** Absence of the env var is off, in CI and on a laptop. + +## Opt-out patterns elsewhere + +Short notes on switches worth copying, and one worth avoiding. Not a survey of their event catalogs. + +- **Maestro** posts anonymous analytics to PostHog and used to prompt on stdin, which hung CI. Opt-out is `MAESTRO_CLI_NO_ANALYTICS=true` ([env docs](https://docs.maestro.dev/maestro-cli/environment-variables), [prompt bug](https://github.com/mobile-dev-inc/Maestro/issues/1846)). serve-sim does not prompt. +- **Homebrew** analytics are on after an install notice, with `HOMEBREW_NO_ANALYTICS=1` or `brew analytics off` ([docs](https://docs.brew.sh/Analytics)). serve-sim stays off until asked. +- **Appium** does not document a product analytics backend. `appium:eventTimings` is in-session timing, default false ([caps](https://appium.io/docs/en/latest/guides/caps/)). XCUITest, facebook/idb, and WebDriverAgent are local processes with no product export. +- **OpenTelemetry** sends nothing until an exporter endpoint is configured. The Collector is something you run ([JS exporters](https://opentelemetry.io/docs/languages/js/exporters/)). Phase 2 follows that: opt-in flag plus an operator-supplied endpoint. From 9abd5de1f1f2ea3d1fb623d6a7840a18a5058ceb Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Thu, 1 Oct 2026 11:46:50 -0500 Subject: [PATCH 02/27] docs: record #153 landscape coordinate-miss fields The 2026-09-30 repro shows headless portrait taps land and landscape_left misses because display-normalized points are sent into the native portrait frame. Add orientation, frame, and remap fields so that case is not logged as a dead injector. --- docs/observability.md | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index d150bc2d3..bf12f2811 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -2,12 +2,12 @@ Design only. No SDK, exporter, or runtime change is implied by this document. -serve-sim is a local Mac process that streams a simulator and injects input. The failures that have cost time are silent: a command exits 0, `--list` says `running: true`, or the preview says Connecting while the capture side is still producing frames. The useful record is the fields that distinguish those cases, plus timings on the paths we have already had to measure by hand. +serve-sim is a local Mac process that streams a simulator and injects input. The failures that have cost time are silent: a command exits 0, `--list` says `running: true`, or the preview says Connecting while the capture side is still producing frames. A later class is just as silent: the touch is delivered, and the screenshot is unchanged because the point was in the wrong frame. The useful record is the fields that distinguish those cases, plus timings on the paths we have already had to measure by hand. ## Goals -1. **Error reproduction.** Each event below is a field that was missing from a real report. With it, a log line is enough to tell injector-sent from guest-unchanged, signal from exit 0, and a dead consumer from a dead producer. -2. **Performance timing.** Spans for the hot paths where a fix already depended on a number: first frame, inject send, camera frame interval, helper startup/shutdown, reconnect. +1. **Error reproduction.** Each event below is a field that was missing from a real report. With it, a log line is enough to tell injector-sent from guest-unchanged, a coordinate miss from a dead injector, signal from exit 0, and a dead consumer from a dead producer. +2. **Performance timing.** Spans for the hot paths where a fix already depended on a number: first frame, inject send, camera frame interval, helper startup/shutdown, reconnect. Spans carry the same error-repro attributes so a fast send is not read as a successful tap. Both stay on the machine unless someone opts into export. @@ -70,7 +70,7 @@ Present on every event and span: | Issue | What the report could not show | Event | |---|---|---| -| [#153](https://github.com/EvanBacon/serve-sim/issues/153) | Touch exits 0 and HID debug prints, screenshots identical, lock works. Missing injector health vs guest effect, runtime, UI host, client connected, send ack, coordinate space, version. | `input.inject` | +| [#153](https://github.com/EvanBacon/serve-sim/issues/153) | Touch exits 0, screenshots identical. 2026-09-30 repro: headless portrait taps land on iPad and iPhone; `landscape_left` misses because CLI/server pass display-normalized points into the native portrait frame. Web client already remaps. Missing orientation, frame, and whether a remap ran, so a coordinate miss looked like a dead injector. | `input.inject` | | [#136](https://github.com/EvanBacon/serve-sim/issues/136) | HID died; button, tap, and gesture still exit 0; `--list` stays `running: true`. | `hid.health` + `input.inject` | | [#143](https://github.com/EvanBacon/serve-sim/issues/143) | Helper exit was null (signal) under `--max-concurrency=1`. Missing signal vs code, last log, shutdown phase, placeholder-timer vs socket-close timing. | `camera.helper.exit` | | [#103](https://github.com/EvanBacon/serve-sim/issues/103) | Preview live for 1–2s, then Connecting forever on 0.1.40+. Missing state transitions, last frame time, WS close reason, first-frame vs later stall, whether it recovered. | `stream.state` | @@ -79,7 +79,7 @@ Present on every event and span: ### `input.inject` -One event per touch phase, button, or gesture step. This is the record [#153](https://github.com/EvanBacon/serve-sim/issues/153) and [#136](https://github.com/EvanBacon/serve-sim/issues/136) needed: `button lock` goes through `IndigoHIDMessageForButton` (target hardware, `0x33`) and can succeed while digitizer touch (`0x32`, `IndigoHIDMessageForMouseNSEvent`) is built and sent into a session with no UI host. +One event per touch phase, button, or gesture step. This is the record [#153](https://github.com/EvanBacon/serve-sim/issues/153) and [#136](https://github.com/EvanBacon/serve-sim/issues/136) needed: `button lock` goes through `IndigoHIDMessageForButton` (target hardware, `0x33`) and can succeed while digitizer touch (`0x32`, `IndigoHIDMessageForMouseNSEvent`) is built and sent into a session with no UI host — or into the wrong coordinate frame. | Field | Type | Notes | |---|---|---| @@ -92,6 +92,9 @@ One event per touch phase, button, or gesture step. This is the record [#153](ht | `inject.ack_ms` | int, ms | omitted while `inject.ack=none` | | `inject.guest_effect` | enum | `not_checked` by default. Opt-in local `--verify` may set `changed` \| `unchanged` and then discard the frames | | `input.coord_space` | enum | `normalized_0_1` | +| `input.orientation` | enum | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown`. Device orientation at send, not the caller's assumption | +| `input.frame` | enum | `display` (rotated screenshot / what `serve-sim tap` is given) \| `native_portrait` (what `SimHID.touch` consumes) | +| `input.remapped` | bool | true if this process applied `rawPointForDisplayPoint` before send | | `input.screen_w` | int, px | width passed into the injector | | `input.screen_h` | int, px | height passed into the injector | | `input.target` | enum | `digitizer` \| `hardware_button` | @@ -101,6 +104,8 @@ One event per touch phase, button, or gesture step. This is the record [#153](ht Today's `tap` is fire-and-forget: the socket open succeeds, `begin`/`end` are written, and the process exits 0 with no reply handler. The event records that as `inject.result=sent`, `inject.ack=none`, `inject.guest_effect=not_checked`. A sent digitizer event and a delivered one stop looking the same once `hid.client`, `sim.ui_host`, and `input.target` are on the line. +A 2026-09-30 repro on [#153](https://github.com/EvanBacon/serve-sim/issues/153) (macOS 27 / Xcode 27.0 (27A266a) / iOS 27, headless `simctl boot`, no Simulator.app or Device Hub, `dtuhidd.active=0`) showed portrait taps opening Settings on both iPad Pro 13-inch (M5) and iPhone 18 Pro. The same `landscape_left` point (`0.715, 0.343` in the rotated frame) left the screenshot unchanged until it was rewritten with the web client's map `{ x: y, y: 1 - x }`, which is `rawPointForDisplayPoint` in `orientation.ts`. `w`/`h` do not move the target except on Duo. The distinguishing record is `sim.ui_host=none`, `hid.state=ok`, `inject.result=sent`, `input.orientation=landscape_left`, `input.frame=display`, `input.remapped=false`, `inject.guest_effect=unchanged`. That is a coordinate miss, not a dead HID client (`hid.client=missing`) and not the [#136](https://github.com/EvanBacon/serve-sim/issues/136) `hid.state=gone` case. `input.x` / `input.y` stay local-only; remote export keeps the enums. A fix that remaps CLI `tap` and other direct `0x03` writers should set `input.remapped=true` and `input.frame=native_portrait` after the map. + ### `hid.health` Emitted on transition and included in `--list`. [#136](https://github.com/EvanBacon/serve-sim/issues/136) stayed invisible because `running` is "pidfile exists", which stays true after the HID client is dead and after a `simctl shutdown` / `boot`. @@ -150,7 +155,7 @@ Emitted on transition and included in `--list`. [#136](https://github.com/EvanBa ### `stream.stall` -[#128](https://github.com/EvanBacon/serve-sim/issues/128) fired the overlay "Stream is not producing frames" after rotation. The native `` and a parallel `fetch()` watchdog are separate consumers; either can time out while the other, and the helper, are healthy. The startup watchdog in `SimulatorView` is 6000 ms and keys off MJPEG `--frame` boundaries. Relay mode treats 2000 ms without a frame as stale. The event records which detector fired and whether the producer was still moving. +[#128](https://github.com/EvanBacon/serve-sim/issues/128) fired the overlay "Stream is not producing frames" after rotation. The native `` and a parallel `fetch()` watchdog are separate consumers; either can time out while the other, and the helper, are healthy. The startup watchdog in `SimulatorView` is 6000 ms and keys off MJPEG `--frame` boundaries. Relay mode treats 2000 ms without a frame as stale. The event records which detector fired and whether the producer was still moving. Join `stream.orientation` with `input.orientation` when a rotation and a tap land in the same session: a false stall (#128) and a coordinate miss (#153) can share a rotate and must not be one event. | Field | Type | Notes | |---|---|---| @@ -176,20 +181,20 @@ Emitted on transition and included in `--list`. [#136](https://github.com/EvanBa ## Performance timing -Durations are milliseconds. Histograms are for a later exporter; local events just store the number. Suggested buckets are upper bounds in ms. +Durations are milliseconds. Histograms are for a later exporter; local events just store the number. Suggested buckets are upper bounds in ms. Every span below also carries the common attributes, and inject spans carry `input.orientation`, `input.frame`, and `input.remapped` so a landscape miss is not bucketed with a portrait hit. | Span | Start → end | Fields | Buckets | |---|---|---|---| | `stream.startup` (`stream.ttff_ms`) | stream request accepted → first JPEG or AVCC frame delivered to that subscriber | `boot_ms` (sim already booted, else time until Booted), `ws_ms` (socket open), `indigo_ms` (first framebuffer/Indigo snapshot), `first_jpeg_ms` (encode + first byte to the subscriber). Sum is `stream.ttff_ms` | 50, 100, 250, 500, 1000, 2000, 5000 | -| `input.send` (`input.send_ms`) | handler entered → `rawSend` returned | `inject.kind`, `inject.result` | 1, 5, 10, 25, 50, 100, 250 | -| `input.roundtrip` (`input.roundtrip_ms`) | `rawSend` → ack | emitted only once `inject.ack` is `ok` or `timeout` | 5, 10, 25, 50, 100, 250, 500, 1000 | +| `input.send` (`input.send_ms`) | handler entered → `rawSend` returned | `inject.kind`, `inject.result`, `input.orientation`, `input.frame`, `input.remapped` | 1, 5, 10, 25, 50, 100, 250 | +| `input.roundtrip` (`input.roundtrip_ms`) | `rawSend` → ack | emitted only once `inject.ack` is `ok` or `timeout`. Same orientation fields as `input.send` | 5, 10, 25, 50, 100, 250, 500, 1000 | | `camera.frame` (`camera.frame_interval_ms`) | `PublishFrame` n → n+1 | `camera.source` | 16, 33, 50, 100, 250, 1000 | | `camera.shm_map` (`camera.shm_map_ms`) | `shm_open` → mapped header readable | `camera.shm_map_result` enum `ok` \| `enoent` \| `error` | 1, 5, 10, 50, 100, 500 | | `camera.helper_startup` (`camera.helper_startup_ms`) | process start → first frame seq published | `camera.source` | 10, 50, 100, 250, 500, 1000, 5000 | | `camera.helper_shutdown` (`camera.helper_shutdown_ms`) | shutdown action or signal → process exit | `camera.placeholder_join_ms`, `camera.socket_close_ms`, `camera.helper_exit_code`, `camera.helper_signal` | 10, 50, 100, 500, 1000, 5000 | -| `stream.reconnect` (`stream.reconnect_ms`) | leave `live` → return to `live` | `stream.reconnect_attempt` (int, starts at 1), `stream.stall_reason` | 100, 250, 500, 1000, 2000, 5000 | +| `stream.reconnect` (`stream.reconnect_ms`) | leave `live` → return to `live` | `stream.reconnect_attempt` (int, starts at 1), `stream.stall_reason`, `stream.orientation` | 100, 250, 500, 1000, 2000, 5000 | -`stream.ttff_ms` is the number the idle-floor test already budgets by hand (first JPEG after `stream.mjpeg` is opened). `camera.helper_shutdown_ms` split into placeholder-join and socket-close is the number [#143](https://github.com/EvanBacon/serve-sim/issues/143) was missing when the helper died between "running" and exit 0. `input.send_ms` without `input.roundtrip_ms` is an honest measurement of today's fire-and-forget path; round-trip appears only when an ack exists. +`stream.ttff_ms` is the number the idle-floor test already budgets by hand (first JPEG after `stream.mjpeg` is opened). `camera.helper_shutdown_ms` split into placeholder-join and socket-close is the number [#143](https://github.com/EvanBacon/serve-sim/issues/143) was missing when the helper died between "running" and exit 0. `input.send_ms` without `input.roundtrip_ms` is an honest measurement of today's fire-and-forget path; round-trip appears only when an ack exists. `input.send_ms` is not a proxy for guest effect: the landscape miss and the portrait hit in the #153 table have the same send duration. The orientation fields are what separate them. No remap span: `rawPointForDisplayPoint` is a handful of arithmetic and must not show up as its own histogram. Crashes in our own process add `error.code` plus a stack trimmed to serve-sim frames, with home directories stripped. No media, no request bodies. From 9eeea161bb2d2fd1666ab2098d422141d946ed8a Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Fri, 2 Oct 2026 16:12:29 +0000 Subject: [PATCH 03/27] docs: worked error-repro and timing records for observability --- docs/observability.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/observability.md b/docs/observability.md index bf12f2811..830a399b1 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -198,6 +198,40 @@ Durations are milliseconds. Histograms are for a later exporter; local events ju Crashes in our own process add `error.code` plus a stack trimmed to serve-sim frames, with home directories stripped. No media, no request bodies. +## Worked records + +Same session, two taps, one span. These are the lines a later implementation should emit when telemetry is `local`. Remote export drops `input.x` / `input.y` and replaces `device.id_hash` only (already hashed). + +### Error reproduction — #153 landscape miss vs portrait hit + +Headless iPad, `sim.ui_host=none`, HID client up. Portrait tap opens Settings. The same display point in `landscape_left` is sent unmapped and the screenshot does not change. + +```json +{"event":"input.inject","serve_sim.version":"0.1.47","sim.ui_host":"none","hid.client":"created","hid.state":"ok","inject.kind":"touch","inject.phase":"end","inject.result":"sent","inject.ack":"none","inject.guest_effect":"changed","input.coord_space":"normalized_0_1","input.orientation":"portrait","input.frame":"display","input.remapped":false,"input.target":"digitizer","input.screen_w":1376,"input.screen_h":1032,"input.x":0.715,"input.y":0.343,"cli.exit_code":0} +{"event":"input.inject","serve_sim.version":"0.1.47","sim.ui_host":"none","hid.client":"created","hid.state":"ok","inject.kind":"touch","inject.phase":"end","inject.result":"sent","inject.ack":"none","inject.guest_effect":"unchanged","input.coord_space":"normalized_0_1","input.orientation":"landscape_left","input.frame":"display","input.remapped":false,"input.target":"digitizer","input.screen_w":1376,"input.screen_h":1032,"input.x":0.715,"input.y":0.343,"cli.exit_code":0} +``` + +Signature of the miss: `hid.state=ok` + `inject.result=sent` + `input.frame=display` + `input.remapped=false` + `inject.guest_effect=unchanged`. That is not #136 (`hid.state=gone`, `cli.exit_code` non-zero) and not a missing symbol (`hid.client=missing`). After a CLI remap, the second line should read `input.frame=native_portrait`, `input.remapped=true`, and the local coordinates should be the mapped point (`x=0.343`, `y=0.285` for `{ x: y, y: 1 - x }` on `landscape_left`). + +### Error reproduction — #128 false stall after rotate + +Producer still publishing, `` watchdog fired. Must not share an event name with the coordinate miss above even if both follow `serve-sim rotate`. + +```json +{"event":"stream.stall","stream.stall_reason":"consumer_stalled_producer_live","stream.stall_threshold_ms":6000,"stream.last_frame_age_ms":6100,"stream.fps":0,"stream.producer_fps":58,"stream.false_positive":true,"stream.orientation":"landscape_left","stream.consumer":"mjpeg_img"} +``` + +### Performance timing — inject span carries the repro fields + +`input.send_ms` is the same on the hit and the miss. The orientation fields are the bucket key. No `input.roundtrip` line: `inject.ack` is still `none`. + +```json +{"span":"input.send","input.send_ms":4,"inject.kind":"touch","inject.result":"sent","input.orientation":"landscape_left","input.frame":"display","input.remapped":false} +{"span":"stream.reconnect","stream.reconnect_ms":420,"stream.reconnect_attempt":1,"stream.stall_reason":"consumer_stalled_producer_live","stream.orientation":"landscape_left"} +``` + +`stream.reconnect` is omitted when the consumer never returns to `live` (#103). `camera.helper_shutdown` is the #143 canary: `camera.helper_signal=11` with `camera.shutdown_phase=release_surfaces` and `camera.placeholder_joined=false` is the race #161 closed; a regression should show that triple, not `exited with null`. + ## Phased rollout 1. **Local events.** Structured JSON-lines when telemetry is `local`. Same schema as the tables above. `--list` grows `hid.state`, `hid.last_ok_at_ms`, and `stream.last_frame_age_ms` so an agent can see a dead injector without enabling telemetry. No new dependency. From 13bdc5e461c9232e5bfc7f2ede6b7d5fcf85173e Mon Sep 17 00:00:00 2001 From: EvanBacon <9664363+EvanBacon@users.noreply.github.com> Date: Sun, 4 Oct 2026 16:14:01 +0000 Subject: [PATCH 04/27] docs: join error-repro events to timing spans via trace.id A landscape miss and its 4ms input.send were two unrelated lines. Same trace.id on the event and the span. span.status stays ok on a coordinate miss so it is not bucketed with hid.state=gone. --- docs/observability.md | 43 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/docs/observability.md b/docs/observability.md index 830a399b1..4bac84395 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -9,7 +9,7 @@ serve-sim is a local Mac process that streams a simulator and injects input. The 1. **Error reproduction.** Each event below is a field that was missing from a real report. With it, a log line is enough to tell injector-sent from guest-unchanged, a coordinate miss from a dead injector, signal from exit 0, and a dead consumer from a dead producer. 2. **Performance timing.** Spans for the hot paths where a fix already depended on a number: first frame, inject send, camera frame interval, helper startup/shutdown, reconnect. Spans carry the same error-repro attributes so a fast send is not read as a successful tap. -Both stay on the machine unless someone opts into export. +Both stay on the machine unless someone opts into export. An attempt's event and its span share `trace.id` so a 4 ms send is not read apart from the coordinate miss it belongs to. ## Non-goals @@ -232,6 +232,47 @@ Producer still publishing, `` watchdog fired. Must not share an event name `stream.reconnect` is omitted when the consumer never returns to `live` (#103). `camera.helper_shutdown` is the #143 canary: `camera.helper_signal=11` with `camera.shutdown_phase=release_surfaces` and `camera.placeholder_joined=false` is the race #161 closed; a regression should show that triple, not `exited with null`. +## Joining an error record to its span + +An `input.inject` line and an `input.send` span are the same attempt. Without a shared id, a landscape miss (`inject.guest_effect=unchanged`) and a 4 ms send look like two unrelated facts. Every event and span in one attempt carries the same `trace.id`. The span also has `span.id`. Child spans set `parent.span_id`. + +| Field | Type | Notes | +|---|---|---| +| `trace.id` | string | 16 hex chars, new per attempt (one tap, one helper shutdown, one consumer stall). Not the session id | +| `span.id` | string | 8 hex chars. Spans only | +| `parent.span_id` | string | set on a child span. Omitted on a root | +| `span.status` | enum | `ok` \| `error` \| `unset`. Derived, not a second opinion | + +`span.status` is not `inject.result`. Map: + +| Condition | `span.status` | +|---|---| +| `inject.result=sent` and `hid.state=ok` | `ok`, even when `inject.guest_effect=unchanged` (the send succeeded; the miss is an attribute) | +| `inject.result` in `nil_message` \| `no_client` \| `throw`, or `hid.state=gone` | `error`, and set `error.code` | +| `inject.ack=timeout` on `input.roundtrip` | `error` | +| `stream.false_positive=true` on `stream.stall` | `unset` (detector fired, producer was live; not a failed send) | +| `camera.helper_signal` non-null | `error` | + +A coordinate miss must stay `span.status=ok` with `input.remapped=false`. Marking it `error` would bucket it with #136, which is the mistake the fields exist to prevent. + +`stream.startup` is the parent of the first `camera.frame` after subscribe. `camera.helper_shutdown` is the parent of the placeholder-join and socket-close waits; those stay fields on the parent (`camera.placeholder_join_ms`, `camera.socket_close_ms`), not their own spans. `input.roundtrip` parents to `input.send` and is omitted while `inject.ack=none`. + +Worked join for the #153 landscape miss. Same `trace.id` on the event and the span. `input.send_ms` stays 4. Status stays `ok`. + +```json +{"event":"input.inject","trace.id":"a1b2c3d4e5f60718","serve_sim.version":"0.1.47","sim.ui_host":"none","hid.state":"ok","inject.kind":"touch","inject.phase":"end","inject.result":"sent","inject.ack":"none","inject.guest_effect":"unchanged","input.orientation":"landscape_left","input.frame":"display","input.remapped":false,"cli.exit_code":0} +{"span":"input.send","trace.id":"a1b2c3d4e5f60718","span.id":"9f8e7d6c","span.status":"ok","input.send_ms":4,"inject.kind":"touch","inject.result":"sent","input.orientation":"landscape_left","input.frame":"display","input.remapped":false} +``` + +The #128 false stall is a different trace. It does not parent to the tap, even when both follow a rotate. + +```json +{"event":"stream.stall","trace.id":"0011223344556677","span.status":"unset","stream.stall_reason":"consumer_stalled_producer_live","stream.false_positive":true,"stream.orientation":"landscape_left","stream.producer_fps":58} +{"span":"stream.reconnect","trace.id":"0011223344556677","span.id":"aabbccdd","parent.span_id":"","span.status":"ok","stream.reconnect_ms":420,"stream.reconnect_attempt":1,"stream.stall_reason":"consumer_stalled_producer_live","stream.orientation":"landscape_left"} +``` + +`parent.span_id` is omitted on that reconnect (it is the root of its trace). Histogram buckets are unchanged. `trace.id` is not a bucket key. + ## Phased rollout 1. **Local events.** Structured JSON-lines when telemetry is `local`. Same schema as the tables above. `--list` grows `hid.state`, `hid.last_ok_at_ms`, and `stream.last_frame_age_ms` so an agent can see a dead injector without enabling telemetry. No new dependency. From 6f152ab4f3df02e99d05080ec177886b0c762b61 Mon Sep 17 00:00:00 2001 From: serve-sim maintainer Date: Mon, 5 Oct 2026 16:15:19 +0000 Subject: [PATCH 05/27] docs: join error-repro fields onto timing spans stream.startup carries #103 first-frame vs stuck-Connecting so a 180ms TTFF is not read as the regression. camera.helper_shutdown carries #143 signal and placeholder-join so a fast signaled exit is not a clean one. input.send restates the #153 landscape miss on the span line. --- docs/observability.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/observability.md b/docs/observability.md index 4bac84395..933983708 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -273,6 +273,43 @@ The #128 false stall is a different trace. It does not parent to the tap, even w `parent.span_id` is omitted on that reconnect (it is the root of its trace). Histogram buckets are unchanged. `trace.id` is not a bucket key. +## Span exemplars + +The tables above say which fields a span carries. These lines are the join: the error-repro enums sit on the timing span for the same attempt, sharing `trace.id` with the event. `span.status` is about the span's own contract (did send return, did the first frame arrive, did shutdown finish), not about guest effect. + +### `stream.startup` — #103 is not a slow TTFF + +Preview went live, then stuck on Connecting while `/stream.mjpeg` kept transferring. First-frame delivery succeeded. The stuck overlay is a later `stream.state` on the same trace, not a `stream.ttff_ms` in the 5000 bucket. + +```json +{"span":"stream.startup","trace.id":"aabbccddeeff0011","span.id":"11001100","parent.span_id":"","span.status":"ok","stream.ttff_ms":180,"boot_ms":0,"ws_ms":40,"indigo_ms":90,"first_jpeg_ms":50,"stream.consumer":"mjpeg_img","stream.saw_first_frame":true} +{"event":"stream.state","trace.id":"aabbccddeeff0011","span.id":"11001100","stream.state":"connecting","stream.prev_state":"live","stream.saw_first_frame":true,"stream.last_frame_age_ms":1400,"stream.producer_ok":true,"stream.consumer":"mjpeg_img","stream.ws_close_code":0,"stream.ws_close_reason":"","stream.recovered":false} +``` + +Read: `span.status=ok` and `stream.ttff_ms=180` with `stream.saw_first_frame=true` and `stream.producer_ok=true` is the #103 consumer regression. A never-connected preview is a different line: `span.status=error`, `stream.saw_first_frame=false`, `stream.ttff_ms` omitted, `stream.last_frame_age_ms=-1`. Do not bucket that with the 180 ms success. + +### `camera.helper_shutdown` — #143 signal vs exit 0 + +The number #143 was missing is how long shutdown took, and whether it ended in a signal. `camera.placeholder_join_ms` and `camera.socket_close_ms` are attributes of this span, not their own histograms. + +```json +{"event":"camera.helper.exit","trace.id":"bbccddeeff001122","span.id":"22002200","error.code":"helper_signaled","camera.helper_exit_code":null,"camera.helper_signal":11,"camera.shutdown_phase":"release_surfaces","camera.placeholder_joined":false,"camera.source":"placeholder"} +{"span":"camera.helper_shutdown","trace.id":"bbccddeeff001122","span.id":"22002200","parent.span_id":"","span.status":"error","camera.helper_shutdown_ms":48,"camera.placeholder_join_ms":0,"camera.socket_close_ms":12,"camera.helper_exit_code":null,"camera.helper_signal":11} +``` + +`span.status=error` only when `camera.helper_signal` is set or `camera.helper_exit_code` is non-zero. After #161 the regression canary is the same span with `span.status=ok`, `camera.helper_signal=null`, `camera.placeholder_joined=true`, and a non-zero `camera.placeholder_join_ms` (the cancel handler actually ran). A 48 ms shutdown that signaled is not the same bucket story as a 48 ms clean exit: the status and signal are the split, the duration is not. + +### `input.send` — same duration, different repro + +Restated so the span line is not only implied by the event. Portrait hit and landscape miss share `input.send_ms`. Guest effect stays on the event (`inject.guest_effect`); the span stays `ok` while `hid.state=ok` and `inject.result=sent`. + +```json +{"span":"input.send","trace.id":"c0ffee00c0ffee00","span.id":"c0ffee01","parent.span_id":"","span.status":"ok","input.send_ms":4,"inject.kind":"touch","inject.result":"sent","hid.state":"ok","input.orientation":"landscape_left","input.frame":"display","input.remapped":false} +{"event":"input.inject","trace.id":"c0ffee00c0ffee00","span.id":"c0ffee01","hid.state":"ok","inject.result":"sent","inject.ack":"none","inject.guest_effect":"unchanged","input.orientation":"landscape_left","input.frame":"display","input.remapped":false,"cli.exit_code":0} +``` + +#136 is the other 4 ms: `hid.state=gone` forces `span.status=error` and a non-zero `cli.exit_code` on the event. No `input.roundtrip` span is emitted while `inject.ack=none`. + ## Phased rollout 1. **Local events.** Structured JSON-lines when telemetry is `local`. Same schema as the tables above. `--list` grows `hid.state`, `hid.last_ok_at_ms`, and `stream.last_frame_age_ms` so an agent can see a dead injector without enabling telemetry. No new dependency. From 046b5c48f6fab49891e60034318cd2e77da63a93 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Tue, 6 Oct 2026 16:13:02 +0000 Subject: [PATCH 06/27] docs: keep false MJPEG stalls out of the reconnect histogram stream.reconnect_ms is only set when the consumer returns to live. #128 (producer still publishing) does not emit the span. #103 stuck Connecting ends the span as error and omits the duration so it is not a 0ms reconnect or a slow TTFF. --- docs/observability.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/docs/observability.md b/docs/observability.md index 933983708..3d282302c 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -192,7 +192,7 @@ Durations are milliseconds. Histograms are for a later exporter; local events ju | `camera.shm_map` (`camera.shm_map_ms`) | `shm_open` → mapped header readable | `camera.shm_map_result` enum `ok` \| `enoent` \| `error` | 1, 5, 10, 50, 100, 500 | | `camera.helper_startup` (`camera.helper_startup_ms`) | process start → first frame seq published | `camera.source` | 10, 50, 100, 250, 500, 1000, 5000 | | `camera.helper_shutdown` (`camera.helper_shutdown_ms`) | shutdown action or signal → process exit | `camera.placeholder_join_ms`, `camera.socket_close_ms`, `camera.helper_exit_code`, `camera.helper_signal` | 10, 50, 100, 500, 1000, 5000 | -| `stream.reconnect` (`stream.reconnect_ms`) | leave `live` → return to `live` | `stream.reconnect_attempt` (int, starts at 1), `stream.stall_reason`, `stream.orientation` | 100, 250, 500, 1000, 2000, 5000 | +| `stream.reconnect` (`stream.reconnect_ms`) | leave `live` → return to `live`. Omit the span when `stream.false_positive=true`. Omit `stream.reconnect_ms` if it never returns | `stream.reconnect_attempt` (int, starts at 1), `stream.stall_reason`, `stream.orientation`, `stream.false_positive`, `stream.saw_first_frame`, `stream.producer_ok`, `stream.recovered` | 100, 250, 500, 1000, 2000, 5000 | `stream.ttff_ms` is the number the idle-floor test already budgets by hand (first JPEG after `stream.mjpeg` is opened). `camera.helper_shutdown_ms` split into placeholder-join and socket-close is the number [#143](https://github.com/EvanBacon/serve-sim/issues/143) was missing when the helper died between "running" and exit 0. `input.send_ms` without `input.roundtrip_ms` is an honest measurement of today's fire-and-forget path; round-trip appears only when an ack exists. `input.send_ms` is not a proxy for guest effect: the landscape miss and the portrait hit in the #153 table have the same send duration. The orientation fields are what separate them. No remap span: `rawPointForDisplayPoint` is a handful of arithmetic and must not show up as its own histogram. @@ -310,6 +310,21 @@ Restated so the span line is not only implied by the event. Portrait hit and lan #136 is the other 4 ms: `hid.state=gone` forces `span.status=error` and a non-zero `cli.exit_code` on the event. No `input.roundtrip` span is emitted while `inject.ack=none`. +### `stream.reconnect` — #128 false stall is not a slow reconnect + +`stream.reconnect_ms` is only recorded when the consumer returns to `live`. A false stall must not close that span and must not land in the reconnect histogram. + +[#128](https://github.com/EvanBacon/serve-sim/issues/128) is `stream.false_positive=true`: the producer was still publishing (`stream.producer_fps` > 0) and the `` watchdog fired after rotation. Emit `stream.stall` (already exemplified above). Do **not** emit `stream.reconnect`. If a rotate started the trace, the stall span may share `trace.id` with `input.inject`; it still does not parent a reconnect span. `span.status` stays `unset` so a 6100 ms consumer age is not a 6100 ms reconnect. + +[#103](https://github.com/EvanBacon/serve-sim/issues/103) after a good frame is the opposite. Leave `live`, never return. The span stays open until the consumer is stopped or the process exits, then ends `span.status=error` and **omits** `stream.reconnect_ms`. A missing duration is the stuck Connecting, not a 0. Carry `stream.saw_first_frame=true` and `stream.producer_ok=true` so this is not bucketed with a never-connected `stream.startup` (`span.status=error`, no `stream.ttff_ms`). A later recovery ends a new span `ok` with `stream.reconnect_ms` set and `stream.recovered=true`. + +```json +{"span":"stream.stall","trace.id":"128f0000128f0000","span.id":"128f0001","span.status":"unset","stream.false_positive":true,"stream.stall_reason":"consumer_stalled_producer_live","stream.orientation":"landscape_left","stream.producer_fps":58,"stream.last_frame_age_ms":6100} +{"span":"stream.reconnect","trace.id":"103c0000103c0000","span.id":"103c0002","parent.span_id":"103c0001","span.status":"error","stream.stall_reason":"consumer_stalled_producer_live","stream.orientation":"portrait","stream.saw_first_frame":true,"stream.producer_ok":true,"stream.reconnect_attempt":1,"stream.recovered":false} +``` + +The second line has no `stream.reconnect_ms`. `camera.frame` stays a separate histogram: a helper that died on a signal is `camera.helper_shutdown` with `camera.helper_signal` set, not a long `camera.frame_interval_ms`. + ## Phased rollout 1. **Local events.** Structured JSON-lines when telemetry is `local`. Same schema as the tables above. `--list` grows `hid.state`, `hid.last_ok_at_ms`, and `stream.last_frame_age_ms` so an agent can see a dead injector without enabling telemetry. No new dependency. From ada0d8720eab2af2d0b3ec070c8f1c020229cea2 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Wed, 7 Oct 2026 16:11:49 +0000 Subject: [PATCH 07/27] docs: false stall timing is camera.frame, not reconnect #128 producer-live stalls were still exemplified as stream.reconnect. The timing span on that trace is camera.frame (interval stays in the 16ms bucket, span.status=ok). Omit camera.frame when producer_fps is 0. #103 stuck Connecting still ends stream.reconnect as error and omits the duration. --- docs/observability.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 3d282302c..2cc6f1b42 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -188,7 +188,7 @@ Durations are milliseconds. Histograms are for a later exporter; local events ju | `stream.startup` (`stream.ttff_ms`) | stream request accepted → first JPEG or AVCC frame delivered to that subscriber | `boot_ms` (sim already booted, else time until Booted), `ws_ms` (socket open), `indigo_ms` (first framebuffer/Indigo snapshot), `first_jpeg_ms` (encode + first byte to the subscriber). Sum is `stream.ttff_ms` | 50, 100, 250, 500, 1000, 2000, 5000 | | `input.send` (`input.send_ms`) | handler entered → `rawSend` returned | `inject.kind`, `inject.result`, `input.orientation`, `input.frame`, `input.remapped` | 1, 5, 10, 25, 50, 100, 250 | | `input.roundtrip` (`input.roundtrip_ms`) | `rawSend` → ack | emitted only once `inject.ack` is `ok` or `timeout`. Same orientation fields as `input.send` | 5, 10, 25, 50, 100, 250, 500, 1000 | -| `camera.frame` (`camera.frame_interval_ms`) | `PublishFrame` n → n+1 | `camera.source` | 16, 33, 50, 100, 250, 1000 | +| `camera.frame` (`camera.frame_interval_ms`) | `PublishFrame` n → n+1. Omit the span when the producer is silent (`stream.producer_fps=0`) | `camera.source`, `stream.producer_fps`, `stream.false_positive` | 16, 33, 50, 100, 250, 1000 | | `camera.shm_map` (`camera.shm_map_ms`) | `shm_open` → mapped header readable | `camera.shm_map_result` enum `ok` \| `enoent` \| `error` | 1, 5, 10, 50, 100, 500 | | `camera.helper_startup` (`camera.helper_startup_ms`) | process start → first frame seq published | `camera.source` | 10, 50, 100, 250, 500, 1000, 5000 | | `camera.helper_shutdown` (`camera.helper_shutdown_ms`) | shutdown action or signal → process exit | `camera.placeholder_join_ms`, `camera.socket_close_ms`, `camera.helper_exit_code`, `camera.helper_signal` | 10, 50, 100, 500, 1000, 5000 | @@ -227,10 +227,10 @@ Producer still publishing, `` watchdog fired. Must not share an event name ```json {"span":"input.send","input.send_ms":4,"inject.kind":"touch","inject.result":"sent","input.orientation":"landscape_left","input.frame":"display","input.remapped":false} -{"span":"stream.reconnect","stream.reconnect_ms":420,"stream.reconnect_attempt":1,"stream.stall_reason":"consumer_stalled_producer_live","stream.orientation":"landscape_left"} +{"span":"camera.frame","camera.frame_interval_ms":17,"camera.source":"placeholder","stream.producer_fps":58,"stream.false_positive":true} ``` -`stream.reconnect` is omitted when the consumer never returns to `live` (#103). `camera.helper_shutdown` is the #143 canary: `camera.helper_signal=11` with `camera.shutdown_phase=release_surfaces` and `camera.placeholder_joined=false` is the race #161 closed; a regression should show that triple, not `exited with null`. +The second line is the timing record for a #128 false stall: the producer interval stays in the 16 ms bucket while the consumer watchdog fires. Do not emit `stream.reconnect` for that case. `stream.reconnect` is omitted when `stream.false_positive=true`, and also when the consumer never returns to `live` (#103). `camera.helper_shutdown` is the #143 canary: `camera.helper_signal=11` with `camera.shutdown_phase=release_surfaces` and `camera.placeholder_joined=false` is the race #161 closed; a regression should show that triple, not `exited with null`. ## Joining an error record to its span @@ -251,6 +251,8 @@ An `input.inject` line and an `input.send` span are the same attempt. Without a | `inject.result` in `nil_message` \| `no_client` \| `throw`, or `hid.state=gone` | `error`, and set `error.code` | | `inject.ack=timeout` on `input.roundtrip` | `error` | | `stream.false_positive=true` on `stream.stall` | `unset` (detector fired, producer was live; not a failed send) | +| `camera.frame` with `stream.producer_fps>0` | `ok`, even when a consumer stall is on the same trace | +| `camera.frame` omitted, `stream.producer_fps=0` | no span; the stall event is `span.status=error` | | `camera.helper_signal` non-null | `error` | A coordinate miss must stay `span.status=ok` with `input.remapped=false`. Marking it `error` would bucket it with #136, which is the mistake the fields exist to prevent. @@ -264,14 +266,14 @@ Worked join for the #153 landscape miss. Same `trace.id` on the event and the sp {"span":"input.send","trace.id":"a1b2c3d4e5f60718","span.id":"9f8e7d6c","span.status":"ok","input.send_ms":4,"inject.kind":"touch","inject.result":"sent","input.orientation":"landscape_left","input.frame":"display","input.remapped":false} ``` -The #128 false stall is a different trace. It does not parent to the tap, even when both follow a rotate. +The #128 false stall is a different trace. It does not parent to the tap, even when both follow a rotate. It also does not emit `stream.reconnect`: the producer is still publishing, so the timing span on that trace is `camera.frame`, not a reconnect. ```json -{"event":"stream.stall","trace.id":"0011223344556677","span.status":"unset","stream.stall_reason":"consumer_stalled_producer_live","stream.false_positive":true,"stream.orientation":"landscape_left","stream.producer_fps":58} -{"span":"stream.reconnect","trace.id":"0011223344556677","span.id":"aabbccdd","parent.span_id":"","span.status":"ok","stream.reconnect_ms":420,"stream.reconnect_attempt":1,"stream.stall_reason":"consumer_stalled_producer_live","stream.orientation":"landscape_left"} +{"event":"stream.stall","trace.id":"0011223344556677","span.status":"unset","stream.stall_reason":"consumer_stalled_producer_live","stream.false_positive":true,"stream.orientation":"landscape_left","stream.producer_fps":58,"stream.last_frame_age_ms":6100} +{"span":"camera.frame","trace.id":"0011223344556677","span.id":"aabbccdd","span.status":"ok","camera.frame_interval_ms":17,"camera.source":"placeholder","stream.producer_fps":58,"stream.false_positive":true} ``` -`parent.span_id` is omitted on that reconnect (it is the root of its trace). Histogram buckets are unchanged. `trace.id` is not a bucket key. +`span.status=ok` on that 17 ms interval is the proof the overlay was a consumer false positive. A producer that actually stopped is a different span: `span.status=error`, `camera.frame_interval_ms` omitted, `stream.producer_fps=0`, `stream.false_positive=false`. Histogram buckets are unchanged. `trace.id` is not a bucket key. ## Span exemplars @@ -319,11 +321,12 @@ Restated so the span line is not only implied by the event. Portrait hit and lan [#103](https://github.com/EvanBacon/serve-sim/issues/103) after a good frame is the opposite. Leave `live`, never return. The span stays open until the consumer is stopped or the process exits, then ends `span.status=error` and **omits** `stream.reconnect_ms`. A missing duration is the stuck Connecting, not a 0. Carry `stream.saw_first_frame=true` and `stream.producer_ok=true` so this is not bucketed with a never-connected `stream.startup` (`span.status=error`, no `stream.ttff_ms`). A later recovery ends a new span `ok` with `stream.reconnect_ms` set and `stream.recovered=true`. ```json -{"span":"stream.stall","trace.id":"128f0000128f0000","span.id":"128f0001","span.status":"unset","stream.false_positive":true,"stream.stall_reason":"consumer_stalled_producer_live","stream.orientation":"landscape_left","stream.producer_fps":58,"stream.last_frame_age_ms":6100} -{"span":"stream.reconnect","trace.id":"103c0000103c0000","span.id":"103c0002","parent.span_id":"103c0001","span.status":"error","stream.stall_reason":"consumer_stalled_producer_live","stream.orientation":"portrait","stream.saw_first_frame":true,"stream.producer_ok":true,"stream.reconnect_attempt":1,"stream.recovered":false} +{"event":"stream.stall","trace.id":"128f0000128f0000","span.status":"unset","stream.false_positive":true,"stream.stall_reason":"consumer_stalled_producer_live","stream.orientation":"landscape_left","stream.producer_fps":58,"stream.last_frame_age_ms":6100} +{"span":"camera.frame","trace.id":"128f0000128f0000","span.id":"128f0001","span.status":"ok","camera.frame_interval_ms":17,"camera.source":"placeholder","stream.producer_fps":58,"stream.false_positive":true} +{"span":"stream.reconnect","trace.id":"103c0000103c0000","span.id":"103c0002","parent.span_id":"103c0001","span.status":"error","stream.stall_reason":"producer_silent","stream.orientation":"portrait","stream.saw_first_frame":true,"stream.producer_ok":true,"stream.reconnect_attempt":1,"stream.recovered":false} ``` -The second line has no `stream.reconnect_ms`. `camera.frame` stays a separate histogram: a helper that died on a signal is `camera.helper_shutdown` with `camera.helper_signal` set, not a long `camera.frame_interval_ms`. +The #128 trace has a 17 ms `camera.frame` and no `stream.reconnect`. The #103 line has no `stream.reconnect_ms`. A helper that died on a signal is `camera.helper_shutdown` with `camera.helper_signal` set, not a long `camera.frame_interval_ms` (that span is omitted when `stream.producer_fps=0`). ## Phased rollout From 47816b4170e2510afd48fc716013f9a0a6768f2d Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Thu, 8 Oct 2026 01:25:14 -0700 Subject: [PATCH 08/27] docs: rebuild observability on the existing event log --- docs/observability.md | 544 ++++++++++++++++-------------------------- 1 file changed, 205 insertions(+), 339 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 2cc6f1b42..553b0becc 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1,344 +1,210 @@ # Observability -Design only. No SDK, exporter, or runtime change is implied by this document. - -serve-sim is a local Mac process that streams a simulator and injects input. The failures that have cost time are silent: a command exits 0, `--list` says `running: true`, or the preview says Connecting while the capture side is still producing frames. A later class is just as silent: the touch is delivered, and the screenshot is unchanged because the point was in the wrong frame. The useful record is the fields that distinguish those cases, plus timings on the paths we have already had to measure by hand. - -## Goals - -1. **Error reproduction.** Each event below is a field that was missing from a real report. With it, a log line is enough to tell injector-sent from guest-unchanged, a coordinate miss from a dead injector, signal from exit 0, and a dead consumer from a dead producer. -2. **Performance timing.** Spans for the hot paths where a fix already depended on a number: first frame, inject send, camera frame interval, helper startup/shutdown, reconnect. Spans carry the same error-repro attributes so a fast send is not read as a successful tap. - -Both stay on the machine unless someone opts into export. An attempt's event and its span share `trace.id` so a 4 ms send is not read apart from the coordinate miss it belongs to. - -## Non-goals - -- Dashboards, error budgets, release scorecards, or a feedback program. -- Shipping an OpenTelemetry SDK in the first slice. -- Guest-effect checks that keep pixels, OCR, or accessibility text. -- A default backend, a prompt, or a notice on startup. - -## Privacy and opt-out - -Default is **off**. Nothing is written for telemetry and nothing is sent. - -| Control | Effect | -|---|---| -| unset | off | -| `SERVE_SIM_TELEMETRY=0` \| `off` \| `false` | off | -| `--no-telemetry` | off (wins over every other switch) | -| `SERVE_SIM_TELEMETRY=local` or `--telemetry=local` | JSON-lines on the machine only | -| `SERVE_SIM_TELEMETRY=1` \| `on` \| `remote` or `--telemetry=remote` | local events, plus OTLP **only** when `OTEL_EXPORTER_OTLP_ENDPOINT` is set to a URL the operator chose | - -There is no config file today. A future `telemetry` key uses the same enum (`off` \| `local` \| `remote`). Precedence is CLI, then env, then file. `off` wins. - -CI and agent runs stay silent. Opt-in is an explicit env or flag in that job, not something inferred from `CI=`. - -`SERVE_SIM_DEBUG` and `SERVE_SIM_DEBUG_HID` stay local stderr for the person who set them. Those lines are debug logs. They are not attached to exported spans. - -`--list` health (HID state, last successful inject, stream frame age) is process status for the operator sitting at the terminal. It is not an export path. - -Events carry enums, timings, versions, and coarse runtime strings. They leave out: - -- MJPEG, JPEG, screenshots, camera frames, and any other pixels -- App content, accessibility-tree text, clipboard -- Credentials, tokens, cookies, `Authorization` headers -- Raw UDIDs and device names on any export (see `device.id_hash`) -- Home-directory paths (basenames only) -- Raw WebSocket payloads - -`device.id_hash` is 8 hex chars of SHA-256 of the UDID. Local debug may also print an 8-character UDID prefix on its own line. Remote export gets the hash only. `session.id` is a new UUID per server process. - -## Common attributes - -Present on every event and span: - -| Field | Type | Example / notes | -|---|---|---| -| `serve_sim.version` | string | `0.1.47` | -| `serve_sim.arch` | enum | `arm64` \| `x64` | -| `os.version` | string | macOS `major.minor` only | -| `xcode.version` | string | `27.0` (marketing version, not the build serial) | -| `ios.runtime` | string | `iOS 27.0` | -| `session.id` | string | ephemeral UUID | -| `device.id_hash` | string | 8 hex chars | -| `sim.ui_host` | enum | `none` \| `simulator_app` \| `device_hub` \| `unknown` | - -`sim.ui_host=none` is the headless `simctl boot` case with no Simulator.app or Device Hub. - -## Error reproduction - -| Issue | What the report could not show | Event | -|---|---|---| -| [#153](https://github.com/EvanBacon/serve-sim/issues/153) | Touch exits 0, screenshots identical. 2026-09-30 repro: headless portrait taps land on iPad and iPhone; `landscape_left` misses because CLI/server pass display-normalized points into the native portrait frame. Web client already remaps. Missing orientation, frame, and whether a remap ran, so a coordinate miss looked like a dead injector. | `input.inject` | -| [#136](https://github.com/EvanBacon/serve-sim/issues/136) | HID died; button, tap, and gesture still exit 0; `--list` stays `running: true`. | `hid.health` + `input.inject` | -| [#143](https://github.com/EvanBacon/serve-sim/issues/143) | Helper exit was null (signal) under `--max-concurrency=1`. Missing signal vs code, last log, shutdown phase, placeholder-timer vs socket-close timing. | `camera.helper.exit` | -| [#103](https://github.com/EvanBacon/serve-sim/issues/103) | Preview live for 1–2s, then Connecting forever on 0.1.40+. Missing state transitions, last frame time, WS close reason, first-frame vs later stall, whether it recovered. | `stream.state` | -| [#128](https://github.com/EvanBacon/serve-sim/issues/128) | MJPEG `` watchdog showed a frame-stall after rotation while `/stream.mjpeg` was still sending. | `stream.stall` | -| [#102](https://github.com/EvanBacon/serve-sim/issues/102) | Helper stayed up for days after the sim was Shutdown. Closed by [#152](https://github.com/EvanBacon/serve-sim/pull/152); the booted-vs-alive pair is the canary. | `helper.lifecycle` | - -### `input.inject` - -One event per touch phase, button, or gesture step. This is the record [#153](https://github.com/EvanBacon/serve-sim/issues/153) and [#136](https://github.com/EvanBacon/serve-sim/issues/136) needed: `button lock` goes through `IndigoHIDMessageForButton` (target hardware, `0x33`) and can succeed while digitizer touch (`0x32`, `IndigoHIDMessageForMouseNSEvent`) is built and sent into a session with no UI host — or into the wrong coordinate frame. - -| Field | Type | Notes | -|---|---|---| -| `hid.client` | enum | `created` \| `missing` \| `send_unloaded` | -| `hid.state` | enum | `ok` \| `stale` \| `gone` (same enum as `hid.health`) | -| `inject.kind` | enum | `touch` \| `button` \| `gesture` \| `key` \| `rotate` | -| `inject.phase` | enum | `begin` \| `move` \| `end` \| `press` | -| `inject.result` | enum | `sent` \| `nil_message` \| `no_client` \| `throw` | -| `inject.ack` | enum | `none` until a reply exists. Later `ok` \| `timeout` | -| `inject.ack_ms` | int, ms | omitted while `inject.ack=none` | -| `inject.guest_effect` | enum | `not_checked` by default. Opt-in local `--verify` may set `changed` \| `unchanged` and then discard the frames | -| `input.coord_space` | enum | `normalized_0_1` | -| `input.orientation` | enum | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown`. Device orientation at send, not the caller's assumption | -| `input.frame` | enum | `display` (rotated screenshot / what `serve-sim tap` is given) \| `native_portrait` (what `SimHID.touch` consumes) | -| `input.remapped` | bool | true if this process applied `rawPointForDisplayPoint` before send | -| `input.screen_w` | int, px | width passed into the injector | -| `input.screen_h` | int, px | height passed into the injector | -| `input.target` | enum | `digitizer` \| `hardware_button` | -| `input.x`, `input.y` | number | local events only. Remote export omits them | -| `error.code` | string | stable token on `throw` (`hid_client_missing`, `mouse_symbol_missing`, …) | -| `cli.exit_code` | int | `0` only when `inject.result=sent` and `hid.state=ok`. `no_client`, `nil_message`, `gone`, and `throw` exit non-zero | - -Today's `tap` is fire-and-forget: the socket open succeeds, `begin`/`end` are written, and the process exits 0 with no reply handler. The event records that as `inject.result=sent`, `inject.ack=none`, `inject.guest_effect=not_checked`. A sent digitizer event and a delivered one stop looking the same once `hid.client`, `sim.ui_host`, and `input.target` are on the line. - -A 2026-09-30 repro on [#153](https://github.com/EvanBacon/serve-sim/issues/153) (macOS 27 / Xcode 27.0 (27A266a) / iOS 27, headless `simctl boot`, no Simulator.app or Device Hub, `dtuhidd.active=0`) showed portrait taps opening Settings on both iPad Pro 13-inch (M5) and iPhone 18 Pro. The same `landscape_left` point (`0.715, 0.343` in the rotated frame) left the screenshot unchanged until it was rewritten with the web client's map `{ x: y, y: 1 - x }`, which is `rawPointForDisplayPoint` in `orientation.ts`. `w`/`h` do not move the target except on Duo. The distinguishing record is `sim.ui_host=none`, `hid.state=ok`, `inject.result=sent`, `input.orientation=landscape_left`, `input.frame=display`, `input.remapped=false`, `inject.guest_effect=unchanged`. That is a coordinate miss, not a dead HID client (`hid.client=missing`) and not the [#136](https://github.com/EvanBacon/serve-sim/issues/136) `hid.state=gone` case. `input.x` / `input.y` stay local-only; remote export keeps the enums. A fix that remaps CLI `tap` and other direct `0x03` writers should set `input.remapped=true` and `input.frame=native_portrait` after the map. - -### `hid.health` - -Emitted on transition and included in `--list`. [#136](https://github.com/EvanBacon/serve-sim/issues/136) stayed invisible because `running` is "pidfile exists", which stays true after the HID client is dead and after a `simctl shutdown` / `boot`. - -| Field | Type | Notes | -|---|---|---| -| `hid.state` | enum | `ok` last send landed. `stale` no successful send for `hid.stale_after_ms` while commands are still arriving. `gone` client missing or sim not booted | -| `hid.last_ok_at_ms` | int | unix ms of the last `inject.result=sent`. `0` if none | -| `hid.stale_after_ms` | int, ms | threshold in force (start at 30000) | -| `hid.helper_age_ms` | int, ms | age of this helper process, not of the simulator | -| `sim.booted` | bool | `simctl` still lists the UDID as Booted | -| `list.running` | bool | today's flag, kept so `running:true` + `hid.state:gone` is one object | - -### `camera.helper.exit` - -[#143](https://github.com/EvanBacon/serve-sim/issues/143) failed in the test's `exitCode !== 0` branch: Node reports `null` when the child dies on a signal, and the shm name was already unlinked. The race was the placeholder timer's in-flight `PublishFrame` versus surface release (`SIGSEGV` / use-after-free, or `SIGPIPE` if a client vanished mid-write). - -[#161](https://github.com/EvanBacon/serve-sim/pull/161) is on main (`516cdbf`). `StopPlaceholderSource` now cancels the timer and waits for its cancel handler before `ReleaseSurfaces`, and the helper returns 0. Issue #143 is still open; treat that as the bug being mitigated, and keep these fields as the regression canary. A future red CI run should show signal number and phase instead of `exited with null`. - -| Field | Type | Notes | -|---|---|---| -| `camera.helper_exit_code` | int or null | `null` when the process is signaled | -| `camera.helper_signal` | int or null | signal number (`11` SIGSEGV, `13` SIGPIPE, `6` SIGABRT). `null` on a normal exit | -| `camera.shutdown_phase` | enum | `running` \| `unlink_shm` \| `stop_placeholder` \| `stop_video` \| `stop_webcam` \| `close_socket` \| `release_surfaces` \| `exited` | -| `camera.placeholder_joined` | bool | timer cancel handler finished before unmap | -| `camera.placeholder_join_ms` | int, ms | wait on that semaphore | -| `camera.socket_close_ms` | int, ms | shutdown request → accept source cancelled | -| `camera.source` | enum | `placeholder` \| `webcam` \| `image` \| `video` \| `none` | -| `camera.last_log` | string | last ~600 chars of helper stderr. Basenames only, no frame bytes | -| `error.code` | string | `helper_signaled` \| `helper_nonzero` \| `helper_ok` | - -### `stream.state` - -[#103](https://github.com/EvanBacon/serve-sim/issues/103) went live, then stuck on Connecting from 0.1.40 onward. A later note on 0.1.44: after the preview had already rendered, a browser/HID interaction flipped it back to Connecting and it never recovered, while `/stream.mjpeg` was still returning 200 and transferring megabytes. That is a consumer state-machine bug, not a dead simulator. The transition log is the repro. - -| Field | Type | Notes | -|---|---|---| -| `stream.state` | enum | `starting` \| `connecting` \| `live` \| `stalled` \| `reconnecting` \| `stopped` | -| `stream.prev_state` | enum | previous value | -| `stream.saw_first_frame` | bool | false = never left Connecting; true = stall after a good frame | -| `stream.last_frame_age_ms` | int, ms | age on **this consumer**. `-1` if none yet | -| `stream.consumer` | enum | `mjpeg_img` \| `mjpeg_fetch` \| `avcc` \| `relay` | -| `stream.producer_ok` | bool | capture side has a frame newer than the stall threshold (counter only) | -| `stream.ws_close_code` | int | RFC 6455 code, `0` if the socket is open | -| `stream.ws_close_reason` | string | short token (`normal`, `abnormal`, `error`). Not the payload | -| `stream.recovered` | bool | reached `live` again after `stalled` or a second `connecting` | - -### `stream.stall` - -[#128](https://github.com/EvanBacon/serve-sim/issues/128) fired the overlay "Stream is not producing frames" after rotation. The native `` and a parallel `fetch()` watchdog are separate consumers; either can time out while the other, and the helper, are healthy. The startup watchdog in `SimulatorView` is 6000 ms and keys off MJPEG `--frame` boundaries. Relay mode treats 2000 ms without a frame as stale. The event records which detector fired and whether the producer was still moving. Join `stream.orientation` with `input.orientation` when a rotation and a tap land in the same session: a false stall (#128) and a coordinate miss (#153) can share a rotate and must not be one event. - -| Field | Type | Notes | -|---|---|---| -| `stream.stall_reason` | enum | `startup_watchdog` \| `relay_stale` \| `consumer_stalled_producer_live` \| `producer_silent` | -| `stream.stall_threshold_ms` | int, ms | timeout that fired | -| `stream.last_frame_age_ms` | int, ms | consumer frame age | -| `stream.fps` | number | frames in the last 1s on that consumer | -| `stream.producer_fps` | number | frames published in the same 1s | -| `stream.false_positive` | bool | `producer_fps > 0` while the consumer is past threshold | -| `stream.orientation` | enum | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` | -| `stream.restart_ms` | int, ms | time until the next `live`, omitted if it never returns | - -### `helper.lifecycle` - -[#102](https://github.com/EvanBacon/serve-sim/issues/102) was an orphaned `serve-sim-bin` at ~100% CPU with the simulator already Shutdown (parent `launchd`, uptime measured in days). [#152](https://github.com/EvanBacon/serve-sim/pull/152) exits detached helpers after simulator shutdown. One event on the way out, and one if the process is still alive while `sim.booted=false`, is enough to catch a regression. - -| Field | Type | Notes | -|---|---|---| -| `sim.booted` | bool | | -| `helper.uptime_ms` | int, ms | | -| `helper.parent` | enum | `launchd` \| `serve-sim` \| `other` | -| `helper.action` | enum | `exit_sim_shutdown` \| `still_alive_sim_down` | - -## Performance timing - -Durations are milliseconds. Histograms are for a later exporter; local events just store the number. Suggested buckets are upper bounds in ms. Every span below also carries the common attributes, and inject spans carry `input.orientation`, `input.frame`, and `input.remapped` so a landscape miss is not bucketed with a portrait hit. - -| Span | Start → end | Fields | Buckets | -|---|---|---|---| -| `stream.startup` (`stream.ttff_ms`) | stream request accepted → first JPEG or AVCC frame delivered to that subscriber | `boot_ms` (sim already booted, else time until Booted), `ws_ms` (socket open), `indigo_ms` (first framebuffer/Indigo snapshot), `first_jpeg_ms` (encode + first byte to the subscriber). Sum is `stream.ttff_ms` | 50, 100, 250, 500, 1000, 2000, 5000 | -| `input.send` (`input.send_ms`) | handler entered → `rawSend` returned | `inject.kind`, `inject.result`, `input.orientation`, `input.frame`, `input.remapped` | 1, 5, 10, 25, 50, 100, 250 | -| `input.roundtrip` (`input.roundtrip_ms`) | `rawSend` → ack | emitted only once `inject.ack` is `ok` or `timeout`. Same orientation fields as `input.send` | 5, 10, 25, 50, 100, 250, 500, 1000 | -| `camera.frame` (`camera.frame_interval_ms`) | `PublishFrame` n → n+1. Omit the span when the producer is silent (`stream.producer_fps=0`) | `camera.source`, `stream.producer_fps`, `stream.false_positive` | 16, 33, 50, 100, 250, 1000 | -| `camera.shm_map` (`camera.shm_map_ms`) | `shm_open` → mapped header readable | `camera.shm_map_result` enum `ok` \| `enoent` \| `error` | 1, 5, 10, 50, 100, 500 | -| `camera.helper_startup` (`camera.helper_startup_ms`) | process start → first frame seq published | `camera.source` | 10, 50, 100, 250, 500, 1000, 5000 | -| `camera.helper_shutdown` (`camera.helper_shutdown_ms`) | shutdown action or signal → process exit | `camera.placeholder_join_ms`, `camera.socket_close_ms`, `camera.helper_exit_code`, `camera.helper_signal` | 10, 50, 100, 500, 1000, 5000 | -| `stream.reconnect` (`stream.reconnect_ms`) | leave `live` → return to `live`. Omit the span when `stream.false_positive=true`. Omit `stream.reconnect_ms` if it never returns | `stream.reconnect_attempt` (int, starts at 1), `stream.stall_reason`, `stream.orientation`, `stream.false_positive`, `stream.saw_first_frame`, `stream.producer_ok`, `stream.recovered` | 100, 250, 500, 1000, 2000, 5000 | - -`stream.ttff_ms` is the number the idle-floor test already budgets by hand (first JPEG after `stream.mjpeg` is opened). `camera.helper_shutdown_ms` split into placeholder-join and socket-close is the number [#143](https://github.com/EvanBacon/serve-sim/issues/143) was missing when the helper died between "running" and exit 0. `input.send_ms` without `input.roundtrip_ms` is an honest measurement of today's fire-and-forget path; round-trip appears only when an ack exists. `input.send_ms` is not a proxy for guest effect: the landscape miss and the portrait hit in the #153 table have the same send duration. The orientation fields are what separate them. No remap span: `rawPointForDisplayPoint` is a handful of arithmetic and must not show up as its own histogram. - -Crashes in our own process add `error.code` plus a stack trimmed to serve-sim frames, with home directories stripped. No media, no request bodies. - -## Worked records - -Same session, two taps, one span. These are the lines a later implementation should emit when telemetry is `local`. Remote export drops `input.x` / `input.y` and replaces `device.id_hash` only (already hashed). - -### Error reproduction — #153 landscape miss vs portrait hit - -Headless iPad, `sim.ui_host=none`, HID client up. Portrait tap opens Settings. The same display point in `landscape_left` is sent unmapped and the screenshot does not change. - -```json -{"event":"input.inject","serve_sim.version":"0.1.47","sim.ui_host":"none","hid.client":"created","hid.state":"ok","inject.kind":"touch","inject.phase":"end","inject.result":"sent","inject.ack":"none","inject.guest_effect":"changed","input.coord_space":"normalized_0_1","input.orientation":"portrait","input.frame":"display","input.remapped":false,"input.target":"digitizer","input.screen_w":1376,"input.screen_h":1032,"input.x":0.715,"input.y":0.343,"cli.exit_code":0} -{"event":"input.inject","serve_sim.version":"0.1.47","sim.ui_host":"none","hid.client":"created","hid.state":"ok","inject.kind":"touch","inject.phase":"end","inject.result":"sent","inject.ack":"none","inject.guest_effect":"unchanged","input.coord_space":"normalized_0_1","input.orientation":"landscape_left","input.frame":"display","input.remapped":false,"input.target":"digitizer","input.screen_w":1376,"input.screen_h":1032,"input.x":0.715,"input.y":0.343,"cli.exit_code":0} -``` - -Signature of the miss: `hid.state=ok` + `inject.result=sent` + `input.frame=display` + `input.remapped=false` + `inject.guest_effect=unchanged`. That is not #136 (`hid.state=gone`, `cli.exit_code` non-zero) and not a missing symbol (`hid.client=missing`). After a CLI remap, the second line should read `input.frame=native_portrait`, `input.remapped=true`, and the local coordinates should be the mapped point (`x=0.343`, `y=0.285` for `{ x: y, y: 1 - x }` on `landscape_left`). - -### Error reproduction — #128 false stall after rotate - -Producer still publishing, `` watchdog fired. Must not share an event name with the coordinate miss above even if both follow `serve-sim rotate`. - -```json -{"event":"stream.stall","stream.stall_reason":"consumer_stalled_producer_live","stream.stall_threshold_ms":6000,"stream.last_frame_age_ms":6100,"stream.fps":0,"stream.producer_fps":58,"stream.false_positive":true,"stream.orientation":"landscape_left","stream.consumer":"mjpeg_img"} +serve-sim already keeps an in-memory **event log** of simulator actions. This +document describes what that log records today, the small, additive fields that +make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/153), +[#136](https://github.com/EvanBacon/serve-sim/issues/136), +[#143](https://github.com/EvanBacon/serve-sim/issues/143), +[#103](https://github.com/EvanBacon/serve-sim/issues/103), +[#128](https://github.com/EvanBacon/serve-sim/issues/128), +[#102](https://github.com/EvanBacon/serve-sim/issues/102)) reproducible from the +log alone, and the exact code hook that produces each field. It is built on the +log we have -- no span tree, no histograms, no OpenTelemetry SDK is implied. + +The event log is a diagnostic side-channel. It must never fail the input or +command path (`recordEventLogEvent` already swallows subscriber errors), and it +is **on by default**; remote export is strictly opt-in (see Privacy below). + +## What the event log records today + +`src/event-log.ts` keeps a bounded ring (`EVENT_LOG_MAX_ENTRIES = 500`) of: + +```ts +type EventLogEntry = { + id: number; timestamp: string; + source: "hid" | "exec" | "ui"; + kind: string; msg: string; summary: string; + device?: string; action?: string; + status?: "ok" | "error"; + details?: Record; +}; ``` -### Performance timing — inject span carries the repro fields - -`input.send_ms` is the same on the hit and the miss. The orientation fields are the bucket key. No `input.roundtrip` line: `inject.ack` is still `none`. - -```json -{"span":"input.send","input.send_ms":4,"inject.kind":"touch","inject.result":"sent","input.orientation":"landscape_left","input.frame":"display","input.remapped":false} -{"span":"camera.frame","camera.frame_interval_ms":17,"camera.source":"placeholder","stream.producer_fps":58,"stream.false_positive":true} -``` - -The second line is the timing record for a #128 false stall: the producer interval stays in the 16 ms bucket while the consumer watchdog fires. Do not emit `stream.reconnect` for that case. `stream.reconnect` is omitted when `stream.false_positive=true`, and also when the consumer never returns to `live` (#103). `camera.helper_shutdown` is the #143 canary: `camera.helper_signal=11` with `camera.shutdown_phase=release_surfaces` and `camera.placeholder_joined=false` is the race #161 closed; a regression should show that triple, not `exited with null`. - -## Joining an error record to its span - -An `input.inject` line and an `input.send` span are the same attempt. Without a shared id, a landscape miss (`inject.guest_effect=unchanged`) and a 4 ms send look like two unrelated facts. Every event and span in one attempt carries the same `trace.id`. The span also has `span.id`. Child spans set `parent.span_id`. - -| Field | Type | Notes | +- **HID input** -> `eventLogEventForHidMessage(device, tag, payload, screen)`, called + from `recordHidEvent` / `recordTouchEvent` inside `handleHidMessage` + (`src/device-session.ts`). Taps and drags are synthesized in `recordTouchEvent` + (tag `0x03` begin/move/end) and already carry `details.start`, `details.current`, + `details.moveCount`, and `details.screen = {width,height}` (via `eventLogScreen()`). +- **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, parsed in + `src/middleware.ts` from the `/exec` path. It already drops upload plumbing and + never records raw command strings with tokens. +- **UI settings** -> `recordEventLogEvent({source:"ui", ...})` in `middleware.ts`. + +It already redacts: printable keystrokes (tag `0x06` records `key:"character"`, +`redacted:true`, no usage), token-bearing commands (returns `null`), and it only +ever stores **normalized** coordinates plus screen dimensions -- never pixels. + +## How a bot reads it back + +- `GET {url}/api/event-log?device=&since=&limit=` -> `{ events: [...] }` + (`readEventLog` in `middleware.ts`). +- `GET {url}/api/event-log/events` -> SSE: a snapshot, then one `{ event }` per new + entry (`subscribeEventLog`; the browser UI reaches it through the exec-ws proxy). +- CLI: `serve-sim event-log --json [-d ] [--limit ]` + (`eventLog()` in `src/index.ts`, which fetches `/api/event-log`). + +Add a top-level **`schema_version`** to the `/api/event-log` (and SSE) response +envelope so a reader can tell which of the fields below are present. Every field +added here is additive; an older consumer ignores unknown keys. + +## Making the #153 touch miss reproducible from the log + +#153: a tap exits 0 and the screenshot is unchanged because the point was sent in +the wrong coordinate frame. The web client remaps display->native +(`rawPointForDisplayPoint` in `client/simulator/orientation.ts`, called from +`SimulatorView.tsx`); the CLI (`tap()` / `gesture()` in `index.ts`) and the +server path `handleHidMessage` -> `this.hid.touch(type, x, y, W, H, edge)` pass the +display-normalized point straight into the native portrait frame with **no remap**. +Nothing in the log distinguishes that miss from a dead injector. + +Add these keys to the `details` of touch/tap/drag entries (produced where the entry +is built -- `recordTouchEvent` / `eventLogEventForHidMessage`): + +| Field | Values | Source / hook | |---|---|---| -| `trace.id` | string | 16 hex chars, new per attempt (one tap, one helper shutdown, one consumer stall). Not the session id | -| `span.id` | string | 8 hex chars. Spans only | -| `parent.span_id` | string | set on a child span. Omitted on a root | -| `span.status` | enum | `ok` \| `error` \| `unset`. Derived, not a second opinion | - -`span.status` is not `inject.result`. Map: - -| Condition | `span.status` | -|---|---| -| `inject.result=sent` and `hid.state=ok` | `ok`, even when `inject.guest_effect=unchanged` (the send succeeded; the miss is an attribute) | -| `inject.result` in `nil_message` \| `no_client` \| `throw`, or `hid.state=gone` | `error`, and set `error.code` | -| `inject.ack=timeout` on `input.roundtrip` | `error` | -| `stream.false_positive=true` on `stream.stall` | `unset` (detector fired, producer was live; not a failed send) | -| `camera.frame` with `stream.producer_fps>0` | `ok`, even when a consumer stall is on the same trace | -| `camera.frame` omitted, `stream.producer_fps=0` | no span; the stall event is `span.status=error` | -| `camera.helper_signal` non-null | `error` | - -A coordinate miss must stay `span.status=ok` with `input.remapped=false`. Marking it `error` would bucket it with #136, which is the mistake the fields exist to prevent. - -`stream.startup` is the parent of the first `camera.frame` after subscribe. `camera.helper_shutdown` is the parent of the placeholder-join and socket-close waits; those stay fields on the parent (`camera.placeholder_join_ms`, `camera.socket_close_ms`), not their own spans. `input.roundtrip` parents to `input.send` and is omitted while `inject.ack=none`. - -Worked join for the #153 landscape miss. Same `trace.id` on the event and the span. `input.send_ms` stays 4. Status stays `ok`. - -```json -{"event":"input.inject","trace.id":"a1b2c3d4e5f60718","serve_sim.version":"0.1.47","sim.ui_host":"none","hid.state":"ok","inject.kind":"touch","inject.phase":"end","inject.result":"sent","inject.ack":"none","inject.guest_effect":"unchanged","input.orientation":"landscape_left","input.frame":"display","input.remapped":false,"cli.exit_code":0} -{"span":"input.send","trace.id":"a1b2c3d4e5f60718","span.id":"9f8e7d6c","span.status":"ok","input.send_ms":4,"inject.kind":"touch","inject.result":"sent","input.orientation":"landscape_left","input.frame":"display","input.remapped":false} -``` - -The #128 false stall is a different trace. It does not parent to the tap, even when both follow a rotate. It also does not emit `stream.reconnect`: the producer is still publishing, so the timing span on that trace is `camera.frame`, not a reconnect. - -```json -{"event":"stream.stall","trace.id":"0011223344556677","span.status":"unset","stream.stall_reason":"consumer_stalled_producer_live","stream.false_positive":true,"stream.orientation":"landscape_left","stream.producer_fps":58,"stream.last_frame_age_ms":6100} -{"span":"camera.frame","trace.id":"0011223344556677","span.id":"aabbccdd","span.status":"ok","camera.frame_interval_ms":17,"camera.source":"placeholder","stream.producer_fps":58,"stream.false_positive":true} -``` - -`span.status=ok` on that 17 ms interval is the proof the overlay was a consumer false positive. A producer that actually stopped is a different span: `span.status=error`, `camera.frame_interval_ms` omitted, `stream.producer_fps=0`, `stream.false_positive=false`. Histogram buckets are unchanged. `trace.id` is not a bucket key. - -## Span exemplars - -The tables above say which fields a span carries. These lines are the join: the error-repro enums sit on the timing span for the same attempt, sharing `trace.id` with the event. `span.status` is about the span's own contract (did send return, did the first frame arrive, did shutdown finish), not about guest effect. - -### `stream.startup` — #103 is not a slow TTFF - -Preview went live, then stuck on Connecting while `/stream.mjpeg` kept transferring. First-frame delivery succeeded. The stuck overlay is a later `stream.state` on the same trace, not a `stream.ttff_ms` in the 5000 bucket. - -```json -{"span":"stream.startup","trace.id":"aabbccddeeff0011","span.id":"11001100","parent.span_id":"","span.status":"ok","stream.ttff_ms":180,"boot_ms":0,"ws_ms":40,"indigo_ms":90,"first_jpeg_ms":50,"stream.consumer":"mjpeg_img","stream.saw_first_frame":true} -{"event":"stream.state","trace.id":"aabbccddeeff0011","span.id":"11001100","stream.state":"connecting","stream.prev_state":"live","stream.saw_first_frame":true,"stream.last_frame_age_ms":1400,"stream.producer_ok":true,"stream.consumer":"mjpeg_img","stream.ws_close_code":0,"stream.ws_close_reason":"","stream.recovered":false} -``` - -Read: `span.status=ok` and `stream.ttff_ms=180` with `stream.saw_first_frame=true` and `stream.producer_ok=true` is the #103 consumer regression. A never-connected preview is a different line: `span.status=error`, `stream.saw_first_frame=false`, `stream.ttff_ms` omitted, `stream.last_frame_age_ms=-1`. Do not bucket that with the 180 ms success. - -### `camera.helper_shutdown` — #143 signal vs exit 0 - -The number #143 was missing is how long shutdown took, and whether it ended in a signal. `camera.placeholder_join_ms` and `camera.socket_close_ms` are attributes of this span, not their own histograms. - -```json -{"event":"camera.helper.exit","trace.id":"bbccddeeff001122","span.id":"22002200","error.code":"helper_signaled","camera.helper_exit_code":null,"camera.helper_signal":11,"camera.shutdown_phase":"release_surfaces","camera.placeholder_joined":false,"camera.source":"placeholder"} -{"span":"camera.helper_shutdown","trace.id":"bbccddeeff001122","span.id":"22002200","parent.span_id":"","span.status":"error","camera.helper_shutdown_ms":48,"camera.placeholder_join_ms":0,"camera.socket_close_ms":12,"camera.helper_exit_code":null,"camera.helper_signal":11} -``` - -`span.status=error` only when `camera.helper_signal` is set or `camera.helper_exit_code` is non-zero. After #161 the regression canary is the same span with `span.status=ok`, `camera.helper_signal=null`, `camera.placeholder_joined=true`, and a non-zero `camera.placeholder_join_ms` (the cancel handler actually ran). A 48 ms shutdown that signaled is not the same bucket story as a 48 ms clean exit: the status and signal are the split, the duration is not. - -### `input.send` — same duration, different repro - -Restated so the span line is not only implied by the event. Portrait hit and landscape miss share `input.send_ms`. Guest effect stays on the event (`inject.guest_effect`); the span stays `ok` while `hid.state=ok` and `inject.result=sent`. - -```json -{"span":"input.send","trace.id":"c0ffee00c0ffee00","span.id":"c0ffee01","parent.span_id":"","span.status":"ok","input.send_ms":4,"inject.kind":"touch","inject.result":"sent","hid.state":"ok","input.orientation":"landscape_left","input.frame":"display","input.remapped":false} -{"event":"input.inject","trace.id":"c0ffee00c0ffee00","span.id":"c0ffee01","hid.state":"ok","inject.result":"sent","inject.ack":"none","inject.guest_effect":"unchanged","input.orientation":"landscape_left","input.frame":"display","input.remapped":false,"cli.exit_code":0} -``` - -#136 is the other 4 ms: `hid.state=gone` forces `span.status=error` and a non-zero `cli.exit_code` on the event. No `input.roundtrip` span is emitted while `inject.ack=none`. - -### `stream.reconnect` — #128 false stall is not a slow reconnect - -`stream.reconnect_ms` is only recorded when the consumer returns to `live`. A false stall must not close that span and must not land in the reconnect histogram. - -[#128](https://github.com/EvanBacon/serve-sim/issues/128) is `stream.false_positive=true`: the producer was still publishing (`stream.producer_fps` > 0) and the `` watchdog fired after rotation. Emit `stream.stall` (already exemplified above). Do **not** emit `stream.reconnect`. If a rotate started the trace, the stall span may share `trace.id` with `input.inject`; it still does not parent a reconnect span. `span.status` stays `unset` so a 6100 ms consumer age is not a 6100 ms reconnect. - -[#103](https://github.com/EvanBacon/serve-sim/issues/103) after a good frame is the opposite. Leave `live`, never return. The span stays open until the consumer is stopped or the process exits, then ends `span.status=error` and **omits** `stream.reconnect_ms`. A missing duration is the stuck Connecting, not a 0. Carry `stream.saw_first_frame=true` and `stream.producer_ok=true` so this is not bucketed with a never-connected `stream.startup` (`span.status=error`, no `stream.ttff_ms`). A later recovery ends a new span `ok` with `stream.reconnect_ms` set and `stream.recovered=true`. - -```json -{"event":"stream.stall","trace.id":"128f0000128f0000","span.status":"unset","stream.false_positive":true,"stream.stall_reason":"consumer_stalled_producer_live","stream.orientation":"landscape_left","stream.producer_fps":58,"stream.last_frame_age_ms":6100} -{"span":"camera.frame","trace.id":"128f0000128f0000","span.id":"128f0001","span.status":"ok","camera.frame_interval_ms":17,"camera.source":"placeholder","stream.producer_fps":58,"stream.false_positive":true} -{"span":"stream.reconnect","trace.id":"103c0000103c0000","span.id":"103c0002","parent.span_id":"103c0001","span.status":"error","stream.stall_reason":"producer_silent","stream.orientation":"portrait","stream.saw_first_frame":true,"stream.producer_ok":true,"stream.reconnect_attempt":1,"stream.recovered":false} -``` - -The #128 trace has a 17 ms `camera.frame` and no `stream.reconnect`. The #103 line has no `stream.reconnect_ms`. A helper that died on a signal is `camera.helper_shutdown` with `camera.helper_signal` set, not a long `camera.frame_interval_ms` (that span is omitted when `stream.producer_fps=0`). - -## Phased rollout - -1. **Local events.** Structured JSON-lines when telemetry is `local`. Same schema as the tables above. `--list` grows `hid.state`, `hid.last_ok_at_ms`, and `stream.last_frame_age_ms` so an agent can see a dead injector without enabling telemetry. No new dependency. -2. **Optional OTLP.** When telemetry is `remote` and `OTEL_EXPORTER_OTLP_ENDPOINT` is set, export the same spans and events. Map `serve_sim.version` → `service.version`, `os.version` → `os.version`, `os.type=darwin`. Histograms use the buckets above. No endpoint is built in. Unset endpoint means stay on phase 1 even if the flag says remote. -3. **Default remains off.** Absence of the env var is off, in CI and on a laptop. - -## Opt-out patterns elsewhere - -Short notes on switches worth copying, and one worth avoiding. Not a survey of their event catalogs. - -- **Maestro** posts anonymous analytics to PostHog and used to prompt on stdin, which hung CI. Opt-out is `MAESTRO_CLI_NO_ANALYTICS=true` ([env docs](https://docs.maestro.dev/maestro-cli/environment-variables), [prompt bug](https://github.com/mobile-dev-inc/Maestro/issues/1846)). serve-sim does not prompt. -- **Homebrew** analytics are on after an install notice, with `HOMEBREW_NO_ANALYTICS=1` or `brew analytics off` ([docs](https://docs.brew.sh/Analytics)). serve-sim stays off until asked. -- **Appium** does not document a product analytics backend. `appium:eventTimings` is in-session timing, default false ([caps](https://appium.io/docs/en/latest/guides/caps/)). XCUITest, facebook/idb, and WebDriverAgent are local processes with no product export. -- **OpenTelemetry** sends nothing until an exporter endpoint is configured. The Collector is something you run ([JS exporters](https://opentelemetry.io/docs/languages/js/exporters/)). Phase 2 follows that: opt-in flag plus an operator-supplied endpoint. +| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always normalized to the displayed frame | +| `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `screenConfig().orientation` (i.e. `this.orientation`) read **at record time** | +| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `display` for the web preview and `serve-sim tap`; `native_portrait` after a remap | +| `remapped` | bool | whether this sender applied `rawPointForDisplayPoint` before the native call; the web client sets `true`, today's CLI/server path sets `false` | +| `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | +| `screen` | `{width,height}` | already recorded via `eventLogScreen()` | +| `screen_changed` | bool | guest-effect signal (see below) | + +**`source` (cli vs web), compatibly.** Both CLI and browser touches arrive over the +same socket (`attachHidSocket` -> `handleHidMessage`) and are recorded as +`source:"hid"`, so the log can't say which client sent them. Add an extra key to the +`0x03` JSON payload -- `{type,x,y,src:"cli"}` from `tap()`/`gesture()` in `index.ts`, +`{...,src:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`. It is an +extra JSON field: the native `touch()` call ignores it and older clients that omit it +record `source:"unknown"`. (A dedicated tag is unnecessary; the payload is already JSON.) + +**Keeping `orientation` current when rotation happens outside `0x07`.** For non-Duo +devices `this.orientation` is only updated when a `0x07` rotate message is handled +(`device-session.ts`: `if (this.duo.hinge == null) this.orientation = m.orientation`). +A rotation triggered from the Simulator menu or forced by the app is **not** reflected. +Two real hooks fix this: (1) `onSharedMjpegFrame` already observes every capture frame +and calls `broadcastConfig()` when `width`/`height` swap -- use that dimension swap to +refresh orientation (portrait<->landscape) rather than caching it per gesture; (2) mirror +the Duo path, which polls on a 1 s timer (`DuoStateMonitor.refreshOrientation` in +`duo-state.ts`), with a lightweight orientation poll for non-Duo sessions. Either way, +read the orientation at the moment the entry is recorded. + +**Did the screen change after the tap, without keeping pixels.** `onSharedMjpegFrame` +receives every frame's bytes and dimensions. Maintain a monotonic `frameSeq` and a +cheap content **seed** (e.g. a hash of JPEG length plus a few sampled bytes, or the +IOSurface seed if the capture exposes it) -- not the image. When a tap entry is +recorded, remember the current seed; when the next frame with a different seed arrives +within a short window, `updateEventLogEvent(tapId, { details: { ..., screen_changed: true }})`. +Store only the boolean (and optionally the seq delta). This is the guest-effect evidence +that an ack can't provide. + +## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) + +Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's +`0x0e {ok}` frame, times out at 10 s, and rejects on failure (`index.ts`). `tap`, +`gesture`, `rotate`, and `button` are **fire-and-forget** -- they write the frame, +wait a 40-50 ms timer, close the socket, and exit 0 unless the socket itself failed. + +The default we want is the `fold` pattern for all input: send, wait for a server ack +frame, and exit non-zero when it reports failure. Record the outcome on the entry +(`details.ack = ok | timeout | rejected`, plus `status`). Be explicit about scope: + +- An ack **proves** the server received the frame, `handleHidMessage` ran, and the + native injector returned. The injector boundary is `NativeHid.touch` -> + `SimHIDHandle.touch` (N-API), wrapped by `guard()` in `native.ts`. `guard()` + currently **swallows** N-API coercion errors and returns a fallback, so a malformed + send looks successful -- record whether `guard()` swallowed an error + (`inject.result = sent | swallowed | threw`) so #136's dead-injector case is visible. +- An ack **does not prove** the guest UI reacted. That is exactly the #153 gap, and why + `screen_changed` above is the separate, stronger signal. + +## HID health and Device Hub shadowing (#136, and a #153 look-alike) + +- **`hid.state`** (`ok` | `stale` | `gone`): HID can die while the pidfile and + `--list` still say `running: true`. Derive it from the last successful native call + (`SimHIDHandle` constructed in `native.ts`; sends go through `this.hid.*` in + `handleHidMessage`) and surface it in `--list` so `running:true` + `hid.state:gone` + is one object. +- **Device Hub input shadowing.** `src/device-hub-input.ts` checks + `com.apple.coredevice.dtuhidd.active` (`isDeviceHubInputShadowed` / + `warnDeviceHubInput`). When Device Hub owns input, legacy HID touches silently no-op -- + which looks exactly like a #153 coordinate miss. Record the shadow state on the + session (and on any inject that produced no `screen_changed`) so the two are + distinguishable; `serve-sim repair-input` is the human-only recovery (it restarts + SpringBoard and closes apps). + +## Helper lifecycle (#102) + +The orphan in #102 was a detached helper still running after the simulator was +`Shutdown`. The real hooks are `classifyStaleState(state, bootedUdids, pid)` -> +`keep | recycle-self | recycle-helper` and the pure-state watchdog +`startSimulatorShutdownWatch(targets, ...)` (no timers; driven by `tick`), both in +`src/helper-lifecycle.ts`. Record one `helper.lifecycle` entry when the watchdog +decides to recycle/exit after shutdown, carrying `sim.booted` and helper uptime. +(There is no separate `serve-sim-bin` process today -- HID and capture run in-process +via the native addon; don't name a binary that doesn't exist.) + +## Stream and camera (correcting earlier mistakes) + +- **#128 is a consumer false-stall, not a dead producer.** The watchdog that fires + "Stream is not producing frames" after a rotation is on the **`fetch()` reader** + path (the `raw=1` MJPEG consumer read through `body.getReader()`), not the `` + element. Record a stall entry with the reason and a `producer_live` boolean (the + capture side is still publishing) so a false stall isn't read as a dead stream. +- **Camera helper exit (#143).** Run the camera helper under a **supervising parent** + that captures both exit code and signal -- Node reports a `null` exit code when the + child dies on a signal. Record `camera.helper.exit` with `exit_code`, `signal`, and + `shutdown_phase`. **SIGPIPE is already ignored**, so a client vanishing mid-write is + not the cause; the real race was the placeholder timer vs. surface release (mitigated + by [#161](https://github.com/EvanBacon/serve-sim/pull/161)). Don't attribute it to SIGPIPE. + +Keep both as ordinary event-log entries -- no span parenting, no `stream.startup` -> +`camera.frame` tree, no reconnect histograms. + +## Privacy and export (Evan's defaults) + +- **Local event log is ON by default.** It is the channel bots read. It stores enums, + normalized coordinates, screen dimensions, versions, and coarse status -- never pixels, + app/accessibility text, clipboard, credentials, or raw command strings with tokens. +- **Remote export is opt-in and serve-sim-specific.** Enable export only through a + serve-sim variable (e.g. `SERVE_SIM_TELEMETRY=remote` with `SERVE_SIM_OTLP_ENDPOINT`). + It must **never** inherit a stray generic `OTEL_EXPORTER_OTLP_ENDPOINT` from the + environment -- that is how CI and agents would leak without asking. Unset serve-sim + variable = no export, in CI and on a laptop. +- **On export, drop**: pixels/frames, app & accessibility text, clipboard, + credentials/tokens/`Authorization`, raw UDIDs and device names (export a hash), + home-directory paths (basenames only), raw WebSocket payloads, and the local-only + absolute `x`/`y`. +- **Debug logs are separate and local.** `DEBUG=serve-sim:*` enables the namespaced + `debug()` streams (`serve-sim:cli|helper|state|mw` in `src/debug.ts`); + `SERVE_SIM_DEBUG_HID=1` enables per-event HID stderr. These go to the terminal of + whoever set them and are never attached to an export. (There is no `SERVE_SIM_DEBUG` + variable.) + +## Test plan + +Docs-only; the field additions above are small and additive, each at a named hook. +Prove recordability with unit tests (`bun test`) and the maintainer loop in +`AGENTS.md` (`build.ts` -> run from `dist/` -> reproduce): + +- Extend `src/__tests__/event-log.test.ts`: `eventLogEventForHidMessage("UDID", 0x03, + {type:"end", x, y, src:"cli"})` records `source:"cli"`; omitting `src` -> `"unknown"`; + assert `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. +- A `device-session` test: record a tap, deliver a differing `onSharedMjpegFrame` + seed -> `updateEventLogEvent` sets `screen_changed:true`; an identical seed -> `false`. +- `src/__tests__/device-hub-input.test.ts` already covers `isDeviceHubInputShadowed`; + add that a shadowed session stamps the shadow state on the session/inject entry. +- `src/__tests__/helper-lifecycle.test.ts` already covers `classifyStaleState`; assert + the recycle-after-shutdown path records a `helper.lifecycle` entry. +- Read-back: `GET /api/event-log` and `serve-sim event-log --json` return the new + fields and a `schema_version`. From 53fe9e7f0251424ad49f49ea685a6cb87c0aa0fd Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Thu, 8 Oct 2026 09:12:38 -0700 Subject: [PATCH 09/27] docs: put timing durations on the event-log entries Error-repro fields already land on the existing log. Timing is the same entries, not a span tree: input.send_ms, stream.ttff_ms, stream.reconnect_ms, camera.frame_interval_ms, camera.helper_shutdown_ms. Omit a duration when it would bucket a #128 false stall or a #103 stuck Connecting with a real reconnect, and omit input.send_ms when hid.state=gone so a dead injector is not a 4ms send. --- docs/observability.md | 66 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/docs/observability.md b/docs/observability.md index 553b0becc..6be39f328 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -11,6 +11,12 @@ make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/1 log alone, and the exact code hook that produces each field. It is built on the log we have -- no span tree, no histograms, no OpenTelemetry SDK is implied. +Timing is the same entries. A duration is an additive key (`input.send_ms`, +`stream.ttff_ms`, `stream.reconnect_ms`, `camera.frame_interval_ms`, +`camera.helper_shutdown_ms`). Omit the key when the number would be a lie +(false stall, stuck Connecting, dead injector). Do not invent a parent span +to carry it. + The event log is a diagnostic side-channel. It must never fail the input or command path (`recordEventLogEvent` already swallows subscriber errors), and it is **on by default**; remote export is strictly opt-in (see Privacy below). @@ -79,6 +85,7 @@ is built -- `recordTouchEvent` / `eventLogEventForHidMessage`): | `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | | `screen` | `{width,height}` | already recorded via `eventLogScreen()` | | `screen_changed` | bool | guest-effect signal (see below) | +| `input.send_ms` | number, omitted on a dead injector | socket write to ack or fire-and-forget close (see Timing) | **`source` (cli vs web), compatibly.** Both CLI and browser touches arrive over the same socket (`attachHidSocket` -> `handleHidMessage`) and are recorded as @@ -108,6 +115,17 @@ within a short window, `updateEventLogEvent(tapId, { details: { ..., screen_chan Store only the boolean (and optionally the seq delta). This is the guest-effect evidence that an ack can't provide. +Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields: + +```json +{"kind":"tap","status":"ok","details":{"orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} +{"kind":"tap","status":"ok","details":{"orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}} +``` + +The miss is `hid.state=ok` + `inject.result=sent` + `frame=display` + `remapped=false` + +`screen_changed=false`. Status stays `ok`: the injector did what it was asked. A later +remap expectation is the same entry shape with `frame=native_portrait` and `remapped=true`. + ## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's @@ -143,6 +161,18 @@ frame, and exit non-zero when it reports failure. Record the outcome on the entr distinguishable; `serve-sim repair-input` is the human-only recovery (it restarts SpringBoard and closes apps). +#136 is the other 4 ms, and it is not a send duration. `hid.state=gone` sets +`status=error` and **omits** `input.send_ms`, so a dead injector is not bucketed with +the landscape miss: + +```json +{"kind":"tap","status":"error","details":{"hid.state":"gone","inject.result":"threw","ack":"rejected","screen_changed":false}} +``` + +Device Hub shadowing is the third look-alike: `hid.state=ok`, `inject.result=sent`, +`remapped=true`, `screen_changed=false`, plus `input.shadowed=true`. Same omit rule +does not apply -- the send completed -- but the shadow key is what separates it from #153. + ## Helper lifecycle (#102) The orphan in #102 was a detached helper still running after the simulator was @@ -171,6 +201,39 @@ via the native addon; don't name a binary that doesn't exist.) Keep both as ordinary event-log entries -- no span parenting, no `stream.startup` -> `camera.frame` tree, no reconnect histograms. +## Timing on those entries + +Durations are measured at the hook that already builds the entry, then written with +`updateEventLogEvent` if the end is later than the start (ack, next frame, helper +exit). Units are milliseconds. Missing key means "not a duration", not zero. + +| Key | Entry | Record when | Omit when | +|---|---|---|---| +| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136). A landscape miss (#153) still records it. | +| `stream.ttff_ms` | first `stream.frame` | session start to first published frame | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | +| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | #128 `producer_live=true` (false stall). #103 stuck Connecting after a good frame (`producer_live=true`, never returns). | +| `camera.frame_interval_ms` | `camera.frame` | gap since previous published camera frame | `producer_fps=0` / capture stopped. A long gap is not an interval. | +| `camera.helper_shutdown_ms` | `camera.helper.exit` | supervising parent saw exit, including signal | never. A 48 ms SIGTERM is still `status=error`. | + +Worked lines (one entry each; no second span line): + +```json +{"kind":"stream.stall","status":"ok","details":{"producer_live":true,"orientation":"landscape_left","producer_fps":58,"last_frame_age_ms":6100,"camera.frame_interval_ms":17}} +{"kind":"stream.state","status":"error","details":{"producer_live":true,"saw_first_frame":true,"phase":"connecting"}} +{"kind":"stream.state","status":"ok","details":{"phase":"live","saw_first_frame":true,"stream.reconnect_ms":840}} +{"kind":"stream.frame","status":"ok","details":{"stream.ttff_ms":180,"saw_first_frame":true}} +{"kind":"camera.helper.exit","status":"error","details":{"signal":"SIGTERM","exit_code":null,"shutdown_phase":"placeholder","camera.helper_shutdown_ms":48,"placeholder_joined":false}} +{"kind":"camera.helper.exit","status":"ok","details":{"signal":null,"exit_code":0,"shutdown_phase":"surface_released","camera.helper_shutdown_ms":48,"placeholder_joined":true}} +``` + +#128 stays a stall entry. `camera.frame_interval_ms=17` says the producer was fine; +`stream.reconnect_ms` is absent, so the 6100 ms consumer age is not a reconnect. +#103 after a good frame is `stream.state` with `status=error` and no `stream.reconnect_ms` +and no `stream.ttff_ms` (TTFF already happened on the first-frame entry). A later +recovery is a new `stream.state` with `stream.reconnect_ms` set. #143 records the +shutdown duration on both the signaled exit and the post-#161 canary; status, not the +milliseconds, says which one failed. + ## Privacy and export (Evan's defaults) - **Local event log is ON by default.** It is the channel bots read. It stores enums, @@ -200,11 +263,14 @@ Prove recordability with unit tests (`bun test`) and the maintainer loop in - Extend `src/__tests__/event-log.test.ts`: `eventLogEventForHidMessage("UDID", 0x03, {type:"end", x, y, src:"cli"})` records `source:"cli"`; omitting `src` -> `"unknown"`; assert `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. + A landscape miss records `input.send_ms`; `hid.state=gone` omits it. - A `device-session` test: record a tap, deliver a differing `onSharedMjpegFrame` seed -> `updateEventLogEvent` sets `screen_changed:true`; an identical seed -> `false`. - `src/__tests__/device-hub-input.test.ts` already covers `isDeviceHubInputShadowed`; add that a shadowed session stamps the shadow state on the session/inject entry. - `src/__tests__/helper-lifecycle.test.ts` already covers `classifyStaleState`; assert the recycle-after-shutdown path records a `helper.lifecycle` entry. +- Stream: a `producer_live` stall does not set `stream.reconnect_ms`; a return to + `live` does. First frame sets `stream.ttff_ms`; a never-connected preview does not. - Read-back: `GET /api/event-log` and `serve-sim event-log --json` return the new fields and a `schema_version`. From 28c49d2bad3e4032c07bfa337e6e5365a40e6ccf Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Fri, 9 Oct 2026 09:12:12 -0700 Subject: [PATCH 10/27] docs: pair timing keys with error-repro fields A duration is only recorded when the repro fields say the interval happened. #102 helper recycle now carries helper.uptime_ms; Device Hub shadow keeps input.send_ms and is not a #153 miss. --- docs/observability.md | 51 ++++++++++++++++++++++++++++--------------- 1 file changed, 34 insertions(+), 17 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 6be39f328..86de71f26 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -13,9 +13,9 @@ log we have -- no span tree, no histograms, no OpenTelemetry SDK is implied. Timing is the same entries. A duration is an additive key (`input.send_ms`, `stream.ttff_ms`, `stream.reconnect_ms`, `camera.frame_interval_ms`, -`camera.helper_shutdown_ms`). Omit the key when the number would be a lie -(false stall, stuck Connecting, dead injector). Do not invent a parent span -to carry it. +`camera.helper_shutdown_ms`, `helper.uptime_ms`). Omit the key when the number +would be a lie (false stall, stuck Connecting, dead injector, still-booted +helper). Do not invent a parent span to carry it. The event log is a diagnostic side-channel. It must never fail the input or command path (`recordEventLogEvent` already swallows subscriber errors), and it @@ -115,16 +115,17 @@ within a short window, `updateEventLogEvent(tapId, { details: { ..., screen_chan Store only the boolean (and optionally the seq delta). This is the guest-effect evidence that an ack can't provide. -Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields: +Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields. +`source` is part of the signature: today's miss is the CLI/server path, not the web client. ```json -{"kind":"tap","status":"ok","details":{"orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} -{"kind":"tap","status":"ok","details":{"orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}} +{"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} +{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}} ``` The miss is `hid.state=ok` + `inject.result=sent` + `frame=display` + `remapped=false` + -`screen_changed=false`. Status stays `ok`: the injector did what it was asked. A later -remap expectation is the same entry shape with `frame=native_portrait` and `remapped=true`. +`screen_changed=false` + `source=cli`. Status stays `ok`: the injector did what it was asked. +A later remap expectation is the same entry shape with `frame=native_portrait` and `remapped=true`. ## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) @@ -166,12 +167,16 @@ frame, and exit non-zero when it reports failure. Record the outcome on the entr the landscape miss: ```json -{"kind":"tap","status":"error","details":{"hid.state":"gone","inject.result":"threw","ack":"rejected","screen_changed":false}} +{"kind":"tap","status":"error","details":{"source":"cli","hid.state":"gone","inject.result":"threw","ack":"rejected","screen_changed":false}} ``` -Device Hub shadowing is the third look-alike: `hid.state=ok`, `inject.result=sent`, -`remapped=true`, `screen_changed=false`, plus `input.shadowed=true`. Same omit rule -does not apply -- the send completed -- but the shadow key is what separates it from #153. +Device Hub shadowing is the third look-alike: the send completed, so `input.send_ms` +stays. `input.shadowed=true` is what separates it from #153 (which has `remapped=false` +and `input.shadowed` absent or false): + +```json +{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","input.shadowed":true,"screen_changed":false,"input.send_ms":4}} +``` ## Helper lifecycle (#102) @@ -184,6 +189,10 @@ decides to recycle/exit after shutdown, carrying `sim.booted` and helper uptime. (There is no separate `serve-sim-bin` process today -- HID and capture run in-process via the native addon; don't name a binary that doesn't exist.) +`classifyStaleState` returning `keep` does **not** emit the entry and does not record +`helper.uptime_ms`. A still-booted sim is not a recycle duration. The orphan is the +entry with `sim.booted=false` and `decision=recycle-helper`. + ## Stream and camera (correcting earlier mistakes) - **#128 is a consumer false-stall, not a dead producer.** The watchdog that fires @@ -207,13 +216,17 @@ Durations are measured at the hook that already builds the entry, then written w `updateEventLogEvent` if the end is later than the start (ack, next frame, helper exit). Units are milliseconds. Missing key means "not a duration", not zero. +A reader pairs each duration with the repro fields on the **same** entry. The fields +decide whether the key is present; they do not live on a second line. + | Key | Entry | Record when | Omit when | |---|---|---|---| -| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136). A landscape miss (#153) still records it. | +| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136). A landscape miss (#153) and a Device Hub shadow still record it. | | `stream.ttff_ms` | first `stream.frame` | session start to first published frame | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | | `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | #128 `producer_live=true` (false stall). #103 stuck Connecting after a good frame (`producer_live=true`, never returns). | -| `camera.frame_interval_ms` | `camera.frame` | gap since previous published camera frame | `producer_fps=0` / capture stopped. A long gap is not an interval. | +| `camera.frame_interval_ms` | `camera.frame` or `stream.stall` | gap since previous published camera frame | `producer_fps=0` / capture stopped. A long gap is not an interval. #128 may carry the last good interval on the stall entry. | | `camera.helper_shutdown_ms` | `camera.helper.exit` | supervising parent saw exit, including signal | never. A 48 ms SIGTERM is still `status=error`. | +| `helper.uptime_ms` | `helper.lifecycle` | watchdog decision is `recycle-helper` or `recycle-self` after shutdown | `classifyStaleState` returned `keep`, or pid unknown. #102 is the recycle, not a heartbeat. | Worked lines (one entry each; no second span line): @@ -224,6 +237,7 @@ Worked lines (one entry each; no second span line): {"kind":"stream.frame","status":"ok","details":{"stream.ttff_ms":180,"saw_first_frame":true}} {"kind":"camera.helper.exit","status":"error","details":{"signal":"SIGTERM","exit_code":null,"shutdown_phase":"placeholder","camera.helper_shutdown_ms":48,"placeholder_joined":false}} {"kind":"camera.helper.exit","status":"ok","details":{"signal":null,"exit_code":0,"shutdown_phase":"surface_released","camera.helper_shutdown_ms":48,"placeholder_joined":true}} +{"kind":"helper.lifecycle","status":"ok","details":{"decision":"recycle-helper","sim.booted":false,"helper.uptime_ms":86400000}} ``` #128 stays a stall entry. `camera.frame_interval_ms=17` says the producer was fine; @@ -232,7 +246,8 @@ Worked lines (one entry each; no second span line): and no `stream.ttff_ms` (TTFF already happened on the first-frame entry). A later recovery is a new `stream.state` with `stream.reconnect_ms` set. #143 records the shutdown duration on both the signaled exit and the post-#161 canary; status, not the -milliseconds, says which one failed. +milliseconds, says which one failed. #102 records `helper.uptime_ms` only on the recycle +entry; a still-booted `keep` is silence, not a zero. ## Privacy and export (Evan's defaults) @@ -263,13 +278,15 @@ Prove recordability with unit tests (`bun test`) and the maintainer loop in - Extend `src/__tests__/event-log.test.ts`: `eventLogEventForHidMessage("UDID", 0x03, {type:"end", x, y, src:"cli"})` records `source:"cli"`; omitting `src` -> `"unknown"`; assert `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. - A landscape miss records `input.send_ms`; `hid.state=gone` omits it. + A landscape miss records `input.send_ms`; `hid.state=gone` omits it; `input.shadowed=true` + still records it. - A `device-session` test: record a tap, deliver a differing `onSharedMjpegFrame` seed -> `updateEventLogEvent` sets `screen_changed:true`; an identical seed -> `false`. - `src/__tests__/device-hub-input.test.ts` already covers `isDeviceHubInputShadowed`; add that a shadowed session stamps the shadow state on the session/inject entry. - `src/__tests__/helper-lifecycle.test.ts` already covers `classifyStaleState`; assert - the recycle-after-shutdown path records a `helper.lifecycle` entry. + the recycle-after-shutdown path records a `helper.lifecycle` entry with `helper.uptime_ms`, + and `keep` records nothing. - Stream: a `producer_live` stall does not set `stream.reconnect_ms`; a return to `live` does. First frame sets `stream.ttff_ms`; a never-connected preview does not. - Read-back: `GET /api/event-log` and `serve-sim event-log --json` return the new From 04e683ea7597a1956ed61f153e228769097fbec6 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Sat, 10 Oct 2026 09:14:59 -0700 Subject: [PATCH 11/27] docs(observability): omit input.send_ms on ack timeout --- docs/observability.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 86de71f26..41f26169c 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -221,7 +221,7 @@ decide whether the key is present; they do not live on a second line. | Key | Entry | Record when | Omit when | |---|---|---|---| -| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136). A landscape miss (#153) and a Device Hub shadow still record it. | +| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136), or `ack=timeout` (the 10 s wait is not a send). A landscape miss (#153) and a Device Hub shadow still record it. | | `stream.ttff_ms` | first `stream.frame` | session start to first published frame | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | | `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | #128 `producer_live=true` (false stall). #103 stuck Connecting after a good frame (`producer_live=true`, never returns). | | `camera.frame_interval_ms` | `camera.frame` or `stream.stall` | gap since previous published camera frame | `producer_fps=0` / capture stopped. A long gap is not an interval. #128 may carry the last good interval on the stall entry. | @@ -278,8 +278,8 @@ Prove recordability with unit tests (`bun test`) and the maintainer loop in - Extend `src/__tests__/event-log.test.ts`: `eventLogEventForHidMessage("UDID", 0x03, {type:"end", x, y, src:"cli"})` records `source:"cli"`; omitting `src` -> `"unknown"`; assert `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. - A landscape miss records `input.send_ms`; `hid.state=gone` omits it; `input.shadowed=true` - still records it. + A landscape miss records `input.send_ms`; `hid.state=gone` and `ack=timeout` omit it; + `input.shadowed=true` still records it. - A `device-session` test: record a tap, deliver a differing `onSharedMjpegFrame` seed -> `updateEventLogEvent` sets `screen_changed:true`; an identical seed -> `false`. - `src/__tests__/device-hub-input.test.ts` already covers `isDeviceHubInputShadowed`; From 4514e58516afb4f4b0be0bf0fb2cd56a293d9b06 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Sun, 11 Oct 2026 09:13:21 -0700 Subject: [PATCH 12/27] docs: clarify error-repro fields gate timing keys (2026-10-11 maintainer note) --- docs/observability.md | 294 +----------------------------------------- 1 file changed, 1 insertion(+), 293 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 41f26169c..d4b9d3ddb 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1,293 +1 @@ -# Observability - -serve-sim already keeps an in-memory **event log** of simulator actions. This -document describes what that log records today, the small, additive fields that -make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/153), -[#136](https://github.com/EvanBacon/serve-sim/issues/136), -[#143](https://github.com/EvanBacon/serve-sim/issues/143), -[#103](https://github.com/EvanBacon/serve-sim/issues/103), -[#128](https://github.com/EvanBacon/serve-sim/issues/128), -[#102](https://github.com/EvanBacon/serve-sim/issues/102)) reproducible from the -log alone, and the exact code hook that produces each field. It is built on the -log we have -- no span tree, no histograms, no OpenTelemetry SDK is implied. - -Timing is the same entries. A duration is an additive key (`input.send_ms`, -`stream.ttff_ms`, `stream.reconnect_ms`, `camera.frame_interval_ms`, -`camera.helper_shutdown_ms`, `helper.uptime_ms`). Omit the key when the number -would be a lie (false stall, stuck Connecting, dead injector, still-booted -helper). Do not invent a parent span to carry it. - -The event log is a diagnostic side-channel. It must never fail the input or -command path (`recordEventLogEvent` already swallows subscriber errors), and it -is **on by default**; remote export is strictly opt-in (see Privacy below). - -## What the event log records today - -`src/event-log.ts` keeps a bounded ring (`EVENT_LOG_MAX_ENTRIES = 500`) of: - -```ts -type EventLogEntry = { - id: number; timestamp: string; - source: "hid" | "exec" | "ui"; - kind: string; msg: string; summary: string; - device?: string; action?: string; - status?: "ok" | "error"; - details?: Record; -}; -``` - -- **HID input** -> `eventLogEventForHidMessage(device, tag, payload, screen)`, called - from `recordHidEvent` / `recordTouchEvent` inside `handleHidMessage` - (`src/device-session.ts`). Taps and drags are synthesized in `recordTouchEvent` - (tag `0x03` begin/move/end) and already carry `details.start`, `details.current`, - `details.moveCount`, and `details.screen = {width,height}` (via `eventLogScreen()`). -- **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, parsed in - `src/middleware.ts` from the `/exec` path. It already drops upload plumbing and - never records raw command strings with tokens. -- **UI settings** -> `recordEventLogEvent({source:"ui", ...})` in `middleware.ts`. - -It already redacts: printable keystrokes (tag `0x06` records `key:"character"`, -`redacted:true`, no usage), token-bearing commands (returns `null`), and it only -ever stores **normalized** coordinates plus screen dimensions -- never pixels. - -## How a bot reads it back - -- `GET {url}/api/event-log?device=&since=&limit=` -> `{ events: [...] }` - (`readEventLog` in `middleware.ts`). -- `GET {url}/api/event-log/events` -> SSE: a snapshot, then one `{ event }` per new - entry (`subscribeEventLog`; the browser UI reaches it through the exec-ws proxy). -- CLI: `serve-sim event-log --json [-d ] [--limit ]` - (`eventLog()` in `src/index.ts`, which fetches `/api/event-log`). - -Add a top-level **`schema_version`** to the `/api/event-log` (and SSE) response -envelope so a reader can tell which of the fields below are present. Every field -added here is additive; an older consumer ignores unknown keys. - -## Making the #153 touch miss reproducible from the log - -#153: a tap exits 0 and the screenshot is unchanged because the point was sent in -the wrong coordinate frame. The web client remaps display->native -(`rawPointForDisplayPoint` in `client/simulator/orientation.ts`, called from -`SimulatorView.tsx`); the CLI (`tap()` / `gesture()` in `index.ts`) and the -server path `handleHidMessage` -> `this.hid.touch(type, x, y, W, H, edge)` pass the -display-normalized point straight into the native portrait frame with **no remap**. -Nothing in the log distinguishes that miss from a dead injector. - -Add these keys to the `details` of touch/tap/drag entries (produced where the entry -is built -- `recordTouchEvent` / `eventLogEventForHidMessage`): - -| Field | Values | Source / hook | -|---|---|---| -| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always normalized to the displayed frame | -| `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `screenConfig().orientation` (i.e. `this.orientation`) read **at record time** | -| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `display` for the web preview and `serve-sim tap`; `native_portrait` after a remap | -| `remapped` | bool | whether this sender applied `rawPointForDisplayPoint` before the native call; the web client sets `true`, today's CLI/server path sets `false` | -| `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | -| `screen` | `{width,height}` | already recorded via `eventLogScreen()` | -| `screen_changed` | bool | guest-effect signal (see below) | -| `input.send_ms` | number, omitted on a dead injector | socket write to ack or fire-and-forget close (see Timing) | - -**`source` (cli vs web), compatibly.** Both CLI and browser touches arrive over the -same socket (`attachHidSocket` -> `handleHidMessage`) and are recorded as -`source:"hid"`, so the log can't say which client sent them. Add an extra key to the -`0x03` JSON payload -- `{type,x,y,src:"cli"}` from `tap()`/`gesture()` in `index.ts`, -`{...,src:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`. It is an -extra JSON field: the native `touch()` call ignores it and older clients that omit it -record `source:"unknown"`. (A dedicated tag is unnecessary; the payload is already JSON.) - -**Keeping `orientation` current when rotation happens outside `0x07`.** For non-Duo -devices `this.orientation` is only updated when a `0x07` rotate message is handled -(`device-session.ts`: `if (this.duo.hinge == null) this.orientation = m.orientation`). -A rotation triggered from the Simulator menu or forced by the app is **not** reflected. -Two real hooks fix this: (1) `onSharedMjpegFrame` already observes every capture frame -and calls `broadcastConfig()` when `width`/`height` swap -- use that dimension swap to -refresh orientation (portrait<->landscape) rather than caching it per gesture; (2) mirror -the Duo path, which polls on a 1 s timer (`DuoStateMonitor.refreshOrientation` in -`duo-state.ts`), with a lightweight orientation poll for non-Duo sessions. Either way, -read the orientation at the moment the entry is recorded. - -**Did the screen change after the tap, without keeping pixels.** `onSharedMjpegFrame` -receives every frame's bytes and dimensions. Maintain a monotonic `frameSeq` and a -cheap content **seed** (e.g. a hash of JPEG length plus a few sampled bytes, or the -IOSurface seed if the capture exposes it) -- not the image. When a tap entry is -recorded, remember the current seed; when the next frame with a different seed arrives -within a short window, `updateEventLogEvent(tapId, { details: { ..., screen_changed: true }})`. -Store only the boolean (and optionally the seq delta). This is the guest-effect evidence -that an ack can't provide. - -Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields. -`source` is part of the signature: today's miss is the CLI/server path, not the web client. - -```json -{"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} -{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}} -``` - -The miss is `hid.state=ok` + `inject.result=sent` + `frame=display` + `remapped=false` + -`screen_changed=false` + `source=cli`. Status stays `ok`: the injector did what it was asked. -A later remap expectation is the same entry shape with `frame=native_portrait` and `remapped=true`. - -## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) - -Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's -`0x0e {ok}` frame, times out at 10 s, and rejects on failure (`index.ts`). `tap`, -`gesture`, `rotate`, and `button` are **fire-and-forget** -- they write the frame, -wait a 40-50 ms timer, close the socket, and exit 0 unless the socket itself failed. - -The default we want is the `fold` pattern for all input: send, wait for a server ack -frame, and exit non-zero when it reports failure. Record the outcome on the entry -(`details.ack = ok | timeout | rejected`, plus `status`). Be explicit about scope: - -- An ack **proves** the server received the frame, `handleHidMessage` ran, and the - native injector returned. The injector boundary is `NativeHid.touch` -> - `SimHIDHandle.touch` (N-API), wrapped by `guard()` in `native.ts`. `guard()` - currently **swallows** N-API coercion errors and returns a fallback, so a malformed - send looks successful -- record whether `guard()` swallowed an error - (`inject.result = sent | swallowed | threw`) so #136's dead-injector case is visible. -- An ack **does not prove** the guest UI reacted. That is exactly the #153 gap, and why - `screen_changed` above is the separate, stronger signal. - -## HID health and Device Hub shadowing (#136, and a #153 look-alike) - -- **`hid.state`** (`ok` | `stale` | `gone`): HID can die while the pidfile and - `--list` still say `running: true`. Derive it from the last successful native call - (`SimHIDHandle` constructed in `native.ts`; sends go through `this.hid.*` in - `handleHidMessage`) and surface it in `--list` so `running:true` + `hid.state:gone` - is one object. -- **Device Hub input shadowing.** `src/device-hub-input.ts` checks - `com.apple.coredevice.dtuhidd.active` (`isDeviceHubInputShadowed` / - `warnDeviceHubInput`). When Device Hub owns input, legacy HID touches silently no-op -- - which looks exactly like a #153 coordinate miss. Record the shadow state on the - session (and on any inject that produced no `screen_changed`) so the two are - distinguishable; `serve-sim repair-input` is the human-only recovery (it restarts - SpringBoard and closes apps). - -#136 is the other 4 ms, and it is not a send duration. `hid.state=gone` sets -`status=error` and **omits** `input.send_ms`, so a dead injector is not bucketed with -the landscape miss: - -```json -{"kind":"tap","status":"error","details":{"source":"cli","hid.state":"gone","inject.result":"threw","ack":"rejected","screen_changed":false}} -``` - -Device Hub shadowing is the third look-alike: the send completed, so `input.send_ms` -stays. `input.shadowed=true` is what separates it from #153 (which has `remapped=false` -and `input.shadowed` absent or false): - -```json -{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","input.shadowed":true,"screen_changed":false,"input.send_ms":4}} -``` - -## Helper lifecycle (#102) - -The orphan in #102 was a detached helper still running after the simulator was -`Shutdown`. The real hooks are `classifyStaleState(state, bootedUdids, pid)` -> -`keep | recycle-self | recycle-helper` and the pure-state watchdog -`startSimulatorShutdownWatch(targets, ...)` (no timers; driven by `tick`), both in -`src/helper-lifecycle.ts`. Record one `helper.lifecycle` entry when the watchdog -decides to recycle/exit after shutdown, carrying `sim.booted` and helper uptime. -(There is no separate `serve-sim-bin` process today -- HID and capture run in-process -via the native addon; don't name a binary that doesn't exist.) - -`classifyStaleState` returning `keep` does **not** emit the entry and does not record -`helper.uptime_ms`. A still-booted sim is not a recycle duration. The orphan is the -entry with `sim.booted=false` and `decision=recycle-helper`. - -## Stream and camera (correcting earlier mistakes) - -- **#128 is a consumer false-stall, not a dead producer.** The watchdog that fires - "Stream is not producing frames" after a rotation is on the **`fetch()` reader** - path (the `raw=1` MJPEG consumer read through `body.getReader()`), not the `` - element. Record a stall entry with the reason and a `producer_live` boolean (the - capture side is still publishing) so a false stall isn't read as a dead stream. -- **Camera helper exit (#143).** Run the camera helper under a **supervising parent** - that captures both exit code and signal -- Node reports a `null` exit code when the - child dies on a signal. Record `camera.helper.exit` with `exit_code`, `signal`, and - `shutdown_phase`. **SIGPIPE is already ignored**, so a client vanishing mid-write is - not the cause; the real race was the placeholder timer vs. surface release (mitigated - by [#161](https://github.com/EvanBacon/serve-sim/pull/161)). Don't attribute it to SIGPIPE. - -Keep both as ordinary event-log entries -- no span parenting, no `stream.startup` -> -`camera.frame` tree, no reconnect histograms. - -## Timing on those entries - -Durations are measured at the hook that already builds the entry, then written with -`updateEventLogEvent` if the end is later than the start (ack, next frame, helper -exit). Units are milliseconds. Missing key means "not a duration", not zero. - -A reader pairs each duration with the repro fields on the **same** entry. The fields -decide whether the key is present; they do not live on a second line. - -| Key | Entry | Record when | Omit when | -|---|---|---|---| -| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136), or `ack=timeout` (the 10 s wait is not a send). A landscape miss (#153) and a Device Hub shadow still record it. | -| `stream.ttff_ms` | first `stream.frame` | session start to first published frame | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | -| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | #128 `producer_live=true` (false stall). #103 stuck Connecting after a good frame (`producer_live=true`, never returns). | -| `camera.frame_interval_ms` | `camera.frame` or `stream.stall` | gap since previous published camera frame | `producer_fps=0` / capture stopped. A long gap is not an interval. #128 may carry the last good interval on the stall entry. | -| `camera.helper_shutdown_ms` | `camera.helper.exit` | supervising parent saw exit, including signal | never. A 48 ms SIGTERM is still `status=error`. | -| `helper.uptime_ms` | `helper.lifecycle` | watchdog decision is `recycle-helper` or `recycle-self` after shutdown | `classifyStaleState` returned `keep`, or pid unknown. #102 is the recycle, not a heartbeat. | - -Worked lines (one entry each; no second span line): - -```json -{"kind":"stream.stall","status":"ok","details":{"producer_live":true,"orientation":"landscape_left","producer_fps":58,"last_frame_age_ms":6100,"camera.frame_interval_ms":17}} -{"kind":"stream.state","status":"error","details":{"producer_live":true,"saw_first_frame":true,"phase":"connecting"}} -{"kind":"stream.state","status":"ok","details":{"phase":"live","saw_first_frame":true,"stream.reconnect_ms":840}} -{"kind":"stream.frame","status":"ok","details":{"stream.ttff_ms":180,"saw_first_frame":true}} -{"kind":"camera.helper.exit","status":"error","details":{"signal":"SIGTERM","exit_code":null,"shutdown_phase":"placeholder","camera.helper_shutdown_ms":48,"placeholder_joined":false}} -{"kind":"camera.helper.exit","status":"ok","details":{"signal":null,"exit_code":0,"shutdown_phase":"surface_released","camera.helper_shutdown_ms":48,"placeholder_joined":true}} -{"kind":"helper.lifecycle","status":"ok","details":{"decision":"recycle-helper","sim.booted":false,"helper.uptime_ms":86400000}} -``` - -#128 stays a stall entry. `camera.frame_interval_ms=17` says the producer was fine; -`stream.reconnect_ms` is absent, so the 6100 ms consumer age is not a reconnect. -#103 after a good frame is `stream.state` with `status=error` and no `stream.reconnect_ms` -and no `stream.ttff_ms` (TTFF already happened on the first-frame entry). A later -recovery is a new `stream.state` with `stream.reconnect_ms` set. #143 records the -shutdown duration on both the signaled exit and the post-#161 canary; status, not the -milliseconds, says which one failed. #102 records `helper.uptime_ms` only on the recycle -entry; a still-booted `keep` is silence, not a zero. - -## Privacy and export (Evan's defaults) - -- **Local event log is ON by default.** It is the channel bots read. It stores enums, - normalized coordinates, screen dimensions, versions, and coarse status -- never pixels, - app/accessibility text, clipboard, credentials, or raw command strings with tokens. -- **Remote export is opt-in and serve-sim-specific.** Enable export only through a - serve-sim variable (e.g. `SERVE_SIM_TELEMETRY=remote` with `SERVE_SIM_OTLP_ENDPOINT`). - It must **never** inherit a stray generic `OTEL_EXPORTER_OTLP_ENDPOINT` from the - environment -- that is how CI and agents would leak without asking. Unset serve-sim - variable = no export, in CI and on a laptop. -- **On export, drop**: pixels/frames, app & accessibility text, clipboard, - credentials/tokens/`Authorization`, raw UDIDs and device names (export a hash), - home-directory paths (basenames only), raw WebSocket payloads, and the local-only - absolute `x`/`y`. -- **Debug logs are separate and local.** `DEBUG=serve-sim:*` enables the namespaced - `debug()` streams (`serve-sim:cli|helper|state|mw` in `src/debug.ts`); - `SERVE_SIM_DEBUG_HID=1` enables per-event HID stderr. These go to the terminal of - whoever set them and are never attached to an export. (There is no `SERVE_SIM_DEBUG` - variable.) - -## Test plan - -Docs-only; the field additions above are small and additive, each at a named hook. -Prove recordability with unit tests (`bun test`) and the maintainer loop in -`AGENTS.md` (`build.ts` -> run from `dist/` -> reproduce): - -- Extend `src/__tests__/event-log.test.ts`: `eventLogEventForHidMessage("UDID", 0x03, - {type:"end", x, y, src:"cli"})` records `source:"cli"`; omitting `src` -> `"unknown"`; - assert `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. - A landscape miss records `input.send_ms`; `hid.state=gone` and `ack=timeout` omit it; - `input.shadowed=true` still records it. -- A `device-session` test: record a tap, deliver a differing `onSharedMjpegFrame` - seed -> `updateEventLogEvent` sets `screen_changed:true`; an identical seed -> `false`. -- `src/__tests__/device-hub-input.test.ts` already covers `isDeviceHubInputShadowed`; - add that a shadowed session stamps the shadow state on the session/inject entry. -- `src/__tests__/helper-lifecycle.test.ts` already covers `classifyStaleState`; assert - the recycle-after-shutdown path records a `helper.lifecycle` entry with `helper.uptime_ms`, - and `keep` records nothing. -- Stream: a `producer_live` stall does not set `stream.reconnect_ms`; a return to - `live` does. First frame sets `stream.ttff_ms`; a never-connected preview does not. -- Read-back: `GET /api/event-log` and `serve-sim event-log --json` return the new - fields and a `schema_version`. +$(cat /tmp/observability.md) \ No newline at end of file From d5450ec84d7585dbe3efd244aa709674986e9508 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Mon, 12 Oct 2026 09:39:08 -0700 Subject: [PATCH 13/27] docs(observability): address Claude review objections Restore the broken tip file, clarify remapped vs observed_orientation, avoid writing poll results into broadcast orientation, fix screen_changed timing, stall field (producer_frame_age), helper lifecycle hooks, and note the PR #166 dependency for remote event-log readback. --- docs/observability.md | 293 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 292 insertions(+), 1 deletion(-) diff --git a/docs/observability.md b/docs/observability.md index d4b9d3ddb..b2d7aaadd 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1 +1,292 @@ -$(cat /tmp/observability.md) \ No newline at end of file +# Observability + +> Remote readback of `/api/event-log` needs the auth gate from [PR #166](https://github.com/EvanBacon/serve-sim/pull/166). Merge that before relying on off-loopback log access. + +serve-sim already keeps an in-memory **event log** of simulator actions. This +document describes what that log records today, the small, additive fields that +make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/153), +[#136](https://github.com/EvanBacon/serve-sim/issues/136), +[#143](https://github.com/EvanBacon/serve-sim/issues/143), +[#103](https://github.com/EvanBacon/serve-sim/issues/103), +[#128](https://github.com/EvanBacon/serve-sim/issues/128), +[#102](https://github.com/EvanBacon/serve-sim/issues/102)) reproducible from the +log alone, and the exact code hook that produces each field. It is built on the +log we have -- no span tree, no histograms, no OpenTelemetry SDK is implied. + +Timing is the same entries. A duration is an additive key (`input.send_ms`, +`stream.ttff_ms`, `stream.reconnect_ms`, `stream.producer_frame_age_ms`, +`camera.helper_shutdown_ms`, `helper.uptime_ms`). Omit the key when the number +would be a lie (false stall, stuck Connecting, dead injector, still-booted +helper). Do not invent a parent span to carry it. + +The event log is a diagnostic side-channel. It must never fail the input or +command path (`recordEventLogEvent` already swallows subscriber errors), and it +is **on by default**; remote export is strictly opt-in (see Privacy below). + +## What the event log records today + +`src/event-log.ts` keeps a bounded ring (`EVENT_LOG_MAX_ENTRIES = 500`) of: + +```ts +type EventLogEntry = { + id: number; timestamp: string; + source: "hid" | "exec" | "ui"; + kind: string; msg: string; summary: string; + device?: string; action?: string; + status?: "ok" | "error"; + details?: Record; +}; +``` + +- **HID input** -> `eventLogEventForHidMessage(device, tag, payload, screen)`, called + from `recordHidEvent` / `recordTouchEvent` inside `handleHidMessage` + (`src/device-session.ts`). Taps and drags are synthesized in `recordTouchEvent` + (tag `0x03` begin/move/end) and already carry `details.start`, `details.current`, + `details.moveCount`, and `details.screen = {width,height}` (via `eventLogScreen()`). +- **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, parsed in + `src/middleware.ts` from the `/exec` path. It already drops upload plumbing and + never records raw command strings with tokens. +- **UI settings** -> `recordEventLogEvent({source:"ui", ...})` in `middleware.ts`. + +It already redacts: printable keystrokes (tag `0x06` records `key:"character"`, +`redacted:true`, no usage), token-bearing commands (returns `null`), and it only +ever stores **normalized** coordinates plus screen dimensions -- never pixels. + +## How a bot reads it back + +- `GET {url}/api/event-log?device=&since=&limit=` -> `{ events: [...] }` + (`readEventLog` in `middleware.ts`). +- `GET {url}/api/event-log/events` -> SSE: a snapshot, then one `{ event }` per new + entry (`subscribeEventLog`; the browser UI reaches it through the exec-ws proxy). +- CLI: `serve-sim event-log --json [-d ] [--limit ]` + (`eventLog()` in `src/index.ts`, which fetches `/api/event-log`). + +Add a top-level **`schema_version`** to the `/api/event-log` (and SSE) response +envelope so a reader can tell which of the fields below are present. Every field +added here is additive; an older consumer ignores unknown keys. + +## Making the #153 touch miss reproducible from the log + +#153: a tap exits 0 and the screenshot is unchanged because the point was sent in +the wrong coordinate frame. The web client remaps display->native +(`rawPointForDisplayPoint` in `client/simulator/orientation.ts`, called from +`SimulatorView.tsx`); the CLI (`tap()` / `gesture()` in `index.ts`) and the +server path `handleHidMessage` -> `this.hid.touch(type, x, y, W, H, edge)` pass the +display-normalized point straight into the native portrait frame with **no remap**. +Nothing in the log distinguishes that miss from a dead injector. + +Add these keys to the `details` of touch/tap/drag entries (produced where the entry +is built -- `recordTouchEvent` / `eventLogEventForHidMessage`): + +| Field | Values | Source / hook | +|---|---|---| +| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always normalized to the displayed frame | +| `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `screenConfig().orientation` (i.e. `this.orientation`) read **at record time** | +| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `display` for the web preview and `serve-sim tap`; `native_portrait` after a remap | +| `remapped` | bool | `input_source === "web"` && the `needsCssRotation` rule applied to the **broadcast** `orientation` and raw size (what the browser actually used). Omitted for Duo. | +| `observed_orientation` | string | Polled UI orientation (`simctl io enumerate`); can lag ~1s. Do **not** write into `this.orientation` / `broadcastConfig()`. | +| `remap_expected` | bool | Rule applied to `observed_orientation` when present, else `orientation`. Omitted for Duo (`hingeDegrees != null`): Duo web touches use 3D projection, not `rawPointForDisplayPoint`. | +| `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | + +A web miss is `remap_expected=true`, `remapped=false`: the browser hadn't been told about the rotation. +| `screen` | `{width,height}` | already recorded via `eventLogScreen()` | +| `screen_changed` | bool | guest-effect signal (see below) | +| `input.send_ms` | number, omitted on a dead injector | socket write to ack or fire-and-forget close (see Timing) | + +**`source` (cli vs web), compatibly.** Both CLI and browser touches arrive over the +separate WebSocket connections to the same handler (`attachHidInProcess` -> `attachHidSocket` -> `handleHidMessage`) and are recorded as +`source:"hid"`, so the log can't say which client sent them. Add an extra key to the +`0x03` JSON payload -- `{type,x,y,src:"cli"}` from `tap()`/`gesture()` in `index.ts`, +`{...,src:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`. It is an +extra JSON field: the native `touch()` call ignores it and older clients that omit it +record `source:"unknown"`. (A dedicated tag is unnecessary; the payload is already JSON.) + +**Keeping `orientation` current when rotation happens outside `0x07`.** For non-Duo +devices `this.orientation` is only updated when a `0x07` rotate message is handled +(`device-session.ts`: `if (this.duo.hinge == null) this.orientation = m.orientation`). +A rotation triggered from the Simulator menu or forced by the app is **not** reflected. +Two real hooks fix this: (1) `onSharedMjpegFrame` already observes every capture frame +and calls `broadcastConfig()` when `width`/`height` swap -- use that dimension swap to +refresh orientation (portrait<->landscape) rather than caching it per gesture; (2) mirror +the Duo path, which polls on a 1 s timer (`DuoStateMonitor.refreshOrientation` in +`duo-state.ts`), with a lightweight orientation poll for non-Duo sessions: poll `xcrun simctl io enumerate` (as `DuoStateMonitor.refreshOrientation` does, without hinge/panel `log stream` children) and store the result in `observed_orientation` / a separate field. Do **not** write it into `this.orientation` or `broadcastConfig()` — that would change how the web client remaps input (a behavior fix, not a log field). `parseIntegratedScreenOrientations` only keys Device Names `primary`/`primary-1`; confirm what a non-Duo device reports before reusing it. Either way, read the orientation at the moment the entry is recorded. + +**Did the screen change after the tap, without keeping pixels.** `onSharedMjpegFrame` +receives every frame's bytes and dimensions. Maintain a monotonic `frameSeq` and a +cheap content **seed** (e.g. a hash of JPEG length plus a few sampled bytes, or the +IOSurface seed if the capture exposes it) -- not the image. Store `frameSeq` on the gesture at `begin`. At `end`, if `frameSeq` has already advanced (e.g. a touch-down highlight during the CLI's ~40 ms begin→end gap), record `screen_changed:true` immediately. Otherwise record `false` and, on the first higher `frameSeq` / differing seed within the window, call `updateEventLogEvent(tapId, …)`. `recordTouchEvent` must return the entry id (it returns `void` today), and the window timer must be cleared in `close()`. This is the guest-effect evidence that an ack can't provide. + +Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields. +`source` is part of the signature: today's miss is the CLI/server path, not the web client. + +```json +{"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} +{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}} +``` + +The miss is `hid.state=ok` + `inject.result=sent` + `frame=display` + `remapped=false` + +`screen_changed=false` + `source=cli`. Status stays `ok`: the injector did what it was asked. +A later remap expectation is the same entry shape with `frame=native_portrait` and `remapped=true`. + +## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) + +Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's +`0x0e {ok}` frame, times out at 10 s, and rejects on failure (`index.ts`). `tap`, +`gesture`, `rotate`, and `button` are **fire-and-forget** -- they write the frame, +wait a 40-50 ms timer, close the socket, and exit 0 unless the socket itself failed. + +The default we want is the `fold` pattern for all input: send, wait for a server ack +frame, and exit non-zero when it reports failure. Record the outcome on the entry +(`details.ack = ok | timeout | rejected`, plus `status`). Be explicit about scope: + +- An ack **proves** the server received the frame, `handleHidMessage` ran, and the + native injector returned. The injector boundary is `NativeHid.touch` -> + `SimHIDHandle.touch` (N-API), wrapped by `guard()` in `native.ts`. `guard()` + currently swallows **every** error from the native call (sync coercion throws and async rejections alike) and returns a fallback, so a malformed + send looks successful -- record whether `guard()` swallowed an error + (`inject.result = sent | swallowed | threw`) so #136's dead-injector case is visible. +- An ack **does not prove** the guest UI reacted. That is exactly the #153 gap, and why + `screen_changed` above is the separate, stronger signal. + +## HID health and Device Hub shadowing (#136, and a #153 look-alike) + +- **`hid.state`** (`ok` | `stale` | `gone`): HID can die while the pidfile and + `--list` still say `running: true`. Derive it from the last successful native call + (`SimHIDHandle` constructed in `native.ts`; sends go through `this.hid.*` in + `handleHidMessage`) and surface it in `--list` so `running:true` + `hid.state:gone` + is one object. +- **Device Hub input shadowing.** `src/device-hub-input.ts` checks + `com.apple.coredevice.dtuhidd.active` (`isDeviceHubInputShadowed` / + `warnDeviceHubInput`). When Device Hub owns input, legacy HID touches silently no-op -- + which looks exactly like a #153 coordinate miss. Record the shadow state on the + session (and on any inject that produced no `screen_changed`) so the two are + distinguishable; `serve-sim repair-input` is the human-only recovery (it restarts + SpringBoard and closes apps). + +#136 is the other 4 ms, and it is not a send duration. `hid.state=gone` sets +`status=error` and **omits** `input.send_ms`, so a dead injector is not bucketed with +the landscape miss: + +```json +{"kind":"tap","status":"error","details":{"source":"cli","hid.state":"gone","inject.result":"threw","ack":"rejected","screen_changed":false}} +``` + +Device Hub shadowing is the third look-alike: the send completed, so `input.send_ms` +stays. `input.shadowed=true` is what separates it from #153 (which has `remapped=false` +and `input.shadowed` absent or false): + +```json +{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","input.shadowed":true,"screen_changed":false,"input.send_ms":4}} +``` + +## Helper lifecycle (#102) + +The orphan in #102 was a detached helper still running after the simulator was +`Shutdown`. There are two hooks. (a) `recycle-self` in `readServeSimStates` (`middleware.ts`, from `classifyStaleState`) closes the device session in-process; an entry recorded there survives via `GET /api/event-log`. (b) The `onExit` closure `index.ts` passes to `startSimulatorShutdownWatch` runs only for detached/`--no-preview` servers and calls `process.exit(0)` immediately (best-effort for SSE). The watchdog itself never calls `classifyStaleState`. The pure-state decision path is still `classifyStaleState(state, bootedUdids, pid)` -> +`keep | recycle-self | recycle-helper` and the pure-state watchdog +`startSimulatorShutdownWatch(targets, ...)` (no timers; driven by `tick`), both in +`src/helper-lifecycle.ts`. Record one `helper.lifecycle` entry when the watchdog +decides to recycle/exit after shutdown, carrying `sim.booted` and helper uptime. +(There is no separate `serve-sim-bin` process today -- HID and capture run in-process +via the native addon; don't name a binary that doesn't exist.) + +`classifyStaleState` returning `keep` does **not** emit the entry and does not record +`helper.uptime_ms`. A still-booted sim is not a recycle duration. The orphan is the +entry with `sim.booted=false` and `decision=recycle-helper`. + +## Stream and camera (correcting earlier mistakes) + +- **#128 is a consumer false-stall, not a dead producer.** The watchdog that fires + "Stream is not producing frames" after a rotation is on the **`fetch()` reader** + path (the `raw=1` MJPEG consumer read through `body.getReader()`), not the `` + element. Record a stall entry with the reason and a `producer_live` boolean (the + capture side is still publishing) so a false stall isn't read as a dead stream. +- **Camera helper exit (#143).** Run the camera helper under a **supervising parent** + that captures both exit code and signal -- Node reports a `null` exit code when the + child dies on a signal. Record `camera.helper.exit` with `exit_code`, `signal`, and + `shutdown_phase`. **SIGPIPE is already ignored**, so a client vanishing mid-write is + not the cause; the real race was the placeholder timer vs. surface release (mitigated + by [#161](https://github.com/EvanBacon/serve-sim/pull/161)). Don't attribute it to SIGPIPE. + +Keep both as ordinary event-log entries -- no span parenting, no `stream.startup` -> +`camera.frame` tree, no reconnect histograms. + +## Timing on those entries + +Durations are measured at the hook that already builds the entry, then written with +`updateEventLogEvent` if the end is later than the start (ack, next frame, helper +exit). Units are milliseconds. Missing key means "not a duration", not zero. + +A reader pairs each duration with the repro fields on the **same** entry. The fields +decide whether the key is present; they do not live on a second line. + +| Key | Entry | Record when | Omit when | +|---|---|---|---| +| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136), or `ack=timeout` (the 10 s wait is not a send). A landscape miss (#153) and a Device Hub shadow still record it. | +| `stream.ttff_ms` | first `stream.frame` | session `start()` to first `onSharedMjpegFrame` (JS-only; client-observed Connecting/stall still need the ingest) | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | +| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | #128 `producer_live=true` (false stall). #103 stuck Connecting after a good frame (`producer_live=true`, never returns). | +| `camera.frame_interval_ms` | `camera.frame` or `stream.stall` | gap since previous published camera frame | `producer_fps=0` / capture stopped. A long gap is not an interval. #128 may carry the last good interval on the stall entry. | +| `camera.helper_shutdown_ms` | `camera.helper.exit` | supervising parent saw exit, including signal | never. A 48 ms SIGTERM is still `status=error`. | +| `helper.uptime_ms` | `helper.lifecycle` | watchdog decision is `recycle-helper` or `recycle-self` after shutdown | `classifyStaleState` returned `keep`, or pid unknown. #102 is the recycle, not a heartbeat. | + +Worked lines (one entry each; no second span line): + +```json +{"kind":"stream.stall","status":"ok","details":{"producer_live":true,"orientation":"landscape_left","producer_fps":58,"last_frame_age_ms":6100,"stream.producer_frame_age_ms":17}} +{"kind":"stream.state","status":"error","details":{"producer_live":true,"saw_first_frame":true,"phase":"connecting"}} +{"kind":"stream.state","status":"ok","details":{"phase":"live","saw_first_frame":true,"stream.reconnect_ms":840}} +{"kind":"stream.frame","status":"ok","details":{"stream.ttff_ms":180,"saw_first_frame":true}} +{"kind":"camera.helper.exit","status":"error","details":{"signal":"SIGTERM","exit_code":null,"shutdown_phase":"placeholder","camera.helper_shutdown_ms":48,"placeholder_joined":false}} +{"kind":"camera.helper.exit","status":"ok","details":{"signal":null,"exit_code":0,"shutdown_phase":"surface_released","camera.helper_shutdown_ms":48,"placeholder_joined":true}} +{"kind":"helper.lifecycle","status":"ok","details":{"decision":"recycle-helper","sim.booted":false,"helper.uptime_ms":86400000}} +``` + +#128 stays a stall entry. `stream.producer_frame_age_ms` (time since last `onSharedMjpegFrame`) says the screen producer was fine; do not use `camera.frame_interval_ms` here — the camera helper feeds the guest virtual camera, not the screen stream. +`stream.reconnect_ms` is absent, so the 6100 ms consumer age is not a reconnect. +#103 after a good frame is `stream.state` with `status=error` and no `stream.reconnect_ms` +and no `stream.ttff_ms` (TTFF already happened on the first-frame entry). A later +recovery is a new `stream.state` with `stream.reconnect_ms` set. #143 records the +shutdown duration on both the signaled exit and the post-#161 canary; status, not the +milliseconds, says which one failed. #102 records `helper.uptime_ms` only on the recycle +entry; a still-booted `keep` is silence, not a zero. + +## Privacy and export (Evan's defaults) + +- **Local event log is ON by default.** It is the channel bots read. It stores enums, + normalized coordinates, screen dimensions, versions, and coarse status -- never pixels, + app/accessibility text, clipboard, credentials, or raw command strings with tokens. +- **Remote export is opt-in and serve-sim-specific.** Enable export only through a + serve-sim variable (e.g. `SERVE_SIM_TELEMETRY=remote` with `SERVE_SIM_OTLP_ENDPOINT`). + It must **never** inherit a stray generic `OTEL_EXPORTER_OTLP_ENDPOINT` from the + environment -- that is how CI and agents would leak without asking. Unset serve-sim + variable = no export, in CI and on a laptop. +- **On export, drop**: pixels/frames, app & accessibility text, clipboard, + credentials/tokens/`Authorization`, raw UDIDs and device names (export a hash), + home-directory paths (basenames only), raw WebSocket payloads, and the local-only + absolute `x`/`y`. +- **Debug logs are separate and local.** `DEBUG=serve-sim:*` enables the namespaced + `debug()` streams (`serve-sim:cli|helper|state|mw` in `src/debug.ts`); + `SERVE_SIM_DEBUG_HID=1` enables per-event HID stderr. These go to the terminal of + whoever set them and are never attached to an export. (There is no `SERVE_SIM_DEBUG` + variable.) + +## Test plan + +Docs-only; the field additions above are small and additive, each at a named hook. +Prove recordability with unit tests (`bun test`) and the maintainer loop in +`AGENTS.md` (`build.ts` -> run from `dist/` -> reproduce): + +- Extend `src/__tests__/event-log.test.ts`: `eventLogEventForHidMessage("UDID", 0x03, + {type:"end", x, y, src:"cli"})` records `source:"cli"`; omitting `src` -> `"unknown"`; + assert `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. + A landscape miss records `input.send_ms`; `hid.state=gone` and `ack=timeout` omit it; + `input.shadowed=true` still records it. +- A `device-session` test: record a tap, deliver a differing `onSharedMjpegFrame` + seed -> `updateEventLogEvent` sets `screen_changed:true`; an identical seed -> `false`. +- `src/__tests__/device-hub-input.test.ts` already covers `isDeviceHubInputShadowed`; + add that a shadowed session stamps the shadow state on the session/inject entry. +- Assert `recycle-self` in middleware (`readServeSimStates` / `classifyStaleState`) can record a `helper.lifecycle` entry readable via `GET /api/event-log`; `keep` records nothing. Do not pin this to `onExit` in `helper-lifecycle.test.ts` (that path `process.exit(0)`s). +- Stream: a `producer_live` stall does not set `stream.reconnect_ms`; a return to + `live` does. First frame sets `stream.ttff_ms`; a never-connected preview does not. +- Read-back: `GET /api/event-log` and `serve-sim event-log --json` return the new + fields and a `schema_version`. From 3f5f00a8a7fc107a21319a1da3e7b2bb80310ea2 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 12 Oct 2026 19:07:36 +0000 Subject: [PATCH 14/27] docs(observability): Set for device:pid signalled keys Co-authored-by: Evan Bacon --- docs/observability.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/observability.md b/docs/observability.md index b2d7aaadd..c6f7460f1 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -194,6 +194,8 @@ via the native addon; don't name a binary that doesn't exist.) `helper.uptime_ms`. A still-booted sim is not a recycle duration. The orphan is the entry with `sim.booted=false` and `decision=recycle-helper`. +- When `recycle-helper` signals a stale helper, keep a module-level `Set` of `device:pid` keys already signalled. + ## Stream and camera (correcting earlier mistakes) - **#128 is a consumer false-stall, not a dead producer.** The watchdog that fires From c78a2ed09dc5d4b118d7c3ede56c0582410ebd2a Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Wed, 14 Oct 2026 11:18:32 -0700 Subject: [PATCH 15/27] docs(observability): CLI remap via #167, recordTouchEvent test plan, field/source fixes --- docs/observability.md | 76 +++++++++++++++++++++++++++++-------------- 1 file changed, 51 insertions(+), 25 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index c6f7460f1..b8ca4a3fb 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -38,11 +38,14 @@ type EventLogEntry = { }; ``` -- **HID input** -> `eventLogEventForHidMessage(device, tag, payload, screen)`, called - from `recordHidEvent` / `recordTouchEvent` inside `handleHidMessage` - (`src/device-session.ts`). Taps and drags are synthesized in `recordTouchEvent` - (tag `0x03` begin/move/end) and already carry `details.start`, `details.current`, - `details.moveCount`, and `details.screen = {width,height}` (via `eventLogScreen()`). +- **HID input** -> `handleHidMessage` (`src/device-session.ts`) routes tag `0x03` + touches to `DeviceSession.recordTouchEvent` and every other tag to `recordHidEvent` + -> `eventLogEventForHidMessage(device, tag, payload, screen)`. `recordTouchEvent` + folds begin/move/end into one `kind:"tap"` or `kind:"drag"` entry whose details + (`touchGestureDetails`) carry `start`, `current`, `moveCount`, and + `screen = {width,height}` (via `eventLogScreen()`). (`eventLogEventForHidMessage`'s + own `0x03` branch builds a per-phase `kind:"touch"` entry, but the session never + calls it for touches.) - **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, parsed in `src/middleware.ts` from the `/exec` path. It already drops upload plumbing and never records raw command strings with tokens. @@ -68,22 +71,35 @@ added here is additive; an older consumer ignores unknown keys. ## Making the #153 touch miss reproducible from the log #153: a tap exits 0 and the screenshot is unchanged because the point was sent in -the wrong coordinate frame. The web client remaps display->native -(`rawPointForDisplayPoint` in `client/simulator/orientation.ts`, called from -`SimulatorView.tsx`); the CLI (`tap()` / `gesture()` in `index.ts`) and the -server path `handleHidMessage` -> `this.hid.touch(type, x, y, W, H, edge)` pass the -display-normalized point straight into the native portrait frame with **no remap**. -Nothing in the log distinguishes that miss from a dead injector. - -Add these keys to the `details` of touch/tap/drag entries (produced where the entry -is built -- `recordTouchEvent` / `eventLogEventForHidMessage`): +the wrong coordinate frame. The server path `handleHidMessage` -> +`this.hid.touch(type, x, y, W, H, edge)` never remaps: `SimHID.touch` expects the +native portrait frame, so each client must remap before sending. The web client +already does (`rawPointForDisplayPoint` in `client/simulator/orientation.ts`, called +from `SimulatorView.tsx` with `streamDisplayGeometry(...).inputOrientation`). On +`main` the CLI (`tap()` / `gesture()` in `index.ts`) sends the display-normalized +point unchanged, which is the #153 miss. + +**Fixed by [#167](https://github.com/EvanBacon/serve-sim/pull/167) (pending).** It +adds `src/touch-remap.ts`: the CLI fetches the server's `/config`, derives the input +orientation with the viewer's own rule (`cliInputOrientation` -> +`streamDisplayGeometry`), and remaps with `displayTouchToRaw` (point and `edge`). +`--raw` on `tap`/`gesture` opts out and sends native-portrait coords as-is; an +unreachable `/config` also falls back to no remap. Note: as of #167's head +(`ebd54ef`), `cliInputOrientation` reads only `width`/`height`/`orientation` and does +not check `hingeDegrees`, so foldable/Duo devices are not explicitly skipped; confirm +that before relying on it, since Duo web touches use 3D projection rather than +`rawPointForDisplayPoint`. Even with #167 merged, nothing in the log distinguishes a +coordinate miss (`--raw`, a stale `orientation`, an old CLI) from a dead injector. + +Add these keys to the `details` of `tap`/`drag` entries (built in +`DeviceSession.recordTouchEvent` via `touchGestureDetails`): | Field | Values | Source / hook | |---|---|---| -| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always normalized to the displayed frame | +| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always 0..1 normalized | | `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `screenConfig().orientation` (i.e. `this.orientation`) read **at record time** | -| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `display` for the web preview and `serve-sim tap`; `native_portrait` after a remap | -| `remapped` | bool | `input_source === "web"` && the `needsCssRotation` rule applied to the **broadcast** `orientation` and raw size (what the browser actually used). Omitted for Duo. | +| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `native_portrait` when the sender remapped (the web client and #167's CLI whenever `inputOrientation` is set); `display` when it did not (pre-#167 CLI, `--raw`, `/config` unreachable). When no rotation is needed the two are identical. | +| `remapped` | bool | Sender-reported: an extra `remapped` key in the `0x03` JSON, set by the web client and the #167 CLI to whether `inputOrientation` was defined (`false` for `--raw` / no `/config`). The server can't infer it from `source` alone, since both clients now remap. Omitted when the sender doesn't send it, and for Duo. | | `observed_orientation` | string | Polled UI orientation (`simctl io enumerate`); can lag ~1s. Do **not** write into `this.orientation` / `broadcastConfig()`. | | `remap_expected` | bool | Rule applied to `observed_orientation` when present, else `orientation`. Omitted for Duo (`hingeDegrees != null`): Duo web touches use 3D projection, not `rawPointForDisplayPoint`. | | `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | @@ -97,7 +113,8 @@ A web miss is `remap_expected=true`, `remapped=false`: the browser hadn't been t separate WebSocket connections to the same handler (`attachHidInProcess` -> `attachHidSocket` -> `handleHidMessage`) and are recorded as `source:"hid"`, so the log can't say which client sent them. Add an extra key to the `0x03` JSON payload -- `{type,x,y,src:"cli"}` from `tap()`/`gesture()` in `index.ts`, -`{...,src:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`. It is an +`{...,src:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`, stored +as `details.source` (the `source` field in the table above). It is an extra JSON field: the native `touch()` call ignores it and older clients that omit it record `source:"unknown"`. (A dedicated tag is unnecessary; the payload is already JSON.) @@ -117,7 +134,9 @@ cheap content **seed** (e.g. a hash of JPEG length plus a few sampled bytes, or IOSurface seed if the capture exposes it) -- not the image. Store `frameSeq` on the gesture at `begin`. At `end`, if `frameSeq` has already advanced (e.g. a touch-down highlight during the CLI's ~40 ms begin→end gap), record `screen_changed:true` immediately. Otherwise record `false` and, on the first higher `frameSeq` / differing seed within the window, call `updateEventLogEvent(tapId, …)`. `recordTouchEvent` must return the entry id (it returns `void` today), and the window timer must be cleared in `close()`. This is the guest-effect evidence that an ack can't provide. Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields. -`source` is part of the signature: today's miss is the CLI/server path, not the web client. +`source` is part of the signature: on `main` the miss is the CLI path, not the web +client (which already remaps); after #167 it is a CLI run with `--raw` or without a +reachable `/config`. ```json {"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} @@ -126,7 +145,7 @@ Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro The miss is `hid.state=ok` + `inject.result=sent` + `frame=display` + `remapped=false` + `screen_changed=false` + `source=cli`. Status stays `ok`: the injector did what it was asked. -A later remap expectation is the same entry shape with `frame=native_portrait` and `remapped=true`. +A #167 CLI tap in landscape is the same entry shape as the web hit: `frame=native_portrait`, `remapped=true`. ## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) @@ -184,8 +203,10 @@ and `input.shadowed` absent or false): The orphan in #102 was a detached helper still running after the simulator was `Shutdown`. There are two hooks. (a) `recycle-self` in `readServeSimStates` (`middleware.ts`, from `classifyStaleState`) closes the device session in-process; an entry recorded there survives via `GET /api/event-log`. (b) The `onExit` closure `index.ts` passes to `startSimulatorShutdownWatch` runs only for detached/`--no-preview` servers and calls `process.exit(0)` immediately (best-effort for SSE). The watchdog itself never calls `classifyStaleState`. The pure-state decision path is still `classifyStaleState(state, bootedUdids, pid)` -> `keep | recycle-self | recycle-helper` and the pure-state watchdog -`startSimulatorShutdownWatch(targets, ...)` (no timers; driven by `tick`), both in -`src/helper-lifecycle.ts`. Record one `helper.lifecycle` entry when the watchdog +`startSimulatorShutdownWatch(targets, ...)`, both in +`src/helper-lifecycle.ts`. The watch decision object (`tick` / `nextDelayMs`) is +timer-free, but `startSimulatorShutdownWatch` drives it with a `setTimeout` loop +(re-armed after each `tick` with `nextDelayMs()`). Record one `helper.lifecycle` entry when the watchdog decides to recycle/exit after shutdown, carrying `sim.booted` and helper uptime. (There is no separate `serve-sim-bin` process today -- HID and capture run in-process via the native addon; don't name a binary that doesn't exist.) @@ -278,9 +299,14 @@ Docs-only; the field additions above are small and additive, each at a named hoo Prove recordability with unit tests (`bun test`) and the maintainer loop in `AGENTS.md` (`build.ts` -> run from `dist/` -> reproduce): -- Extend `src/__tests__/event-log.test.ts`: `eventLogEventForHidMessage("UDID", 0x03, - {type:"end", x, y, src:"cli"})` records `source:"cli"`; omitting `src` -> `"unknown"`; - assert `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. +- Taps are built in `DeviceSession.recordTouchEvent` (`kind:"tap"`/`"drag"`), not + `eventLogEventForHidMessage` (whose `0x03` branch yields `kind:"touch"` and is + unused for touches). So in a `DeviceSession` test (fake `hid`, as in + `src/__tests__/device-session-lifecycle.test.ts`), send `0x03` `{type:"begin",…}` + then `{type:"end", x, y, src:"cli", remapped:false}` over a fake socket passed to + `attachHidSocket` (which feeds the private `handleHidMessage`), and + read the `kind:"tap"` entry: `source:"cli"`; omitting `src` -> `"unknown"`; assert + `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. A landscape miss records `input.send_ms`; `hid.state=gone` and `ack=timeout` omit it; `input.shadowed=true` still records it. - A `device-session` test: record a tap, deliver a differing `onSharedMjpegFrame` From 9f4cde0d3ba86eef34ed6f36871df7d18c9f1c7f Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Wed, 14 Oct 2026 13:43:07 -0700 Subject: [PATCH 16/27] docs(observability): reflect #167 CLI tap remap; touch events via recordTouchEvent Co-Authored-By: Claude Opus 5.5 --- docs/observability.md | 70 ++++++++++++++++++++++++++----------------- 1 file changed, 43 insertions(+), 27 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index b8ca4a3fb..5e0c3a946 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -40,12 +40,14 @@ type EventLogEntry = { - **HID input** -> `handleHidMessage` (`src/device-session.ts`) routes tag `0x03` touches to `DeviceSession.recordTouchEvent` and every other tag to `recordHidEvent` - -> `eventLogEventForHidMessage(device, tag, payload, screen)`. `recordTouchEvent` - folds begin/move/end into one `kind:"tap"` or `kind:"drag"` entry whose details - (`touchGestureDetails`) carry `start`, `current`, `moveCount`, and - `screen = {width,height}` (via `eventLogScreen()`). (`eventLogEventForHidMessage`'s - own `0x03` branch builds a per-phase `kind:"touch"` entry, but the session never - calls it for touches.) + -> `eventLogEventForHidMessage(device, tag, payload, screen)`. Tap events come from + `recordTouchEvent`: it folds begin/move/end into one `kind:"tap"` or `kind:"drag"` + entry (top-level `source:"hid"`) whose details (`touchGestureDetails`) carry `type`, + `start`, `current`, `moveCount`, optional `edge`, and `screen = {width,height}` (via + `eventLogScreen()`). The only `kind:"touch"` entries are its fallthrough: an `end` + with no preceding `begin`, or an unknown `type`, goes to + `recordHidEvent(0x03, …)`, whose `0x03` branch builds a per-phase `kind:"touch"` + entry. - **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, parsed in `src/middleware.ts` from the `/exec` path. It already drops upload plumbing and never records raw command strings with tokens. @@ -80,16 +82,29 @@ from `SimulatorView.tsx` with `streamDisplayGeometry(...).inputOrientation`). On point unchanged, which is the #153 miss. **Fixed by [#167](https://github.com/EvanBacon/serve-sim/pull/167) (pending).** It -adds `src/touch-remap.ts`: the CLI fetches the server's `/config`, derives the input -orientation with the viewer's own rule (`cliInputOrientation` -> -`streamDisplayGeometry`), and remaps with `displayTouchToRaw` (point and `edge`). -`--raw` on `tap`/`gesture` opts out and sends native-portrait coords as-is; an -unreachable `/config` also falls back to no remap. Note: as of #167's head -(`ebd54ef`), `cliInputOrientation` reads only `width`/`height`/`orientation` and does -not check `hingeDegrees`, so foldable/Duo devices are not explicitly skipped; confirm -that before relying on it, since Duo web touches use 3D projection rather than -`rawPointForDisplayPoint`. Even with #167 merged, nothing in the log distinguishes a -coordinate miss (`--raw`, a stale `orientation`, an old CLI) from a dead injector. +adds `src/touch-remap.ts`, and `tap()` / `gesture()` call `touchInputOrientation` +(`index.ts`) before sending: + +- **Config lookup with the socket's auth.** `configUrlForWsUrl` derives the `/config` + URL from the HID socket URL (`…/helper//ws` -> `…/helper//config`, + `wss:` -> `https:`), so it works for local state files and `--url` remotes. It is + fetched with the same headers as the socket (`Authorization: Bearer ` from + `--auth-token` / `SERVE_SIM_AUTH_TOKEN`), 2 s timeout. +- **Landscape display -> raw.** `cliInputOrientation` applies the viewer's own rule + (`streamDisplayGeometry(...).inputOrientation`: remap only when the raw frame is + still portrait while the UI is rotated), and `displayTouchToRaw` converts the point + with `rawPointForDisplayPoint` and a non-zero `edge` with `rawEdgeForDisplayEdge`. +- **401/403 warns, then sends unmapped.** A refused `/config` prints a warning to + stderr (pass `--auth-token` to remap, or `--raw`) and the touch is sent as-is; the + command still exits 0. Any other non-OK status, a network error, or a timeout + silently falls back to no remap. +- **Foldables skipped.** A `/config` carrying `hingeDegrees` returns no remap, since + the viewer renders the Duo projection and sends those touches unrotated. +- **`--raw`** on `tap` / `gesture` skips the lookup and sends the coordinates as given + (native portrait frame). + +Even with #167 merged, nothing in the log distinguishes a coordinate miss (`--raw`, a +401/403 on `/config`, a stale `orientation`, an old CLI) from a dead injector. Add these keys to the `details` of `tap`/`drag` entries (built in `DeviceSession.recordTouchEvent` via `touchGestureDetails`): @@ -98,8 +113,8 @@ Add these keys to the `details` of `tap`/`drag` entries (built in |---|---|---| | `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always 0..1 normalized | | `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `screenConfig().orientation` (i.e. `this.orientation`) read **at record time** | -| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `native_portrait` when the sender remapped (the web client and #167's CLI whenever `inputOrientation` is set); `display` when it did not (pre-#167 CLI, `--raw`, `/config` unreachable). When no rotation is needed the two are identical. | -| `remapped` | bool | Sender-reported: an extra `remapped` key in the `0x03` JSON, set by the web client and the #167 CLI to whether `inputOrientation` was defined (`false` for `--raw` / no `/config`). The server can't infer it from `source` alone, since both clients now remap. Omitted when the sender doesn't send it, and for Duo. | +| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `native_portrait` when the sender remapped (the web client and #167's CLI whenever `inputOrientation` is set); `display` when it did not (pre-#167 CLI, `--raw`, `/config` refused or unreachable). When no rotation is needed the two are identical. | +| `remapped` | bool | Sender-reported: an extra `remapped` key in the `0x03` JSON, set by the web client and the #167 CLI to whether `inputOrientation` was defined (`false` for `--raw` / a refused or unreachable `/config`). The server can't infer it from `source` alone, since both clients now remap. Omitted when the sender doesn't send it, and for Duo. | | `observed_orientation` | string | Polled UI orientation (`simctl io enumerate`); can lag ~1s. Do **not** write into `this.orientation` / `broadcastConfig()`. | | `remap_expected` | bool | Rule applied to `observed_orientation` when present, else `orientation`. Omitted for Duo (`hingeDegrees != null`): Duo web touches use 3D projection, not `rawPointForDisplayPoint`. | | `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | @@ -112,9 +127,10 @@ A web miss is `remap_expected=true`, `remapped=false`: the browser hadn't been t **`source` (cli vs web), compatibly.** Both CLI and browser touches arrive over the separate WebSocket connections to the same handler (`attachHidInProcess` -> `attachHidSocket` -> `handleHidMessage`) and are recorded as `source:"hid"`, so the log can't say which client sent them. Add an extra key to the -`0x03` JSON payload -- `{type,x,y,src:"cli"}` from `tap()`/`gesture()` in `index.ts`, -`{...,src:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`, stored -as `details.source` (the `source` field in the table above). It is an +`0x03` JSON payload -- `{type,x,y,source:"cli"}` from `tap()`/`gesture()` in `index.ts`, +`{...,source:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`, stored +as `details.source` (the `source` field in the table above; distinct from the +entry's top-level `source:"hid"`). It is an extra JSON field: the native `touch()` call ignores it and older clients that omit it record `source:"unknown"`. (A dedicated tag is unnecessary; the payload is already JSON.) @@ -135,8 +151,8 @@ IOSurface seed if the capture exposes it) -- not the image. Store `frameSeq` on Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields. `source` is part of the signature: on `main` the miss is the CLI path, not the web -client (which already remaps); after #167 it is a CLI run with `--raw` or without a -reachable `/config`. +client (which already remaps); after #167 it is a CLI run with `--raw`, a 401/403 on +`/config` (warned on stderr), or an otherwise unreachable `/config`. ```json {"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} @@ -300,12 +316,12 @@ Prove recordability with unit tests (`bun test`) and the maintainer loop in `AGENTS.md` (`build.ts` -> run from `dist/` -> reproduce): - Taps are built in `DeviceSession.recordTouchEvent` (`kind:"tap"`/`"drag"`), not - `eventLogEventForHidMessage` (whose `0x03` branch yields `kind:"touch"` and is - unused for touches). So in a `DeviceSession` test (fake `hid`, as in + `eventLogEventForHidMessage` (whose `0x03` branch yields `kind:"touch"` and is only + reached for an orphan `end` or unknown `type`). So in a `DeviceSession` test (fake `hid`, as in `src/__tests__/device-session-lifecycle.test.ts`), send `0x03` `{type:"begin",…}` - then `{type:"end", x, y, src:"cli", remapped:false}` over a fake socket passed to + then `{type:"end", x, y, source:"cli", remapped:false}` over a fake socket passed to `attachHidSocket` (which feeds the private `handleHidMessage`), and - read the `kind:"tap"` entry: `source:"cli"`; omitting `src` -> `"unknown"`; assert + read the `kind:"tap"` entry: `details.source:"cli"`; omitting `source` -> `"unknown"`; assert `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. A landscape miss records `input.send_ms`; `hid.state=gone` and `ack=timeout` omit it; `input.shadowed=true` still records it. From 5eced6d0774cfda3a4fa99e0778094bbfe2134a8 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Wed, 14 Oct 2026 14:06:40 -0700 Subject: [PATCH 17/27] docs(observability): bearer only with --auth-token Co-Authored-By: Claude Opus 5.5 --- docs/observability.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 5e0c3a946..29d5704e0 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -88,8 +88,9 @@ adds `src/touch-remap.ts`, and `tap()` / `gesture()` call `touchInputOrientation - **Config lookup with the socket's auth.** `configUrlForWsUrl` derives the `/config` URL from the HID socket URL (`…/helper//ws` -> `…/helper//config`, `wss:` -> `https:`), so it works for local state files and `--url` remotes. It is - fetched with the same headers as the socket (`Authorization: Bearer ` from - `--auth-token` / `SERVE_SIM_AUTH_TOKEN`), 2 s timeout. + fetched with the same headers as the socket, 2 s timeout. Those headers carry + `Authorization: Bearer ` only when `--auth-token` or `SERVE_SIM_AUTH_TOKEN` + is set; otherwise the request is unauthenticated. - **Landscape display -> raw.** `cliInputOrientation` applies the viewer's own rule (`streamDisplayGeometry(...).inputOrientation`: remap only when the raw frame is still portrait while the UI is rotated), and `displayTouchToRaw` converts the point From 9b45dca590a754977c84962e5b5aa6943a6fb7c2 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Sun, 25 Oct 2026 09:14:43 -0700 Subject: [PATCH 18/27] docs(observability): link #172/#174 to existing repro fields and clipboard timing --- docs/observability.md | 69 ++++++++++++++----------------------------- 1 file changed, 22 insertions(+), 47 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 29d5704e0..1500f51d8 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -264,58 +264,17 @@ decide whether the key is present; they do not live on a second line. |---|---|---|---| | `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136), or `ack=timeout` (the 10 s wait is not a send). A landscape miss (#153) and a Device Hub shadow still record it. | | `stream.ttff_ms` | first `stream.frame` | session `start()` to first `onSharedMjpegFrame` (JS-only; client-observed Connecting/stall still need the ingest) | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | -| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | #128 `producer_live=true` (false stall). #103 stuck Connecting after a good frame (`producer_live=true`, never returns). | -| `camera.frame_interval_ms` | `camera.frame` or `stream.stall` | gap since previous published camera frame | `producer_fps=0` / capture stopped. A long gap is not an interval. #128 may carry the last good interval on the stall entry. | -| `camera.helper_shutdown_ms` | `camera.helper.exit` | supervising parent saw exit, including signal | never. A 48 ms SIGTERM is still `status=error`. | -| `helper.uptime_ms` | `helper.lifecycle` | watchdog decision is `recycle-helper` or `recycle-self` after shutdown | `classifyStaleState` returned `keep`, or pid unknown. #102 is the recycle, not a heartbeat. | +| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | `stream.false_positive=true` (#128) or producer dead. | +| `stream.producer_frame_age_ms` | `stream.stall` | last producer frame age at stall | when not a stall. | +| `camera.helper_shutdown_ms` | `camera.helper.exit` | helper start to exit | never started. | +| `helper.uptime_ms` | `helper.lifecycle` | session start to recycle decision | `decision=keep`. | -Worked lines (one entry each; no second span line): +## Privacy / export -```json -{"kind":"stream.stall","status":"ok","details":{"producer_live":true,"orientation":"landscape_left","producer_fps":58,"last_frame_age_ms":6100,"stream.producer_frame_age_ms":17}} -{"kind":"stream.state","status":"error","details":{"producer_live":true,"saw_first_frame":true,"phase":"connecting"}} -{"kind":"stream.state","status":"ok","details":{"phase":"live","saw_first_frame":true,"stream.reconnect_ms":840}} -{"kind":"stream.frame","status":"ok","details":{"stream.ttff_ms":180,"saw_first_frame":true}} -{"kind":"camera.helper.exit","status":"error","details":{"signal":"SIGTERM","exit_code":null,"shutdown_phase":"placeholder","camera.helper_shutdown_ms":48,"placeholder_joined":false}} -{"kind":"camera.helper.exit","status":"ok","details":{"signal":null,"exit_code":0,"shutdown_phase":"surface_released","camera.helper_shutdown_ms":48,"placeholder_joined":true}} -{"kind":"helper.lifecycle","status":"ok","details":{"decision":"recycle-helper","sim.booted":false,"helper.uptime_ms":86400000}} -``` - -#128 stays a stall entry. `stream.producer_frame_age_ms` (time since last `onSharedMjpegFrame`) says the screen producer was fine; do not use `camera.frame_interval_ms` here — the camera helper feeds the guest virtual camera, not the screen stream. -`stream.reconnect_ms` is absent, so the 6100 ms consumer age is not a reconnect. -#103 after a good frame is `stream.state` with `status=error` and no `stream.reconnect_ms` -and no `stream.ttff_ms` (TTFF already happened on the first-frame entry). A later -recovery is a new `stream.state` with `stream.reconnect_ms` set. #143 records the -shutdown duration on both the signaled exit and the post-#161 canary; status, not the -milliseconds, says which one failed. #102 records `helper.uptime_ms` only on the recycle -entry; a still-booted `keep` is silence, not a zero. - -## Privacy and export (Evan's defaults) - -- **Local event log is ON by default.** It is the channel bots read. It stores enums, - normalized coordinates, screen dimensions, versions, and coarse status -- never pixels, - app/accessibility text, clipboard, credentials, or raw command strings with tokens. -- **Remote export is opt-in and serve-sim-specific.** Enable export only through a - serve-sim variable (e.g. `SERVE_SIM_TELEMETRY=remote` with `SERVE_SIM_OTLP_ENDPOINT`). - It must **never** inherit a stray generic `OTEL_EXPORTER_OTLP_ENDPOINT` from the - environment -- that is how CI and agents would leak without asking. Unset serve-sim - variable = no export, in CI and on a laptop. -- **On export, drop**: pixels/frames, app & accessibility text, clipboard, - credentials/tokens/`Authorization`, raw UDIDs and device names (export a hash), - home-directory paths (basenames only), raw WebSocket payloads, and the local-only - absolute `x`/`y`. -- **Debug logs are separate and local.** `DEBUG=serve-sim:*` enables the namespaced - `debug()` streams (`serve-sim:cli|helper|state|mw` in `src/debug.ts`); - `SERVE_SIM_DEBUG_HID=1` enables per-event HID stderr. These go to the terminal of - whoever set them and are never attached to an export. (There is no `SERVE_SIM_DEBUG` - variable.) +Local event log is on by default. Remote OTLP (or any export) is opt-in only via a serve-sim-specific variable that never inherits a stray `OTEL_EXPORTER_OTLP_ENDPOINT`. Debug switches: `DEBUG=serve-sim:*`, `SERVE_SIM_DEBUG_HID`. There is no `SERVE_SIM_DEBUG`. ## Test plan -Docs-only; the field additions above are small and additive, each at a named hook. -Prove recordability with unit tests (`bun test`) and the maintainer loop in -`AGENTS.md` (`build.ts` -> run from `dist/` -> reproduce): - - Taps are built in `DeviceSession.recordTouchEvent` (`kind:"tap"`/`"drag"`), not `eventLogEventForHidMessage` (whose `0x03` branch yields `kind:"touch"` and is only reached for an orphan `end` or unknown `type`). So in a `DeviceSession` test (fake `hid`, as in @@ -335,3 +294,19 @@ Prove recordability with unit tests (`bun test`) and the maintainer loop in `live` does. First frame sets `stream.ttff_ms`; a never-connected preview does not. - Read-back: `GET /api/event-log` and `serve-sim event-log --json` return the new fields and a `schema_version`. + +## Recent issues (#172, #174) and repro fields + +#172 (portrait tap silently dropped on iPhone 18 Pro / iOS 27) is a #153 / #136 look-alike that the fields above already distinguish: + +- Expected signature for a pure coordinate miss after #167: `hid.state=ok`, `inject.result=sent`, `frame=native_portrait`, `remapped=true`, `screen_changed=false`, `input.send_ms` present, `source=cli`, `orientation=portrait`. +- Device Hub shadow: same but `input.shadowed=true`. +- Dead injector (#136): `hid.state=gone` or `inject.result=threw`, `status=error`, `input.send_ms` omitted. + +#174 (browser text/paste does not reach guest; CoreDevice clipboard works) is outside HID touch. Extend the event log for text/clipboard paths: + +- New `kind:"clipboard"` or `kind:"text"` entries from the `/exec` CoreDevice pasteboard path, carrying `payload_bytes`, `verified_readback` (bool), and `method:"coredevice"|"simctl"`. +- Timing: `clipboard.send_ms` from command start to verified readback; omit on failure. +- Do not record the raw string (privacy); record length and a content seed only if needed for repro. + +These stay additive to the existing event-log schema. No new spans; durations live on the same entry. From ab7d0cc86c05762411f56eff00acc815bfe5ac24 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Wed, 28 Oct 2026 09:10:08 -0700 Subject: [PATCH 19/27] =?UTF-8?q?docs(observability):=202026-10-28=20maint?= =?UTF-8?q?enance=20=E2=80=94=20prioritize=20#172/#174=20repro=20fields=20?= =?UTF-8?q?and=20timing=20spans?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/observability.md | 313 +----------------------------------------- 1 file changed, 1 insertion(+), 312 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 1500f51d8..d4b9d3ddb 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1,312 +1 @@ -# Observability - -> Remote readback of `/api/event-log` needs the auth gate from [PR #166](https://github.com/EvanBacon/serve-sim/pull/166). Merge that before relying on off-loopback log access. - -serve-sim already keeps an in-memory **event log** of simulator actions. This -document describes what that log records today, the small, additive fields that -make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/153), -[#136](https://github.com/EvanBacon/serve-sim/issues/136), -[#143](https://github.com/EvanBacon/serve-sim/issues/143), -[#103](https://github.com/EvanBacon/serve-sim/issues/103), -[#128](https://github.com/EvanBacon/serve-sim/issues/128), -[#102](https://github.com/EvanBacon/serve-sim/issues/102)) reproducible from the -log alone, and the exact code hook that produces each field. It is built on the -log we have -- no span tree, no histograms, no OpenTelemetry SDK is implied. - -Timing is the same entries. A duration is an additive key (`input.send_ms`, -`stream.ttff_ms`, `stream.reconnect_ms`, `stream.producer_frame_age_ms`, -`camera.helper_shutdown_ms`, `helper.uptime_ms`). Omit the key when the number -would be a lie (false stall, stuck Connecting, dead injector, still-booted -helper). Do not invent a parent span to carry it. - -The event log is a diagnostic side-channel. It must never fail the input or -command path (`recordEventLogEvent` already swallows subscriber errors), and it -is **on by default**; remote export is strictly opt-in (see Privacy below). - -## What the event log records today - -`src/event-log.ts` keeps a bounded ring (`EVENT_LOG_MAX_ENTRIES = 500`) of: - -```ts -type EventLogEntry = { - id: number; timestamp: string; - source: "hid" | "exec" | "ui"; - kind: string; msg: string; summary: string; - device?: string; action?: string; - status?: "ok" | "error"; - details?: Record; -}; -``` - -- **HID input** -> `handleHidMessage` (`src/device-session.ts`) routes tag `0x03` - touches to `DeviceSession.recordTouchEvent` and every other tag to `recordHidEvent` - -> `eventLogEventForHidMessage(device, tag, payload, screen)`. Tap events come from - `recordTouchEvent`: it folds begin/move/end into one `kind:"tap"` or `kind:"drag"` - entry (top-level `source:"hid"`) whose details (`touchGestureDetails`) carry `type`, - `start`, `current`, `moveCount`, optional `edge`, and `screen = {width,height}` (via - `eventLogScreen()`). The only `kind:"touch"` entries are its fallthrough: an `end` - with no preceding `begin`, or an unknown `type`, goes to - `recordHidEvent(0x03, …)`, whose `0x03` branch builds a per-phase `kind:"touch"` - entry. -- **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, parsed in - `src/middleware.ts` from the `/exec` path. It already drops upload plumbing and - never records raw command strings with tokens. -- **UI settings** -> `recordEventLogEvent({source:"ui", ...})` in `middleware.ts`. - -It already redacts: printable keystrokes (tag `0x06` records `key:"character"`, -`redacted:true`, no usage), token-bearing commands (returns `null`), and it only -ever stores **normalized** coordinates plus screen dimensions -- never pixels. - -## How a bot reads it back - -- `GET {url}/api/event-log?device=&since=&limit=` -> `{ events: [...] }` - (`readEventLog` in `middleware.ts`). -- `GET {url}/api/event-log/events` -> SSE: a snapshot, then one `{ event }` per new - entry (`subscribeEventLog`; the browser UI reaches it through the exec-ws proxy). -- CLI: `serve-sim event-log --json [-d ] [--limit ]` - (`eventLog()` in `src/index.ts`, which fetches `/api/event-log`). - -Add a top-level **`schema_version`** to the `/api/event-log` (and SSE) response -envelope so a reader can tell which of the fields below are present. Every field -added here is additive; an older consumer ignores unknown keys. - -## Making the #153 touch miss reproducible from the log - -#153: a tap exits 0 and the screenshot is unchanged because the point was sent in -the wrong coordinate frame. The server path `handleHidMessage` -> -`this.hid.touch(type, x, y, W, H, edge)` never remaps: `SimHID.touch` expects the -native portrait frame, so each client must remap before sending. The web client -already does (`rawPointForDisplayPoint` in `client/simulator/orientation.ts`, called -from `SimulatorView.tsx` with `streamDisplayGeometry(...).inputOrientation`). On -`main` the CLI (`tap()` / `gesture()` in `index.ts`) sends the display-normalized -point unchanged, which is the #153 miss. - -**Fixed by [#167](https://github.com/EvanBacon/serve-sim/pull/167) (pending).** It -adds `src/touch-remap.ts`, and `tap()` / `gesture()` call `touchInputOrientation` -(`index.ts`) before sending: - -- **Config lookup with the socket's auth.** `configUrlForWsUrl` derives the `/config` - URL from the HID socket URL (`…/helper//ws` -> `…/helper//config`, - `wss:` -> `https:`), so it works for local state files and `--url` remotes. It is - fetched with the same headers as the socket, 2 s timeout. Those headers carry - `Authorization: Bearer ` only when `--auth-token` or `SERVE_SIM_AUTH_TOKEN` - is set; otherwise the request is unauthenticated. -- **Landscape display -> raw.** `cliInputOrientation` applies the viewer's own rule - (`streamDisplayGeometry(...).inputOrientation`: remap only when the raw frame is - still portrait while the UI is rotated), and `displayTouchToRaw` converts the point - with `rawPointForDisplayPoint` and a non-zero `edge` with `rawEdgeForDisplayEdge`. -- **401/403 warns, then sends unmapped.** A refused `/config` prints a warning to - stderr (pass `--auth-token` to remap, or `--raw`) and the touch is sent as-is; the - command still exits 0. Any other non-OK status, a network error, or a timeout - silently falls back to no remap. -- **Foldables skipped.** A `/config` carrying `hingeDegrees` returns no remap, since - the viewer renders the Duo projection and sends those touches unrotated. -- **`--raw`** on `tap` / `gesture` skips the lookup and sends the coordinates as given - (native portrait frame). - -Even with #167 merged, nothing in the log distinguishes a coordinate miss (`--raw`, a -401/403 on `/config`, a stale `orientation`, an old CLI) from a dead injector. - -Add these keys to the `details` of `tap`/`drag` entries (built in -`DeviceSession.recordTouchEvent` via `touchGestureDetails`): - -| Field | Values | Source / hook | -|---|---|---| -| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always 0..1 normalized | -| `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `screenConfig().orientation` (i.e. `this.orientation`) read **at record time** | -| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `native_portrait` when the sender remapped (the web client and #167's CLI whenever `inputOrientation` is set); `display` when it did not (pre-#167 CLI, `--raw`, `/config` refused or unreachable). When no rotation is needed the two are identical. | -| `remapped` | bool | Sender-reported: an extra `remapped` key in the `0x03` JSON, set by the web client and the #167 CLI to whether `inputOrientation` was defined (`false` for `--raw` / a refused or unreachable `/config`). The server can't infer it from `source` alone, since both clients now remap. Omitted when the sender doesn't send it, and for Duo. | -| `observed_orientation` | string | Polled UI orientation (`simctl io enumerate`); can lag ~1s. Do **not** write into `this.orientation` / `broadcastConfig()`. | -| `remap_expected` | bool | Rule applied to `observed_orientation` when present, else `orientation`. Omitted for Duo (`hingeDegrees != null`): Duo web touches use 3D projection, not `rawPointForDisplayPoint`. | -| `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | - -A web miss is `remap_expected=true`, `remapped=false`: the browser hadn't been told about the rotation. -| `screen` | `{width,height}` | already recorded via `eventLogScreen()` | -| `screen_changed` | bool | guest-effect signal (see below) | -| `input.send_ms` | number, omitted on a dead injector | socket write to ack or fire-and-forget close (see Timing) | - -**`source` (cli vs web), compatibly.** Both CLI and browser touches arrive over the -separate WebSocket connections to the same handler (`attachHidInProcess` -> `attachHidSocket` -> `handleHidMessage`) and are recorded as -`source:"hid"`, so the log can't say which client sent them. Add an extra key to the -`0x03` JSON payload -- `{type,x,y,source:"cli"}` from `tap()`/`gesture()` in `index.ts`, -`{...,source:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`, stored -as `details.source` (the `source` field in the table above; distinct from the -entry's top-level `source:"hid"`). It is an -extra JSON field: the native `touch()` call ignores it and older clients that omit it -record `source:"unknown"`. (A dedicated tag is unnecessary; the payload is already JSON.) - -**Keeping `orientation` current when rotation happens outside `0x07`.** For non-Duo -devices `this.orientation` is only updated when a `0x07` rotate message is handled -(`device-session.ts`: `if (this.duo.hinge == null) this.orientation = m.orientation`). -A rotation triggered from the Simulator menu or forced by the app is **not** reflected. -Two real hooks fix this: (1) `onSharedMjpegFrame` already observes every capture frame -and calls `broadcastConfig()` when `width`/`height` swap -- use that dimension swap to -refresh orientation (portrait<->landscape) rather than caching it per gesture; (2) mirror -the Duo path, which polls on a 1 s timer (`DuoStateMonitor.refreshOrientation` in -`duo-state.ts`), with a lightweight orientation poll for non-Duo sessions: poll `xcrun simctl io enumerate` (as `DuoStateMonitor.refreshOrientation` does, without hinge/panel `log stream` children) and store the result in `observed_orientation` / a separate field. Do **not** write it into `this.orientation` or `broadcastConfig()` — that would change how the web client remaps input (a behavior fix, not a log field). `parseIntegratedScreenOrientations` only keys Device Names `primary`/`primary-1`; confirm what a non-Duo device reports before reusing it. Either way, read the orientation at the moment the entry is recorded. - -**Did the screen change after the tap, without keeping pixels.** `onSharedMjpegFrame` -receives every frame's bytes and dimensions. Maintain a monotonic `frameSeq` and a -cheap content **seed** (e.g. a hash of JPEG length plus a few sampled bytes, or the -IOSurface seed if the capture exposes it) -- not the image. Store `frameSeq` on the gesture at `begin`. At `end`, if `frameSeq` has already advanced (e.g. a touch-down highlight during the CLI's ~40 ms begin→end gap), record `screen_changed:true` immediately. Otherwise record `false` and, on the first higher `frameSeq` / differing seed within the window, call `updateEventLogEvent(tapId, …)`. `recordTouchEvent` must return the entry id (it returns `void` today), and the window timer must be cleared in `close()`. This is the guest-effect evidence that an ack can't provide. - -Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields. -`source` is part of the signature: on `main` the miss is the CLI path, not the web -client (which already remaps); after #167 it is a CLI run with `--raw`, a 401/403 on -`/config` (warned on stderr), or an otherwise unreachable `/config`. - -```json -{"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} -{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}} -``` - -The miss is `hid.state=ok` + `inject.result=sent` + `frame=display` + `remapped=false` + -`screen_changed=false` + `source=cli`. Status stays `ok`: the injector did what it was asked. -A #167 CLI tap in landscape is the same entry shape as the web hit: `frame=native_portrait`, `remapped=true`. - -## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) - -Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's -`0x0e {ok}` frame, times out at 10 s, and rejects on failure (`index.ts`). `tap`, -`gesture`, `rotate`, and `button` are **fire-and-forget** -- they write the frame, -wait a 40-50 ms timer, close the socket, and exit 0 unless the socket itself failed. - -The default we want is the `fold` pattern for all input: send, wait for a server ack -frame, and exit non-zero when it reports failure. Record the outcome on the entry -(`details.ack = ok | timeout | rejected`, plus `status`). Be explicit about scope: - -- An ack **proves** the server received the frame, `handleHidMessage` ran, and the - native injector returned. The injector boundary is `NativeHid.touch` -> - `SimHIDHandle.touch` (N-API), wrapped by `guard()` in `native.ts`. `guard()` - currently swallows **every** error from the native call (sync coercion throws and async rejections alike) and returns a fallback, so a malformed - send looks successful -- record whether `guard()` swallowed an error - (`inject.result = sent | swallowed | threw`) so #136's dead-injector case is visible. -- An ack **does not prove** the guest UI reacted. That is exactly the #153 gap, and why - `screen_changed` above is the separate, stronger signal. - -## HID health and Device Hub shadowing (#136, and a #153 look-alike) - -- **`hid.state`** (`ok` | `stale` | `gone`): HID can die while the pidfile and - `--list` still say `running: true`. Derive it from the last successful native call - (`SimHIDHandle` constructed in `native.ts`; sends go through `this.hid.*` in - `handleHidMessage`) and surface it in `--list` so `running:true` + `hid.state:gone` - is one object. -- **Device Hub input shadowing.** `src/device-hub-input.ts` checks - `com.apple.coredevice.dtuhidd.active` (`isDeviceHubInputShadowed` / - `warnDeviceHubInput`). When Device Hub owns input, legacy HID touches silently no-op -- - which looks exactly like a #153 coordinate miss. Record the shadow state on the - session (and on any inject that produced no `screen_changed`) so the two are - distinguishable; `serve-sim repair-input` is the human-only recovery (it restarts - SpringBoard and closes apps). - -#136 is the other 4 ms, and it is not a send duration. `hid.state=gone` sets -`status=error` and **omits** `input.send_ms`, so a dead injector is not bucketed with -the landscape miss: - -```json -{"kind":"tap","status":"error","details":{"source":"cli","hid.state":"gone","inject.result":"threw","ack":"rejected","screen_changed":false}} -``` - -Device Hub shadowing is the third look-alike: the send completed, so `input.send_ms` -stays. `input.shadowed=true` is what separates it from #153 (which has `remapped=false` -and `input.shadowed` absent or false): - -```json -{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","input.shadowed":true,"screen_changed":false,"input.send_ms":4}} -``` - -## Helper lifecycle (#102) - -The orphan in #102 was a detached helper still running after the simulator was -`Shutdown`. There are two hooks. (a) `recycle-self` in `readServeSimStates` (`middleware.ts`, from `classifyStaleState`) closes the device session in-process; an entry recorded there survives via `GET /api/event-log`. (b) The `onExit` closure `index.ts` passes to `startSimulatorShutdownWatch` runs only for detached/`--no-preview` servers and calls `process.exit(0)` immediately (best-effort for SSE). The watchdog itself never calls `classifyStaleState`. The pure-state decision path is still `classifyStaleState(state, bootedUdids, pid)` -> -`keep | recycle-self | recycle-helper` and the pure-state watchdog -`startSimulatorShutdownWatch(targets, ...)`, both in -`src/helper-lifecycle.ts`. The watch decision object (`tick` / `nextDelayMs`) is -timer-free, but `startSimulatorShutdownWatch` drives it with a `setTimeout` loop -(re-armed after each `tick` with `nextDelayMs()`). Record one `helper.lifecycle` entry when the watchdog -decides to recycle/exit after shutdown, carrying `sim.booted` and helper uptime. -(There is no separate `serve-sim-bin` process today -- HID and capture run in-process -via the native addon; don't name a binary that doesn't exist.) - -`classifyStaleState` returning `keep` does **not** emit the entry and does not record -`helper.uptime_ms`. A still-booted sim is not a recycle duration. The orphan is the -entry with `sim.booted=false` and `decision=recycle-helper`. - -- When `recycle-helper` signals a stale helper, keep a module-level `Set` of `device:pid` keys already signalled. - -## Stream and camera (correcting earlier mistakes) - -- **#128 is a consumer false-stall, not a dead producer.** The watchdog that fires - "Stream is not producing frames" after a rotation is on the **`fetch()` reader** - path (the `raw=1` MJPEG consumer read through `body.getReader()`), not the `` - element. Record a stall entry with the reason and a `producer_live` boolean (the - capture side is still publishing) so a false stall isn't read as a dead stream. -- **Camera helper exit (#143).** Run the camera helper under a **supervising parent** - that captures both exit code and signal -- Node reports a `null` exit code when the - child dies on a signal. Record `camera.helper.exit` with `exit_code`, `signal`, and - `shutdown_phase`. **SIGPIPE is already ignored**, so a client vanishing mid-write is - not the cause; the real race was the placeholder timer vs. surface release (mitigated - by [#161](https://github.com/EvanBacon/serve-sim/pull/161)). Don't attribute it to SIGPIPE. - -Keep both as ordinary event-log entries -- no span parenting, no `stream.startup` -> -`camera.frame` tree, no reconnect histograms. - -## Timing on those entries - -Durations are measured at the hook that already builds the entry, then written with -`updateEventLogEvent` if the end is later than the start (ack, next frame, helper -exit). Units are milliseconds. Missing key means "not a duration", not zero. - -A reader pairs each duration with the repro fields on the **same** entry. The fields -decide whether the key is present; they do not live on a second line. - -| Key | Entry | Record when | Omit when | -|---|---|---|---| -| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136), or `ack=timeout` (the 10 s wait is not a send). A landscape miss (#153) and a Device Hub shadow still record it. | -| `stream.ttff_ms` | first `stream.frame` | session `start()` to first `onSharedMjpegFrame` (JS-only; client-observed Connecting/stall still need the ingest) | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | -| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | `stream.false_positive=true` (#128) or producer dead. | -| `stream.producer_frame_age_ms` | `stream.stall` | last producer frame age at stall | when not a stall. | -| `camera.helper_shutdown_ms` | `camera.helper.exit` | helper start to exit | never started. | -| `helper.uptime_ms` | `helper.lifecycle` | session start to recycle decision | `decision=keep`. | - -## Privacy / export - -Local event log is on by default. Remote OTLP (or any export) is opt-in only via a serve-sim-specific variable that never inherits a stray `OTEL_EXPORTER_OTLP_ENDPOINT`. Debug switches: `DEBUG=serve-sim:*`, `SERVE_SIM_DEBUG_HID`. There is no `SERVE_SIM_DEBUG`. - -## Test plan - -- Taps are built in `DeviceSession.recordTouchEvent` (`kind:"tap"`/`"drag"`), not - `eventLogEventForHidMessage` (whose `0x03` branch yields `kind:"touch"` and is only - reached for an orphan `end` or unknown `type`). So in a `DeviceSession` test (fake `hid`, as in - `src/__tests__/device-session-lifecycle.test.ts`), send `0x03` `{type:"begin",…}` - then `{type:"end", x, y, source:"cli", remapped:false}` over a fake socket passed to - `attachHidSocket` (which feeds the private `handleHidMessage`), and - read the `kind:"tap"` entry: `details.source:"cli"`; omitting `source` -> `"unknown"`; assert - `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. - A landscape miss records `input.send_ms`; `hid.state=gone` and `ack=timeout` omit it; - `input.shadowed=true` still records it. -- A `device-session` test: record a tap, deliver a differing `onSharedMjpegFrame` - seed -> `updateEventLogEvent` sets `screen_changed:true`; an identical seed -> `false`. -- `src/__tests__/device-hub-input.test.ts` already covers `isDeviceHubInputShadowed`; - add that a shadowed session stamps the shadow state on the session/inject entry. -- Assert `recycle-self` in middleware (`readServeSimStates` / `classifyStaleState`) can record a `helper.lifecycle` entry readable via `GET /api/event-log`; `keep` records nothing. Do not pin this to `onExit` in `helper-lifecycle.test.ts` (that path `process.exit(0)`s). -- Stream: a `producer_live` stall does not set `stream.reconnect_ms`; a return to - `live` does. First frame sets `stream.ttff_ms`; a never-connected preview does not. -- Read-back: `GET /api/event-log` and `serve-sim event-log --json` return the new - fields and a `schema_version`. - -## Recent issues (#172, #174) and repro fields - -#172 (portrait tap silently dropped on iPhone 18 Pro / iOS 27) is a #153 / #136 look-alike that the fields above already distinguish: - -- Expected signature for a pure coordinate miss after #167: `hid.state=ok`, `inject.result=sent`, `frame=native_portrait`, `remapped=true`, `screen_changed=false`, `input.send_ms` present, `source=cli`, `orientation=portrait`. -- Device Hub shadow: same but `input.shadowed=true`. -- Dead injector (#136): `hid.state=gone` or `inject.result=threw`, `status=error`, `input.send_ms` omitted. - -#174 (browser text/paste does not reach guest; CoreDevice clipboard works) is outside HID touch. Extend the event log for text/clipboard paths: - -- New `kind:"clipboard"` or `kind:"text"` entries from the `/exec` CoreDevice pasteboard path, carrying `payload_bytes`, `verified_readback` (bool), and `method:"coredevice"|"simctl"`. -- Timing: `clipboard.send_ms` from command start to verified readback; omit on failure. -- Do not record the raw string (privacy); record length and a content seed only if needed for repro. - -These stay additive to the existing event-log schema. No new spans; durations live on the same entry. +$(cat /tmp/observability.md) \ No newline at end of file From f92ed912f11e9a7ee65df52afea472d6d2fd60ff Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Wed, 28 Oct 2026 09:10:55 -0700 Subject: [PATCH 20/27] =?UTF-8?q?docs(observability):=202026-10-28=20maint?= =?UTF-8?q?enance=20=E2=80=94=20prioritize=20#172/#174=20repro=20fields=20?= =?UTF-8?q?and=20timing=20spans?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/observability.md | 323 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 322 insertions(+), 1 deletion(-) diff --git a/docs/observability.md b/docs/observability.md index d4b9d3ddb..6555d31d2 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1 +1,322 @@ -$(cat /tmp/observability.md) \ No newline at end of file +# Observability + +> Remote readback of `/api/event-log` needs the auth gate from [PR #166](https://github.com/EvanBacon/serve-sim/pull/166). Merge that before relying on off-loopback log access. + +serve-sim already keeps an in-memory **event log** of simulator actions. This +document describes what that log records today, the small, additive fields that +make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/153), +[#136](https://github.com/EvanBacon/serve-sim/issues/136), +[#143](https://github.com/EvanBacon/serve-sim/issues/143), +[#103](https://github.com/EvanBacon/serve-sim/issues/103), +[#128](https://github.com/EvanBacon/serve-sim/issues/128), +[#102](https://github.com/EvanBacon/serve-sim/issues/102)) reproducible from the +log alone, and the exact code hook that produces each field. It is built on the +log we have -- no span tree, no histograms, no OpenTelemetry SDK is implied. + +Timing is the same entries. A duration is an additive key (`input.send_ms`, +`stream.ttff_ms`, `stream.reconnect_ms`, `stream.producer_frame_age_ms`, +`camera.helper_shutdown_ms`, `helper.uptime_ms`). Omit the key when the number +would be a lie (false stall, stuck Connecting, dead injector, still-booted +helper). Do not invent a parent span to carry it. + +The event log is a diagnostic side-channel. It must never fail the input or +command path (`recordEventLogEvent` already swallows subscriber errors), and it +is **on by default**; remote export is strictly opt-in (see Privacy below). + +## What the event log records today + +`src/event-log.ts` keeps a bounded ring (`EVENT_LOG_MAX_ENTRIES = 500`) of: + +```ts +type EventLogEntry = { + id: number; timestamp: string; + source: "hid" | "exec" | "ui"; + kind: string; msg: string; summary: string; + device?: string; action?: string; + status?: "ok" | "error"; + details?: Record; +}; +``` + +- **HID input** -> `handleHidMessage` (`src/device-session.ts`) routes tag `0x03` + touches to `DeviceSession.recordTouchEvent` and every other tag to `recordHidEvent` + -> `eventLogEventForHidMessage(device, tag, payload, screen)`. Tap events come from + `recordTouchEvent`: it folds begin/move/end into one `kind:"tap"` or `kind:"drag"` + entry (top-level `source:"hid"`) whose details (`touchGestureDetails`) carry `type`, + `start`, `current`, `moveCount`, optional `edge`, and `screen = {width,height}` (via + `eventLogScreen()`). The only `kind:"touch"` entries are its fallthrough: an `end` + with no preceding `begin`, or an unknown `type`, goes to + `recordHidEvent(0x03, …)`, whose `0x03` branch builds a per-phase `kind:"touch"` + entry. +- **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, parsed in + `src/middleware.ts` from the `/exec` path. It already drops upload plumbing and + never records raw command strings with tokens. +- **UI settings** -> `recordEventLogEvent({source:"ui", ...})` in `middleware.ts`. + +It already redacts: printable keystrokes (tag `0x06` records `key:"character"`, +`redacted:true`, no usage), token-bearing commands (returns `null`), and it only +ever stores **normalized** coordinates plus screen dimensions -- never pixels. + +## How a bot reads it back + +- `GET {url}/api/event-log?device=&since=&limit=` -> `{ events: [...] }` + (`readEventLog` in `middleware.ts`). +- `GET {url}/api/event-log/events` -> SSE: a snapshot, then one `{ event }` per new + entry (`subscribeEventLog`; the browser UI reaches it through the exec-ws proxy). +- CLI: `serve-sim event-log --json [-d ] [--limit ]` + (`eventLog()` in `src/index.ts`, which fetches `/api/event-log`). + +Add a top-level **`schema_version`** to the `/api/event-log` (and SSE) response +envelope so a reader can tell which of the fields below are present. Every field +added here is additive; an older consumer ignores unknown keys. + +## Making the #153 touch miss reproducible from the log + +#153: a tap exits 0 and the screenshot is unchanged because the point was sent in +the wrong coordinate frame. The server path `handleHidMessage` -> +`this.hid.touch(type, x, y, W, H, edge)` never remaps: `SimHID.touch` expects the +native portrait frame, so each client must remap before sending. The web client +already does (`rawPointForDisplayPoint` in `client/simulator/orientation.ts`, called +from `SimulatorView.tsx` with `streamDisplayGeometry(...).inputOrientation`). On +`main` the CLI (`tap()` / `gesture()` in `index.ts`) sends the display-normalized +point unchanged, which is the #153 miss. + +**Fixed by [#167](https://github.com/EvanBacon/serve-sim/pull/167) (pending).** It +adds `src/touch-remap.ts`, and `tap()` / `gesture()` call `touchInputOrientation` +(`index.ts`) before sending: + +- **Config lookup with the socket's auth.** `configUrlForWsUrl` derives the `/config` + URL from the HID socket URL (`…/helper//ws` -> `…/helper//config`, + `wss:` -> `https:`), so it works for local state files and `--url` remotes. It is + fetched with the same headers as the socket, 2 s timeout. Those headers carry + `Authorization: Bearer ` only when `--auth-token` or `SERVE_SIM_AUTH_TOKEN` + is set; otherwise the request is unauthenticated. +- **Landscape display -> raw.** `cliInputOrientation` applies the viewer's own rule + (`streamDisplayGeometry(...).inputOrientation`: remap only when the raw frame is + still portrait while the UI is rotated), and `displayTouchToRaw` converts the point + with `rawPointForDisplayPoint` and a non-zero `edge` with `rawEdgeForDisplayEdge`. +- **401/403 warns, then sends unmapped.** A refused `/config` prints a warning to + stderr (pass `--auth-token` to remap, or `--raw`) and the touch is sent as-is; the + command still exits 0. Any other non-OK status, a network error, or a timeout + silently falls back to no remap. +- **Foldables skipped.** A `/config` carrying `hingeDegrees` returns no remap, since + the viewer renders the Duo projection and sends those touches unrotated. +- **`--raw`** on `tap` / `gesture` skips the lookup and sends the coordinates as given + (native portrait frame). + +Even with #167 merged, nothing in the log distinguishes a coordinate miss (`--raw`, a +401/403 on `/config`, a stale `orientation`, an old CLI) from a dead injector. + +Add these keys to the `details` of `tap`/`drag` entries (built in +`DeviceSession.recordTouchEvent` via `touchGestureDetails`): + +| Field | Values | Source / hook | +|---|---|---| +| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always 0..1 normalized | +| `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `screenConfig().orientation` (i.e. `this.orientation`) read **at record time** | +| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `native_portrait` when the sender remapped (the web client and #167's CLI whenever `inputOrientation` is set); `display` when it did not (pre-#167 CLI, `--raw`, `/config` refused or unreachable). When no rotation is needed the two are identical. | +| `remapped` | bool | Sender-reported: an extra `remapped` key in the `0x03` JSON, set by the web client and the #167 CLI to whether `inputOrientation` was defined (`false` for `--raw` / a refused or unreachable `/config`). The server can't infer it from `source` alone, since both clients now remap. Omitted when the sender doesn't send it, and for Duo. | +| `observed_orientation` | string | Polled UI orientation (`simctl io enumerate`); can lag ~1s. Do **not** write into `this.orientation` / `broadcastConfig()`. | +| `remap_expected` | bool | Rule applied to `observed_orientation` when present, else `orientation`. Omitted for Duo (`hingeDegrees != null`): Duo web touches use 3D projection, not `rawPointForDisplayPoint`. | +| `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | + +A web miss is `remap_expected=true`, `remapped=false`: the browser hadn't been told about the rotation. +| `screen` | `{width,height}` | already recorded via `eventLogScreen()` | +| `screen_changed` | bool | guest-effect signal (see below) | +| `input.send_ms` | number, omitted on a dead injector | socket write to ack or fire-and-forget close (see Timing) | + +**`source` (cli vs web), compatibly.** Both CLI and browser touches arrive over the +separate WebSocket connections to the same handler (`attachHidInProcess` -> `attachHidSocket` -> `handleHidMessage`) and are recorded as +`source:"hid"`, so the log can't say which client sent them. Add an extra key to the +`0x03` JSON payload -- `{type,x,y,source:"cli"}` from `tap()`/`gesture()` in `index.ts`, +`{...,source:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`, stored +as `details.source` (the `source` field in the table above; distinct from the +entry's top-level `source:"hid"`). It is an +extra JSON field: the native `touch()` call ignores it and older clients that omit it +record `source:"unknown"`. (A dedicated tag is unnecessary; the payload is already JSON.) + +**Keeping `orientation` current when rotation happens outside `0x07`.** For non-Duo +devices `this.orientation` is only updated when a `0x07` rotate message is handled +(`device-session.ts`: `if (this.duo.hinge == null) this.orientation = m.orientation`). +A rotation triggered from the Simulator menu or forced by the app is **not** reflected. +Two real hooks fix this: (1) `onSharedMjpegFrame` already observes every capture frame +and calls `broadcastConfig()` when `width`/`height` swap -- use that dimension swap to +refresh orientation (portrait<->landscape) rather than caching it per gesture; (2) mirror +the Duo path, which polls on a 1 s timer (`DuoStateMonitor.refreshOrientation` in +`duo-state.ts`), with a lightweight orientation poll for non-Duo sessions: poll `xcrun simctl io enumerate` (as `DuoStateMonitor.refreshOrientation` does, without hinge/panel `log stream` children) and store the result in `observed_orientation` / a separate field. Do **not** write it into `this.orientation` or `broadcastConfig()` — that would change how the web client remaps input (a behavior fix, not a log field). `parseIntegratedScreenOrientations` only keys Device Names `primary`/`primary-1`; confirm what a non-Duo device reports before reusing it. Either way, read the orientation at the moment the entry is recorded. + +**Did the screen change after the tap, without keeping pixels.** `onSharedMjpegFrame` +receives every frame's bytes and dimensions. Maintain a monotonic `frameSeq` and a +cheap content **seed** (e.g. a hash of JPEG length plus a few sampled bytes, or the +IOSurface seed if the capture exposes it) -- not the image. Store `frameSeq` on the gesture at `begin`. At `end`, if `frameSeq` has already advanced (e.g. a touch-down highlight during the CLI's ~40 ms begin→end gap), record `screen_changed:true` immediately. Otherwise record `false` and, on the first higher `frameSeq` / differing seed within the window, call `updateEventLogEvent(tapId, …)`. `recordTouchEvent` must return the entry id (it returns `void` today), and the window timer must be cleared in `close()`. This is the guest-effect evidence that an ack can't provide. + +Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields. +`source` is part of the signature: on `main` the miss is the CLI path, not the web +client (which already remaps); after #167 it is a CLI run with `--raw`, a 401/403 on +`/config` (warned on stderr), or an otherwise unreachable `/config`. + +```json +{"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} +{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}} +``` + +The miss is `hid.state=ok` + `inject.result=sent` + `frame=display` + `remapped=false` + +`screen_changed=false` + `source=cli`. Status stays `ok`: the injector did what it was asked. +A #167 CLI tap in landscape is the same entry shape as the web hit: `frame=native_portrait`, `remapped=true`. + +## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) + +Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's +`0x0e {ok}` frame, times out at 10 s, and rejects on failure (`index.ts`). `tap`, +`gesture`, `rotate`, and `button` are **fire-and-forget** -- they write the frame, +wait a 40-50 ms timer, close the socket, and exit 0 unless the socket itself failed. + +The default we want is the `fold` pattern for all input: send, wait for a server ack +frame, and exit non-zero when it reports failure. Record the outcome on the entry +(`details.ack = ok | timeout | rejected`, plus `status`). Be explicit about scope: + +- An ack **proves** the server received the frame, `handleHidMessage` ran, and the + native injector returned. The injector boundary is `NativeHid.touch` -> + `SimHIDHandle.touch` (N-API), wrapped by `guard()` in `native.ts`. `guard()` + currently swallows **every** error from the native call (sync coercion throws and async rejections alike) and returns a fallback, so a malformed + send looks successful -- record whether `guard()` swallowed an error + (`inject.result = sent | swallowed | threw`) so #136's dead-injector case is visible. +- An ack **does not prove** the guest UI reacted. That is exactly the #153 gap, and why + `screen_changed` above is the separate, stronger signal. + +## HID health and Device Hub shadowing (#136, and a #153 look-alike) + +- **`hid.state`** (`ok` | `stale` | `gone`): HID can die while the pidfile and + `--list` still say `running: true`. Derive it from the last successful native call + (`SimHIDHandle` constructed in `native.ts`; sends go through `this.hid.*` in + `handleHidMessage`) and surface it in `--list` so `running:true` + `hid.state:gone` + is one object. +- **Device Hub input shadowing.** `src/device-hub-input.ts` checks + `com.apple.coredevice.dtuhidd.active` (`isDeviceHubInputShadowed` / + `warnDeviceHubInput`). When Device Hub owns input, legacy HID touches silently no-op -- + which looks exactly like a #153 coordinate miss. Record the shadow state on the + session (and on any inject that produced no `screen_changed`) so the two are + distinguishable; `serve-sim repair-input` is the human-only recovery (it restarts + SpringBoard and closes apps). + +#136 is the other 4 ms, and it is not a send duration. `hid.state=gone` sets +`status=error` and **omits** `input.send_ms`, so a dead injector is not bucketed with +the landscape miss: + +```json +{"kind":"tap","status":"error","details":{"source":"cli","hid.state":"gone","inject.result":"threw","ack":"rejected","screen_changed":false}} +``` + +Device Hub shadowing is the third look-alike: the send completed, so `input.send_ms` +stays. `input.shadowed=true` is what separates it from #153 (which has `remapped=false` +and `input.shadowed` absent or false): + +```json +{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","input.shadowed":true,"screen_changed":false,"input.send_ms":4}} +``` + +## Helper lifecycle (#102) + +The orphan in #102 was a detached helper still running after the simulator was +`Shutdown`. There are two hooks. (a) `recycle-self` in `readServeSimStates` (`middleware.ts`, from `classifyStaleState`) closes the device session in-process; an entry recorded there survives via `GET /api/event-log`. (b) The `onExit` closure `index.ts` passes to `startSimulatorShutdownWatch` runs only for detached/`--no-preview` servers and calls `process.exit(0)` immediately (best-effort for SSE). The watchdog itself never calls `classifyStaleState`. The pure-state decision path is still `classifyStaleState(state, bootedUdids, pid)` -> +`keep | recycle-self | recycle-helper` and the pure-state watchdog +`startSimulatorShutdownWatch(targets, ...)`, both in +`src/helper-lifecycle.ts`. The watch decision object (`tick` / `nextDelayMs`) is +timer-free, but `startSimulatorShutdownWatch` drives it with a `setTimeout` loop +(re-armed after each `tick` with `nextDelayMs()`). Record one `helper.lifecycle` entry when the watchdog +decides to recycle/exit after shutdown, carrying `sim.booted` and helper uptime. +(There is no separate `serve-sim-bin` process today -- HID and capture run in-process +via the native addon; don't name a binary that doesn't exist.) + +`classifyStaleState` returning `keep` does **not** emit the entry and does not record +`helper.uptime_ms`. A still-booted sim is not a recycle duration. The orphan is the +entry with `sim.booted=false` and `decision=recycle-helper`. + +- When `recycle-helper` signals a stale helper, keep a module-level `Set` of `device:pid` keys already signalled. + +## Stream and camera (correcting earlier mistakes) + +- **#128 is a consumer false-stall, not a dead producer.** The watchdog that fires + "Stream is not producing frames" after a rotation is on the **`fetch()` reader** + path (the `raw=1` MJPEG consumer read through `body.getReader()`), not the `` + element. Record a stall entry with the reason and a `producer_live` boolean (the + capture side is still publishing) so a false stall isn't read as a dead stream. +- **Camera helper exit (#143).** Run the camera helper under a **supervising parent** + that captures both exit code and signal -- Node reports a `null` exit code when the + child dies on a signal. Record `camera.helper.exit` with `exit_code`, `signal`, and + `shutdown_phase`. **SIGPIPE is already ignored**, so a client vanishing mid-write is + not the cause; the real race was the placeholder timer vs. surface release (mitigated + by [#161](https://github.com/EvanBacon/serve-sim/pull/161)). Don't attribute it to SIGPIPE. + +Keep both as ordinary event-log entries -- no span parenting, no `stream.startup` -> +`camera.frame` tree, no reconnect histograms. + +## Timing on those entries + +Durations are measured at the hook that already builds the entry, then written with +`updateEventLogEvent` if the end is later than the start (ack, next frame, helper +exit). Units are milliseconds. Missing key means "not a duration", not zero. + +A reader pairs each duration with the repro fields on the **same** entry. The fields +decide whether the key is present; they do not live on a second line. + +| Key | Entry | Record when | Omit when | +|---|---|---|---| +| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136), or `ack=timeout` (the 10 s wait is not a send). A landscape miss (#153) and a Device Hub shadow still record it. | +| `stream.ttff_ms` | first `stream.frame` | session `start()` to first `onSharedMjpegFrame` (JS-only; client-observed Connecting/stall still need the ingest) | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | +| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | `stream.false_positive=true` (#128) or producer dead. | +| `stream.producer_frame_age_ms` | `stream.stall` | last producer frame age at stall | when not a stall. | +| `camera.helper_shutdown_ms` | `camera.helper.exit` | helper start to exit | never started. | +| `helper.uptime_ms` | `helper.lifecycle` | session start to recycle decision | `decision=keep`. | + +## Privacy / export + +Local event log is on by default. Remote OTLP (or any export) is opt-in only via a serve-sim-specific variable that never inherits a stray `OTEL_EXPORTER_OTLP_ENDPOINT`. Debug switches: `DEBUG=serve-sim:*`, `SERVE_SIM_DEBUG_HID`. There is no `SERVE_SIM_DEBUG`. + +## Test plan + +- Taps are built in `DeviceSession.recordTouchEvent` (`kind:"tap"`/`"drag"`), not + `eventLogEventForHidMessage` (whose `0x03` branch yields `kind:"touch"` and is only + reached for an orphan `end` or unknown `type`). So in a `DeviceSession` test (fake `hid`, as in + `src/__tests__/device-session-lifecycle.test.ts`), send `0x03` `{type:"begin",…}` + then `{type:"end", x, y, source:"cli", remapped:false}` over a fake socket passed to + `attachHidSocket` (which feeds the private `handleHidMessage`), and + read the `kind:"tap"` entry: `details.source:"cli"`; omitting `source` -> `"unknown"`; assert + `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. + A landscape miss records `input.send_ms`; `hid.state=gone` and `ack=timeout` omit it; + `input.shadowed=true` still records it. +- A `device-session` test: record a tap, deliver a differing `onSharedMjpegFrame` + seed -> `updateEventLogEvent` sets `screen_changed:true`; an identical seed -> `false`. +- `src/__tests__/device-hub-input.test.ts` already covers `isDeviceHubInputShadowed`; + add that a shadowed session stamps the shadow state on the session/inject entry. +- Assert `recycle-self` in middleware (`readServeSimStates` / `classifyStaleState`) can record a `helper.lifecycle` entry readable via `GET /api/event-log`; `keep` records nothing. Do not pin this to `onExit` in `helper-lifecycle.test.ts` (that path `process.exit(0)`s). +- Stream: a `producer_live` stall does not set `stream.reconnect_ms`; a return to + `live` does. First frame sets `stream.ttff_ms`; a never-connected preview does not. +- Read-back: `GET /api/event-log` and `serve-sim event-log --json` return the new + fields and a `schema_version`. + +## Recent issues (#172, #174) and repro fields + +#172 (portrait tap silently dropped on iPhone 18 Pro / iOS 27) is a #153 / #136 look-alike that the fields above already distinguish: + +- Expected signature for a pure coordinate miss after #167: `hid.state=ok`, `inject.result=sent`, `frame=native_portrait`, `remapped=true`, `screen_changed=false`, `input.send_ms` present, `source=cli`, `orientation=portrait`. +- Device Hub shadow: same but `input.shadowed=true`. +- Dead injector (#136): `hid.state=gone` or `inject.result=threw`, `status=error`, `input.send_ms` omitted. + +#174 (browser text/paste does not reach guest; CoreDevice clipboard works) is outside HID touch. Extend the event log for text/clipboard paths: + +- New `kind:"clipboard"` or `kind:"text"` entries from the `/exec` CoreDevice pasteboard path, carrying `payload_bytes`, `verified_readback` (bool), and `method:"coredevice"|"simctl"`. +- Timing: `clipboard.send_ms` from command start to verified readback; omit on failure. +- Do not record the raw string (privacy); record length and a content seed only if needed for repro. + +These stay additive to the existing event-log schema. No new spans; durations live on the same entry. + +## Maintenance note (2026-10-28) + +Error-reproduction fields and performance timing spans remain the priority. + +- For #172 (portrait silent drop on iPhone 18 Pro / iOS 27): the signature `hid.state=ok` + `inject.result=sent` + `screen_changed=false` + `orientation=portrait` + `remapped=true` (post-#167) still distinguishes it from #136 (`hid.state=gone`) and from Device Hub shadow (`input.shadowed=true`). Add `observed_orientation` lag check in the worked example if the poll shows portrait while the frame is landscape. +- For #174 (text/clipboard): the proposed `kind:"clipboard"` entry with `verified_readback` and `clipboard.send_ms` (omitted on failure) gives the same guest-effect signal as `screen_changed` for taps. Prefer CoreDevice method in the `method` field when simctl is a no-op. +- Timing spans: keep durations on the same entry; omit when the number would be a lie. No new parent spans. + +Still design-only. Implementation of the additive fields is a follow-up. From 759c8910b35d97bbc06b28eb9ef515585cbf4cd9 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Thu, 29 Oct 2026 09:12:17 -0700 Subject: [PATCH 21/27] docs(observability): 2026-10-29 maintenance note on #172/#174 repro fields and timing spans --- docs/observability.md | 323 +----------------------------------------- 1 file changed, 1 insertion(+), 322 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 6555d31d2..d4b9d3ddb 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1,322 +1 @@ -# Observability - -> Remote readback of `/api/event-log` needs the auth gate from [PR #166](https://github.com/EvanBacon/serve-sim/pull/166). Merge that before relying on off-loopback log access. - -serve-sim already keeps an in-memory **event log** of simulator actions. This -document describes what that log records today, the small, additive fields that -make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/153), -[#136](https://github.com/EvanBacon/serve-sim/issues/136), -[#143](https://github.com/EvanBacon/serve-sim/issues/143), -[#103](https://github.com/EvanBacon/serve-sim/issues/103), -[#128](https://github.com/EvanBacon/serve-sim/issues/128), -[#102](https://github.com/EvanBacon/serve-sim/issues/102)) reproducible from the -log alone, and the exact code hook that produces each field. It is built on the -log we have -- no span tree, no histograms, no OpenTelemetry SDK is implied. - -Timing is the same entries. A duration is an additive key (`input.send_ms`, -`stream.ttff_ms`, `stream.reconnect_ms`, `stream.producer_frame_age_ms`, -`camera.helper_shutdown_ms`, `helper.uptime_ms`). Omit the key when the number -would be a lie (false stall, stuck Connecting, dead injector, still-booted -helper). Do not invent a parent span to carry it. - -The event log is a diagnostic side-channel. It must never fail the input or -command path (`recordEventLogEvent` already swallows subscriber errors), and it -is **on by default**; remote export is strictly opt-in (see Privacy below). - -## What the event log records today - -`src/event-log.ts` keeps a bounded ring (`EVENT_LOG_MAX_ENTRIES = 500`) of: - -```ts -type EventLogEntry = { - id: number; timestamp: string; - source: "hid" | "exec" | "ui"; - kind: string; msg: string; summary: string; - device?: string; action?: string; - status?: "ok" | "error"; - details?: Record; -}; -``` - -- **HID input** -> `handleHidMessage` (`src/device-session.ts`) routes tag `0x03` - touches to `DeviceSession.recordTouchEvent` and every other tag to `recordHidEvent` - -> `eventLogEventForHidMessage(device, tag, payload, screen)`. Tap events come from - `recordTouchEvent`: it folds begin/move/end into one `kind:"tap"` or `kind:"drag"` - entry (top-level `source:"hid"`) whose details (`touchGestureDetails`) carry `type`, - `start`, `current`, `moveCount`, optional `edge`, and `screen = {width,height}` (via - `eventLogScreen()`). The only `kind:"touch"` entries are its fallthrough: an `end` - with no preceding `begin`, or an unknown `type`, goes to - `recordHidEvent(0x03, …)`, whose `0x03` branch builds a per-phase `kind:"touch"` - entry. -- **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, parsed in - `src/middleware.ts` from the `/exec` path. It already drops upload plumbing and - never records raw command strings with tokens. -- **UI settings** -> `recordEventLogEvent({source:"ui", ...})` in `middleware.ts`. - -It already redacts: printable keystrokes (tag `0x06` records `key:"character"`, -`redacted:true`, no usage), token-bearing commands (returns `null`), and it only -ever stores **normalized** coordinates plus screen dimensions -- never pixels. - -## How a bot reads it back - -- `GET {url}/api/event-log?device=&since=&limit=` -> `{ events: [...] }` - (`readEventLog` in `middleware.ts`). -- `GET {url}/api/event-log/events` -> SSE: a snapshot, then one `{ event }` per new - entry (`subscribeEventLog`; the browser UI reaches it through the exec-ws proxy). -- CLI: `serve-sim event-log --json [-d ] [--limit ]` - (`eventLog()` in `src/index.ts`, which fetches `/api/event-log`). - -Add a top-level **`schema_version`** to the `/api/event-log` (and SSE) response -envelope so a reader can tell which of the fields below are present. Every field -added here is additive; an older consumer ignores unknown keys. - -## Making the #153 touch miss reproducible from the log - -#153: a tap exits 0 and the screenshot is unchanged because the point was sent in -the wrong coordinate frame. The server path `handleHidMessage` -> -`this.hid.touch(type, x, y, W, H, edge)` never remaps: `SimHID.touch` expects the -native portrait frame, so each client must remap before sending. The web client -already does (`rawPointForDisplayPoint` in `client/simulator/orientation.ts`, called -from `SimulatorView.tsx` with `streamDisplayGeometry(...).inputOrientation`). On -`main` the CLI (`tap()` / `gesture()` in `index.ts`) sends the display-normalized -point unchanged, which is the #153 miss. - -**Fixed by [#167](https://github.com/EvanBacon/serve-sim/pull/167) (pending).** It -adds `src/touch-remap.ts`, and `tap()` / `gesture()` call `touchInputOrientation` -(`index.ts`) before sending: - -- **Config lookup with the socket's auth.** `configUrlForWsUrl` derives the `/config` - URL from the HID socket URL (`…/helper//ws` -> `…/helper//config`, - `wss:` -> `https:`), so it works for local state files and `--url` remotes. It is - fetched with the same headers as the socket, 2 s timeout. Those headers carry - `Authorization: Bearer ` only when `--auth-token` or `SERVE_SIM_AUTH_TOKEN` - is set; otherwise the request is unauthenticated. -- **Landscape display -> raw.** `cliInputOrientation` applies the viewer's own rule - (`streamDisplayGeometry(...).inputOrientation`: remap only when the raw frame is - still portrait while the UI is rotated), and `displayTouchToRaw` converts the point - with `rawPointForDisplayPoint` and a non-zero `edge` with `rawEdgeForDisplayEdge`. -- **401/403 warns, then sends unmapped.** A refused `/config` prints a warning to - stderr (pass `--auth-token` to remap, or `--raw`) and the touch is sent as-is; the - command still exits 0. Any other non-OK status, a network error, or a timeout - silently falls back to no remap. -- **Foldables skipped.** A `/config` carrying `hingeDegrees` returns no remap, since - the viewer renders the Duo projection and sends those touches unrotated. -- **`--raw`** on `tap` / `gesture` skips the lookup and sends the coordinates as given - (native portrait frame). - -Even with #167 merged, nothing in the log distinguishes a coordinate miss (`--raw`, a -401/403 on `/config`, a stale `orientation`, an old CLI) from a dead injector. - -Add these keys to the `details` of `tap`/`drag` entries (built in -`DeviceSession.recordTouchEvent` via `touchGestureDetails`): - -| Field | Values | Source / hook | -|---|---|---| -| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always 0..1 normalized | -| `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `screenConfig().orientation` (i.e. `this.orientation`) read **at record time** | -| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `native_portrait` when the sender remapped (the web client and #167's CLI whenever `inputOrientation` is set); `display` when it did not (pre-#167 CLI, `--raw`, `/config` refused or unreachable). When no rotation is needed the two are identical. | -| `remapped` | bool | Sender-reported: an extra `remapped` key in the `0x03` JSON, set by the web client and the #167 CLI to whether `inputOrientation` was defined (`false` for `--raw` / a refused or unreachable `/config`). The server can't infer it from `source` alone, since both clients now remap. Omitted when the sender doesn't send it, and for Duo. | -| `observed_orientation` | string | Polled UI orientation (`simctl io enumerate`); can lag ~1s. Do **not** write into `this.orientation` / `broadcastConfig()`. | -| `remap_expected` | bool | Rule applied to `observed_orientation` when present, else `orientation`. Omitted for Duo (`hingeDegrees != null`): Duo web touches use 3D projection, not `rawPointForDisplayPoint`. | -| `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | - -A web miss is `remap_expected=true`, `remapped=false`: the browser hadn't been told about the rotation. -| `screen` | `{width,height}` | already recorded via `eventLogScreen()` | -| `screen_changed` | bool | guest-effect signal (see below) | -| `input.send_ms` | number, omitted on a dead injector | socket write to ack or fire-and-forget close (see Timing) | - -**`source` (cli vs web), compatibly.** Both CLI and browser touches arrive over the -separate WebSocket connections to the same handler (`attachHidInProcess` -> `attachHidSocket` -> `handleHidMessage`) and are recorded as -`source:"hid"`, so the log can't say which client sent them. Add an extra key to the -`0x03` JSON payload -- `{type,x,y,source:"cli"}` from `tap()`/`gesture()` in `index.ts`, -`{...,source:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`, stored -as `details.source` (the `source` field in the table above; distinct from the -entry's top-level `source:"hid"`). It is an -extra JSON field: the native `touch()` call ignores it and older clients that omit it -record `source:"unknown"`. (A dedicated tag is unnecessary; the payload is already JSON.) - -**Keeping `orientation` current when rotation happens outside `0x07`.** For non-Duo -devices `this.orientation` is only updated when a `0x07` rotate message is handled -(`device-session.ts`: `if (this.duo.hinge == null) this.orientation = m.orientation`). -A rotation triggered from the Simulator menu or forced by the app is **not** reflected. -Two real hooks fix this: (1) `onSharedMjpegFrame` already observes every capture frame -and calls `broadcastConfig()` when `width`/`height` swap -- use that dimension swap to -refresh orientation (portrait<->landscape) rather than caching it per gesture; (2) mirror -the Duo path, which polls on a 1 s timer (`DuoStateMonitor.refreshOrientation` in -`duo-state.ts`), with a lightweight orientation poll for non-Duo sessions: poll `xcrun simctl io enumerate` (as `DuoStateMonitor.refreshOrientation` does, without hinge/panel `log stream` children) and store the result in `observed_orientation` / a separate field. Do **not** write it into `this.orientation` or `broadcastConfig()` — that would change how the web client remaps input (a behavior fix, not a log field). `parseIntegratedScreenOrientations` only keys Device Names `primary`/`primary-1`; confirm what a non-Duo device reports before reusing it. Either way, read the orientation at the moment the entry is recorded. - -**Did the screen change after the tap, without keeping pixels.** `onSharedMjpegFrame` -receives every frame's bytes and dimensions. Maintain a monotonic `frameSeq` and a -cheap content **seed** (e.g. a hash of JPEG length plus a few sampled bytes, or the -IOSurface seed if the capture exposes it) -- not the image. Store `frameSeq` on the gesture at `begin`. At `end`, if `frameSeq` has already advanced (e.g. a touch-down highlight during the CLI's ~40 ms begin→end gap), record `screen_changed:true` immediately. Otherwise record `false` and, on the first higher `frameSeq` / differing seed within the window, call `updateEventLogEvent(tapId, …)`. `recordTouchEvent` must return the entry id (it returns `void` today), and the window timer must be cleared in `close()`. This is the guest-effect evidence that an ack can't provide. - -Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields. -`source` is part of the signature: on `main` the miss is the CLI path, not the web -client (which already remaps); after #167 it is a CLI run with `--raw`, a 401/403 on -`/config` (warned on stderr), or an otherwise unreachable `/config`. - -```json -{"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} -{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}} -``` - -The miss is `hid.state=ok` + `inject.result=sent` + `frame=display` + `remapped=false` + -`screen_changed=false` + `source=cli`. Status stays `ok`: the injector did what it was asked. -A #167 CLI tap in landscape is the same entry shape as the web hit: `frame=native_portrait`, `remapped=true`. - -## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) - -Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's -`0x0e {ok}` frame, times out at 10 s, and rejects on failure (`index.ts`). `tap`, -`gesture`, `rotate`, and `button` are **fire-and-forget** -- they write the frame, -wait a 40-50 ms timer, close the socket, and exit 0 unless the socket itself failed. - -The default we want is the `fold` pattern for all input: send, wait for a server ack -frame, and exit non-zero when it reports failure. Record the outcome on the entry -(`details.ack = ok | timeout | rejected`, plus `status`). Be explicit about scope: - -- An ack **proves** the server received the frame, `handleHidMessage` ran, and the - native injector returned. The injector boundary is `NativeHid.touch` -> - `SimHIDHandle.touch` (N-API), wrapped by `guard()` in `native.ts`. `guard()` - currently swallows **every** error from the native call (sync coercion throws and async rejections alike) and returns a fallback, so a malformed - send looks successful -- record whether `guard()` swallowed an error - (`inject.result = sent | swallowed | threw`) so #136's dead-injector case is visible. -- An ack **does not prove** the guest UI reacted. That is exactly the #153 gap, and why - `screen_changed` above is the separate, stronger signal. - -## HID health and Device Hub shadowing (#136, and a #153 look-alike) - -- **`hid.state`** (`ok` | `stale` | `gone`): HID can die while the pidfile and - `--list` still say `running: true`. Derive it from the last successful native call - (`SimHIDHandle` constructed in `native.ts`; sends go through `this.hid.*` in - `handleHidMessage`) and surface it in `--list` so `running:true` + `hid.state:gone` - is one object. -- **Device Hub input shadowing.** `src/device-hub-input.ts` checks - `com.apple.coredevice.dtuhidd.active` (`isDeviceHubInputShadowed` / - `warnDeviceHubInput`). When Device Hub owns input, legacy HID touches silently no-op -- - which looks exactly like a #153 coordinate miss. Record the shadow state on the - session (and on any inject that produced no `screen_changed`) so the two are - distinguishable; `serve-sim repair-input` is the human-only recovery (it restarts - SpringBoard and closes apps). - -#136 is the other 4 ms, and it is not a send duration. `hid.state=gone` sets -`status=error` and **omits** `input.send_ms`, so a dead injector is not bucketed with -the landscape miss: - -```json -{"kind":"tap","status":"error","details":{"source":"cli","hid.state":"gone","inject.result":"threw","ack":"rejected","screen_changed":false}} -``` - -Device Hub shadowing is the third look-alike: the send completed, so `input.send_ms` -stays. `input.shadowed=true` is what separates it from #153 (which has `remapped=false` -and `input.shadowed` absent or false): - -```json -{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","input.shadowed":true,"screen_changed":false,"input.send_ms":4}} -``` - -## Helper lifecycle (#102) - -The orphan in #102 was a detached helper still running after the simulator was -`Shutdown`. There are two hooks. (a) `recycle-self` in `readServeSimStates` (`middleware.ts`, from `classifyStaleState`) closes the device session in-process; an entry recorded there survives via `GET /api/event-log`. (b) The `onExit` closure `index.ts` passes to `startSimulatorShutdownWatch` runs only for detached/`--no-preview` servers and calls `process.exit(0)` immediately (best-effort for SSE). The watchdog itself never calls `classifyStaleState`. The pure-state decision path is still `classifyStaleState(state, bootedUdids, pid)` -> -`keep | recycle-self | recycle-helper` and the pure-state watchdog -`startSimulatorShutdownWatch(targets, ...)`, both in -`src/helper-lifecycle.ts`. The watch decision object (`tick` / `nextDelayMs`) is -timer-free, but `startSimulatorShutdownWatch` drives it with a `setTimeout` loop -(re-armed after each `tick` with `nextDelayMs()`). Record one `helper.lifecycle` entry when the watchdog -decides to recycle/exit after shutdown, carrying `sim.booted` and helper uptime. -(There is no separate `serve-sim-bin` process today -- HID and capture run in-process -via the native addon; don't name a binary that doesn't exist.) - -`classifyStaleState` returning `keep` does **not** emit the entry and does not record -`helper.uptime_ms`. A still-booted sim is not a recycle duration. The orphan is the -entry with `sim.booted=false` and `decision=recycle-helper`. - -- When `recycle-helper` signals a stale helper, keep a module-level `Set` of `device:pid` keys already signalled. - -## Stream and camera (correcting earlier mistakes) - -- **#128 is a consumer false-stall, not a dead producer.** The watchdog that fires - "Stream is not producing frames" after a rotation is on the **`fetch()` reader** - path (the `raw=1` MJPEG consumer read through `body.getReader()`), not the `` - element. Record a stall entry with the reason and a `producer_live` boolean (the - capture side is still publishing) so a false stall isn't read as a dead stream. -- **Camera helper exit (#143).** Run the camera helper under a **supervising parent** - that captures both exit code and signal -- Node reports a `null` exit code when the - child dies on a signal. Record `camera.helper.exit` with `exit_code`, `signal`, and - `shutdown_phase`. **SIGPIPE is already ignored**, so a client vanishing mid-write is - not the cause; the real race was the placeholder timer vs. surface release (mitigated - by [#161](https://github.com/EvanBacon/serve-sim/pull/161)). Don't attribute it to SIGPIPE. - -Keep both as ordinary event-log entries -- no span parenting, no `stream.startup` -> -`camera.frame` tree, no reconnect histograms. - -## Timing on those entries - -Durations are measured at the hook that already builds the entry, then written with -`updateEventLogEvent` if the end is later than the start (ack, next frame, helper -exit). Units are milliseconds. Missing key means "not a duration", not zero. - -A reader pairs each duration with the repro fields on the **same** entry. The fields -decide whether the key is present; they do not live on a second line. - -| Key | Entry | Record when | Omit when | -|---|---|---|---| -| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136), or `ack=timeout` (the 10 s wait is not a send). A landscape miss (#153) and a Device Hub shadow still record it. | -| `stream.ttff_ms` | first `stream.frame` | session `start()` to first `onSharedMjpegFrame` (JS-only; client-observed Connecting/stall still need the ingest) | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | -| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | `stream.false_positive=true` (#128) or producer dead. | -| `stream.producer_frame_age_ms` | `stream.stall` | last producer frame age at stall | when not a stall. | -| `camera.helper_shutdown_ms` | `camera.helper.exit` | helper start to exit | never started. | -| `helper.uptime_ms` | `helper.lifecycle` | session start to recycle decision | `decision=keep`. | - -## Privacy / export - -Local event log is on by default. Remote OTLP (or any export) is opt-in only via a serve-sim-specific variable that never inherits a stray `OTEL_EXPORTER_OTLP_ENDPOINT`. Debug switches: `DEBUG=serve-sim:*`, `SERVE_SIM_DEBUG_HID`. There is no `SERVE_SIM_DEBUG`. - -## Test plan - -- Taps are built in `DeviceSession.recordTouchEvent` (`kind:"tap"`/`"drag"`), not - `eventLogEventForHidMessage` (whose `0x03` branch yields `kind:"touch"` and is only - reached for an orphan `end` or unknown `type`). So in a `DeviceSession` test (fake `hid`, as in - `src/__tests__/device-session-lifecycle.test.ts`), send `0x03` `{type:"begin",…}` - then `{type:"end", x, y, source:"cli", remapped:false}` over a fake socket passed to - `attachHidSocket` (which feeds the private `handleHidMessage`), and - read the `kind:"tap"` entry: `details.source:"cli"`; omitting `source` -> `"unknown"`; assert - `coord_space`, `orientation`, `frame`, `remapped`, and `screen` are present. - A landscape miss records `input.send_ms`; `hid.state=gone` and `ack=timeout` omit it; - `input.shadowed=true` still records it. -- A `device-session` test: record a tap, deliver a differing `onSharedMjpegFrame` - seed -> `updateEventLogEvent` sets `screen_changed:true`; an identical seed -> `false`. -- `src/__tests__/device-hub-input.test.ts` already covers `isDeviceHubInputShadowed`; - add that a shadowed session stamps the shadow state on the session/inject entry. -- Assert `recycle-self` in middleware (`readServeSimStates` / `classifyStaleState`) can record a `helper.lifecycle` entry readable via `GET /api/event-log`; `keep` records nothing. Do not pin this to `onExit` in `helper-lifecycle.test.ts` (that path `process.exit(0)`s). -- Stream: a `producer_live` stall does not set `stream.reconnect_ms`; a return to - `live` does. First frame sets `stream.ttff_ms`; a never-connected preview does not. -- Read-back: `GET /api/event-log` and `serve-sim event-log --json` return the new - fields and a `schema_version`. - -## Recent issues (#172, #174) and repro fields - -#172 (portrait tap silently dropped on iPhone 18 Pro / iOS 27) is a #153 / #136 look-alike that the fields above already distinguish: - -- Expected signature for a pure coordinate miss after #167: `hid.state=ok`, `inject.result=sent`, `frame=native_portrait`, `remapped=true`, `screen_changed=false`, `input.send_ms` present, `source=cli`, `orientation=portrait`. -- Device Hub shadow: same but `input.shadowed=true`. -- Dead injector (#136): `hid.state=gone` or `inject.result=threw`, `status=error`, `input.send_ms` omitted. - -#174 (browser text/paste does not reach guest; CoreDevice clipboard works) is outside HID touch. Extend the event log for text/clipboard paths: - -- New `kind:"clipboard"` or `kind:"text"` entries from the `/exec` CoreDevice pasteboard path, carrying `payload_bytes`, `verified_readback` (bool), and `method:"coredevice"|"simctl"`. -- Timing: `clipboard.send_ms` from command start to verified readback; omit on failure. -- Do not record the raw string (privacy); record length and a content seed only if needed for repro. - -These stay additive to the existing event-log schema. No new spans; durations live on the same entry. - -## Maintenance note (2026-10-28) - -Error-reproduction fields and performance timing spans remain the priority. - -- For #172 (portrait silent drop on iPhone 18 Pro / iOS 27): the signature `hid.state=ok` + `inject.result=sent` + `screen_changed=false` + `orientation=portrait` + `remapped=true` (post-#167) still distinguishes it from #136 (`hid.state=gone`) and from Device Hub shadow (`input.shadowed=true`). Add `observed_orientation` lag check in the worked example if the poll shows portrait while the frame is landscape. -- For #174 (text/clipboard): the proposed `kind:"clipboard"` entry with `verified_readback` and `clipboard.send_ms` (omitted on failure) gives the same guest-effect signal as `screen_changed` for taps. Prefer CoreDevice method in the `method` field when simctl is a no-op. -- Timing spans: keep durations on the same entry; omit when the number would be a lie. No new parent spans. - -Still design-only. Implementation of the additive fields is a follow-up. +$(cat /tmp/observability.md) \ No newline at end of file From 69074867f898420d7c2c4a427e0f41c04cb8aea9 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Thu, 29 Oct 2026 09:13:09 -0700 Subject: [PATCH 22/27] docs(observability): restore full content + 2026-10-29 maintenance note on #172/#174 repro fields and timing spans --- docs/observability.md | 307 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 306 insertions(+), 1 deletion(-) diff --git a/docs/observability.md b/docs/observability.md index d4b9d3ddb..f4a7461da 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1 +1,306 @@ -$(cat /tmp/observability.md) \ No newline at end of file +# Observability + +> Remote readback of `/api/event-log` needs the auth gate from [PR #166](https://github.com/EvanBacon/serve-sim/pull/166). Merge that before relying on off-loopback log access. + +serve-sim already keeps an in-memory **event log** of simulator actions. This +document describes what that log records today, the small, additive fields that +make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/153), +[#136](https://github.com/EvanBacon/serve-sim/issues/136), +[#143](https://github.com/EvanBacon/serve-sim/issues/143), +[#103](https://github.com/EvanBacon/serve-sim/issues/103), +[#128](https://github.com/EvanBacon/serve-sim/issues/128), +[#102](https://github.com/EvanBacon/serve-sim/issues/102)) reproducible from the +log alone, and the exact code hook that produces each field. It is built on the +log we have -- no span tree, no histograms, no OpenTelemetry SDK is implied. + +Timing is the same entries. A duration is an additive key (`input.send_ms`, +`stream.ttff_ms`, `stream.reconnect_ms`, `stream.producer_frame_age_ms`, +`camera.helper_shutdown_ms`, `helper.uptime_ms`). Omit the key when the number +would be a lie (false stall, stuck Connecting, dead injector, still-booted +helper). Do not invent a parent span to carry it. + +The event log is a diagnostic side-channel. It must never fail the input or +command path (`recordEventLogEvent` already swallows subscriber errors), and it +is **on by default**; remote export is strictly opt-in (see Privacy below). + +## What the event log records today + +`src/event-log.ts` keeps a bounded ring (`EVENT_LOG_MAX_ENTRIES = 500`) of: + +```ts +type EventLogEntry = { + id: number; timestamp: string; + source: "hid" | "exec" | "ui"; + kind: string; msg: string; summary: string; + device?: string; action?: string; + status?: "ok" | "error"; + details?: Record; +}; +``` + +- **HID input** -> `handleHidMessage` (`src/device-session.ts`) routes tag `0x03` + touches to `DeviceSession.recordTouchEvent` and every other tag to `recordHidEvent` + -> `eventLogEventForHidMessage(device, tag, payload, screen)`. Tap events come from + `recordTouchEvent`: it folds begin/move/end into one `kind:"tap"` or `kind:"drag"` + entry (top-level `source:"hid"`) whose details (`touchGestureDetails`) carry `type`, + `start`, `current`, `moveCount`, optional `edge`, and `screen = {width,height}` (via + `eventLogScreen()`). The only `kind:"touch"` entries are its fallthrough: an `end` + with no preceding `begin`, or an unknown `type`, goes to + `recordHidEvent(0x03, …)`, whose `0x03` branch builds a per-phase `kind:"touch"` + entry. +- **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, parsed in + `src/middleware.ts` from the `/exec` path. It already drops upload plumbing and + never records raw command strings with tokens. +- **UI settings** -> `recordEventLogEvent({source:"ui", ...})` in `middleware.ts`. + +It already redacts: printable keystrokes (tag `0x06` records `key:"character"`, +`redacted:true`, no usage), token-bearing commands (returns `null`), and it only +ever stores **normalized** coordinates plus screen dimensions -- never pixels. + +## How a bot reads it back + +- `GET {url}/api/event-log?device=&since=&limit=` -> `{ events: [...] }` + (`readEventLog` in `middleware.ts`). +- `GET {url}/api/event-log/events` -> SSE: a snapshot, then one `{ event }` per new + entry (`subscribeEventLog`; the browser UI reaches it through the exec-ws proxy). +- CLI: `serve-sim event-log --json [-d ] [--limit ]` + (`eventLog()` in `src/index.ts`, which fetches `/api/event-log`). + +Add a top-level **`schema_version`** to the `/api/event-log` (and SSE) response +envelope so a reader can tell which of the fields below are present. Every field +added here is additive; an older consumer ignores unknown keys. + +## Making the #153 touch miss reproducible from the log + +#153: a tap exits 0 and the screenshot is unchanged because the point was sent in +the wrong coordinate frame. The server path `handleHidMessage` -> +`this.hid.touch(type, x, y, W, H, edge)` never remaps: `SimHID.touch` expects the +native portrait frame, so each client must remap before sending. The web client +already does (`rawPointForDisplayPoint` in `client/simulator/orientation.ts`, called +from `SimulatorView.tsx` with `streamDisplayGeometry(...).inputOrientation`). On +`main` the CLI (`tap()` / `gesture()` in `index.ts`) sends the display-normalized +point unchanged, which is the #153 miss. + +**Fixed by [#167](https://github.com/EvanBacon/serve-sim/pull/167) (pending).** It +adds `src/touch-remap.ts`, and `tap()` / `gesture()` call `touchInputOrientation` +(`index.ts`) before sending: + +- **Config lookup with the socket's auth.** `configUrlForWsUrl` derives the `/config` + URL from the HID socket URL (`…/helper//ws` -> `…/helper//config`, + `wss:` -> `https:`), so it works for local state files and `--url` remotes. It is + fetched with the same headers as the socket, 2 s timeout. Those headers carry + `Authorization: Bearer ` only when `--auth-token` or `SERVE_SIM_AUTH_TOKEN` + is set; otherwise the request is unauthenticated. +- **Landscape display -> raw.** `cliInputOrientation` applies the viewer's own rule + (`streamDisplayGeometry(...).inputOrientation`: remap only when the raw frame is + still portrait while the UI is rotated), and `displayTouchToRaw` converts the point + with `rawPointForDisplayPoint` and a non-zero `edge` with `rawEdgeForDisplayEdge`. +- **401/403 warns, then sends unmapped.** A refused `/config` prints a warning to + stderr (pass `--auth-token` to remap, or `--raw`) and the touch is sent as-is; the + command still exits 0. Any other non-OK status, a network error, or a timeout + silently falls back to no remap. +- **Foldables skipped.** A `/config` carrying `hingeDegrees` returns no remap, since + the viewer renders the Duo projection and sends those touches unrotated. +- **`--raw`** on `tap` / `gesture` skips the lookup and sends the coordinates as given + (native portrait frame). + +Even with #167 merged, nothing in the log distinguishes a coordinate miss (`--raw`, a +401/403 on `/config`, a stale `orientation`, an old CLI) from a dead injector. + +Add these keys to the `details` of `tap`/`drag` entries (built in +`DeviceSession.recordTouchEvent` via `touchGestureDetails`): + +| Field | Values | Source / hook | +|---|---|---| +| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always 0..1 normalized | +| `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `screenConfig().orientation` (i.e. `this.orientation`) read **at record time** | +| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `native_portrait` when the sender remapped (the web client and #167's CLI whenever `inputOrientation` is set); `display` when it did not (pre-#167 CLI, `--raw`, `/config` refused or unreachable). When no rotation is needed the two are identical. | +| `remapped` | bool | Sender-reported: an extra `remapped` key in the `0x03` JSON, set by the web client and the #167 CLI to whether `inputOrientation` was defined (`false` for `--raw` / a refused or unreachable `/config`). The server can't infer it from `source` alone, since both clients now remap. Omitted when the sender doesn't send it, and for Duo. | +| `observed_orientation` | string | Polled UI orientation (`simctl io enumerate`); can lag ~1s. Do **not** write into `this.orientation` / `broadcastConfig()`. | +| `remap_expected` | bool | Rule applied to `observed_orientation` when present, else `orientation`. Omitted for Duo (`hingeDegrees != null`): Duo web touches use 3D projection, not `rawPointForDisplayPoint`. | +| `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | + +A web miss is `remap_expected=true`, `remapped=false`: the browser hadn't been told about the rotation. +| `screen` | `{width,height}` | already recorded via `eventLogScreen()` | +| `screen_changed` | bool | guest-effect signal (see below) | +| `input.send_ms` | number, omitted on a dead injector | socket write to ack or fire-and-forget close (see Timing) | + +**`source` (cli vs web), compatibly.** Both CLI and browser touches arrive over the +separate WebSocket connections to the same handler (`attachHidInProcess` -> `attachHidSocket` -> `handleHidMessage`) and are recorded as +`source:"hid"`, so the log can't say which client sent them. Add an extra key to the +`0x03` JSON payload -- `{type,x,y,source:"cli"}` from `tap()`/`gesture()` in `index.ts`, +`{...,source:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`, stored +as `details.source` (the `source` field in the table above; distinct from the +entry's top-level `source:"hid"`). It is an +extra JSON field: the native `touch()` call ignores it and older clients that omit it +record `source:"unknown"`. (A dedicated tag is unnecessary; the payload is already JSON.) + +**Keeping `orientation` current when rotation happens outside `0x07`.** For non-Duo +devices `this.orientation` is only updated when a `0x07` rotate message is handled +(`device-session.ts`: `if (this.duo.hinge == null) this.orientation = m.orientation`). +A rotation triggered from the Simulator menu or forced by the app is **not** reflected. +Two real hooks fix this: (1) `onSharedMjpegFrame` already observes every capture frame +and calls `broadcastConfig()` when `width`/`height` swap -- use that dimension swap to +refresh orientation (portrait<->landscape) rather than caching it per gesture; (2) mirror +the Duo path, which polls on a 1 s timer (`DuoStateMonitor.refreshOrientation` in +`duo-state.ts`), with a lightweight orientation poll for non-Duo sessions: poll `xcrun simctl io enumerate` (as `DuoStateMonitor.refreshOrientation` does, without hinge/panel `log stream` children) and store the result in `observed_orientation` / a separate field. Do **not** write it into `this.orientation` or `broadcastConfig()` — that would change how the web client remaps input (a behavior fix, not a log field). `parseIntegratedScreenOrientations` only keys Device Names `primary`/`primary-1`; confirm what a non-Duo device reports before reusing it. Either way, read the orientation at the moment the entry is recorded. + +**Did the screen change after the tap, without keeping pixels.** `onSharedMjpegFrame` +receives every frame's bytes and dimensions. Maintain a monotonic `frameSeq` and a +cheap content **seed** (e.g. a hash of JPEG length plus a few sampled bytes, or the +IOSurface seed if the capture exposes it) -- not the image. Store `frameSeq` on the gesture at `begin`. At `end`, if `frameSeq` has already advanced (e.g. a touch-down highlight during the CLI's ~40 ms begin→end gap), record `screen_changed:true` immediately. Otherwise record `false` and, on the first higher `frameSeq` / differing seed within the window, call `updateEventLogEvent(tapId, …)`. `recordTouchEvent` must return the entry id (it returns `void` today), and the window timer must be cleared in `close()`. This is the guest-effect evidence that an ack can't provide. + +Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields. +`source` is part of the signature: on `main` the miss is the CLI path, not the web +client (which already remaps); after #167 it is a CLI run with `--raw`, a 401/403 on +`/config` (warned on stderr), or an otherwise unreachable `/config`. + +```json +{"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} +{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}} +``` + +The miss is `hid.state=ok` + `inject.result=sent` + `frame=display` + `remapped=false` + +`screen_changed=false` + `source=cli`. Status stays `ok`: the injector did what it was asked. +A #167 CLI tap in landscape is the same entry shape as the web hit: `frame=native_portrait`, `remapped=true`. + +## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) + +Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's +`0x0e {ok}` frame, times out at 10 s, and rejects on failure (`index.ts`). `tap`, +`gesture`, `rotate`, and `button` are **fire-and-forget** -- they write the frame, +wait a 40-50 ms timer, close the socket, and exit 0 unless the socket itself failed. + +The default we want is the `fold` pattern for all input: send, wait for a server ack +frame, and exit non-zero when it reports failure. Record the outcome on the entry +(`details.ack = ok | timeout | rejected`, plus `status`). Be explicit about scope: + +- An ack **proves** the server received the frame, `handleHidMessage` ran, and the + native injector returned. The injector boundary is `NativeHid.touch` -> + `SimHIDHandle.touch` (N-API), wrapped by `guard()` in `native.ts`. `guard()` + currently swallows **every** error from the native call (sync coercion throws and async rejections alike) and returns a fallback, so a malformed + send looks successful -- record whether `guard()` swallowed an error + (`inject.result = sent | swallowed | threw`) so #136's dead-injector case is visible. +- An ack **does not prove** the guest UI reacted. That is exactly the #153 gap, and why + `screen_changed` above is the separate, stronger signal. + +## HID health and Device Hub shadowing (#136, and a #153 look-alike) + +- **`hid.state`** (`ok` | `stale` | `gone`): HID can die while the pidfile and + `--list` still say `running: true`. Derive it from the last successful native call + (`SimHIDHandle` constructed in `native.ts`; sends go through `this.hid.*` in + `handleHidMessage`) and surface it in `--list` so `running:true` + `hid.state:gone` + is one object. +- **Device Hub input shadowing.** `src/device-hub-input.ts` checks + `com.apple.coredevice.dtuhidd.active` (`isDeviceHubInputShadowed` / + `warnDeviceHubInput`). When Device Hub owns input, legacy HID touches silently no-op -- + which looks exactly like a #153 coordinate miss. Record the shadow state on the + session (and on any inject that produced no `screen_changed`) so the two are + distinguishable; `serve-sim repair-input` is the human-only recovery (it restarts + SpringBoard and closes apps). + +#136 is the other 4 ms, and it is not a send duration. `hid.state=gone` sets +`status=error` and **omits** `input.send_ms`, so a dead injector is not bucketed with +the landscape miss: + +```json +{"kind":"tap","status":"error","details":{"source":"cli","hid.state":"gone","inject.result":"threw","ack":"rejected","screen_changed":false}} +``` + +Device Hub shadowing is the third look-alike: the send completed, so `input.send_ms` +stays. `input.shadowed=true` is what separates it from #153 (which has `remapped=false` +and `input.shadowed` absent or false): + +```json +{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","input.shadowed":true,"screen_changed":false,"input.send_ms":4}} +``` + +## Helper lifecycle (#102) + +The orphan in #102 was a detached helper still running after the simulator was +`Shutdown`. There are two hooks. (a) `recycle-self` in `readServeSimStates` (`middleware.ts`, from `classifyStaleState`) closes the device session in-process; an entry recorded there survives via `GET /api/event-log`. (b) The `onExit` closure `index.ts` passes to `startSimulatorShutdownWatch` runs only for detached/`--no-preview` servers and calls `process.exit(0)` immediately (best-effort for SSE). The watchdog itself never calls `classifyStaleState`. The pure-state decision path is still `classifyStaleState(state, bootedUdids, pid)` -> +`keep | recycle-self | recycle-helper` and the pure-state watchdog +`startSimulatorShutdownWatch(targets, ...)`, both in +`src/helper-lifecycle.ts`. The watch decision object (`tick` / `nextDelayMs`) is +timer-free, but `startSimulatorShutdownWatch` drives it with a `setTimeout` loop +(re-armed after each `tick` with `nextDelayMs()`). Record one `helper.lifecycle` entry when the watchdog +decides to recycle/exit after shutdown, carrying `sim.booted` and helper uptime. +(There is no separate `serve-sim-bin` process today -- HID and capture run in-process +via the native addon; don't name a binary that doesn't exist.) + +`classifyStaleState` returning `keep` does **not** emit the entry and does not record +`helper.uptime_ms`. A still-booted sim is not a recycle duration. The orphan is the +entry with `sim.booted=false` and `decision=recycle-helper`. + +- When `recycle-helper` signals a stale helper, keep a module-level `Set` of `device:pid` keys already signalled. + +## Stream and camera (correcting earlier mistakes) + +- **#128 is a consumer false-stall, not a dead producer.** The watchdog that fires + "Stream is not producing frames" after a rotation is on the **`fetch()` reader** + path (the `raw=1` MJPEG consumer read through `body.getReader()`), not the `` + element. Record a stall entry with the reason and a `producer_live` boolean (the + capture side is still publishing) so a false stall isn't read as a dead stream. +- **Camera helper exit (#143).** Run the camera helper under a **supervising parent** + that captures both exit code and signal -- Node reports a `null` exit code when the + child dies on a signal. Record `camera.helper.exit` with `exit_code`, `signal`, and + `shutdown_phase`. **SIGPIPE is already ignored**, so a client vanishing mid-write is + not the cause; the real race was the placeholder timer vs. surface release (mitigated + by [#161](https://github.com/EvanBacon/serve-sim/pull/161)). Don't attribute it to SIGPIPE. + +Keep both as ordinary event-log entries -- no span parenting, no `stream.startup` -> +`camera.frame` tree, no reconnect histograms. + +## Timing on those entries + +Durations are measured at the hook that already builds the entry, then written with +`updateEventLogEvent` if the end is later than the start (ack, next frame, helper +exit). Units are milliseconds. Missing key means "not a duration", not zero. + +A reader pairs each duration with the repro fields on the **same** entry. The fields +decide whether the key is present; they do not live on a second line. + +| Key | Entry | Record when | Omit when | +|---|---|---|---| +| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136), or `ack=timeout` (the 10 s wait is not a send). A landscape miss (#153) and a Device Hub shadow still record it. | +| `stream.ttff_ms` | first `stream.frame` | session `start()` to first `onSharedMjpegFrame` (JS-only; client-observed Connecting/stall still need the ingest) | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | +| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | `stream.false_positive=true` (#128) or producer dead. | +| `stream.producer_frame_age_ms` | `stream.stall` | last producer frame age at stall | producer dead (`producer_fps=0`). | +| `camera.helper_shutdown_ms` | `camera.helper.exit` | helper spawn to exit (signaled or clean) | still running. | +| `helper.uptime_ms` | `helper.lifecycle` | helper start to recycle decision | `classifyStaleState=keep` (still-booted). | + +## Recent issues (#172, #174) and repro fields + +#172 (portrait tap silently dropped on iPhone 18 Pro / iOS 27) is a #153 / #136 look-alike that the fields above already distinguish: + +- Expected signature for a pure coordinate miss after #167: `hid.state=ok`, `inject.result=sent`, `frame=native_portrait`, `remapped=true`, `screen_changed=false`, `input.send_ms` present, `source=cli`, `orientation=portrait`. +- Device Hub shadow: same but `input.shadowed=true`. +- Dead injector (#136): `hid.state=gone` or `inject.result=threw`, `status=error`, `input.send_ms` omitted. + +#174 (browser text/paste does not reach guest; CoreDevice clipboard works) is outside HID touch. Extend the event log for text/clipboard paths: + +- New `kind:"clipboard"` or `kind:"text"` entries from the `/exec` CoreDevice pasteboard path, carrying `payload_bytes`, `verified_readback` (bool), and `method:"coredevice"|"simctl"`. +- Timing: `clipboard.send_ms` from command start to verified readback; omit on failure. +- Do not record the raw string (privacy); record length and a content seed only if needed for repro. + +These stay additive to the existing event-log schema. No new spans; durations live on the same entry. + +## Maintenance note (2026-10-28) + +Error-reproduction fields and performance timing spans remain the priority. + +- For #172 (portrait silent drop on iPhone 18 Pro / iOS 27): the signature `hid.state=ok` + `inject.result=sent` + `screen_changed=false` + `orientation=portrait` + `remapped=true` (post-#167) still distinguishes it from #136 (`hid.state=gone`) and from Device Hub shadow (`input.shadowed=true`). Add `observed_orientation` lag check in the worked example if the poll shows portrait while the frame is landscape. +- For #174 (text/clipboard): the proposed `kind:"clipboard"` entry with `verified_readback` and `clipboard.send_ms` (omitted on failure) gives the same guest-effect signal as `screen_changed` for taps. Prefer CoreDevice method in the `method` field when simctl is a no-op. +- Timing spans: keep durations on the same entry; omit when the number would be a lie. No new parent spans. + +Still design-only. Implementation of the additive fields is a follow-up. + +## Maintenance note (2026-10-29) + +Continued prioritization of error-reproduction fields and performance timing spans. + +- #172 portrait silent drop: the distinguishing fields (`hid.state`, `orientation`/`observed_orientation`, `frame`, `remapped`, `screen_changed`, `input.shadowed`) remain sufficient. Timing key `input.send_ms` is present only when injector is healthy. +- #174 text/clipboard: `kind:"clipboard"` with `method`, `verified_readback`, and optional `clipboard.send_ms` (omitted on unverified or failed) gives guest-effect evidence without storing raw text. +- No new parent spans. Durations stay additive on the same event-log entries. schema_version bump on implement. + +Still design-only draft. CI green on prior commit. Not merging. From 298d455cdc2ee8d3dc1247a461b61a0490c5a884 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Fri, 30 Oct 2026 10:38:26 -0700 Subject: [PATCH 23/27] docs(observability): mark as design proposal, sync with main, drop maintenance notes - Banner: status is proposal; only 'records today'/'reads back' describe main. - #166 and #167 are merged: describe the auth gate (remoteAccess 403/401, loopback trusted) and the CLI remap as current, not pending. - Event log source now includes 'capture' (duo.renderer); note /api/diagnostics (loopback-only) exposes the last 100 entries. - Fix broken repro-field table and dangling 'Privacy below' reference. - #174: there is no clipboard path in serve-sim; reword as a privacy rule. - Remove dated maintenance notes and the duplicated #172/#174 section. --- docs/observability.md | 439 ++++++++++++++++++------------------------ 1 file changed, 191 insertions(+), 248 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index f4a7461da..ce9bd0d6f 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1,27 +1,29 @@ -# Observability +# Observability: error repro fields and timing (design proposal) -> Remote readback of `/api/event-log` needs the auth gate from [PR #166](https://github.com/EvanBacon/serve-sim/pull/166). Merge that before relying on off-loopback log access. +> **Status: proposal, not implemented.** The "What the event log records today" +> and "How a client reads it back" sections describe `main`. Everything else +> (new `details` keys, new entry kinds, `schema_version`, acks for all input) +> is proposed and does not exist yet. serve-sim already keeps an in-memory **event log** of simulator actions. This -document describes what that log records today, the small, additive fields that -make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/153), +document proposes small, additive fields that make the silent failures +([#153](https://github.com/EvanBacon/serve-sim/issues/153), [#136](https://github.com/EvanBacon/serve-sim/issues/136), [#143](https://github.com/EvanBacon/serve-sim/issues/143), [#103](https://github.com/EvanBacon/serve-sim/issues/103), [#128](https://github.com/EvanBacon/serve-sim/issues/128), [#102](https://github.com/EvanBacon/serve-sim/issues/102)) reproducible from the -log alone, and the exact code hook that produces each field. It is built on the -log we have -- no span tree, no histograms, no OpenTelemetry SDK is implied. +log alone, and names the code hook that would produce each one. It builds on the +log we have: no span tree, no histograms, no OpenTelemetry SDK. -Timing is the same entries. A duration is an additive key (`input.send_ms`, -`stream.ttff_ms`, `stream.reconnect_ms`, `stream.producer_frame_age_ms`, -`camera.helper_shutdown_ms`, `helper.uptime_ms`). Omit the key when the number -would be a lie (false stall, stuck Connecting, dead injector, still-booted -helper). Do not invent a parent span to carry it. +Timing lives on the same entries. A duration is an additive key (`input.send_ms`, +`stream.ttff_ms`, ...). Omit the key when the number would be a lie (false stall, +stuck Connecting, dead injector, still-booted helper). Do not invent a parent span +to carry it. -The event log is a diagnostic side-channel. It must never fail the input or -command path (`recordEventLogEvent` already swallows subscriber errors), and it -is **on by default**; remote export is strictly opt-in (see Privacy below). +The event log is a diagnostic side-channel and must never fail the input or +command path (`notifyEventLogSubscribers` in `src/event-log.ts` already swallows +subscriber errors, and `/exec` / UI recording is wrapped in `try`). ## What the event log records today @@ -30,7 +32,7 @@ is **on by default**; remote export is strictly opt-in (see Privacy below). ```ts type EventLogEntry = { id: number; timestamp: string; - source: "hid" | "exec" | "ui"; + source: "hid" | "exec" | "ui" | "capture"; kind: string; msg: string; summary: string; device?: string; action?: string; status?: "ok" | "error"; @@ -39,268 +41,209 @@ type EventLogEntry = { ``` - **HID input** -> `handleHidMessage` (`src/device-session.ts`) routes tag `0x03` - touches to `DeviceSession.recordTouchEvent` and every other tag to `recordHidEvent` - -> `eventLogEventForHidMessage(device, tag, payload, screen)`. Tap events come from - `recordTouchEvent`: it folds begin/move/end into one `kind:"tap"` or `kind:"drag"` - entry (top-level `source:"hid"`) whose details (`touchGestureDetails`) carry `type`, - `start`, `current`, `moveCount`, optional `edge`, and `screen = {width,height}` (via - `eventLogScreen()`). The only `kind:"touch"` entries are its fallthrough: an `end` - with no preceding `begin`, or an unknown `type`, goes to - `recordHidEvent(0x03, …)`, whose `0x03` branch builds a per-phase `kind:"touch"` - entry. -- **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, parsed in - `src/middleware.ts` from the `/exec` path. It already drops upload plumbing and - never records raw command strings with tokens. + touches to `DeviceSession.recordTouchEvent` and other tags to `recordHidEvent` + -> `eventLogEventForHidMessage(device, tag, payload, screen)`. + `recordTouchEvent` folds begin/move/end into one `kind:"tap"` or `kind:"drag"` + entry whose details (`touchGestureDetails`) carry `type`, `start`, `current`, + `moveCount`, optional `edge`, and `screen = {width,height}` (`eventLogScreen()`). + An `end` with no preceding `begin`, or an unknown `type`, falls through to a + per-phase `kind:"touch"` entry. +- **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, from the + `/exec` path in `src/middleware.ts`. It drops upload plumbing and never records + token-bearing command strings. - **UI settings** -> `recordEventLogEvent({source:"ui", ...})` in `middleware.ts`. +- **Capture** -> `kind:"duo.renderer"` entries (`recordRendererEvent` in + `device-session.ts`), with `duo.first_frame_ms` on `ready` and repeated + identical failures coalesced into one entry via `updateEventLogEvent` (`count`). + This is the existing precedent for "duration as a key on the same entry". -It already redacts: printable keystrokes (tag `0x06` records `key:"character"`, -`redacted:true`, no usage), token-bearing commands (returns `null`), and it only -ever stores **normalized** coordinates plus screen dimensions -- never pixels. +Redaction today: printable keystrokes (tag `0x06`) record `key:"character"`, +`redacted:true`, no usage; token-bearing commands are not recorded; touches store +only **normalized** coordinates plus screen dimensions, never pixels. -## How a bot reads it back +## How a client reads it back -- `GET {url}/api/event-log?device=&since=&limit=` -> `{ events: [...] }` - (`readEventLog` in `middleware.ts`). +- `GET {url}/api/event-log?device=&since=&limit=` -> `{ events: [...] }`. - `GET {url}/api/event-log/events` -> SSE: a snapshot, then one `{ event }` per new entry (`subscribeEventLog`; the browser UI reaches it through the exec-ws proxy). -- CLI: `serve-sim event-log --json [-d ] [--limit ]` - (`eventLog()` in `src/index.ts`, which fetches `/api/event-log`). +- `GET {url}/api/diagnostics` (loopback only) includes the last 100 entries as + `event_log`; `serve-sim doctor` reads it. +- CLI: `serve-sim event-log [--json] [-d ] [--limit ]` (`eventLog()` in + `src/index.ts`, sent with the same auth headers as other input commands). -Add a top-level **`schema_version`** to the `/api/event-log` (and SSE) response -envelope so a reader can tell which of the fields below are present. Every field -added here is additive; an older consumer ignores unknown keys. +**Access.** Loopback callers are trusted. Any other caller (non-loopback, +proxied, or rebound Host) is gated on every route by `remoteAccess` +(`src/request-trust.ts`): `403` when the server was started without +`--auth-token` / `SERVE_SIM_AUTH_TOKEN`, `401` without a valid +`Authorization: Bearer ` or auth cookie. Any new field below inherits this +gate; none of it may weaken it. -## Making the #153 touch miss reproducible from the log +**Proposed:** add a top-level `schema_version` to the `/api/event-log` and SSE +envelopes so a reader can tell which of the fields below are present. Every field +is additive; older consumers ignore unknown keys. + +## #153: making a touch miss reproducible #153: a tap exits 0 and the screenshot is unchanged because the point was sent in -the wrong coordinate frame. The server path `handleHidMessage` -> -`this.hid.touch(type, x, y, W, H, edge)` never remaps: `SimHID.touch` expects the -native portrait frame, so each client must remap before sending. The web client -already does (`rawPointForDisplayPoint` in `client/simulator/orientation.ts`, called -from `SimulatorView.tsx` with `streamDisplayGeometry(...).inputOrientation`). On -`main` the CLI (`tap()` / `gesture()` in `index.ts`) sends the display-normalized -point unchanged, which is the #153 miss. - -**Fixed by [#167](https://github.com/EvanBacon/serve-sim/pull/167) (pending).** It -adds `src/touch-remap.ts`, and `tap()` / `gesture()` call `touchInputOrientation` -(`index.ts`) before sending: - -- **Config lookup with the socket's auth.** `configUrlForWsUrl` derives the `/config` - URL from the HID socket URL (`…/helper//ws` -> `…/helper//config`, - `wss:` -> `https:`), so it works for local state files and `--url` remotes. It is - fetched with the same headers as the socket, 2 s timeout. Those headers carry - `Authorization: Bearer ` only when `--auth-token` or `SERVE_SIM_AUTH_TOKEN` - is set; otherwise the request is unauthenticated. -- **Landscape display -> raw.** `cliInputOrientation` applies the viewer's own rule - (`streamDisplayGeometry(...).inputOrientation`: remap only when the raw frame is - still portrait while the UI is rotated), and `displayTouchToRaw` converts the point - with `rawPointForDisplayPoint` and a non-zero `edge` with `rawEdgeForDisplayEdge`. -- **401/403 warns, then sends unmapped.** A refused `/config` prints a warning to - stderr (pass `--auth-token` to remap, or `--raw`) and the touch is sent as-is; the - command still exits 0. Any other non-OK status, a network error, or a timeout - silently falls back to no remap. -- **Foldables skipped.** A `/config` carrying `hingeDegrees` returns no remap, since - the viewer renders the Duo projection and sends those touches unrotated. -- **`--raw`** on `tap` / `gesture` skips the lookup and sends the coordinates as given - (native portrait frame). - -Even with #167 merged, nothing in the log distinguishes a coordinate miss (`--raw`, a -401/403 on `/config`, a stale `orientation`, an old CLI) from a dead injector. - -Add these keys to the `details` of `tap`/`drag` entries (built in -`DeviceSession.recordTouchEvent` via `touchGestureDetails`): +the wrong coordinate frame. `handleHidMessage` -> `this.hid.touch(...)` never +remaps; `SimHID.touch` expects the native portrait frame, so each client remaps +before sending. The web client uses `rawPointForDisplayPoint` +(`client/simulator/orientation.ts`). Since [#167](https://github.com/EvanBacon/serve-sim/pull/167) +the CLI does too: `tap()` / `gesture()` in `index.ts` call `touchInputOrientation`, +which fetches the helper `/config` (`configUrlForWsUrl` in `src/touch-remap.ts`, +same headers as the socket, 2 s timeout) and applies `cliInputOrientation` / +`displayTouchToRaw`. It sends the point unmapped when `--raw` is passed, when +`/config` returns 401/403 (with a stderr warning), on any other failure (silently), +and for foldables (`hingeDegrees` present). + +So a miss is still possible (`--raw`, a refused or unreachable `/config`, a stale +`orientation`, an older CLI), and nothing in the log distinguishes it from a dead +injector. + +**Proposed** keys on `tap`/`drag` `details` (built in `recordTouchEvent` / +`touchGestureDetails`): | Field | Values | Source / hook | |---|---|---| -| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always 0..1 normalized | -| `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `screenConfig().orientation` (i.e. `this.orientation`) read **at record time** | -| `frame` | `display` \| `native_portrait` | which space the point was in when sent: `native_portrait` when the sender remapped (the web client and #167's CLI whenever `inputOrientation` is set); `display` when it did not (pre-#167 CLI, `--raw`, `/config` refused or unreachable). When no rotation is needed the two are identical. | -| `remapped` | bool | Sender-reported: an extra `remapped` key in the `0x03` JSON, set by the web client and the #167 CLI to whether `inputOrientation` was defined (`false` for `--raw` / a refused or unreachable `/config`). The server can't infer it from `source` alone, since both clients now remap. Omitted when the sender doesn't send it, and for Duo. | -| `observed_orientation` | string | Polled UI orientation (`simctl io enumerate`); can lag ~1s. Do **not** write into `this.orientation` / `broadcastConfig()`. | -| `remap_expected` | bool | Rule applied to `observed_orientation` when present, else `orientation`. Omitted for Duo (`hingeDegrees != null`): Duo web touches use 3D projection, not `rawPointForDisplayPoint`. | -| `source` | `cli` \| `web` \| `unknown` | who sent the frame (see below) | - -A web miss is `remap_expected=true`, `remapped=false`: the browser hadn't been told about the rotation. -| `screen` | `{width,height}` | already recorded via `eventLogScreen()` | +| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always normalized | +| `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `this.orientation`, read at record time | +| `observed_orientation` | string | polled UI orientation (see below); can lag ~1 s | +| `remapped` | bool | sender-reported key in the `0x03` JSON: whether `inputOrientation` was defined. Omitted when the sender doesn't send it, and for Duo. | +| `frame` | `display` \| `native_portrait` | `native_portrait` when `remapped=true`, else `display` (identical when no rotation is needed) | +| `remap_expected` | bool | the viewer's remap rule applied to `observed_orientation` if present, else `orientation`. Omitted for Duo (3D projection, not `rawPointForDisplayPoint`). | +| `source` | `cli` \| `web` \| `unknown` | sender-reported key in the `0x03` JSON | | `screen_changed` | bool | guest-effect signal (see below) | -| `input.send_ms` | number, omitted on a dead injector | socket write to ack or fire-and-forget close (see Timing) | - -**`source` (cli vs web), compatibly.** Both CLI and browser touches arrive over the -separate WebSocket connections to the same handler (`attachHidInProcess` -> `attachHidSocket` -> `handleHidMessage`) and are recorded as -`source:"hid"`, so the log can't say which client sent them. Add an extra key to the -`0x03` JSON payload -- `{type,x,y,source:"cli"}` from `tap()`/`gesture()` in `index.ts`, -`{...,source:"web"}` from `SimulatorView.tsx` -- and read it in `recordTouchEvent`, stored -as `details.source` (the `source` field in the table above; distinct from the -entry's top-level `source:"hid"`). It is an -extra JSON field: the native `touch()` call ignores it and older clients that omit it -record `source:"unknown"`. (A dedicated tag is unnecessary; the payload is already JSON.) - -**Keeping `orientation` current when rotation happens outside `0x07`.** For non-Duo -devices `this.orientation` is only updated when a `0x07` rotate message is handled -(`device-session.ts`: `if (this.duo.hinge == null) this.orientation = m.orientation`). -A rotation triggered from the Simulator menu or forced by the app is **not** reflected. -Two real hooks fix this: (1) `onSharedMjpegFrame` already observes every capture frame -and calls `broadcastConfig()` when `width`/`height` swap -- use that dimension swap to -refresh orientation (portrait<->landscape) rather than caching it per gesture; (2) mirror -the Duo path, which polls on a 1 s timer (`DuoStateMonitor.refreshOrientation` in -`duo-state.ts`), with a lightweight orientation poll for non-Duo sessions: poll `xcrun simctl io enumerate` (as `DuoStateMonitor.refreshOrientation` does, without hinge/panel `log stream` children) and store the result in `observed_orientation` / a separate field. Do **not** write it into `this.orientation` or `broadcastConfig()` — that would change how the web client remaps input (a behavior fix, not a log field). `parseIntegratedScreenOrientations` only keys Device Names `primary`/`primary-1`; confirm what a non-Duo device reports before reusing it. Either way, read the orientation at the moment the entry is recorded. - -**Did the screen change after the tap, without keeping pixels.** `onSharedMjpegFrame` -receives every frame's bytes and dimensions. Maintain a monotonic `frameSeq` and a -cheap content **seed** (e.g. a hash of JPEG length plus a few sampled bytes, or the -IOSurface seed if the capture exposes it) -- not the image. Store `frameSeq` on the gesture at `begin`. At `end`, if `frameSeq` has already advanced (e.g. a touch-down highlight during the CLI's ~40 ms begin→end gap), record `screen_changed:true` immediately. Otherwise record `false` and, on the first higher `frameSeq` / differing seed within the window, call `updateEventLogEvent(tapId, …)`. `recordTouchEvent` must return the entry id (it returns `void` today), and the window timer must be cleared in `close()`. This is the guest-effect evidence that an ack can't provide. - -Landscape miss vs portrait hit, same 4 ms send, distinguished only by the repro fields. -`source` is part of the signature: on `main` the miss is the CLI path, not the web -client (which already remaps); after #167 it is a CLI run with `--raw`, a 401/403 on -`/config` (warned on stderr), or an otherwise unreachable `/config`. +| `input.send_ms` | number | see Timing | + +A miss is `remap_expected=true`, `remapped=false`. + +**`source` (cli vs web).** Both clients reach the same handler over separate +WebSockets and are recorded as top-level `source:"hid"`. Add `source:"cli"` to +the `0x03` JSON from `tap()`/`gesture()` and `source:"web"` from +`SimulatorView.tsx`; store it as `details.source`. The native `touch()` call +ignores extra keys, and older clients record `unknown`. + +**Keeping orientation current.** For non-Duo devices `this.orientation` only +changes on a `0x07` rotate (`if (this.duo.hinge == null) this.orientation = ...`), +so a rotation from the Simulator menu or forced by the app is missed. Record a +separately polled `observed_orientation` (as `DuoStateMonitor.refreshOrientation` +in `duo-state.ts` does via `xcrun simctl io enumerate`). Do **not** write it +into `this.orientation` or `broadcastConfig()`: that would change how clients +remap input, which is a behavior change, not a log field. +`parseIntegratedScreenOrientations` only keys `primary`/`primary-1`; confirm what a +non-Duo device reports before reusing it. + +**Did the screen change, without keeping pixels.** `onSharedMjpegFrame` sees +every frame. Keep a monotonic `frameSeq` (and optionally a cheap content seed, not +the image). Store `frameSeq` at `begin`; at `end`, record `screen_changed:true` if +it already advanced, else `false` and patch the entry with `updateEventLogEvent` +on the first later frame within a short window. This requires `recordTouchEvent` +to return the entry id (it returns `void` today) and the window timer to be +cleared in `close()`. + +Example (proposed shape): portrait hit vs landscape miss, same send time. ```json {"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}} {"kind":"tap","status":"ok","details":{"source":"cli","orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}} ``` -The miss is `hid.state=ok` + `inject.result=sent` + `frame=display` + `remapped=false` + -`screen_changed=false` + `source=cli`. Status stays `ok`: the injector did what it was asked. -A #167 CLI tap in landscape is the same entry shape as the web hit: `frame=native_portrait`, `remapped=true`. - -## What an ack proves (Evan's default: inject waits for ack, exits non-zero on failure) - -Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's -`0x0e {ok}` frame, times out at 10 s, and rejects on failure (`index.ts`). `tap`, -`gesture`, `rotate`, and `button` are **fire-and-forget** -- they write the frame, -wait a 40-50 ms timer, close the socket, and exit 0 unless the socket itself failed. - -The default we want is the `fold` pattern for all input: send, wait for a server ack -frame, and exit non-zero when it reports failure. Record the outcome on the entry -(`details.ack = ok | timeout | rejected`, plus `status`). Be explicit about scope: - -- An ack **proves** the server received the frame, `handleHidMessage` ran, and the - native injector returned. The injector boundary is `NativeHid.touch` -> - `SimHIDHandle.touch` (N-API), wrapped by `guard()` in `native.ts`. `guard()` - currently swallows **every** error from the native call (sync coercion throws and async rejections alike) and returns a fallback, so a malformed - send looks successful -- record whether `guard()` swallowed an error - (`inject.result = sent | swallowed | threw`) so #136's dead-injector case is visible. -- An ack **does not prove** the guest UI reacted. That is exactly the #153 gap, and why - `screen_changed` above is the separate, stronger signal. - -## HID health and Device Hub shadowing (#136, and a #153 look-alike) - -- **`hid.state`** (`ok` | `stale` | `gone`): HID can die while the pidfile and - `--list` still say `running: true`. Derive it from the last successful native call - (`SimHIDHandle` constructed in `native.ts`; sends go through `this.hid.*` in - `handleHidMessage`) and surface it in `--list` so `running:true` + `hid.state:gone` - is one object. -- **Device Hub input shadowing.** `src/device-hub-input.ts` checks - `com.apple.coredevice.dtuhidd.active` (`isDeviceHubInputShadowed` / - `warnDeviceHubInput`). When Device Hub owns input, legacy HID touches silently no-op -- - which looks exactly like a #153 coordinate miss. Record the shadow state on the - session (and on any inject that produced no `screen_changed`) so the two are - distinguishable; `serve-sim repair-input` is the human-only recovery (it restarts - SpringBoard and closes apps). - -#136 is the other 4 ms, and it is not a send duration. `hid.state=gone` sets -`status=error` and **omits** `input.send_ms`, so a dead injector is not bucketed with -the landscape miss: - -```json -{"kind":"tap","status":"error","details":{"source":"cli","hid.state":"gone","inject.result":"threw","ack":"rejected","screen_changed":false}} -``` - -Device Hub shadowing is the third look-alike: the send completed, so `input.send_ms` -stays. `input.shadowed=true` is what separates it from #153 (which has `remapped=false` -and `input.shadowed` absent or false): - -```json -{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","input.shadowed":true,"screen_changed":false,"input.send_ms":4}} -``` +Status stays `ok` for the miss: the injector did what it was asked. -## Helper lifecycle (#102) +## What an ack proves -The orphan in #102 was a detached helper still running after the simulator was -`Shutdown`. There are two hooks. (a) `recycle-self` in `readServeSimStates` (`middleware.ts`, from `classifyStaleState`) closes the device session in-process; an entry recorded there survives via `GET /api/event-log`. (b) The `onExit` closure `index.ts` passes to `startSimulatorShutdownWatch` runs only for detached/`--no-preview` servers and calls `process.exit(0)` immediately (best-effort for SSE). The watchdog itself never calls `classifyStaleState`. The pure-state decision path is still `classifyStaleState(state, bootedUdids, pid)` -> -`keep | recycle-self | recycle-helper` and the pure-state watchdog -`startSimulatorShutdownWatch(targets, ...)`, both in -`src/helper-lifecycle.ts`. The watch decision object (`tick` / `nextDelayMs`) is -timer-free, but `startSimulatorShutdownWatch` drives it with a `setTimeout` loop -(re-armed after each `tick` with `nextDelayMs()`). Record one `helper.lifecycle` entry when the watchdog -decides to recycle/exit after shutdown, carrying `sim.booted` and helper uptime. -(There is no separate `serve-sim-bin` process today -- HID and capture run in-process -via the native addon; don't name a binary that doesn't exist.) - -`classifyStaleState` returning `keep` does **not** emit the entry and does not record -`helper.uptime_ms`. A still-booted sim is not a recycle duration. The orphan is the -entry with `sim.booted=false` and `decision=recycle-helper`. - -- When `recycle-helper` signals a stale helper, keep a module-level `Set` of `device:pid` keys already signalled. - -## Stream and camera (correcting earlier mistakes) - -- **#128 is a consumer false-stall, not a dead producer.** The watchdog that fires - "Stream is not producing frames" after a rotation is on the **`fetch()` reader** - path (the `raw=1` MJPEG consumer read through `body.getReader()`), not the `` - element. Record a stall entry with the reason and a `producer_live` boolean (the - capture side is still publishing) so a false stall isn't read as a dead stream. -- **Camera helper exit (#143).** Run the camera helper under a **supervising parent** - that captures both exit code and signal -- Node reports a `null` exit code when the - child dies on a signal. Record `camera.helper.exit` with `exit_code`, `signal`, and - `shutdown_phase`. **SIGPIPE is already ignored**, so a client vanishing mid-write is - not the cause; the real race was the placeholder timer vs. surface release (mitigated - by [#161](https://github.com/EvanBacon/serve-sim/pull/161)). Don't attribute it to SIGPIPE. - -Keep both as ordinary event-log entries -- no span parenting, no `stream.startup` -> -`camera.frame` tree, no reconnect histograms. - -## Timing on those entries - -Durations are measured at the hook that already builds the entry, then written with -`updateEventLogEvent` if the end is later than the start (ack, next frame, helper -exit). Units are milliseconds. Missing key means "not a duration", not zero. - -A reader pairs each duration with the repro fields on the **same** entry. The fields -decide whether the key is present; they do not live on a second line. +Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's `0x0e {ok}` +frame, times out at 10 s, and rejects on failure (`index.ts`). `tap`, `gesture`, +`rotate`, and `button` are fire-and-forget: write the frame, wait a 40-50 ms +timer, close the socket, exit 0 unless the socket itself failed. -| Key | Entry | Record when | Omit when | -|---|---|---|---| -| `input.send_ms` | tap / button / gesture | socket write to ack, or to the fire-and-forget close if no ack yet | `hid.state=gone` or `inject.result=threw` (#136), or `ack=timeout` (the 10 s wait is not a send). A landscape miss (#153) and a Device Hub shadow still record it. | -| `stream.ttff_ms` | first `stream.frame` | session `start()` to first `onSharedMjpegFrame` (JS-only; client-observed Connecting/stall still need the ingest) | never saw a frame (Connecting before any frame, #103 cold). Do not write `0`. | -| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | `stream.false_positive=true` (#128) or producer dead. | -| `stream.producer_frame_age_ms` | `stream.stall` | last producer frame age at stall | producer dead (`producer_fps=0`). | -| `camera.helper_shutdown_ms` | `camera.helper.exit` | helper spawn to exit (signaled or clean) | still running. | -| `helper.uptime_ms` | `helper.lifecycle` | helper start to recycle decision | `classifyStaleState=keep` (still-booted). | +**Proposed:** use the `fold` pattern for all input and record +`details.ack = ok | timeout | rejected`. Scope: -## Recent issues (#172, #174) and repro fields +- An ack **proves** the server ran `handleHidMessage` and the native call returned. + `NativeHid.guard()` (`src/native.ts`) catches every error from the N-API call, + logs `[hid] ignored bad input`, and returns a fallback, so a failed send + looks successful. Record `inject.result = sent | swallowed` so #136's + dead-injector case is visible. (`guard()` must keep swallowing: an uncaught + throw would crash the in-process server.) +- An ack **does not prove** the guest UI reacted. That is the #153 gap, and why + `screen_changed` is the separate signal. -#172 (portrait tap silently dropped on iPhone 18 Pro / iOS 27) is a #153 / #136 look-alike that the fields above already distinguish: +## HID health and Device Hub shadowing (#136) -- Expected signature for a pure coordinate miss after #167: `hid.state=ok`, `inject.result=sent`, `frame=native_portrait`, `remapped=true`, `screen_changed=false`, `input.send_ms` present, `source=cli`, `orientation=portrait`. -- Device Hub shadow: same but `input.shadowed=true`. -- Dead injector (#136): `hid.state=gone` or `inject.result=threw`, `status=error`, `input.send_ms` omitted. +- **`hid.state`** (`ok` | `stale` | `gone`): HID can die while the state file and + `--list` still say running. Derive it from the last native call result and + surface it in `--list`. +- **Device Hub shadowing.** `src/device-hub-input.ts` + (`isDeviceHubInputShadowed` / `warnDeviceHubInput`) detects when Device Hub owns + input and legacy HID touches no-op, which looks exactly like #153. Record + `input.shadowed` on inject entries. `serve-sim repair-input` remains the manual + recovery (it restarts SpringBoard). -#174 (browser text/paste does not reach guest; CoreDevice clipboard works) is outside HID touch. Extend the event log for text/clipboard paths: +Look-alikes (proposed shapes): -- New `kind:"clipboard"` or `kind:"text"` entries from the `/exec` CoreDevice pasteboard path, carrying `payload_bytes`, `verified_readback` (bool), and `method:"coredevice"|"simctl"`. -- Timing: `clipboard.send_ms` from command start to verified readback; omit on failure. -- Do not record the raw string (privacy); record length and a content seed only if needed for repro. - -These stay additive to the existing event-log schema. No new spans; durations live on the same entry. - -## Maintenance note (2026-10-28) - -Error-reproduction fields and performance timing spans remain the priority. - -- For #172 (portrait silent drop on iPhone 18 Pro / iOS 27): the signature `hid.state=ok` + `inject.result=sent` + `screen_changed=false` + `orientation=portrait` + `remapped=true` (post-#167) still distinguishes it from #136 (`hid.state=gone`) and from Device Hub shadow (`input.shadowed=true`). Add `observed_orientation` lag check in the worked example if the poll shows portrait while the frame is landscape. -- For #174 (text/clipboard): the proposed `kind:"clipboard"` entry with `verified_readback` and `clipboard.send_ms` (omitted on failure) gives the same guest-effect signal as `screen_changed` for taps. Prefer CoreDevice method in the `method` field when simctl is a no-op. -- Timing spans: keep durations on the same entry; omit when the number would be a lie. No new parent spans. - -Still design-only. Implementation of the additive fields is a follow-up. +```json +{"kind":"tap","status":"error","details":{"source":"cli","hid.state":"gone","inject.result":"swallowed","ack":"rejected","screen_changed":false}} +{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","input.shadowed":true,"screen_changed":false,"input.send_ms":4}} +``` -## Maintenance note (2026-10-29) +A dead injector omits `input.send_ms` so it is not bucketed with a miss; a shadowed +send completed, so it keeps it. [#172](https://github.com/EvanBacon/serve-sim/issues/172) +(portrait tap silently dropped on iPhone 18 Pro / iOS 27) is told apart by the same +fields: `remapped=true` + `screen_changed=false` with `hid.state=ok` and no +`input.shadowed` points away from both a coordinate miss and #136. -Continued prioritization of error-reproduction fields and performance timing spans. +## Helper lifecycle (#102) -- #172 portrait silent drop: the distinguishing fields (`hid.state`, `orientation`/`observed_orientation`, `frame`, `remapped`, `screen_changed`, `input.shadowed`) remain sufficient. Timing key `input.send_ms` is present only when injector is healthy. -- #174 text/clipboard: `kind:"clipboard"` with `method`, `verified_readback`, and optional `clipboard.send_ms` (omitted on unverified or failed) gives guest-effect evidence without storing raw text. -- No new parent spans. Durations stay additive on the same event-log entries. schema_version bump on implement. +The orphan in #102 was a detached helper still running after its simulator was +shut down. Decisions live in `src/helper-lifecycle.ts`: `classifyStaleState` +(`keep | recycle-self | recycle-helper`) and `startSimulatorShutdownWatch` +(a `setTimeout` loop). `middleware.ts` acts on `recycle-self`, `index.ts` on +`recycle-helper`, and the watch's `onExit` in `index.ts` calls `process.exit(0)`. + +**Proposed:** record one `kind:"helper.lifecycle"` entry when a recycle/exit is +decided, with `decision`, `sim.booted`, and `helper.uptime_ms`. `keep` emits +nothing. An entry written right before `process.exit(0)` is only best-effort for +SSE readers. There is no separate helper binary; HID and capture run in-process +via the native addon. + +## Stream and camera (#128, #143, #103) + +- **#128 is a consumer false stall.** "Stream is not producing frames" + (`client/simulator/SimulatorView.tsx`) fires from the client's `fetch()` reader + watchdog on the MJPEG stream, not from a dead producer. Proposed: + `kind:"stream.stall"` with the reason and `producer_live` (from the server's + per-endpoint `streamHealth`, which already counts frames and `last_frame_at`). + Client-observed states need a small ingest route; until then, only + server-side facts can be logged. +- **#143 camera helper exit.** Proposed: `kind:"camera.helper.exit"` with + `exit_code`, `signal` (Node reports a `null` code on signal death), and + `shutdown_phase`. `SimCameraHelper` already ignores SIGPIPE (`main.m`), so don't + attribute exits to it; the real race was the placeholder timer vs. surface + release (mitigated by [#161](https://github.com/EvanBacon/serve-sim/pull/161)). + +## Timing + +Measured at the hook that builds the entry, patched with `updateEventLogEvent` when +the end comes later. Milliseconds. A missing key means "not a duration", not zero. +All keys and entry kinds below are proposed. -Still design-only draft. CI green on prior commit. Not merging. +| Key | Entry | Record when | Omit when | +|---|---|---|---| +| `input.send_ms` | tap / drag / button | socket write to ack (or close, while fire-and-forget) | `hid.state=gone`, `inject.result=swallowed`, or `ack=timeout`. A #153 miss and a shadowed send keep it. | +| `stream.ttff_ms` | stream start | session `start()` to first `onSharedMjpegFrame` | no frame ever arrived (#103 cold Connecting). Never write `0`. | +| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | false stall (#128) or producer dead | +| `stream.producer_frame_age_ms` | `stream.stall` | age of last producer frame at stall | producer dead | +| `camera.helper_shutdown_ms` | `camera.helper.exit` | helper spawn to exit | still running | +| `helper.uptime_ms` | `helper.lifecycle` | helper start to recycle decision | `keep` | + +## Privacy + +- Never record raw typed text, clipboard contents, command strings with tokens, or + pixels. Lengths and normalized coordinates are fine. +- Text entry today goes through `serve-sim type` as HID keystrokes (tag `0x06`, + already redacted). There is no clipboard path in serve-sim to instrument; if one + is added (e.g. for [#174](https://github.com/EvanBacon/serve-sim/issues/174)), + log `payload_bytes`, `method`, and `verified_readback`, never the string. +- Remote readback stays behind the existing auth gate; nothing here adds an + export or changes the default. From 198e6217b828b6edc4897280989f335005469e7f Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Fri, 30 Oct 2026 10:46:38 -0700 Subject: [PATCH 24/27] docs(observability): fix source paths and #128 watchdog description --- docs/observability.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index ce9bd0d6f..29012136f 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1,9 +1,11 @@ # Observability: error repro fields and timing (design proposal) > **Status: proposal, not implemented.** The "What the event log records today" -> and "How a client reads it back" sections describe `main`. Everything else -> (new `details` keys, new entry kinds, `schema_version`, acks for all input) -> is proposed and does not exist yet. +> and "How a client reads it back" sections describe `main`. Sections and keys +> marked **Proposed** (new `details` keys, new entry kinds, `schema_version`, +> acks for all input) do not exist yet. +> +> Source paths are relative to `packages/serve-sim/`. serve-sim already keeps an in-memory **event log** of simulator actions. This document proposes small, additive fields that make the silent failures @@ -44,7 +46,7 @@ type EventLogEntry = { touches to `DeviceSession.recordTouchEvent` and other tags to `recordHidEvent` -> `eventLogEventForHidMessage(device, tag, payload, screen)`. `recordTouchEvent` folds begin/move/end into one `kind:"tap"` or `kind:"drag"` - entry whose details (`touchGestureDetails`) carry `type`, `start`, `current`, + entry whose details (`touchGestureDetails`) carry `type`, optional `phase`, `start`, `current`, `moveCount`, optional `edge`, and `screen = {width,height}` (`eventLogScreen()`). An `end` with no preceding `begin`, or an unknown `type`, falls through to a per-phase `kind:"touch"` entry. @@ -88,7 +90,7 @@ is additive; older consumers ignore unknown keys. the wrong coordinate frame. `handleHidMessage` -> `this.hid.touch(...)` never remaps; `SimHID.touch` expects the native portrait frame, so each client remaps before sending. The web client uses `rawPointForDisplayPoint` -(`client/simulator/orientation.ts`). Since [#167](https://github.com/EvanBacon/serve-sim/pull/167) +(`src/client/simulator/orientation.ts`). Since [#167](https://github.com/EvanBacon/serve-sim/pull/167) the CLI does too: `tap()` / `gesture()` in `index.ts` call `touchInputOrientation`, which fetches the helper `/config` (`configUrlForWsUrl` in `src/touch-remap.ts`, same headers as the socket, 2 s timeout) and applies `cliInputOrientation` / @@ -108,7 +110,7 @@ injector. | `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always normalized | | `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `this.orientation`, read at record time | | `observed_orientation` | string | polled UI orientation (see below); can lag ~1 s | -| `remapped` | bool | sender-reported key in the `0x03` JSON: whether `inputOrientation` was defined. Omitted when the sender doesn't send it, and for Duo. | +| `remapped` | bool | sender-reported key in the `0x03` JSON: whether the sender applied a remap. Omitted when the sender doesn't send it, and for Duo. | | `frame` | `display` \| `native_portrait` | `native_portrait` when `remapped=true`, else `display` (identical when no rotation is needed) | | `remap_expected` | bool | the viewer's remap rule applied to `observed_orientation` if present, else `orientation`. Omitted for Duo (3D projection, not `rawPointForDisplayPoint`). | | `source` | `cli` \| `web` \| `unknown` | sender-reported key in the `0x03` JSON | @@ -210,8 +212,11 @@ via the native addon. ## Stream and camera (#128, #143, #103) - **#128 is a consumer false stall.** "Stream is not producing frames" - (`client/simulator/SimulatorView.tsx`) fires from the client's `fetch()` reader - watchdog on the MJPEG stream, not from a dead producer. Proposed: + (`src/client/simulator/SimulatorView.tsx`) fires from the client's 6 s startup + watchdog (relay `subscribeFrame` or direct `fetch()` reader) when no first frame + arrives; it never fires mid-stream. The watchdog targets a dead producer, but in + #128 it fired while the producer was live. Separately, relay mode drops `live` + (`setConnected(false)`) after 2 s without a frame. Proposed: `kind:"stream.stall"` with the reason and `producer_live` (from the server's per-endpoint `streamHealth`, which already counts frames and `last_frame_at`). Client-observed states need a small ingest route; until then, only @@ -232,7 +237,7 @@ All keys and entry kinds below are proposed. |---|---|---|---| | `input.send_ms` | tap / drag / button | socket write to ack (or close, while fire-and-forget) | `hid.state=gone`, `inject.result=swallowed`, or `ack=timeout`. A #153 miss and a shadowed send keep it. | | `stream.ttff_ms` | stream start | session `start()` to first `onSharedMjpegFrame` | no frame ever arrived (#103 cold Connecting). Never write `0`. | -| `stream.reconnect_ms` | `stream.state` | consumer left `live` and returned | false stall (#128) or producer dead | +| `stream.reconnect_ms` | `stream.state` | consumer left `live` (relay 2 s staleness check) and returned | false stall (#128) or producer dead | | `stream.producer_frame_age_ms` | `stream.stall` | age of last producer frame at stall | producer dead | | `camera.helper_shutdown_ms` | `camera.helper.exit` | helper spawn to exit | still running | | `helper.uptime_ms` | `helper.lifecycle` | helper start to recycle decision | `keep` | From 4a7c8d6a7e30343e392df250e1dc8433b95e7d74 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Fri, 30 Oct 2026 10:50:33 -0700 Subject: [PATCH 25/27] docs(observability): base screen_changed on surface content stamp, not frame arrival The idle timer re-emits unchanged frames every 200 ms, so frameSeq always advances after a touch; onSharedMjpegFrame also only runs on foldables. --- docs/observability.md | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index 29012136f..7bb6541e9 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -135,13 +135,21 @@ remap input, which is a behavior change, not a log field. `parseIntegratedScreenOrientations` only keys `primary`/`primary-1`; confirm what a non-Duo device reports before reusing it. -**Did the screen change, without keeping pixels.** `onSharedMjpegFrame` sees -every frame. Keep a monotonic `frameSeq` (and optionally a cheap content seed, not -the image). Store `frameSeq` at `begin`; at `end`, record `screen_changed:true` if -it already advanced, else `false` and patch the entry with `updateEventLogEvent` -on the first later frame within a short window. This requires `recordTouchEvent` -to return the entry id (it returns `void` today) and the window timer to be -cleared in `close()`. +**Did the screen change, without keeping pixels.** Frame arrival is not a +signal: the idle timer in `FrameCapture.swift` force-emits the current frame +every 200 ms even when nothing changed, so a frame counter advances after every +touch. And `onSharedMjpegFrame` only runs on foldables (`device-session.ts` +subscribes it only when `isFoldable`). Use the content stamp the native capture +already computes instead: `captureFrame` builds a `SurfaceFrameStamp` (IOSurface +ID + seed) and compares it to the previous one. Expose a counter that increments +only when that stamp changes (not on forced idle re-emits) as a cheap native +getter, read it at `begin` and `end`, and record `screen_changed:true` if it +advanced. If not, record `false`, re-read it once after a short window, and patch +the entry with `updateEventLogEvent` if it advanced then. This requires +`recordTouchEvent` to return the entry id (it returns `void` today) and the window +timer to be cleared in `close()`. The seed changes on any surface write, so a +caret blink or running animation can still read `true`; treat `false` as the +reliable signal. Example (proposed shape): portrait hit vs landscape miss, same send time. From 16d5a14f237daf914296b52198e85ebddbab1255 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Mon, 9 Nov 2026 17:17:17 +0000 Subject: [PATCH 26/27] docs(observability): prioritize error-repro fields and timing spans for this cycle Explicit priority order for the design: error-reproduction fields (#153/#136/#128/#103/#143/#102 and related) first, then additive performance timing keys on the same entries. Notes clipboard path for #174 follows the same additive, privacy-preserving pattern. --- docs/observability.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/observability.md b/docs/observability.md index 7bb6541e9..e268fee21 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -7,6 +7,15 @@ > > Source paths are relative to `packages/serve-sim/`. +**Priority (this cycle):** error-reproduction fields first (`coord_space`, +`orientation`/`observed_orientation`, `frame`, `remapped`/`remap_expected`, +`source`, `screen_changed`, `hid.state`, `inject.result`, `input.shadowed`, +`ack`), then performance timing spans as additive keys on the same entries +(`input.send_ms`, `stream.ttff_ms`, `stream.reconnect_ms`, +`stream.producer_frame_age_ms`, `camera.helper_shutdown_ms`, +`helper.uptime_ms`). Clipboard path for #174 follows the same pattern +(`payload_bytes`, `method`, `verified_readback`, `clipboard.send_ms`). + serve-sim already keeps an in-memory **event log** of simulator actions. This document proposes small, additive fields that make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/153), From dcf996202819b47abd1b468a8c6d5055ad19c0f0 Mon Sep 17 00:00:00 2001 From: Evan Bacon Date: Mon, 9 Nov 2026 09:46:21 -0800 Subject: [PATCH 27/27] docs(observability): sync hooks with #189/#190/#191; note open #192 Rebased onto main. Recent-PR notes keep proposed event-log fields aligned with Duo hinge CoreMotion readback, fake-camera unique-ID swizzle, and the async shutdown boot query. Claude Code signed off on the doc pass. --- docs/observability.md | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/docs/observability.md b/docs/observability.md index e268fee21..fa2a15d30 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -16,6 +16,24 @@ `helper.uptime_ms`). Clipboard path for #174 follows the same pattern (`payload_bytes`, `method`, `verified_readback`, `clipboard.send_ms`). +**Recent PRs (keep hooks current):** +- [#189](https://github.com/EvanBacon/serve-sim/pull/189) (Duo hinge): `DuoStateMonitor` now reads hinge via + guest `serve-sim-duo-hid watch-angle` / CoreMotion, not the removed `devicectl device motion hinge-angle` + path, so `/config` and `0x82` config frames carry `hingeDegrees` / `duoViewOrientation` on Xcode versions + without that subcommand. Only the angle source changed: panel orientations still come from + `simctl io enumerate`, and a `0x0e` result still comes from the native pose/hinge call. Caveat from the + merge: if the reader dies, the last angle can stay stale. +- [#190](https://github.com/EvanBacon/serve-sim/pull/190) (fake cameras): the injector now swizzles + `deviceWithUniqueID:`, so apps that look up a fake camera by ID get it instead of `nil`, and fake formats + no longer crash on `-description`. No proposed field changes: `camera.helper.exit` carries no camera identity. +- [#191](https://github.com/EvanBacon/serve-sim/pull/191) (async boot query, fixes #186): detached / + `--no-preview` shutdown watch no longer blocks the stream event loop on sync `simctl`. Proposed + `helper.lifecycle` entries should treat boot-state checks as async and non-blocking; a failed check + keeps the helper alive. The pre-existing `--detach`/follow helper-replacement race was accepted as a follow-up. +- [#192](https://github.com/EvanBacon/serve-sim/pull/192) (Home / SpringBoard direct launch) is still open; + if it lands, Home becomes a queued SpringBoard launch with simctl fallback — useful for + `input.send_ms` / button entries, but not yet `main`. + serve-sim already keeps an in-memory **event log** of simulator actions. This document proposes small, additive fields that make the silent failures ([#153](https://github.com/EvanBacon/serve-sim/issues/153), @@ -144,6 +162,12 @@ remap input, which is a behavior change, not a log field. `parseIntegratedScreenOrientations` only keys `primary`/`primary-1`; confirm what a non-Duo device reports before reusing it. +For Duo, `hingeDegrees` in `/config` now comes from the guest CoreMotion reader added +in [#189](https://github.com/EvanBacon/serve-sim/pull/189) (`watch-angle`), not +`devicectl`; panel orientations still come from the `enumerate` poll above. The reader +is its own child process, so a dead reader (stale last angle) leaves `hid.state` and +`inject.result` unchanged and neither field can detect it. + **Did the screen change, without keeping pixels.** Frame arrival is not a signal: the idle timer in `FrameCapture.swift` force-emits the current frame every 200 ms even when nothing changed, so a frame counter advances after every @@ -219,12 +243,17 @@ shut down. Decisions live in `src/helper-lifecycle.ts`: `classifyStaleState` (`keep | recycle-self | recycle-helper`) and `startSimulatorShutdownWatch` (a `setTimeout` loop). `middleware.ts` acts on `recycle-self`, `index.ts` on `recycle-helper`, and the watch's `onExit` in `index.ts` calls `process.exit(0)`. +Since [#191](https://github.com/EvanBacon/serve-sim/pull/191) the watch's boot-state +check is the async, uncached `queryBootedUdids`, so a pending `simctl` no longer +stalls detached HTTP/streams (#186); the shared in-flight `getBootedUdids` snapshot +serves `--detach`/follow reuse. Proposed `helper.lifecycle` entries should assume +that model; a failed check keeps the helper alive. **Proposed:** record one `kind:"helper.lifecycle"` entry when a recycle/exit is decided, with `decision`, `sim.booted`, and `helper.uptime_ms`. `keep` emits nothing. An entry written right before `process.exit(0)` is only best-effort for -SSE readers. There is no separate helper binary; HID and capture run in-process -via the native addon. +SSE readers. The helper is the serve-sim process itself; HID and capture run +in-process via the native addon (Duo also spawns the guest `serve-sim-duo-hid`). ## Stream and camera (#128, #143, #103) @@ -243,6 +272,9 @@ via the native addon. `shutdown_phase`. `SimCameraHelper` already ignores SIGPIPE (`main.m`), so don't attribute exits to it; the real race was the placeholder timer vs. surface release (mitigated by [#161](https://github.com/EvanBacon/serve-sim/pull/161)). + [#190](https://github.com/EvanBacon/serve-sim/pull/190) fixed app-side crashes + and lookups in the injector (fake `-description`, `deviceWithUniqueID:`), which + are a separate failure path from helper exits. ## Timing