Skip to content
Draft
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
1fb9dc8
docs: observability design for error repro and perf timing
cursoragent Sep 25, 2026
9abd5de
docs: record #153 landscape coordinate-miss fields
EvanBacon Oct 1, 2026
9eeea16
docs: worked error-repro and timing records for observability
EvanBacon Oct 2, 2026
13bdc5e
docs: join error-repro events to timing spans via trace.id
EvanBacon Oct 4, 2026
6f152ab
docs: join error-repro fields onto timing spans
Oct 5, 2026
046b5c4
docs: keep false MJPEG stalls out of the reconnect histogram
EvanBacon Oct 6, 2026
ada0d87
docs: false stall timing is camera.frame, not reconnect
EvanBacon Oct 7, 2026
47816b4
docs: rebuild observability on the existing event log
EvanBacon Oct 8, 2026
53fe9e7
docs: put timing durations on the event-log entries
EvanBacon Oct 8, 2026
28c49d2
docs: pair timing keys with error-repro fields
EvanBacon Oct 9, 2026
04e683e
docs(observability): omit input.send_ms on ack timeout
EvanBacon Oct 10, 2026
4514e58
docs: clarify error-repro fields gate timing keys (2026-10-11 maintai…
EvanBacon Oct 11, 2026
d5450ec
docs(observability): address Claude review objections
EvanBacon Oct 12, 2026
3f5f00a
docs(observability): Set<string> for device:pid signalled keys
cursoragent Oct 12, 2026
c78a2ed
docs(observability): CLI remap via #167, recordTouchEvent test plan, …
EvanBacon Oct 14, 2026
9f4cde0
docs(observability): reflect #167 CLI tap remap; touch events via rec…
EvanBacon Oct 14, 2026
5eced6d
docs(observability): bearer only with --auth-token
EvanBacon Oct 14, 2026
9b45dca
docs(observability): link #172/#174 to existing repro fields and clip…
EvanBacon Oct 25, 2026
ab7d0cc
docs(observability): 2026-10-28 maintenance — prioritize #172/#174 re…
EvanBacon Oct 28, 2026
f92ed91
docs(observability): 2026-10-28 maintenance — prioritize #172/#174 re…
EvanBacon Oct 28, 2026
759c891
docs(observability): 2026-10-29 maintenance note on #172/#174 repro f…
EvanBacon Oct 29, 2026
6907486
docs(observability): restore full content + 2026-10-29 maintenance no…
EvanBacon Oct 29, 2026
298d455
docs(observability): mark as design proposal, sync with main, drop ma…
EvanBacon Oct 30, 2026
198e621
docs(observability): fix source paths and #128 watchdog description
EvanBacon Oct 30, 2026
4a7c8d6
docs(observability): base screen_changed on surface content stamp, no…
EvanBacon Oct 30, 2026
16d5a14
docs(observability): prioritize error-repro fields and timing spans f…
EvanBacon Nov 9, 2026
dcf9962
docs(observability): sync hooks with #189/#190/#191; note open #192
EvanBacon Nov 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
303 changes: 303 additions & 0 deletions docs/observability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,303 @@
# Observability: error repro fields and timing (design proposal)

> **Status: proposal, not implemented.** The "What the event log records today"
> and "How a client reads it back" sections describe `main`. Sections and keys
> marked **Proposed** (new `details` keys, new entry kinds, `schema_version`,
> acks for all input) do not exist yet.
>
> Source paths are relative to `packages/serve-sim/`.

**Priority (this cycle):** error-reproduction fields first (`coord_space`,
`orientation`/`observed_orientation`, `frame`, `remapped`/`remap_expected`,
`source`, `screen_changed`, `hid.state`, `inject.result`, `input.shadowed`,
`ack`), then performance timing spans as additive keys on the same entries
(`input.send_ms`, `stream.ttff_ms`, `stream.reconnect_ms`,
`stream.producer_frame_age_ms`, `camera.helper_shutdown_ms`,
`helper.uptime_ms`). Clipboard path for #174 follows the same pattern
(`payload_bytes`, `method`, `verified_readback`, `clipboard.send_ms`).

**Recent PRs (keep hooks current):**
- [#189](https://github.com/EvanBacon/serve-sim/pull/189) (Duo hinge): `DuoStateMonitor` now reads hinge via
guest `serve-sim-duo-hid watch-angle` / CoreMotion, not the removed `devicectl device motion hinge-angle`
path, so `/config` and `0x82` config frames carry `hingeDegrees` / `duoViewOrientation` on Xcode versions
without that subcommand. Only the angle source changed: panel orientations still come from
`simctl io enumerate`, and a `0x0e` result still comes from the native pose/hinge call. Caveat from the
merge: if the reader dies, the last angle can stay stale.
- [#190](https://github.com/EvanBacon/serve-sim/pull/190) (fake cameras): the injector now swizzles
`deviceWithUniqueID:`, so apps that look up a fake camera by ID get it instead of `nil`, and fake formats
no longer crash on `-description`. No proposed field changes: `camera.helper.exit` carries no camera identity.
- [#191](https://github.com/EvanBacon/serve-sim/pull/191) (async boot query, fixes #186): detached /
`--no-preview` shutdown watch no longer blocks the stream event loop on sync `simctl`. Proposed
`helper.lifecycle` entries should treat boot-state checks as async and non-blocking; a failed check
keeps the helper alive. The pre-existing `--detach`/follow helper-replacement race was accepted as a follow-up.
- [#192](https://github.com/EvanBacon/serve-sim/pull/192) (Home / SpringBoard direct launch) is still open;
if it lands, Home becomes a queued SpringBoard launch with simctl fallback — useful for
`input.send_ms` / button entries, but not yet `main`.

serve-sim already keeps an in-memory **event log** of simulator actions. This
document proposes small, additive fields that make the silent failures
([#153](https://github.com/EvanBacon/serve-sim/issues/153),
[#136](https://github.com/EvanBacon/serve-sim/issues/136),
[#143](https://github.com/EvanBacon/serve-sim/issues/143),
[#103](https://github.com/EvanBacon/serve-sim/issues/103),
[#128](https://github.com/EvanBacon/serve-sim/issues/128),
[#102](https://github.com/EvanBacon/serve-sim/issues/102)) reproducible from the
log alone, and names the code hook that would produce each one. It builds on the
log we have: no span tree, no histograms, no OpenTelemetry SDK.

Timing lives on the same entries. A duration is an additive key (`input.send_ms`,
`stream.ttff_ms`, ...). Omit the key when the number would be a lie (false stall,
stuck Connecting, dead injector, still-booted helper). Do not invent a parent span
to carry it.

The event log is a diagnostic side-channel and must never fail the input or
command path (`notifyEventLogSubscribers` in `src/event-log.ts` already swallows
subscriber errors, and `/exec` / UI recording is wrapped in `try`).

## What the event log records today

`src/event-log.ts` keeps a bounded ring (`EVENT_LOG_MAX_ENTRIES = 500`) of:

```ts
type EventLogEntry = {
id: number; timestamp: string;
source: "hid" | "exec" | "ui" | "capture";
kind: string; msg: string; summary: string;
device?: string; action?: string;
status?: "ok" | "error";
details?: Record<string, unknown>;
};
```

- **HID input** -> `handleHidMessage` (`src/device-session.ts`) routes tag `0x03`
touches to `DeviceSession.recordTouchEvent` and other tags to `recordHidEvent`
-> `eventLogEventForHidMessage(device, tag, payload, screen)`.
`recordTouchEvent` folds begin/move/end into one `kind:"tap"` or `kind:"drag"`
entry whose details (`touchGestureDetails`) carry `type`, optional `phase`, `start`, `current`,
`moveCount`, optional `edge`, and `screen = {width,height}` (`eventLogScreen()`).
An `end` with no preceding `begin`, or an unknown `type`, falls through to a
per-phase `kind:"touch"` entry.
- **Shell actions** -> `eventLogEventForCommand(command, {exitCode})`, from the
`/exec` path in `src/middleware.ts`. It drops upload plumbing and never records
token-bearing command strings.
- **UI settings** -> `recordEventLogEvent({source:"ui", ...})` in `middleware.ts`.
- **Capture** -> `kind:"duo.renderer"` entries (`recordRendererEvent` in
`device-session.ts`), with `duo.first_frame_ms` on `ready` and repeated
identical failures coalesced into one entry via `updateEventLogEvent` (`count`).
This is the existing precedent for "duration as a key on the same entry".

Redaction today: printable keystrokes (tag `0x06`) record `key:"character"`,
`redacted:true`, no usage; token-bearing commands are not recorded; touches store
only **normalized** coordinates plus screen dimensions, never pixels.

## How a client reads it back

- `GET {url}/api/event-log?device=<udid>&since=<id>&limit=<n>` -> `{ events: [...] }`.
- `GET {url}/api/event-log/events` -> SSE: a snapshot, then one `{ event }` per new
entry (`subscribeEventLog`; the browser UI reaches it through the exec-ws proxy).
- `GET {url}/api/diagnostics` (loopback only) includes the last 100 entries as
`event_log`; `serve-sim doctor` reads it.
- CLI: `serve-sim event-log [--json] [-d <udid>] [--limit <n>]` (`eventLog()` in
`src/index.ts`, sent with the same auth headers as other input commands).

**Access.** Loopback callers are trusted. Any other caller (non-loopback,
proxied, or rebound Host) is gated on every route by `remoteAccess`
(`src/request-trust.ts`): `403` when the server was started without
`--auth-token` / `SERVE_SIM_AUTH_TOKEN`, `401` without a valid
`Authorization: Bearer <token>` or auth cookie. Any new field below inherits this
gate; none of it may weaken it.

**Proposed:** add a top-level `schema_version` to the `/api/event-log` and SSE
envelopes so a reader can tell which of the fields below are present. Every field
is additive; older consumers ignore unknown keys.

## #153: making a touch miss reproducible

#153: a tap exits 0 and the screenshot is unchanged because the point was sent in
the wrong coordinate frame. `handleHidMessage` -> `this.hid.touch(...)` never
remaps; `SimHID.touch` expects the native portrait frame, so each client remaps
before sending. The web client uses `rawPointForDisplayPoint`
(`src/client/simulator/orientation.ts`). Since [#167](https://github.com/EvanBacon/serve-sim/pull/167)
the CLI does too: `tap()` / `gesture()` in `index.ts` call `touchInputOrientation`,
which fetches the helper `/config` (`configUrlForWsUrl` in `src/touch-remap.ts`,
same headers as the socket, 2 s timeout) and applies `cliInputOrientation` /
`displayTouchToRaw`. It sends the point unmapped when `--raw` is passed, when
`/config` returns 401/403 (with a stderr warning), on any other failure (silently),
and for foldables (`hingeDegrees` present).

So a miss is still possible (`--raw`, a refused or unreachable `/config`, a stale
`orientation`, an older CLI), and nothing in the log distinguishes it from a dead
injector.

**Proposed** keys on `tap`/`drag` `details` (built in `recordTouchEvent` /
`touchGestureDetails`):

| Field | Values | Source / hook |
|---|---|---|
| `coord_space` | `normalized_0_1` | constant; the `0x03` payload is always normalized |
| `orientation` | `portrait` \| `portrait_upside_down` \| `landscape_left` \| `landscape_right` \| `unknown` | `this.orientation`, read at record time |
| `observed_orientation` | string | polled UI orientation (see below); can lag ~1 s |
| `remapped` | bool | sender-reported key in the `0x03` JSON: whether the sender applied a remap. Omitted when the sender doesn't send it, and for Duo. |
| `frame` | `display` \| `native_portrait` | `native_portrait` when `remapped=true`, else `display` (identical when no rotation is needed) |
| `remap_expected` | bool | the viewer's remap rule applied to `observed_orientation` if present, else `orientation`. Omitted for Duo (3D projection, not `rawPointForDisplayPoint`). |
| `source` | `cli` \| `web` \| `unknown` | sender-reported key in the `0x03` JSON |
| `screen_changed` | bool | guest-effect signal (see below) |
| `input.send_ms` | number | see Timing |

A miss is `remap_expected=true`, `remapped=false`.

**`source` (cli vs web).** Both clients reach the same handler over separate
WebSockets and are recorded as top-level `source:"hid"`. Add `source:"cli"` to
the `0x03` JSON from `tap()`/`gesture()` and `source:"web"` from
`SimulatorView.tsx`; store it as `details.source`. The native `touch()` call
ignores extra keys, and older clients record `unknown`.

**Keeping orientation current.** For non-Duo devices `this.orientation` only
changes on a `0x07` rotate (`if (this.duo.hinge == null) this.orientation = ...`),
so a rotation from the Simulator menu or forced by the app is missed. Record a
separately polled `observed_orientation` (as `DuoStateMonitor.refreshOrientation`
in `duo-state.ts` does via `xcrun simctl io <udid> enumerate`). Do **not** write it
into `this.orientation` or `broadcastConfig()`: that would change how clients
remap input, which is a behavior change, not a log field.
`parseIntegratedScreenOrientations` only keys `primary`/`primary-1`; confirm what a
non-Duo device reports before reusing it.

For Duo, `hingeDegrees` in `/config` now comes from the guest CoreMotion reader added
in [#189](https://github.com/EvanBacon/serve-sim/pull/189) (`watch-angle`), not
`devicectl`; panel orientations still come from the `enumerate` poll above. The reader
is its own child process, so a dead reader (stale last angle) leaves `hid.state` and
`inject.result` unchanged and neither field can detect it.

**Did the screen change, without keeping pixels.** Frame arrival is not a
signal: the idle timer in `FrameCapture.swift` force-emits the current frame
every 200 ms even when nothing changed, so a frame counter advances after every
touch. And `onSharedMjpegFrame` only runs on foldables (`device-session.ts`
subscribes it only when `isFoldable`). Use the content stamp the native capture
already computes instead: `captureFrame` builds a `SurfaceFrameStamp` (IOSurface
ID + seed) and compares it to the previous one. Expose a counter that increments
only when that stamp changes (not on forced idle re-emits) as a cheap native
getter, read it at `begin` and `end`, and record `screen_changed:true` if it
advanced. If not, record `false`, re-read it once after a short window, and patch
the entry with `updateEventLogEvent` if it advanced then. This requires
`recordTouchEvent` to return the entry id (it returns `void` today) and the window
timer to be cleared in `close()`. The seed changes on any surface write, so a
caret blink or running animation can still read `true`; treat `false` as the
reliable signal.

Example (proposed shape): portrait hit vs landscape miss, same send time.

```json
{"kind":"tap","status":"ok","details":{"source":"web","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","screen_changed":true,"input.send_ms":4}}
{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"landscape_left","frame":"display","remapped":false,"hid.state":"ok","inject.result":"sent","screen_changed":false,"input.send_ms":4}}
```

Status stays `ok` for the miss: the injector did what it was asked.

## What an ack proves

Today only `fold` (tag `0x0e`) waits for a reply: it reads the server's `0x0e {ok}`
frame, times out at 10 s, and rejects on failure (`index.ts`). `tap`, `gesture`,
`rotate`, and `button` are fire-and-forget: write the frame, wait a 40-50 ms
timer, close the socket, exit 0 unless the socket itself failed.

**Proposed:** use the `fold` pattern for all input and record
`details.ack = ok | timeout | rejected`. Scope:

- An ack **proves** the server ran `handleHidMessage` and the native call returned.
`NativeHid.guard()` (`src/native.ts`) catches every error from the N-API call,
logs `[hid] <op> ignored bad input`, and returns a fallback, so a failed send
looks successful. Record `inject.result = sent | swallowed` so #136's
dead-injector case is visible. (`guard()` must keep swallowing: an uncaught
throw would crash the in-process server.)
- An ack **does not prove** the guest UI reacted. That is the #153 gap, and why
`screen_changed` is the separate signal.

## HID health and Device Hub shadowing (#136)

- **`hid.state`** (`ok` | `stale` | `gone`): HID can die while the state file and
`--list` still say running. Derive it from the last native call result and
surface it in `--list`.
- **Device Hub shadowing.** `src/device-hub-input.ts`
(`isDeviceHubInputShadowed` / `warnDeviceHubInput`) detects when Device Hub owns
input and legacy HID touches no-op, which looks exactly like #153. Record
`input.shadowed` on inject entries. `serve-sim repair-input` remains the manual
recovery (it restarts SpringBoard).

Look-alikes (proposed shapes):

```json
{"kind":"tap","status":"error","details":{"source":"cli","hid.state":"gone","inject.result":"swallowed","ack":"rejected","screen_changed":false}}
{"kind":"tap","status":"ok","details":{"source":"cli","orientation":"portrait","frame":"native_portrait","remapped":true,"hid.state":"ok","inject.result":"sent","input.shadowed":true,"screen_changed":false,"input.send_ms":4}}
```

A dead injector omits `input.send_ms` so it is not bucketed with a miss; a shadowed
send completed, so it keeps it. [#172](https://github.com/EvanBacon/serve-sim/issues/172)
(portrait tap silently dropped on iPhone 18 Pro / iOS 27) is told apart by the same
fields: `remapped=true` + `screen_changed=false` with `hid.state=ok` and no
`input.shadowed` points away from both a coordinate miss and #136.

## Helper lifecycle (#102)

The orphan in #102 was a detached helper still running after its simulator was
shut down. Decisions live in `src/helper-lifecycle.ts`: `classifyStaleState`
(`keep | recycle-self | recycle-helper`) and `startSimulatorShutdownWatch`
(a `setTimeout` loop). `middleware.ts` acts on `recycle-self`, `index.ts` on
`recycle-helper`, and the watch's `onExit` in `index.ts` calls `process.exit(0)`.
Since [#191](https://github.com/EvanBacon/serve-sim/pull/191) the watch's boot-state
check is the async, uncached `queryBootedUdids`, so a pending `simctl` no longer
stalls detached HTTP/streams (#186); the shared in-flight `getBootedUdids` snapshot
serves `--detach`/follow reuse. Proposed `helper.lifecycle` entries should assume
that model; a failed check keeps the helper alive.

**Proposed:** record one `kind:"helper.lifecycle"` entry when a recycle/exit is
decided, with `decision`, `sim.booted`, and `helper.uptime_ms`. `keep` emits
nothing. An entry written right before `process.exit(0)` is only best-effort for
SSE readers. The helper is the serve-sim process itself; HID and capture run
in-process via the native addon (Duo also spawns the guest `serve-sim-duo-hid`).

## Stream and camera (#128, #143, #103)

- **#128 is a consumer false stall.** "Stream is not producing frames"
(`src/client/simulator/SimulatorView.tsx`) fires from the client's 6 s startup
watchdog (relay `subscribeFrame` or direct `fetch()` reader) when no first frame
arrives; it never fires mid-stream. The watchdog targets a dead producer, but in
#128 it fired while the producer was live. Separately, relay mode drops `live`
(`setConnected(false)`) after 2 s without a frame. Proposed:
`kind:"stream.stall"` with the reason and `producer_live` (from the server's
per-endpoint `streamHealth`, which already counts frames and `last_frame_at`).
Client-observed states need a small ingest route; until then, only
server-side facts can be logged.
- **#143 camera helper exit.** Proposed: `kind:"camera.helper.exit"` with
`exit_code`, `signal` (Node reports a `null` code on signal death), and
`shutdown_phase`. `SimCameraHelper` already ignores SIGPIPE (`main.m`), so don't
attribute exits to it; the real race was the placeholder timer vs. surface
release (mitigated by [#161](https://github.com/EvanBacon/serve-sim/pull/161)).
[#190](https://github.com/EvanBacon/serve-sim/pull/190) fixed app-side crashes
and lookups in the injector (fake `-description`, `deviceWithUniqueID:`), which
are a separate failure path from helper exits.

## Timing

Measured at the hook that builds the entry, patched with `updateEventLogEvent` when
the end comes later. Milliseconds. A missing key means "not a duration", not zero.
All keys and entry kinds below are proposed.

| Key | Entry | Record when | Omit when |
|---|---|---|---|
| `input.send_ms` | tap / drag / button | socket write to ack (or close, while fire-and-forget) | `hid.state=gone`, `inject.result=swallowed`, or `ack=timeout`. A #153 miss and a shadowed send keep it. |
| `stream.ttff_ms` | stream start | session `start()` to first `onSharedMjpegFrame` | no frame ever arrived (#103 cold Connecting). Never write `0`. |
| `stream.reconnect_ms` | `stream.state` | consumer left `live` (relay 2 s staleness check) and returned | false stall (#128) or producer dead |
| `stream.producer_frame_age_ms` | `stream.stall` | age of last producer frame at stall | producer dead |
| `camera.helper_shutdown_ms` | `camera.helper.exit` | helper spawn to exit | still running |
| `helper.uptime_ms` | `helper.lifecycle` | helper start to recycle decision | `keep` |

## Privacy

- Never record raw typed text, clipboard contents, command strings with tokens, or
pixels. Lengths and normalized coordinates are fine.
- Text entry today goes through `serve-sim type` as HID keystrokes (tag `0x06`,
already redacted). There is no clipboard path in serve-sim to instrument; if one
is added (e.g. for [#174](https://github.com/EvanBacon/serve-sim/issues/174)),
log `payload_bytes`, `method`, and `verified_readback`, never the string.
- Remote readback stays behind the existing auth gate; nothing here adds an
export or changes the default.
Loading