CR: feat/mcp-stage-models

docs/piezo-processor.md

@@ -200,6 +200,30 @@ Piezo presence only sees vibration energy and rhythm, so strong bed vibration wi
 
 With one side away, readings use the home side. Presence on either side allows vitals, but an empty verdict requires usable, trusted evidence from both sides. Missing or unusable observations produce unknown immediately. A restart requires a new observation before publishing evidence. A missing, unreadable, legacy or stale checkpoint, or per-side evidence more than five minutes old, also means unknown, even if another side keeps rewriting the file. Unknown presence leaves piezo presence in control. Until an unloading has established a trustworthy baseline, empty-bed vibration can therefore still pass the piezo detector.
 
+## 5a. Beat-to-Beat Detection (`beats.py`)
+
+Individual heartbeat times — what HRV and heart-rate-based sleep staging need — detected per side on a 20 s window advancing 5 s; only beats in the central 5 s are committed, so every beat is decided with context on both sides (~12.5 s latency). A side's tracker only runs while the presence detector (§5) says the side is occupied: an empty bed still carries regular vibration from the pod itself, which the detector would otherwise take for a pulse.
+
+| Step | What it does |
+|---|---|
+| Band-pass + decimate | Each piezo channel to the cardiac band (0.8–8.5 Hz, as the rate pipeline), 500 → 100 Hz |
+| Motion blanking | Samples above 3× the typical heartbeat peak (median of per-1.5 s block maxima, so it tracks J-peak height at any heart rate) are zeroed with a 0.5 s margin; no beat is taken within 0.3 s, and a break is recorded, so no interval spans a turn-over |
+| Other-side noise reference | Ridge least-squares FIR (±50 ms) from the other side's two channels, subtracted. Removes vibration common to both sides (pump beat frequencies, building, a partner's coupled heartbeat). The other side can also carry *this* sleeper's heartbeat; subtracting it collapses cardiac periodicity (measured ~0.83 → 0.1–0.26), so the cleaned channel is used unless periodicity drops by more than 0.10 |
+| Two channels | `left1`/`left2` (and right): normalised, the second aligned to the first (lag ±100 ms, polarity), weighted by periodicity²; the combination is used only if it beats the better single channel |
+| Period selection | Autocorrelation of the beat envelope over 40–150 bpm. Candidates are its peaks, minus any lag whose double correlates better (a second wave inside each beat complex, which on a bed can sit ~0.4 s after the main one) or whose half or third correlates at least as well (every other/third cycle of something faster). The winner has the best subharmonic sum — autocorrelation at 1×, 2×, 3× the lag, weighted 0.84ᵏ (Hermes 1988) |
+| Rhythm tracking | As HRTracker (§6) does for the window rate: once a rhythm is tracked, a window's period must come from peaks within 20% of the median of the last 5 (quality is that peak's own correlation, ≥ 0.2 while following), else the window yields no beats — movement or machinery can put a stronger rhythm at another rate on the sensors. Three consistent windows at a new rate (15 s) re-seed it; without a rate hint, tracking starts from two agreeing windows and only at resting rates (≤ 120 bpm, excluding multiples of a faster rhythm), so a fast rhythm present when someone lies down isn't taken for the heart. A tracked rate can climb past 120 bpm; a rhythm unseen for 60 s is forgotten |
+| Rate hint | In some sleeping positions every beat carries a second wave as strong as the first, half a period later: the envelope then repeats every half-beat and periodicity alone can't tell which is the heart (on a real night the tracker followed half-beats at ~2× the true rate for many minutes). The side's window heart rate (HRTracker, a different estimator that corrects half/double readings) is passed to the tracker each minute: a fresh hint (< 2 min) decides where tracking starts — the period nearest it, with the octave filters off — and a tracked rhythm more than 20% from it is dropped |
+| Gated records | A pump-gated record is pushed masked rather than restarting the windows: its samples are blanked like movement (0.5 s margin, a break so no interval spans it), costing that second instead of a 20 s window refill |
+| Template matching | The window's median beat complex is matched against the signal (normalised cross-correlation); peaks ≥ 0.4 at ≥ 0.6 period spacing are beats, gaps > 1.6 periods are searched again at 0.25, positions refined parabolically. With two good channels, a beat neither channel shows (same template) is dropped |
+| Anchor continuity | A beat's time is a fixed point of the complex (J peak or K trough, ~80 ms apart, depending on polarity). Each window's template is aligned to the previous one (both cropped to the shorter, as template length follows the period) and beat times shifted accordingly, so a night keeps one anchor — a switch would put one bad interval per switch into RMSSD |
+| Artifact correction | Intervals > 20% from the median of the last 11 are rejected (classic HRV editing criterion); an extra beat whose two short intervals sum to a normal one is removed; missed beats and ectopy become breaks, never interpolated beats (classification after Lipponen & Tarvainen 2019). Three mutually consistent outliers mean a real rate change and reset the history — unless the new rate is ~½× or ~2× (a detection artifact) |
+
+Output: one `heartbeats` row per side per minute — beat times as ms offsets, `null` for a break (~350 bytes per full minute, ~150 KB per side per night). Leaving the bed stores the minute in progress. In single-sleeper mode each minute keeps the side that saw the sleeper's beats best, stored under the home side: a minute is held until both trackers are past it (at most 3 min) and keeps the home side it was queued for if the mode changes meanwhile; a minute from another sensor than the last one stored under that side starts with a break, as the two trackers' anchors differ. A failed write is retried with the next minutes (up to 30 held).
+
+Validated on synthetic ballistocardiograms with known beat times (`test_beats.py`): ≥ 97% sensitivity, ≥ 99% PPV and < 5 ms RMS timing error on clean signals; 45–100 bpm; a noisy channel rescued by an inverted, delayed second channel; in-band common vibration removed via the other side; the sleeper's own heartbeat on the other side not cancelled; a movement burst leaving no false interval; RMSSD within 12% of ground truth; a second wave at half the period not doubling the rate; a stronger beat-like pulse train (at lie-down or mid-night) not taken for the heart, alone giving no beats; real rate steps and climbs followed. Replaying a short in-bed recording from a Pod 4 against a wrist-worn reference: per-minute heart rate within ~3 bpm once settled, and no beats from an empty bed.
+
+Known limit: a rhythmic disturbance that is stronger than the heartbeat and repeats within ~20% of the heart period (or at a multiple of it) can't be told apart by periodicity; beats in such stretches get the right rate but can land on the disturbance, inflating HRV there. Movement during them is usually blanked. Cost: ~0.1% of one Apple-silicon core per side (estimate ~1–1.5% on the pod's Cortex-A53).
+
 ## 6. Heart Rate Extraction
 
 ### Bandpass: 0.8-8.5 Hz

docs/research/sleep-stage-calibration/README.md

@@ -149,6 +149,110 @@ docs/research/sleep-stage-calibration/feat2.py dataset.csv
 
 It needs numpy and scikit-learn; its shebang pulls them through `uv`.
 
+## 6. Personal models, container side
+
+`personal-models.py` trains the two ojwalch/sleep_classifiers models (logistic
+regression, random forest with 200 trees of depth 8) on the same features the
+iPhone computes for its on-device personal models
+(sleepypod-ios `docs/design/on-device-stage-training.md`), so the container
+run and the phone fit can be cross-checked:
+
+```sh
+docs/research/sleep-stage-calibration/personal-models.py dataset.csv --contract A   # the challenger's four inputs
+docs/research/sleep-stage-calibration/personal-models.py dataset.csv --contract B   # pod native: adds HRV, breathing, 15-min windows
+```
+
+Rows are 5-minute epochs from each night's first Watch-labeled sample, labeled
+by the Watch stage at the epoch midpoint, rare classes oversampled instead of
+class-weighted (Create ML has no class weights), scored leave-one-night-out
+with Cohen's κ against the Watch, pooled and per night. On the ten left-side
+nights to 2026-10-18 (724 labeled epochs):
+
+| contract | logistic κ | forest κ |
+|---|---|---|
+| A | 0.08 | 0.05 |
+| B | 0.11 | 0.20 |
+
+The bundled challenger scores κ 0.12 on these nights and the pod's rules about
+0.02, so contract B's forest is the first model to clear both.
+
+A sweep on the same nights (leave-one-night-out, contract B) chose the forest's
+depth: 200 trees at depth 3/4/6/8/unlimited gave κ 0.18/0.21/0.22/0.26/0.23,
+and 100, 500 or 1000 trees changed nothing at any depth, so the forest is
+pinned at 200 trees, depth 8. Logistic regression was flat (κ 0.10–0.11) across
+C from 0.01 to 100. Histogram gradient boosting at depth 3 reached κ 0.27, the
+best single number but within noise of the depth-8 forest on eleven nights;
+it would need its own export format and is a candidate once more nights exist.
+
+Two parity checks keep the container and the phone honest:
+
+- Features: `--fixture night-fixture.json` writes a synthetic night's raw rows
+  (deterministic, nobody's sleep) and the feature rows under both contracts.
+  Copy it to `sleepypod-ios/SleepypodTests/Fixtures/stage-features-night.json`;
+  the Swift `StageFeatureParityTests` suite recomputes every column from the raw
+  rows and fails on any difference above 1e-6. `--fixture-real` uses the first
+  real labeled night instead, for a local check only: it carries timestamps and
+  per-minute biometrics and must never be committed.
+- Conversion: `--export out/` fits each model on all nights and writes
+  `personal-<kind>.mlmodel` with a card in the challenger's schema (test vectors
+  are the model's own predictions on 12 training rows). The phone's
+  `StageChallenger.load(modelURL:cardURL:)` refuses a model that fails its
+  vectors, so a converter mismatch shows up at load. Needs `coremltools`
+  (`uv run --with coremltools …`).
+
+### One command on the Mac
+
+`scripts/train-personal-models.py` chains the whole thing: snapshot the pod's
+database over SSH, build the dataset, train both models on a contract with
+leave-one-night-out scoring, export the pod form, and upload the model with
+the higher pooled κ. About five seconds on Apple silicon; nothing runs on the
+phone.
+
+```sh
+scripts/train-personal-models.py --pod <pod-ip> --side left            # contract B, best model, uploaded
+scripts/train-personal-models.py --pod <pod-ip> --side left --no-upload --contract A
+scripts/train-personal-models.py --pod <pod-ip> --snapshot poddb/biometrics.db   # reuse a snapshot
+```
+
+The pod needs the model endpoints below, and the Watch stages must already be
+on it (section 2b). The run leaves the dataset and the exported models (about
+half a megabyte) in the work directory and deletes the ~75 MB database
+snapshot unless `--keep-snapshot`; the pod keeps one model row per side,
+replaced on every upload.
+
+### Running a model on the pod
+
+`--export out/` also writes `personal-<kind>.json`, the pod's portable form
+(forest trees as parallel arrays with four-class leaf votes, logistic as
+scaler plus weights, the feature contract, hyperparameters and the test
+vectors), checked against scikit-learn on every training row before it is
+written. The pod evaluates it with `src/lib/personal-stage-model.ts`, which
+computes the same feature contracts from a night's vitals and movement (the
+same synthetic fixture is replayed by `src/lib/tests/personal-stage-model.test.ts`)
+and refuses a model that fails its own test vectors.
+
+```sh
+# store the container's forest on the pod for the left side
+curl -X PUT http://<pod-ip>:3000/api/sleep-stages/model   -H 'content-type: application/json'   -d "$(jq -c '{side:"left", model:.}' out/personal-forest.json)"
+# score it beside the rules against the Watch on the last 14 nights
+curl -X POST http://<pod-ip>:3000/api/sleep-stages/model/preview   -H 'content-type: application/json' -d '{"side":"left","nights":14}'
+```
+
+The stored model can also stage the side's nights: `PUT /api/sleep-stages/model/active`
+with `{"side":"left","active":true}` (or the "Stage nights with this model"
+switch on the Settings → Sleep stages page) makes `getSleepStages` and the
+Watch comparison classify with the model, with the profile's rules as the
+fallback for any night it cannot stage. Every stage read carries `stagedBy`
+(`rules` or `model` plus the version), and the web Night view marks
+model-staged nights. The profile system and its manifest are untouched: the
+rules remain the configured classifier and the model is a per-side switch
+beside it, which is why no manifest-hash migration was needed. On the ten left-side nights to 2026-10-19
+the container's contract-B forest, evaluated by the pod's TypeScript,
+reached in-sample κ 0.33 (leave-one-night-out 0.20 in training).
+
+The Mac analysis container is not checked in here; add its Dockerfile location
+when it is.
+
 ## Tests
 
 ```sh
@@ -159,3 +263,33 @@ The Python Modules CI workflow runs them on every PR that touches
 `scripts/**/*.py`. The tests use small synthetic fixtures, not a real export. When
 `src/lib/sleep-stages.ts` changes, update the port in the script and the
 classifier cases in the test, which mirror `src/lib/tests/sleep-stages.test.ts`.
+
+
+### Saved model library
+
+Settings → Sleep stages now offers **Upload model JSON** even when no model is
+stored. Upload `candidate.json` from a training run, choose an entry in **Saved
+models**, then press **Use this model** to activate it for the current side.
+Choosing a dropdown entry alone does not switch staging. Uploads save candidates
+without changing the active model, and identical uploads reuse the same entry.
+The model version and held-out κ are displayed as inline code; κ includes an
+explanation. The existing toggle returns the selected model to rules-only staging.
+
+The library is separate for each side. Existing models are copied into it during
+migration without changing their active state. It is stored in the configuration
+database and included with its normal backups.
+
+| Method / path | Request | Behavior |
+| --- | --- | --- |
+| `GET /api/sleep-stages/models` | `?side=left` | List summaries, IDs, selected/active state, and recorded metrics |
+| `POST /api/sleep-stages/models` | `{side, model}` | Validate and save a candidate; return its ID without activation |
+| `PUT /api/sleep-stages/models/selected` | `{side, id, active}` | Select a saved model and explicitly set activation |
+
+The original singular `/api/sleep-stages/model` routes remain compatible with
+existing clients. Singular PUT still selects/replaces the per-side slot and
+preserves its active state, but also archives the old and new model in the
+library. Singular PUT with `model: null` clears the selection without deleting
+saved candidates. Use the plural POST route when an upload must not affect
+staging. Models are validated against known feature contracts and their test
+vectors before upload or selection. These retrospective models do not establish
+prospective or clinical validity.