From 95645ddbb5d675489bf015b269dd6efaa0dc8109 Mon Sep 17 00:00:00 2001 From: Alan Tai <46613976+at350@users.noreply.github.com> Date: Sun, 20 Sep 2026 04:57:11 -0400 Subject: [PATCH 1/3] READMEs: lead with the instrument, built on the ESP32-S3-BOX-3 The root README described the pill reading as mocked. It now has a section on the instrument (parts, a run, the verdicts, what the screen can and cannot say), routes the hardware observation through hardware/classify in the architecture diagram, and lists hardware/ and mobile/ in the layout. POST /pill is described as the no-instrument development stand-in it is. hardware/README.md matches: BOX-3 first, firmware named by job, 'case' not 'orange'. --- README.md | 60 +++++++++++++++++++++++++++++++++++++--------- hardware/README.md | 17 +++++++------ 2 files changed, 59 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index b180934..62495ef 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ ## Check the bottle, the imprint, and the pill — then read the evidence. -Peel is an open-source medicine check for places where a tablet and its packaging may not match. Photograph the **bottle**, read the **imprint**, measure the **pill**. The backend turns those three observations into a sourced report against a regulatory corpus, live web evidence, and (when present) a hardware reading. +Peel is an open-source medicine check for places where a tablet and its packaging may not match. It is two things: a low-cost **instrument** that plugs into an Android phone and watches a tablet dissolve through light, and a **backend** that looks up what the phone saw. Photograph the **bottle**, read the **imprint**, measure the **pill**. Those three observations become a sourced report against a regulatory corpus, live web evidence, and the instrument's reading. [![Hardware CI](https://github.com/viktorinkov/HackMIT-2026/actions/workflows/hardware.yml/badge.svg)](https://github.com/viktorinkov/HackMIT-2026/actions/workflows/hardware.yml) [![Python 3.13](https://img.shields.io/badge/python-3.13-3776AB?logo=python&logoColor=white)](https://www.python.org/downloads/) @@ -30,6 +30,7 @@ Peel is an open-source medicine check for places where a tablet and its packagin ## Table of contents - [Why Peel exists](#why-peel-exists) +- [The instrument](#the-instrument) - [Architecture](#architecture) - [Backend](#backend) - [Getting started](#getting-started) @@ -47,9 +48,38 @@ Peel keeps three observations separate, then looks them up: | --- | --- | | **Bottle** | GPT-4o vision reads a photo of the container (name, strength, NDC, lot, manufacturer). | | **Imprint** | GPT-4o vision reads a photo of the tablet (characters, color, shape). | -| **Pill** | A phone-attached instrument measures the physical tablet. The API still mocks this (`POST /pill` → `mock-spectrometry`). | +| **Pill** | The Peel instrument dissolves the tablet in a stirred vial and measures it with six LEDs and two light sensors. `hardware/classify` turns the run into a dose and release verdict. | -The rest of this README is the system that sits behind those three inputs. +A copied label and a faked imprint both pass a photo. Neither changes how much active ingredient comes out of the tablet, which is what the instrument reads. + +## The instrument + +A tablet goes into a vial of water on a magnetic stirrer. LEDs shine through the vial, and the instrument records how the light changes as the tablet breaks up and releases its contents. + +| Part | Job | +| --- | --- | +| **Espressif ESP32-S3-BOX-3** | The instrument's brain and its face. The ESP32-S3 inside drives six LEDs and the stirrer, reads the sensors, and streams one JSON line a second over USB-C to the phone. Its 2.4" touchscreen in the wall of the case shows state, takes a blank, and starts and stops a run. | +| **Six LEDs** (IR 940 nm, red, yellow, green, blue, violet 400 nm) | Green stays on as the fast channel. A periodic sweep through all six gives a coarse absorbance spectrum. | +| **Two TEMT6000 light sensors** | One reads transmission through the vial, one reads scatter. Scatter separates a cloudy vial from a coloured one. | +| **DS18B20 probe** | Water temperature, since dissolution rate depends on it. | +| **TT motor + neodymium magnets** | Magnetic stirrer, PWM-driven through a 2N2222A. | + +A run: + +1. **Blank** on plain water. Every later reading is an absorbance against it. +2. **Drop the tablet in.** The firmware marks t = 0 itself when transmission falls 6 %. +3. **Watch** until the fast channel flattens. `hardware/classify` reads the plateau, % released, the time to 80 % release, turbidity, and four-colour absorbance, then converts absorbance to mg/L through a calibration line fitted on standards read on the same rig. +4. **Verdict**, which becomes the scan's `hardware` observation: + +| Status | When | +| --- | --- | +| `pass_screen` | Dose inside the label band (90–110 %) and every gate held. | +| `refer_to_lab` | The active was seen but the dose is outside the band, it released too slowly, or there was no absorbance at all for the claimed product. | +| `cannot_verify` | Cloudy vial, saturated read, absorbance on colours the active does not absorb, too short a run, or no calibration. | + +This is a screen, not a compendial assay. It reads actives that absorb visible light (the calibrated reference is riboflavin), and it runs room-temperature water rather than a pharmacopoeial medium. `refer_to_lab` means exactly that. + +Firmware, the debug app, a software board you can break on purpose, and real captures from the hardware: [`hardware/README.md`](hardware/README.md). ## Architecture @@ -57,7 +87,8 @@ The rest of this README is the system that sits behind those three inputs. flowchart TB subgraph clients [Clients] phone[Phone / Flutter] - hw[XIAO + BOX-3 instrument] + hw[ESP32-S3-BOX-3 instrument] + classify[hardware/classify] end subgraph runpod [Runpod — FastAPI] @@ -81,9 +112,10 @@ flowchart TB end phone -->|bottle + imprint photos| vision - hw -->|USB serial JSON| phone + hw -->|USB serial JSON, 1 Hz| phone + hw -->|same stream| classify vision --> scans - phone -->|hardware observation| scans + classify -->|hardware observation| scans scans --> pipe pipe -->|exact + hybrid search| reg pipe --> pills @@ -100,7 +132,7 @@ flowchart TB pipe -->|scan_context| voice ``` -A check is one **scan** document in Elasticsearch. The phone (or curl) posts bottle, imprint, and hardware observations to `POST /scans`. That returns `202` immediately with `status: pending`. A background asyncio task then researches the scan and writes back to the same document. The client polls `GET /scans/{id}`. +A check is one **scan** document in Elasticsearch. The bottle and imprint observations come from the phone's photos, the hardware observation from the instrument's run (`classify.cli run --post`), and all three are posted to `POST /scans`. That returns `202` immediately with `status: pending`. A background asyncio task then researches the scan and writes back to the same document. The client polls `GET /scans/{id}`. ``` pending → partial → complete @@ -149,7 +181,7 @@ The API is meant to run as a long-lived process on [Runpod](https://www.runpod.i Locally the same app is `uv run backend` (reload on `127.0.0.1:8000`). Secrets come from a repo-root `.env`, then `backend/.env` (later wins). On boot, `app.py` calls `ensure_indices()` so the five strict Elasticsearch mappings exist before the first `POST /scans`. If the cluster is unreachable at startup, the process logs a warning and later requests return 503. -The hardware spectrometry **model** is also intended to run on Runpod. In this tree `POST /pill` returns a deterministic mock (`hardware/model = mock-spectrometry`). The physical instrument (Seeed XIAO ESP32-S3 + ESP32-S3-BOX-3 face) streams JSON over USB to `hardware/peel_app`. +The pill reading does not run on Runpod. The instrument streams over USB, `hardware/classify` produces the verdict next to it, and the backend only stores and researches the result. `POST /pill` is a deterministic stand-in (`model = mock-spectrometry`) for developing without the instrument; a scan that used it says so in `limitations`. ### Elasticsearch @@ -268,7 +300,7 @@ uv run peel-seed seed --sources all uv run backend # http://127.0.0.1:8000 — OpenAPI at /docs ``` -Hardware firmware, debug app, and simulator: [`hardware/README.md`](hardware/README.md). +To run the instrument (flash the firmware, install the app, calibrate, run a tablet), see [`hardware/README.md`](hardware/README.md) and [`hardware/classify/README.md`](hardware/classify/README.md). With no board plugged in, `hardware/sim/fake_board.py` is the board in software. ```bash cd backend @@ -285,7 +317,7 @@ Interactive docs: [http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs). | --- | --- | --- | | `POST` | `/photo-identification/bottle` | Vision → bottle observation | | `POST` | `/photo-identification/imprint` | Vision → imprint observation | -| `POST` | `/pill` | Mock hardware observation | +| `POST` | `/pill` | Stand-in hardware observation for development without the instrument | | `POST` | `/scans` | Create scan, start pipeline (`202`) | | `GET` | `/scans/{id}` | Poll envelope (`pending` / `partial` / `complete`) | | `GET` | `/scans/{id}/context` | `scan_context` for the voice agent (`?as_string=true`) | @@ -308,7 +340,13 @@ HackMIT-2026/ │ ├── src/backend/ vision, scans, research, knowledge, reports, Deepgram │ ├── scripts/ smoke tests, runpod-start.sh │ └── README.md mappings, retrieval, Agent Builder, Firecrawl budget -├── hardware/ XIAO firmware, BOX-3 face, debug Flutter app, simulator +├── hardware/ the instrument +│ ├── firmware/ 17_stream (measurement), 21_box3_face (the face), self-test and bring-up +│ ├── classify/ serial stream → dose and release verdict → the scan's hardware field +│ ├── peel_app/ Flutter Android app: USB serial → parser → faults → session log +│ ├── sim/ fake_board.py, the board in software +│ └── data/ real captures from this hardware +├── mobile/ Peel Flutter Android app ├── assets/ README images and media placeholders ├── photos-for-testing/ sample bottle and imprint photos └── LICENSE diff --git a/hardware/README.md b/hardware/README.md index 9a42719..8198759 100644 --- a/hardware/README.md +++ b/hardware/README.md @@ -1,16 +1,17 @@ # Peel: hardware -The instrument side of Peel. A Seeed XIAO ESP32-S3 watches a tablet dissolve through light and -streams one reading a second over USB to an Android phone, which also powers it. An -ESP32-S3-BOX-3 in the wall of the orange is its face, linked over ESP-NOW. +The instrument side of Peel, built on the Espressif ESP32-S3-BOX-3. It watches a tablet +dissolve in a stirred vial through six LEDs and two light sensors, and streams one reading a +second over USB to an Android phone. The BOX-3's touchscreen in the wall of the case is its +face: it shows state, takes a blank, and starts and stops a run. ``` -firmware/17_stream/ XIAO firmware. The source of truth for the protocol. +firmware/17_stream/ Measurement firmware: LEDs, sensors, stirrer. The source of truth for the protocol. firmware/18_selftest/ Bench diagnostic: lock-in LED test, diode check, noise. -firmware/21_box3_face/ The BOX-3's face: eyes, moods, one-button blank / start / stop. +firmware/21_box3_face/ The face: eyes, moods, one-button blank / start / stop. firmware/19_box3_link/, 22_box3_probe/ Bring-up tools. peel_app/ Flutter Android app: USB serial → parser → faults → session log. -sim/ fake_board.py: the XIAO in software, breakable on purpose. +sim/ fake_board.py: the board in software, breakable on purpose. tools/ Desktop serial clients: peel_monitor.py, capture.py, flash_when_ready.py. data/ Real captures from this hardware. ``` @@ -19,7 +20,9 @@ The reference docs the code comments name (`PROTOCOL.md`, `SIGNALS.md`, `FAULTS. `BASELINES.md`, `PLUG_IN_DAY.md`) are kept off main: they are in `hardware/docs/` on the `hardware-component` branch. -## Flash the board +## Flash the firmware + +Measurement first, then the face: ```bash arduino-cli compile --upload -p /dev/cu.usbmodemXXXX --fqbn esp32:esp32:XIAO_ESP32S3 firmware/17_stream From cd6537eb1398ca79cb0886660c231fd0990ab589 Mon Sep 17 00:00:00 2001 From: Alan Tai <46613976+at350@users.noreply.github.com> Date: Sun, 20 Sep 2026 10:12:55 -0400 Subject: [PATCH 2/3] README: the full hardware story, and no links to paths that are not on main Parts, power, the wire protocol and commands, the fault engine, a run, the simulator, captures and tools. The classification layer is described through truepill (#26) and its /pill integration (#27) rather than hardware/classify, which never reached main. Two boards, as the firmware has it: an ESP32-S3 sensor board and the ESP32-S3-BOX-3 face over ESP-NOW. --- README.md | 84 ++++++++++++++++++++++++++++++++-------------- hardware/README.md | 14 ++++---- 2 files changed, 66 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index d1fd149..7296589 100644 --- a/README.md +++ b/README.md @@ -47,38 +47,72 @@ Peel keeps three observations separate, then looks them up: | --- | --- | | **Bottle** | GPT-4o vision reads a photo of the container (name, strength, NDC, lot, manufacturer). | | **Imprint** | GPT-4o vision reads a photo of the tablet (characters, color, shape). | -| **Pill** | The Peel instrument dissolves the tablet in a stirred vial and measures it with six LEDs and two light sensors. `hardware/classify` turns the run into a dose and release verdict. | +| **Pill** | The Peel instrument dissolves the tablet in a stirred vial and measures it with six LEDs and two light sensors, once a second. A classification layer turns the run into a verdict on the tablet. | A copied label and a faked imprint both pass a photo. Neither changes how much active ingredient comes out of the tablet, which is what the instrument reads. ## The instrument -A tablet goes into a vial of water on a magnetic stirrer. LEDs shine through the vial, and the instrument records how the light changes as the tablet breaks up and releases its contents. +A tablet goes into a vial of water on a magnetic stirrer. LEDs shine through the vial, and the instrument records how the light changes as the tablet breaks up and releases its contents. The phone powers it, talks to it over USB-C, and is where the numbers are shown. + +### Parts | Part | Job | | --- | --- | -| **Espressif ESP32-S3-BOX-3** | The instrument's brain and its face. The ESP32-S3 inside drives six LEDs and the stirrer, reads the sensors, and streams one JSON line a second over USB-C to the phone. Its 2.4" touchscreen in the wall of the case shows state, takes a blank, and starts and stops a run. | -| **Six LEDs** (IR 940 nm, red, yellow, green, blue, violet 400 nm) | Green stays on as the fast channel. A periodic sweep through all six gives a coarse absorbance spectrum. | -| **Two TEMT6000 light sensors** | One reads transmission through the vial, one reads scatter. Scatter separates a cloudy vial from a coloured one. | -| **DS18B20 probe** | Water temperature, since dissolution rate depends on it. | -| **TT motor + neodymium magnets** | Magnetic stirrer, PWM-driven through a 2N2222A. | +| **ESP32-S3 sensor board** | The measuring half. Drives the LEDs and the stirrer, reads the sensors, streams one JSON line a second over native USB to the phone. `firmware/17_stream` runs unchanged on an Espressif ESP32-S3-DevKitC-1 or a Seeed XIAO ESP32-S3, since pins are raw GPIO numbers. | +| **Espressif ESP32-S3-BOX-3** | The face in the wall of the case. Its 2.4" touchscreen shows state with eyes and moods, and one button takes a blank, starts and stops a run. Linked to the sensor board over ESP-NOW, so it needs no router or pairing. `firmware/21_box3_face`. | +| **Six 5 mm LEDs** | IR 940 nm, red 625, yellow 590, green 525, blue 465, violet 400, each on its own GPIO through a series resistor. Green stays on as the fast channel. A periodic sweep through all six gives a coarse absorbance spectrum. | +| **Two TEMT6000 light sensors** | One looks straight at the LEDs through the liquid (transmission). One sits 90° off the beam (scatter). Scatter separates a cloudy vial from a coloured one. Both are on ADC1, which keeps working with the radio on. | +| **DS18B20 waterproof probe** | Water temperature, since dissolution rate depends on it. | +| **Magnetic stirrer** | A TT gear motor with neodymium discs on the hub, a stack of nine 6 × 2 mm discs as the stir bar, 20 kHz PWM through a low-side transistor with a flyback diode. | -A run: +### Power -1. **Blank** on plain water. Every later reading is an absorbance against it. -2. **Drop the tablet in.** The firmware marks t = 0 itself when transmission falls 6 %. -3. **Watch** until the fast channel flattens. `hardware/classify` reads the plateau, % released, the time to 80 % release, turbidity, and four-colour absorbance, then converts absorbance to mg/L through a calibration line fitted on standards read on the same rig. -4. **Verdict**, which becomes the scan's `hardware` observation: +The phone supplies the sensor board, the LEDs and the sensors over the USB-C data cable. The stirrer motor and the BOX-3 run from a power bank, because a phone's USB port supplies roughly 500 mA and the motor alone draws up to 1 A at start. One rule protects the phone: the power bank's 5 V and the phone's VBUS are never tied together. Grounds are common. + +### What comes down the wire + +One JSON line a second at 115200 baud, plus `# ...` note lines for events. -| Status | When | +| Key | Meaning | | --- | --- | -| `pass_screen` | Dose inside the label band (90–110 %) and every gate held. | -| `refer_to_lab` | The active was seen but the dose is outside the band, it released too slowly, or there was no absorbance at all for the claimed product. | -| `cannot_verify` | Cloudy vial, saturated read, absorbance on colours the active does not absorb, too short a run, or no calibration. | +| `t` | Seconds since t = 0, or −1 when no run is in progress. | +| `trans`, `scat` | Both sensors under the green LED, in mV (mean of 24 ADC reads). | +| `absT`, `absS` | Absorbance against the stored blank, `log10(blank / reading)`. Null until a blank is taken. | +| `tC` | Water temperature in °C. | +| `sweep`, `sweepS` | Transmission and scatter under each of the six LEDs, minus dark. | +| `dark` | Both sensors with every LED off, measured at the start of each sweep, so ambient light is subtracted. | +| `stir`, `swept` | Stirrer duty, and a flag on the one line per sweep whose fast-channel reading the sweep disturbed. | + +Commands are one ASCII byte, over USB or ESP-NOW: `b` store a blank, `z` mark t = 0, `a` toggle automatic t = 0 (on at boot: a 6 % drop in transmission after a blank), `s` stop, `m` toggle the stirrer, `d` print a diagnostics line. The diagnostics line reports every LED's forward voltage, sensor noise, dark levels, the reset reason, heap, the temperature probe's address and the radio's state. + +### Faults are data + +The phone side (`hardware/peel_app`) runs a fault engine that is a pure function of the stream's history, with every threshold traced to a measured baseline. A broken rig shows up as a named fault: `LID_OPEN`, `LED_OPEN`, `LED_SHORT`, `LED_SWAPPED`, `SENSOR_SATURATED`, `SENSOR_NOISY`, `SENSOR_UNPOWERED`, `MOTOR_COUPLING`, `BROWNOUT_RESET`, `BOARD_RESET_MIDRUN`, `PROBE_MISSING`, `STREAM_STALE`, `WRONG_FIRMWARE`, `RADIO_DOWN` and more. + +### A run -This is a screen, not a compendial assay. It reads actives that absorb visible light (the calibrated reference is riboflavin), and it runs room-temperature water rather than a pharmacopoeial medium. `refer_to_lab` means exactly that. +1. **Blank** on plain water. Every later reading is an absorbance against it. +2. **Drop the tablet in.** The firmware marks t = 0 itself. +3. **Watch** the fast channel rise and flatten while sweeps record the spectrum. +4. **Classify.** The run becomes the scan's `hardware` observation: a status of `real`, `substandard`, `fake` or `unknown`, where a reading the rig could not take is always `unknown`. + +### Classification + +The classification layer is [`truepill`](https://github.com/viktorinkov/HackMIT-2026/pull/26): spectral matching of the sweep against a library measured on this rig, first-order dissolution kinetics `A(t) = A∞ · (1 − e^(−kt))` fitted per channel, percent released, an f2 similarity score against a reference profile, and an `INVALID_READING` verdict for readings the optics cannot have produced. [PR #27](https://github.com/viktorinkov/HackMIT-2026/pull/27) wires it to `POST /pill` and the mobile app. Until those merge, `POST /pill` on main returns a labelled stand-in (`model = mock-spectrometry`). + +This is a screen, not a compendial assay. It reads actives that absorb visible light, in room-temperature water rather than a pharmacopoeial medium, and its measured library is small. A failed screen means "send it to a lab". + +### Without the hardware + +| Path | What it is | +| --- | --- | +| `hardware/sim/fake_board.py` | The sensor board in software, standard library only. Same protocol over TCP, a pty or stdout, replay of a real capture, every fault injectable at boot, at a time or live, and a transport that can be broken on purpose (`--chunk 1`, `--junk`, `--disconnect-after`). | +| `hardware/data/` | Real captures from this hardware: a full run, the ESP-NOW link test, LED diode checks, light response, motor coupling, self-test logs. | +| `hardware/tools/` | Desktop serial clients: `peel_monitor.py` (terminal dashboard and CSV), `capture.py`, `flash_when_ready.py`. | +| `firmware/18_selftest` | Bench diagnostic: lock-in LED test, diode check, noise. | -Firmware, the debug app, a software board you can break on purpose, and real captures from the hardware: [`hardware/README.md`](hardware/README.md). +Flashing, the app and the simulator: [`hardware/README.md`](hardware/README.md). ## Architecture @@ -86,8 +120,8 @@ Firmware, the debug app, a software board you can break on purpose, and real cap flowchart TB subgraph clients [Clients] phone[Phone / Flutter] - hw[ESP32-S3-BOX-3 instrument] - classify[hardware/classify] + hw[ESP32-S3 sensor board + BOX-3 face] + classify[Classification layer] end subgraph runpod [Runpod — FastAPI] @@ -131,7 +165,7 @@ flowchart TB pipe -->|scan_context| voice ``` -A check is one **scan** document in Elasticsearch. The bottle and imprint observations come from the phone's photos, the hardware observation from the instrument's run (`classify.cli run --post`), and all three are posted to `POST /scans`. That returns `202` immediately with `status: pending`. A background asyncio task then researches the scan and writes back to the same document. The client polls `GET /scans/{id}`. +A check is one **scan** document in Elasticsearch. The bottle and imprint observations come from the phone's photos, the hardware observation from the instrument's run, and all three are posted to `POST /scans`. That returns `202` immediately with `status: pending`. A background asyncio task then researches the scan and writes back to the same document. The client polls `GET /scans/{id}`. ``` pending → partial → complete @@ -180,7 +214,7 @@ The API is meant to run as a long-lived process on [Runpod](https://www.runpod.i Locally the same app is `uv run backend` (reload on `127.0.0.1:8000`). Secrets come from a repo-root `.env`, then `backend/.env` (later wins). On boot, `app.py` calls `ensure_indices()` so the five strict Elasticsearch mappings exist before the first `POST /scans`. If the cluster is unreachable at startup, the process logs a warning and later requests return 503. -The pill reading does not run on Runpod. The instrument streams over USB, `hardware/classify` produces the verdict next to it, and the backend only stores and researches the result. `POST /pill` is a deterministic stand-in (`model = mock-spectrometry`) for developing without the instrument; a scan that used it says so in `limitations`. +The pill reading does not run on Runpod. The instrument streams over USB to the phone, the classification layer turns the run into a verdict, and the backend stores and researches the result. `POST /pill` is a deterministic stand-in (`model = mock-spectrometry`) for developing without the instrument; a scan that used it says so in `limitations`. ### Elasticsearch @@ -299,7 +333,7 @@ uv run peel-seed seed --sources all uv run backend # http://127.0.0.1:8000 — OpenAPI at /docs ``` -To run the instrument (flash the firmware, install the app, calibrate, run a tablet), see [`hardware/README.md`](hardware/README.md) and [`hardware/classify/README.md`](hardware/classify/README.md). With no board plugged in, `hardware/sim/fake_board.py` is the board in software. +To run the instrument (flash the firmware, install the app, run a tablet), see [`hardware/README.md`](hardware/README.md). With no board plugged in, `hardware/sim/fake_board.py` is the board in software. ```bash cd backend @@ -346,10 +380,10 @@ HackMIT-2026/ │ ├── scripts/ smoke tests, runpod-start.sh │ └── README.md mappings, retrieval, Agent Builder, Firecrawl budget ├── hardware/ the instrument -│ ├── firmware/ 17_stream (measurement), 21_box3_face (the face), self-test and bring-up -│ ├── classify/ serial stream → dose and release verdict → the scan's hardware field +│ ├── firmware/ 17_stream (sensor board), 21_box3_face (BOX-3), 18_selftest, bring-up │ ├── peel_app/ Flutter Android app: USB serial → parser → faults → session log │ ├── sim/ fake_board.py, the board in software +│ ├── tools/ desktop serial clients │ └── data/ real captures from this hardware ├── mobile/ Peel Flutter Android app ├── assets/ README images and media placeholders diff --git a/hardware/README.md b/hardware/README.md index 8198759..e8bcf85 100644 --- a/hardware/README.md +++ b/hardware/README.md @@ -1,14 +1,14 @@ # Peel: hardware -The instrument side of Peel, built on the Espressif ESP32-S3-BOX-3. It watches a tablet -dissolve in a stirred vial through six LEDs and two light sensors, and streams one reading a -second over USB to an Android phone. The BOX-3's touchscreen in the wall of the case is its -face: it shows state, takes a blank, and starts and stops a run. +The instrument side of Peel. An ESP32-S3 sensor board watches a tablet dissolve in a stirred +vial through six LEDs and two light sensors, and streams one reading a second over USB to an +Android phone, which also powers it. An Espressif ESP32-S3-BOX-3 in the wall of the case is +its face, linked over ESP-NOW: it shows state, takes a blank, and starts and stops a run. ``` -firmware/17_stream/ Measurement firmware: LEDs, sensors, stirrer. The source of truth for the protocol. +firmware/17_stream/ Sensor board firmware: LEDs, sensors, stirrer. The source of truth for the protocol. firmware/18_selftest/ Bench diagnostic: lock-in LED test, diode check, noise. -firmware/21_box3_face/ The face: eyes, moods, one-button blank / start / stop. +firmware/21_box3_face/ The BOX-3's face: eyes, moods, one-button blank / start / stop. firmware/19_box3_link/, 22_box3_probe/ Bring-up tools. peel_app/ Flutter Android app: USB serial → parser → faults → session log. sim/ fake_board.py: the board in software, breakable on purpose. @@ -22,7 +22,7 @@ The reference docs the code comments name (`PROTOCOL.md`, `SIGNALS.md`, `FAULTS. ## Flash the firmware -Measurement first, then the face: +Sensor board first, then the BOX-3: ```bash arduino-cli compile --upload -p /dev/cu.usbmodemXXXX --fqbn esp32:esp32:XIAO_ESP32S3 firmware/17_stream From c6b0949dcc37a27b4bcc9e7e9eaad12e2fe4e2bf Mon Sep 17 00:00:00 2001 From: Alan Tai <46613976+at350@users.noreply.github.com> Date: Sun, 20 Sep 2026 10:41:00 -0400 Subject: [PATCH 3/3] Document the local Qwen3-VL photo extraction pipeline Covers the QLoRA post-training setup, reproduction steps on Apple Silicon, the synthetic dataset and evaluation protocol, and the privacy boundaries for the local bottle/imprint endpoints. --- backend/LOCAL_VISION.md | 105 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 backend/LOCAL_VISION.md diff --git a/backend/LOCAL_VISION.md b/backend/LOCAL_VISION.md new file mode 100644 index 0000000..65738c1 --- /dev/null +++ b/backend/LOCAL_VISION.md @@ -0,0 +1,105 @@ +# Local bottle and pill extraction + +The two existing `/photo-identification/{bottle,imprint}` endpoints now use an +in-process Apple Silicon model. There is no OpenAI vision call or cloud fallback. +The mobile response structure stays compatible; personal/free-text fields are +returned as null or empty lists. This is a synthetic-data pilot, not a validated +replacement for GPT's real-photo extraction accuracy. + +Base model: [SmolVLM 500M Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-500M-Instruct), +Apache 2.0, using the +[MLX 4-bit conversion](https://huggingface.co/mlx-community/SmolVLM-500M-Instruct-4bit) +at revision `462811eef5e83cee4df69ef545b31cd6adf0cc50`. +[MLX-VLM](https://github.com/Blaizzy/mlx-vlm) provides inference and QLoRA training. +Only small language-model adapters are trained; base vision weights remain frozen. + +## Reproduce on an Apple Silicon Mac + +From `backend/`: + +```bash +uv sync --extra local-vision +uv run --extra local-vision python scripts/local_vision.py prepare +uv run --extra local-vision python scripts/local_vision.py download +uv run --extra local-vision python scripts/local_vision.py evaluate +uv run --extra local-vision python scripts/local_vision.py train --steps 128 +uv run --extra local-vision python scripts/local_vision.py evaluate --adapter +uv run --extra local-vision backend +``` + +Keep `--extra local-vision` on `uv run` so uv does not remove the optional runtime. +The download is the only step that needs Hugging Face network access. Training and +inference explicitly use local paths and offline mode, with telemetry disabled. +No Hugging Face login or paid inference endpoint is needed. + +Artifacts live in gitignored `backend/data/vision/`: + +- `base/`: downloaded model, tokenizer and pinned provenance. +- `adapter/`: trained `adapters.safetensors`, adapter config, and `run.json`. +- `train.jsonl`, `valid.jsonl`, `test.jsonl`: disjoint synthetic records. +- `evaluation/base.json`, `evaluation/adapter.json`: identical held-out comparison. +- `evaluation/generated-bottle.png`, `evaluation/generated-pill.png`: reviewed + GPT Image examples, when present. + +The training command refuses to overwrite an existing adapter. Move the directory +before starting a new run. Use `VISION_MODEL_PATH` and `VISION_ADAPTER_PATH` to point +the backend at copies of the model and adapter. Defaults are independent of cwd. +Missing weights, missing dependencies, invalid images or invalid model JSON fail +closed with a generic 502; no raw model output is exposed in error responses. +This runtime targets macOS arm64, not iOS/Android or the Linux RunPod deployment. + +## Dataset and evaluation + +`prepare` creates 64 training, 8 validation and 16 test examples from a fixed seed. +Labels are rendered from known text; no real patient image or existing test photo +is used for training. Examples include bottles, round/oval tablets, blank negative +images, and `[REDACTED]` patient placeholders. These are deliberately simple +synthetic examples, with substantial domain differences from camera photos. + +The extra GPT Image bottle and pill are reserved for evaluation and never trained +on. Their ground truth was checked visually. Built-in imagegen bottle prompt: + +> A photorealistic smartphone photo of a single amber bottle, white cap, kitchen +> counter, natural window light. Entire label legible: SYNTHETIC SAMPLE; +> AMOXICILLIN; 500 mg; 30 capsules; LOT: DEMO427; EXP: 2028-06. No other text, +> identifiers, brands, barcodes or pills outside the bottle. + +The pill prompt requests a single white round unscored tablet printed `L484`, on +dark blue fabric in natural light, with no other objects or text. Both generation +prompts are recorded in `data/vision/evaluation/imagegen-prompt.txt`. +On a fresh checkout these optional images are absent, so `prepare` produces 16 tests. +The default comparison evaluates eight procedural images plus the generated photos +when available. It reports JSON validity, exact records and non-null field accuracy; +confidence is excluded from field matching. These tiny synthetic metrics cannot +establish real-photo accuracy, drug identity, authenticity or clinical suitability. +Readable confidence is the model's uncalibrated self-report. + +Before deployment, collect independently labeled, de-identified camera photos, +including tiny imprints, curved labels, blur, glare, partial text and non-medication +objects. Compare field accuracy and hallucinations against the current baseline. +The pilot train command accepts only its procedural synthetic records by design. + +## Privacy boundaries + +- Uploaded pixels stay in backend memory and are processed on that Mac. They are + never written to disk by this extraction path or uploaded to a model provider. + Image metadata is removed by decoding and converting to RGB. +- Patient/prescriber identity, Rx number, pharmacy, directions, warnings, arbitrary + label text and notes are excluded from the model contract and discarded in code, + even if the model emits them. Existing API fields remain null/empty for clients. +- Only product facts (names, strength, lot, expiry, NDC, etc.) and pill appearance + fields are allowed. An additional conservative filter rejects obvious personal + text markers. **This does not guarantee that a model will never place a person's + name inside an allowed product field.** A validated local entity detector or a + product-vocabulary gate would be needed for a stronger output guarantee. +- Synthetic fine-tuning adds no real patient PII. It cannot certify that the upstream + pretrained model contains no PII, nor erase its original training data. +- A phone still uploads its image to the configured backend. Use a trusted Mac + and transport appropriate to your deployment. Running the backend in a cloud VM + would move this privacy boundary to that VM. +- Research, Elasticsearch storage, web search and voice remain external services. + This change is local photo extraction, not an end-to-end offline app. Existing + `SCANS_STORE_SENSITIVE` behavior for separately supplied scan data is unchanged. + +Photos, model outputs and adapters are not automatically uploaded, committed or +used to train on future scans. Generated data and weights are gitignored.