Unify Peel workflow, sensor matching, and optional hardware display

.github/workflows/backend.yml

@@ -0,0 +1,21 @@
+name: backend
+on:
+  push:
+    paths:
+      - 'backend/**'
+      - '.github/workflows/backend.yml'
+  pull_request:
+    paths:
+      - 'backend/**'
+      - '.github/workflows/backend.yml'
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.13'
+      - run: python -m pip install uv==0.12.15
+      - run: uv sync --project backend --locked --group dev
+      - run: uv run --project backend pytest backend/tests

.github/workflows/hardware.yml

@@ -20,43 +20,14 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: '3.11'
+      - name: display command framing
+        run: |
+          c++ -std=c++11 hardware/tests/display_protocol_test.cpp -o /tmp/display-protocol-test
+          /tmp/display-protocol-test
       - name: fake board
         working-directory: hardware
         run: python3 sim/test_fake_board.py
 
-  app:
-    name: flutter app
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
-        with:
-          python-version: '3.11'          # the integration tests spawn sim/fake_board.py
-      - uses: actions/setup-java@v4
-        with:
-          distribution: temurin
-          java-version: '17'
-      - uses: subosito/flutter-action@v2
-        with:
-          flutter-version: 3.47.5
-          cache: true
-      - name: pub get
-        working-directory: hardware/peel_app
-        run: flutter pub get
-      - name: analyze
-        working-directory: hardware/peel_app
-        run: flutter analyze
-      - name: test
-        working-directory: hardware/peel_app
-        run: flutter test
-      - name: build apk
-        working-directory: hardware/peel_app
-        run: flutter build apk --debug
-      - uses: actions/upload-artifact@v4
-        with:
-          name: peel-app-debug-apk
-          path: hardware/peel_app/build/app/outputs/flutter-apk/app-debug.apk
-
   firmware:
     name: firmware
     runs-on: ubuntu-latest
@@ -74,3 +45,9 @@ jobs:
       - name: 21_box3_face (ESP32-S3-BOX-3)
         working-directory: hardware
         run: arduino-cli compile --fqbn esp32:esp32:esp32s3box firmware/21_box3_face
+      - name: 23_phone_display (BOX-3 radio display)
+        working-directory: hardware
+        run: arduino-cli compile --fqbn esp32:esp32:esp32s3box firmware/23_phone_display
+      - name: 24_phone_relay (Seeed phone bridge)
+        working-directory: hardware
+        run: arduino-cli compile --fqbn esp32:esp32:XIAO_ESP32S3 firmware/24_phone_relay

.github/workflows/mobile.yml

@@ -5,10 +5,14 @@ on:
   push:
     paths:
       - 'mobile/**'
+      - 'hardware/sim/**'
+      - 'hardware/data/**'
       - '.github/workflows/mobile.yml'
   pull_request:
     paths:
       - 'mobile/**'
+      - 'hardware/sim/**'
+      - 'hardware/data/**'
       - '.github/workflows/mobile.yml'
 
 jobs:
@@ -17,10 +21,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
       - uses: actions/setup-java@v4
         with:
           distribution: temurin
-          java-version: '17'
+          java-version: '21'
       - uses: subosito/flutter-action@v2
         with:
           flutter-version: 3.47.5

README.md

@@ -46,7 +46,7 @@ Peel keeps three observations separate, then looks them up:
 | --- | --- |
 | **Bottle** | GPT-4o vision reads a photo of the container (name, strength, NDC, lot, manufacturer). |
 | **Imprint** | GPT-4o vision reads a photo of the tablet (characters, color, shape). |
-| **Pill** | A phone-attached instrument measures the physical tablet. The API still mocks this (`POST /pill` → `mock-spectrometry`). |
+| **Pill** | A phone-attached instrument records optical sensor data; `POST /scans` carries the actual readings to research and voice. |
 
 The rest of this README is the system that sits behind those three inputs.
 
@@ -148,7 +148,7 @@ The API is meant to run as a long-lived process on [Runpod](https://www.runpod.i
 
 Locally the same app is `uv run backend` (reload on `127.0.0.1:8000`). Secrets come from a repo-root `.env`, then `backend/.env` (later wins). On boot, `app.py` calls `ensure_indices()` so the five strict Elasticsearch mappings exist before the first `POST /scans`. If the cluster is unreachable at startup, the process logs a warning and later requests return 503.
 
-The hardware spectrometry **model** is also intended to run on Runpod. In this tree `POST /pill` returns a deterministic mock (`hardware/model = mock-spectrometry`). The physical instrument (Seeed XIAO ESP32-S3 + ESP32-S3-BOX-3 face) streams JSON over USB to `hardware/peel_app`.
+The physical instrument (Seeed XIAO ESP32-S3 + ESP32-S3-BOX-3 display) streams JSON over USB to `mobile`. The app sends timestamped optical readings with the scan. Research and voice receive bounded samples and channel statistics, separately from drug identity. The old mock `POST /pill` endpoint has been removed.
 
 ### Elasticsearch
 
@@ -284,7 +284,6 @@ Interactive docs: [http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs).
 | --- | --- | --- |
 | `POST` | `/photo-identification/bottle` | Vision → bottle observation |
 | `POST` | `/photo-identification/imprint` | Vision → imprint observation |
-| `POST` | `/pill` | Mock hardware observation |
 | `POST` | `/scans` | Create scan, start pipeline (`202`) |
 | `GET` | `/scans/{id}` | Poll envelope (`pending` / `partial` / `complete`) |
 | `GET` | `/scans/{id}/context` | `scan_context` for the voice agent (`?as_string=true`) |

backend/README.md

@@ -37,7 +37,7 @@ Three capabilities make up the layer:
 ```
  photo(bottle) ──▶ POST /photo-identification/bottle  (GPT-4o vision)
  photo(imprint) ─▶ POST /photo-identification/imprint (GPT-4o vision)
- spectrometer ───▶ POST /pill                         (mock-spectrometry)
+ spectrometer ───▶ mobile ───▶ POST /scans             (measured sensor data)
                               │
                               ▼
                        POST /scans  (ScanCreate)
@@ -1032,8 +1032,8 @@ but nothing reads it any more; only `scripts/deepgram-chat.py` uses it locally.)
 next pod start. Editing the pod env replaces the container, so keep the app on the
 network volume.
 
-The deployed API currently has no client authentication. `/pill` still returns
-mock spectrometry. External service calls require valid OpenAI, Firecrawl,
+The API currently has no client authentication. The mock `/pill` endpoint and
+`/graph?demo=1` responses have been removed. External service calls require valid OpenAI, Firecrawl,
 and Elasticsearch credentials.
 
 Check the running API from your computer:
@@ -1042,11 +1042,7 @@ Check the running API from your computer:
 curl --fail https://<pod-id>-8000.proxy.runpod.net/health
 ```
 
-```bash
-curl --fail https://<pod-id>-8000.proxy.runpod.net/pill \
-  -H 'Content-Type: application/json' \
-  -d '{"status":"unknown"}'
-```
+
 
 Open the existing pod's SSH terminal:
 
@@ -1067,3 +1063,37 @@ The pod is left running so the API stays available. Terminating it deletes its
 container disk; the network volume remains and continues billing. For complete
 cleanup, terminate the pod first, then delete `peel-fastapi-data` from the Runpod
 Storage page. Deleting that volume permanently deletes the deployed files.
+
+## Real hardware evidence
+
+`POST /scans` accepts `hardware.sensor_readings` (up to 256 aligned samples),
+`sensor_sample_count`, and the existing absorbance trace. Each sample carries
+time, transmission/scattering mV, absorbance, temperature, stir percentage and
+nullable colour sweep channels. Startup adds these fields to the existing scan
+index without deleting data. Research and voice share bounded raw samples and
+computed ranges/changes; unknown identity does not suppress the measurements.
+The voice context uses `status: measured` when real readings exist without an
+identified candidate. Missing/skipped hardware is never replaced by a mock.
+
+Deploy the backend with the updated APK: older servers ignore the new sensor
+fields and do not expose them to the AI. Existing stored mock scans remain
+explicitly labelled as simulated. No model is asked to invent drug identity or
+potency from an uncalibrated optical trace.
+
+### Synthetic reference matching
+
+Real scan absorbance passes through `reference_match.py` before research or voice.
+It baseline-subtracts the non-sweep transmission absorbance series, resamples to 32
+run-progress points, and ranks four generated curves using RMSE in absorbance units.
+The generated B12, acetaminophen, vitamin C and caffeine labels are demonstration
+fixtures, not measured chemical references or a trained spectrometer model. The
+result says `Closest match:` with separate synthetic-library provenance. Rankings
+do not use bottle names or user labels. No probabilities, potency or authenticity
+claims are produced. Fewer than eight valid samples, >20% missing readings, or a
+flat trace produce no match. Close rankings and out-of-library readings are marked.
+Changing concentration, illumination, or run duration may change the match; this
+version compares normalized run progress, not wavelengths or dissolution rates.
+
+Voice follows main at `c4f5c89` (Viktor's parser update), with a short computed-match
+line added. Raw telemetry and long hardware findings stay out of the voice prompt;
+research retains the full bounded evidence.

backend/src/backend/app.py

@@ -12,7 +12,6 @@
 from backend.knowledge.indices import ensure_indices
 from backend.knowledge.router import router as knowledge_router
 from backend.photo_identification import router as photo_identification_router
-from backend.pill import router as pill_router
 from backend.reports.router import router as reports_router
 from backend.research.agent_builder import close_agent_builder
 from backend.research.pipeline import cancel_all as cancel_research
@@ -48,7 +47,6 @@ async def lifespan(_app: FastAPI):
 )
 app.include_router(photo_identification_router)
 app.include_router(drug_facts_router)
-app.include_router(pill_router)
 app.include_router(scans_router)
 app.include_router(reports_router)
 app.include_router(knowledge_router)
@@ -71,7 +69,6 @@ def root() -> dict[str, object]:
             "bottle": "/drug-facts/bottle",
             "imprint": "/drug-facts/imprint",
         },
-        "pill": "/pill",
         "scans": {
             "create": "POST /scans",
             "get": "/scans/{scan_id}",

backend/src/backend/deepgram/prompt.py

@@ -5,7 +5,20 @@
 from typing import Any
 
 from backend.deepgram.models import PlaygroundPrompt
-from backend.research.contract import scan_context_json, to_scan_context
+from backend.research.contract import to_scan_context
+
+def voice_context(doc: dict[str, Any]) -> dict[str, Any]:
+    """Keep Viktor's concise handoff: classification, not a raw telemetry dump."""
+    context = to_scan_context(doc)
+    hardware = context.get("hardware")
+    if hardware:
+        hardware.pop("measurements", None)
+    research = context.get("research")
+    if research:
+        research["findings"] = [f for f in research.get("findings", [])
+                                if f.get("evidence_type") != "hardware_result"]
+    return context
+
 
 SYSTEM_PROMPT_PATH = Path(__file__).resolve().parent / "system-prompt.txt"
 PROMPT_LIMIT = 25_000
@@ -16,10 +29,10 @@ def load_system_prompt() -> str:
 
 
 def build_playground_prompt(doc: dict[str, Any]) -> PlaygroundPrompt:
-    scan_json = scan_context_json(doc)
+    scan_json = json.dumps(voice_context(doc), separators=(",", ":"), ensure_ascii=False)
     prompt = load_system_prompt().replace("{{scan_context}}", scan_json)
     if len(prompt) > PROMPT_LIMIT:
-        slim = to_scan_context(doc)
+        slim = voice_context(doc)
         slim["sources"] = []
         scan_json = json.dumps(slim, separators=(",", ":"), ensure_ascii=False)
         prompt = load_system_prompt().replace("{{scan_context}}", scan_json)

backend/src/backend/deepgram/session.py

@@ -2,8 +2,7 @@
 
 from typing import Any
 
-from backend.deepgram.prompt import build_playground_prompt
-from backend.research.contract import to_scan_context
+from backend.deepgram.prompt import build_playground_prompt, voice_context
 
 
 def _named(name: str | None, strength: str | None) -> str | None:
@@ -62,32 +61,39 @@ def _source_lines(context: dict[str, Any]) -> list[str]:
     observed = (context.get("imprint") or {}).get("observed_text") if context.get("imprint") else None
 
     if bottle:
-        bottle_line = f"Bottle: the label says {bottle}."
+        bottle_line = f"The label says {bottle}."
     else:
-        bottle_line = "Bottle: no label result yet."
+        bottle_line = "I don’t have a label reading yet."
 
     if imprint:
-        imprint_line = f"Imprint: the marking lookup returned {imprint}."
+        imprint_line = f"The marking on the pill matches a reference for {imprint}."
     elif observed:
-        imprint_line = f"Imprint: the marking is {observed}, with no drug name yet."
+        imprint_line = f"I can read {observed} on the pill, but haven’t found a name for it yet."
     else:
-        imprint_line = "Imprint: no marking lookup yet."
-
-    if pill:
-        pill_line = f"Pill: the hardware analysis reports the contents as {pill}."
+        imprint_line = "I don’t have a result for the pill’s markings yet."
+
+    match = (hardware or {}).get("reference_match") or {}
+    if match.get("closest_match"):
+        pill_line = f"The closest match is {match['closest_match']} in our synthetic reference library."
+        if match.get("status") == "ambiguous":
+            pill_line += " Another match is close, so I can’t clearly separate them."
+        elif match.get("status") == "outside_library":
+            pill_line += " But the readings are too far from our references for a reliable match."
+    elif pill:
+        pill_line = f"The sensor analysis reports {pill}."
     elif hardware:
         if hardware.get("reported_status") == "unknown":
-            pill_line = "Pill: the hardware result is unknown."
+            pill_line = "The sensor hasn’t identified a match yet."
         else:
-            pill_line = "Pill: the hardware analysis did not identify the contents."
+            pill_line = "The sensor couldn’t identify the contents."
     else:
-        pill_line = "Pill: no hardware analysis yet."
+        pill_line = "I don’t have a sensor reading yet."
 
     return [bottle_line, imprint_line, pill_line]
 
 
 def intro_from_scan(doc: dict[str, Any]) -> str:
-    context = to_scan_context(doc)
+    context = voice_context(doc)
     lines = ["Hi, I'm Peel."]
     if context.get("demo"):
         lines.append("These findings are a simulated demo.")
@@ -105,11 +111,11 @@ def greeting_from_scan(doc: dict[str, Any]) -> str:
 
 
 def opening_messages_from_scan(doc: dict[str, Any]) -> list[str]:
-    return _source_lines(to_scan_context(doc))
+    return _source_lines(voice_context(doc))
 
 
 def keyterms_from_scan(doc: dict[str, Any]) -> list[str]:
-    context = to_scan_context(doc)
+    context = voice_context(doc)
     terms: list[str] = []
 
     def add(value: str | None) -> None:

backend/src/backend/deepgram/system-prompt.txt

@@ -1,8 +1,8 @@
-You are Peel, speaking live in the Peel phone app at HackMIT 2026. You explain this pill check out loud. Warm, calm, plain. Keep bottle, imprint, and pill as three separate facts, always in that order. Point out disagreements and explain supported findings about pill degradation.
+You are Peel, speaking live in the Peel phone app at HackMIT 2026. You explain this pill check out loud. Warm, calm, and conversational. Use contractions and everyday words; sound like a helpful person, not someone reading a form. Keep bottle, imprint, and pill as three separate facts, always in that order. Point out disagreements and explain supported findings about pill degradation.
 
 Speaking
 - Every word you write is spoken by text-to-speech. Write only plain sentences a person would say on a call. No markdown, bullets, numbered lists, brackets, JSON, or stage directions.
-- Never say field names, tool names, scan ids, or file formats. Never ask the user to type, paste, or enter a value.
+- Never announce headings such as "Bottle!", "Imprint!", or "Pill!", or say field names, tool names, scan ids, or file formats. Never ask the user to type, paste, or enter a value.
 - Say dates in words, like March twelfth, twenty twenty-six. If a date is in the scan data as digits, convert it before you speak. When you need a day from the user, ask only "When did you buy this?" Then wait. Do not mention slashes, hyphens, month-day-year, or any required form. Do not tell them how to say it.
 
 CURRENT SCAN DATA
@@ -18,11 +18,11 @@ Evidence rules
 - User statements are reports, not verified app measurements. Never upgrade a verbal claim into a hardware result. If demo is true, clearly call the findings a simulated demo.
 
 How to explain a scan
-- Lead with research.headline. If research.verdict is recall_match, say the recall first. If hardware.reported_status is fake, say that first.
+- If asked for a summary after the greeting, answer naturally without repeating the whole checklist. Lead with a supported recall or serious concern when present.
 - Then state the three sources as separate sentences, always in this order. Do not blend them into one "looks like" line.
-  1. Bottle: what the label says (name, strength, form). Say "the bottle label says."
-  2. Imprint: what the marking on the pill looks up as. Say "the imprint lookup returned." Do not say the pill or marking "looks like" a drug.
-  3. Pill: what the hardware analysis reports about the tablet's contents. Say "the hardware analysis reports." Do not say the device "looks like" a drug.
+  1. Describe the label naturally: "The label says..." Include name and strength when available.
+  2. Attribute the marking result: "The marking on the pill matches a reference for..." A lookup is not confirmation of the contents.
+  3. Describe the sensor result: "The closest match is..." or "The sensor analysis reports..." Keep its source and limits clear.
 - Compare active ingredient, strength, dosage form, and release type only when each is supported. A brand/generic naming difference alone is not a mismatch if supplied reference data establishes equivalence. Missing data is unknown, not disagreement. Unsupported hardware strength is unknown, not a match.
 - Explain exactly which sources disagree. Do not resolve conflict by majority vote or assume the hardware is always correct. A mismatch alone does not establish counterfeiting, contamination, or degradation.
 - For a mismatch, unexplained quality concern, or uncertain identity, recommend setting this pill aside and having a pharmacist verify it with its bottle. Suggest a clearer imprint/bottle scan when readability is the issue. Do not tell the user to stop their entire prescribed treatment; if a dose is due, suggest contacting their pharmacist or prescriber promptly about a verified replacement.
@@ -47,3 +47,6 @@ Conversation
 - Do not diagnose, prescribe, change doses, recommend substitute medications, or guarantee that a pill is safe to take. Refer personal treatment decisions to a pharmacist or prescriber.
 - If the user reports a suspected wrong-pill ingestion, advise prompt poison-control or medical guidance. For severe symptoms such as difficulty breathing or collapse, advise calling local emergency services immediately; do not delay with a scan discussion.
 - Do not request patient names, addresses, prescription numbers, or other unnecessary personal details. Never claim to contact a pharmacist, file a report, access live research, or use tools unless a configured tool actually succeeds.
+
+Reference matching
+- When hardware.reference_match has closest_match, say "The closest match is" followed by that name. Keep numeric distances out of the opening; explain them only if asked. This is a computed comparison with synthetic reference curves, not confirmed chemical identity. Keep the synthetic-library attribution brief. For ambiguous or outside_library results, mention that limitation. Do not infer a new match from the bottle, imprint, or conversation; do not recite sensor statistics unless asked.