AR Visual Feedback Platform

Real-time visual feedback inside a Microsoft HoloLens 2, driven by any signal you publish over Lab Streaming Layer (LSL).

The headset receives rendered images, not numbers. A compute host resolves your LSL stream, renders a visualization, encodes each frame as JPEG, and streams it to the HoloLens, which decodes and displays it on a head-stabilized panel in the wearer's view. Because the headset only paints pixels, one unmodified headset app can display anything — adapting the platform to your experiment means changing just two things: the signal you publish, and the visualization that draws it.

The platform is signal-agnostic: EMG, motion-capture-derived metrics, metabolic cost, force, joint angles, a symmetry index — it treats them all identically. Newest sample in, rendered image out.

---

The three devices, one direction of flow

  DEVICE A                    DEVICE B  (this repo)                     DEVICE C
  ────────                    ─────────────────────                    ────────
  YOUR SIGNAL         LSL     COMPUTE HOST (Windows PC)     WebSocket   HoloLens 2
  publisher      ──────────►  lsl_input → visualization →  ──────────► decode +
  (you write)                 network  (JPEG frames)                    display
                              ~30 Hz render + encode                    (prebuilt app)

Device	What it is	Who provides it
A — Acquisition	A program that publishes a stream of floats over LSL	You. A fill-in template is included (compute_host/examples/publisher_stub.py).
B — Compute host	Windows PC that renders the visualization and broadcasts frames	This repo (compute_host/).
C — Headset	HoloLens 2 running a prebuilt UWP app that displays the frames	This repo (headset/), sideloaded. No rebuild needed.

You provide: a Windows PC, a HoloLens 2, and one program that publishes your signal over LSL. Everything else is here.

---

Repository layout

ar-visual-feedback-platform/
├── compute_host/            Device B — the Windows render/stream host (the core)
│   ├── config.py            all settings: stream name, channels, port, render size
│   ├── main.py              entry point (LSL → render → encode → WebSocket)
│   ├── lsl_input.py         channel-agnostic LSL input
│   ├── visualization.py     the visualization you replace (default: a bar)
│   ├── network.py           WebSocket frame server
│   ├── demo_publisher.py    synthetic signal, for testing with no hardware
│   ├── requirements.txt
│   ├── README.md            detailed compute-host guide
│   └── examples/
│       ├── publisher_stub.py                 template for your own signal
│       └── visualization_target_example.py   fuller worked visualization
├── headset/                 Device C — the HoloLens 2
│   ├── app/                 the prebuilt .msix + .cer  (sideload this)
│   ├── server_config.txt.example   points the headset at your compute host
│   └── README.md            sideload + configuration steps
└── scripts/
    └── benchmark_client.py  dev tool: stands in for the headset and measures throughput

Note on the headset package: headset/app/ ships the prebuilt HoloLens 2 app (AR Visual Feedback_1.0.0.0_ARM64_Master.msix + its signing .cer), ready to sideload. The compute host works and can be benchmarked entirely without a headset (see below). The Unity source for the app is not included in this repo. See headset/app/README.md and headset/README.md.

---

Quick start — see it work with no hardware

Runs the entire compute-host pipeline end to end with a synthetic signal. No HoloLens or acquisition hardware required.

cd compute_host
python -m venv venv && venv\Scripts\activate          # Windows
pip install -r requirements.txt

Then, in two terminals (from compute_host/):

python demo_publisher.py     # terminal 1: publishes a synthetic sine on LSL
python main.py               # terminal 2: resolves it, renders, and streams

A window opens showing a bar oscillating with the signal. That confirms the LSL → render → encode → WebSocket path works. To measure actual throughput without a headset, run the benchmark client (from the repo root, with both processes above running):

python scripts/benchmark_client.py --seconds 10

To then display it in the headset, sideload the app and set server_config.txt — see headset/README.md.

---

Making it yours — two files

1. Your signal. Copy compute_host/examples/publisher_stub.py, implement compute_value() with your measurement, set CHANNEL_COUNT, and run it. Make sure its STREAM_NAME matches STREAM_NAME in compute_host/config.py. (If you already have a Vicon/EMG/metabolic pipeline speaking LSL, just match the name — no stub needed.)

2. Your visualization. Edit compute_host/visualization.py (or copy examples/visualization_target_example.py over it). Change only how the data is drawn; keep the capture/encode/stream block intact. source.poll() returns the newest sample as a list of floats, so multi-channel signals are sample[0], sample[1], …

   One hard constraint: the HoloLens's additive optical display cannot render black — it only adds light. Keep the background black (the default). Anything relying on dark-on-light will not survive the optics.

3. Point the headset at the compute host via server_config.txt (its IPv4 + port). This is a runtime file, not a rebuild. See headset/README.md.

Full details are in compute_host/README.md.

---

Verified performance

The full compute-host pipeline (LSL → render → JPEG encode → WebSocket) was run end to end and measured with scripts/benchmark_client.py, which connects as a stand-in for the headset. No HoloLens is required for this measurement.

Metric	Result
Target frame rate	30.3 Hz (33 ms interval)
Achieved frame rate	29.1 Hz
Frames received (10 s)	291, 0 gaps
Mean / max frame size	11.8 KB / 12.2 KB
Mean inter-frame interval	34.4 ms
Jitter (stddev)	8.3 ms

Test machine: Windows 11 Home (26200), Python 3.13.7, PyQt5 5.15.11, pylsl 1.18.2 (liblsl 1.17.7), websockets 16.1. Publisher, compute host, and client all on 127.0.0.1. The render window was captured offscreen (QT_QPA_PLATFORM=offscreen); a machine with a real display should meet or exceed these numbers.

Tuning note (Windows). An early run measured only ~19.7 Hz. The cause was Qt's default CoarseTimer, which is bounded by the Windows timer granularity (~15 ms) and rounds the 33 ms render interval up to ~47 ms. Requesting a PreciseTimer in visualization.py restored the full ~30 Hz. This fix is already applied to visualization.py and the target-example visualization.

---

Troubleshooting

• "No LSL stream named 'AR_Feedback' found." No publisher is running, or its name doesn't match config.STREAM_NAME. Start demo_publisher.py to confirm the host works, then check your publisher's name. (The host prints this message and exits cleanly rather than hanging.)
• Headset can't connect. Check server_config.txt (correct IPv4 + port), that main.py is running, that both devices are on the same subnet, and that Windows Firewall allows Python on the WebSocket port. The host prints a "client connected" line when the headset attaches.
• pylsl import fails. pylsl needs the native liblsl. On Windows it ships with the pip wheel; on Linux/macOS it may need separate installation (https://github.com/labstreaminglayer/liblsl).

---

License

Released under the MIT License — see LICENSE. You are free to use, modify, and redistribute this platform, including for other labs and research groups.

Dependency note: the compute host uses PyQt5, which is licensed GPLv3 (dual-licensed GPL/commercial). This MIT license covers this repository's own code; redistributing a combined work that bundles PyQt5 carries PyQt5's GPL obligations. To keep a fully permissive stack, pyqtgraph can run on PySide6 (LGPL) instead — close to a drop-in backend swap.