Sentinel

An offline vision-language assistant on a Qualcomm-powered Rubik Pi 3.

Sentinel combines a local vision-language model, Adreno GPU inference, offline speech recognition, and speech synthesis. It answers questions about an image or a captured frame without a cloud inference service. Model loading runs in a persistent Linux service; the Python application handles inputs, timing, and answers.

What has actually been demonstrated

Component	Verified state
Vision-language inference	SmolVLM-500M Q8_0 with its FP16 visual projector on the actual Pi
GPU acceleration	OpenCL vision encoder and 33/33 language-model layers offloaded to Adreno 643
Offline operation	Image question, local ASR, and WAV synthesis completed with no external network route
Voice input/output	whisper.cpp English ASR and eSpeak NG WAV synthesis; physical audio routing is pending
Capture pipeline	Continuous fresh-frame capture and repeated inference with a synthetic GStreamer source
Deployment	Loopback-only systemd model service enabled on the Pi
NPU	QNN installed; DSP calculator diagnostic fails. No validated NPU inference claim
Physical CSI camera	Software installed; attachment and sensor validation deferred

On a fixed synthetic image, three fresh requests measured 3.45 s median model latency on OpenCL, versus 13.86 s on CPU. Prompt-cache reuse was disabled and checked. This is a small smoke benchmark, not a general accuracy or production-latency result. Method and evidence.

Architecture

flowchart LR
    WAV[WAV or microphone] --> ASR[Local whisper.cpp ASR]
    TEXT[Typed question] --> Q[Question]
    ASR --> Q
    IMAGE[Image or camera frame] --> PRE[CPU resize and JPEG]
    Q --> CLI[Python pipeline]
    PRE --> CLI
    CLI -->|Loopback HTTP| VLM[Persistent llama.cpp service]
    VLM --> GPU[OpenCL vision encoder and language model]
    GPU --> ANSWER[Text answer]
    ANSWER --> TTS[Local eSpeak NG]
    TTS --> OUT[WAV or speaker]
    CLI --> LOG[Stage timings and JSONL records]

Loading

The verified acceleration path is the GPU. QNN presence alone does not establish that a VLM can execute on the NPU.

Run the deployed project

From the Mac:

ssh rubik
cd ~/sentinel
.venv/bin/sentinel doctor --verify-hashes
.venv/bin/sentinel demo

demo creates a synthetic image and spoken question, transcribes the question, runs the VLM, and writes the spoken answer. It needs no camera, microphone, or speaker. It reports the folder containing its inputs, output WAV, and timing record.

Ask your own image question or use the interactive loop:

.venv/bin/sentinel fixtures
.venv/bin/sentinel ask --image data/synthetic/red-rectangle.jpg --question 'What color is the rectangle?'
.venv/bin/sentinel interact --image data/synthetic/red-rectangle.jpg
.venv/bin/sentinel eval

Run fixtures first if the synthetic dataset has not been generated. Each interactive question is independently grounded in the selected image; dialogue history is not retained. Add --speak for device playback or --tts-wav answer.wav to save audio. Physical playback and recording need working Pi audio hardware and routing.

Install on a compatible Pi

Requires Python 3.11+, an Ubuntu image with the appropriate Qualcomm repositories, and Internet access during provisioning:

git clone https://github.com/SiddharthMundra/sentinel.git
cd sentinel
bash scripts/bootstrap-pi.sh opencl
bash scripts/install-service.sh opencl
.venv/bin/sentinel doctor --verify-hashes
.venv/bin/sentinel demo

Bootstrap installs dependencies, fetches fixed native source revisions, builds the runtimes, downloads fixed model revisions, and checks SHA256 hashes. Runtime uses local paths and makes no model downloads. Hardware/runtime compatibility must be rechecked on another board or OS image. Full setup.

Repository map

sentinel/             Python application: CLI, capture, voice, model client, evaluation
config/               Shared runtime settings and local artifact paths
scripts/              Provisioning, service installation, deployment, verification
models/manifest.json  Model revisions and checksums; weights are not in Git
third_party/          Generated local native checkouts and builds; not in Git
data/synthetic/       Generated diagnostic images and labels; not in Git
examples/             Evaluation schema example
benchmarks/           Curated reference measurements and methodology
reports/              Local run output; only curated documentation is published
tests/                Software tests without a Pi or model download
.github/workflows/    Continuous integration on Python 3.11 and 3.12

The Mac checkout is the authoring copy. The Pi checkout, /home/ubuntu/sentinel in the current deployment, executes the project and holds the large artifacts. GitHub holds authored source and curated evidence, not weights or private run data. Directory and execution guide.

Documentation

• Architecture and design decisions
• Installation and reproducibility
• Commands, service operations, and troubleshooting
• Benchmark methodology and reference results
• Project report
• Presentation and portfolio walkthrough
• Compatibility evidence
• Testing
• Remaining work and limits
• Third-party components

Development

python3 -m venv .venv
.venv/bin/pip install -e .
bash scripts/verify.sh

Software tests do not need model weights, a camera, or a GPU. Integration tests run on the Pi. This repository is a measured prototype: natural-scene quality, microphone/speaker behavior, power consumption, and a working QNN model path remain open validation tasks.