demo-voice-hack.mp4

Attune

A self-improving voice agent that runs on a Raspberry Pi, does real on-screen tasks when you talk to it, and gets measurably better at understanding you while it runs. No keyboard, no laptop, no cloud dependency for the common case. You power it on and talk.

This README is long because the interesting part is not "it does voice." The interesting part is how a voice agent improves in production through evaluation, and how much of a capable assistant you can actually fit on a 5 watt computer that costs about as much as a nice dinner.

The idea, in plain words

Most voice assistants are a microphone glued to someone else's data center. Every word you say leaves the room, gets transcribed in the cloud, reasoned about in the cloud, and answered from the cloud. That is fine until the network drops, the bill arrives, or you care about who hears what you say in your own home.

Attune flips that. The common things you ask for run entirely on the device: speech to text, intent understanding, the action itself, and speech back out. The hardware draws roughly 5 watts, about the same as a phone charger. Only the genuinely hard requests, the ones the on-device brain is not confident about, get escalated to a large state of the art model in the cloud. The device knows the difference and routes accordingly.

On top of that, it learns. When it gets something wrong and you correct it, it remembers, and it generalizes the correction to phrasings it has never heard. When you ask it to grade itself against an evaluation suite, it studies its own failures and rewrites how it understands language, and you watch the score climb in real time. That evaluation and improvement loop is the heart of the project, and it is built on Cekura, the evaluation platform.

Demo

A 60 second walkthrough on the Pi: voice to action, the multi step plan drawn live, the self-improvement score climbing from 53 to 87 percent, and a hard case escalating to the cloud.

Why a Raspberry Pi

The Pi is not a gimmick here. It is the honest version of a product thesis: an assistant that is private by default, cheap enough to put in every room, and useful even when the internet is not.

1. Privacy and autonomy. The default path never sends your speech off the device. That is a real property, not a setting you have to trust.
2. Cost and ubiquity. A capable assistant for the price of a board means it can live in places a cloud subscription cannot justify.
3. Accessibility. The framing is an agentic operating system for accessibility. Someone who cannot easily use a keyboard or read small text can run their computer by voice, on a quiet appliance that boots straight into the agent and nothing else.
4. Constraint breeds good engineering. A 5 watt budget forces you to actually decide what needs a large model and what does not, instead of sending everything to a 100 billion parameter model out of laziness.

The Pi runs Raspberry Pi OS Lite with no desktop. On boot it launches a single fullscreen surface (a kiosk) that shows the agent. There is no operating system to get lost in. The screen is the agent.

What it does

You speak. It acts on screen and answers out loud. The supported actions are a deliberately closed set, which is also the security boundary (more on that below):

• Web search and opening a browser view
• Finding and opening local files, including PDFs rendered on screen
• Searching a local mailbox and creating calendar events
• Accessibility settings, for example making all text larger or switching to a dark theme, applied live
• A multi step task that chains several of these together, for example find the flight in my email and add it to my calendar, with the plan drawn step by step on screen

And two things that make it more than a command parser:

• It corrects in the moment. Ask for the wrong thing, correct it in your own words, and a later rephrasing hits the learned target. It generalizes, it does not memorize the exact sentence.
• It improves against an evaluation suite. Ask it to evaluate itself, then to improve, and the score rises as it learns from its own mistakes.

How it works

The loop

Every turn flows through the same pipeline, which runs as a single asynchronous loop on the device:

   microphone
       |
       v
  [ capture ]      push to talk, ALSA arecord, 16 kHz mono
       |
       v
  [   STT   ]      whisper.cpp, tiny.en model, on device
       |
       v
  [ ROUTER  ]      decides who answers, cheapest first
       |               1. learned lexicon, exact match        (about 0 ms)
       |               2. deterministic grammar                (about 0 ms)
       |               3. learned lexicon, generalizing match  (about 0 ms)
       |               4. cloud escalation, NVIDIA Nemotron     (hard cases only)
       v
  [ EXECUTOR ]     allowlisted capabilities only, the security boundary
       |               browser, files, mail, calendar, settings, multi step task
       v
  [   TTS   ]      Piper, lessac medium voice, on device
       |
       v
    speaker   +   the kiosk screen updates live over a WebSocket

In parallel, two systems run alongside the loop:

  MEMORY        an append only event log, preference learning, a learned lexicon
  EVALUATION    Cekura evaluators and metrics, plus a local persona harness for the live demo

The router, and why it matters

The router is the piece that makes 5 watts viable. It is lazy in the good way: it tries the cheapest competent option first and only spends more when it has to.

1. Learned lexicon, exact. If the agent has learned a mapping for this exact phrasing, use it. Costs nothing.
2. Grammar. A deterministic parser handles the common commands directly. Search, open, find, settings, the multi step task. This is offline, instant, and reliable. The majority of real usage never leaves this tier.
3. Learned lexicon, generalizing. If the grammar is unsure, the agent checks what it has learned by signal words rather than exact strings. This is where generalization lives, described in the next section.
4. Cloud escalation. Only when the on-device tiers are not confident does the request go to NVIDIA Nemotron, a large reasoning model, which returns a structured decision. If the network is down or the endpoint is unavailable, a circuit breaker trips and the agent degrades gracefully back to local. It never hangs waiting on the cloud.

The executor is intentionally a closed allowlist. The agent cannot run arbitrary commands. It can only invoke the handlers that exist. That is the security model, and it is a deliberate one: a voice agent with shell access is a great demo right up until someone in the room says the wrong sentence, so this one simply cannot.

How it improves in production, which is the whole point

A voice agent in the real world fails in a specific way: it mishears or misunderstands phrasings its designers did not anticipate. The question that matters for production is not "is it good on day one," it is "does it get better as real people use it, and can you measure that." Attune has two distinct mechanisms, and both are real on-device adaptation. To be precise about the claim: this is online learning of how the agent understands language, not retraining model weights. No gradients run on the Pi.

Mechanism one: correction in the moment

When you correct the agent, it records the mapping from your original phrase to the right target, and it stores it in a way that resolves by the meaningful part of the phrase. So if you correct "open my document" to mean the tax file, a later "pull up my doc" resolves correctly. It learned the reference, not the sentence.

Mechanism two: learn from evaluation failures, and generalize

This is the mechanism the evaluation loop drives, and it is the one to watch.

An evaluation suite is a set of test cases. Each case is a phrasing a real user might say, paired with the action that should result. The agent runs every case through its router and scores how many it got right. On a cold start it passes a little over half. The failures are phrasings the grammar does not cover, for example casual speech like "fire up chrome."

When you ask it to improve, it does not memorize the failing sentences. It extracts the signal words from each failure and learns which words point to which action. "fire up chrome" teaches that the word fire, in this context, signals the open browser action. The proof that this generalizes rather than memorizes is built into the evaluation itself: there are held out probes, phrasings the agent is never taught directly, such as "fire up the browser please." After learning from a different sentence that shares the word fire, the held out probe now routes correctly. The score climbs from about 53 percent to about 87 percent, and a couple of genuinely ambiguous cases are left failing on purpose, because an honest improvement loop has limits and a suspicious 100 percent is a red flag, not a feature.

So the demo is not a number that goes up because it was told to. It is a number that goes up because the agent learned a transferable skill, verified against examples it never saw during learning.

How Cekura is used

Cekura is the evaluation and observability platform, and it is wired in three ways.

1. Evaluators. The test cases described above exist as real Cekura evaluators in the project. Each evaluator has instructions (what the simulated user asks for) and an expected outcome (what a correct agent does). These map one to one onto the agent's capabilities: web search, find a file, open the resume, the flight to calendar multi step task, the accessibility case, and a casual phrasing case that exercises generalization.

2. Metrics. Two LLM judge metrics score every conversation: Intent Correctly Identified (did the agent understand what was asked) and Task Completed Successfully (did it actually do the thing and confirm it). These are real language model judges defined in Cekura, not hand rolled string checks.

3. Observability. When observability is enabled on the device, every live turn is streamed to Cekura as a call log, where the metrics score it. The kiosk shows the live count of turns streamed, and the Cekura dashboard shows the scored conversations. This is the production story made concrete: the same agent, the same metrics, scoring real usage as it happens.

The relationship is the honest one for an edge device. Cekura is excellent at simulating callers against a hosted agent, but Attune is a local appliance, not a hosted endpoint. So Attune uses Cekura for evaluator and metric design and for observability of real turns, while the live on stage score runs through a local harness over the same persona set, so the demo stays fast and works even if the venue network does not.

How NVIDIA is used

Two ways, both for the genuinely hard cases that the on-device path should not pretend to handle.

1. Escalation. When the router is not confident, the request goes to NVIDIA Nemotron, a large reasoning model, which returns a structured intent. The on-device guess and the cloud answer are shown side by side on screen, so you can see the device choosing to defer.

2. Complex task planning. Ask for something genuinely involved, for example plan my morning around my flight, and the agent delegates the planning to Nemotron, which decomposes the request into ordered steps. Those steps animate on screen, labeled as cloud reasoning, because generating a novel multi step plan is exactly the kind of work the on-device path is not trying to do. This showcases the model doing reasoning, not just classification.

Nemotron is served over an OpenAI compatible endpoint, so the integration is a base URL and a model name. Note that "OpenAI compatible" refers to the request format only. There is no OpenAI involvement. The request goes to NVIDIA's endpoint and runs NVIDIA's model.

The models, and why each one

Stage	Model	Where it runs	Why this one
Speech to text	whisper.cpp, tiny.en	On device	The tiny English model transcribes a short clip in under 2 seconds on the Pi. The base model was accurate but took about 20 seconds, which is unusable for a live loop. This was measured, not assumed.
Intent, common case	deterministic grammar	On device	Most commands are regular enough that a parser handles them in about zero milliseconds, offline, with no model at all.
Intent, learned	generalizing lexicon	On device	The self-improvement output. Routes by learned signal words, transfers to unseen phrasings.
Intent, hard case	NVIDIA Nemotron	Cloud	A large reasoning model for the requests the device should not guess at, and for generating multi step plans.
Text to speech	Piper, lessac medium	On device	Natural sounding, fast, fully offline. The medium voice runs at 22 kHz, which the audio path is tuned for.

A small local language model (llama.cpp) is supported as an optional middle tier between the grammar and the cloud, for installations that want more on-device coverage before escalating.

Honest framing

Being clear about what is real matters more than sounding impressive.

• The common path is genuinely on device, sub second, and low power. Hard cases escalate to the cloud. This is real edge routing, not a claim that a 100 billion parameter model runs on a Pi.
• The self-improvement is real online adaptation: correction memory and lexicon evolution from failures. It is not weight training on the device.
• The agent reasons over local and controlled data, a preloaded mailbox, calendar, and files. It does not call your live email or live web services during the demo. The web search renders results in the kiosk, drawn by the agent, not by claiming to be a full live browser.
• The evaluation score is computed, not asserted, and it stops short of a perfect score on purpose.

Running it

On any machine, for development

The backends are abstracted, so on a laptop it uses mock and native backends and needs no models.

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python tools/make_demo_data.py
.venv/bin/python -m app.main

Open the UI at http://localhost:8000. On macOS, speech out uses the native voice and speech in is push to talk text entry, so you can exercise the full loop without any models. Watch the transcript, the plan, the score, the route badge, and the learned aliases update live.

Things to try: find the flight in my email and add it to my calendar. Then open my document, then no the tax pdf, then pull up my doc. Then evaluate yourself, then improve yourself.

On the Raspberry Pi, as an appliance

On a fresh Raspberry Pi OS Lite, with the repo on the device:

bash appliance/setup-pi.sh
sudo reboot

It comes up on the kiosk, running the agent. Real backends are whisper.cpp for speech to text, Piper for speech out, and the kiosk is a minimal Wayland compositor (cage) showing the UI in a lightweight browser (cog). Cloud and evaluation keys live in an environment file and are optional. Without them the agent runs fully offline on the local path.

Tests

.venv/bin/python -m pytest -q

The suite covers the grammar, the router and its tiers, the executor handlers, the memory and preference learning, the generalizing lexicon (including the held out generalization probes), the evaluation loop, the offline behavior, and the cloud and Cekura adapters in their offline safe forms.

Repository layout

app/
  main.py          the voice loop and kiosk server entry point
  agent.py         turn logic: control commands, correction, routing, cloud planning
  router.py        the confidence router and its tiers
  reason/          grammar, the multi step planner, correction parsing, local and cloud models
  executor/        the allowlisted capabilities, the security boundary
  memory/          event log, preference learning, the learned lexicon, the evolve loop
  eval/            the persona suite and the Cekura client
  ui/              the FastAPI server, the WebSocket, and the kiosk HTML
  hardware/        the LED status strip driver, with a mock for development
appliance/         the systemd units and the one shot Pi setup script
data/              the demo mailbox, files, and calendar
scripts/           backend test scripts for the Pi
tests/             the pytest suite

Feedback for the tooling

Sharing is caring, so here is the honest version, the good and the rough.

NVIDIA Nemotron

What it did well. The integration was about as easy as it gets. The endpoint is OpenAI compatible, so wiring it in was a base URL, a model name, and three environment variables, no SDK gymnastics. The public endpoint was reachable from my apartment and not just the venue network, which meant I could actually test the hard-case path at home instead of crossing my fingers on stage. For intent classification it returned clean structured JSON, and when I asked it to plan a multi step task it genuinely decomposed the request into sensible ordered steps. It reasons, which is exactly what I wanted for the cases my on-device path should not be guessing at.

What could be better. nemotron-3-super is a reasoning model, and it spends real tokens thinking before it answers, around 100 completion tokens of chain of thought even on a trivial classify-this-into-one-intent call. With max_tokens at 128 the response truncated with finish_reason length, and message.content came back null while the actual answer sat in a separate reasoning field, so my parser got an empty string and the turn silently failed. I had to raise max_tokens to 512 to let it finish thinking before it would emit the JSON. For short structured outputs that is a lot of overhead. A way to cap or disable the visible reasoning for simple classification, or a lighter non-reasoning variant for the easy cases, would make it noticeably faster and cheaper. Small nit, the content also came back with a leading double newline that I had to account for.

Cekura

What it did well. Setup was genuinely two commands and a browser sign in. Creating an agent, two LLM judge metrics, and six evaluators over the MCP took a few minutes total. The core mental model, that an evaluator is just a test case with instructions and an expected outcome, mapped one to one onto the persona suite I already had, so it clicked immediately. Writing a metric as plain English, "score TRUE if the agent identified the correct intent and acted on it", was the nicest part. No rubric DSL to learn.

What could be better, plus a few real bugs. The recommended /setup-mcp skill is wired for API key auth and 401s with "Check your CEKURA_API_KEY", but the path that actually works is OAuth through /mcp. The recommended skill and the recommended auth disagreed, which cost me a detour. Minting a project key with user_api_key_create over the OAuth MCP also returned 401, so I had to grab the key from the dashboard instead. The OAuth token then expired mid session and my writes started failing with that same "Check your CEKURA_API_KEY" 401, which is misleading, because it was an expired OAuth session and not a missing key. A clear "token expired, re-authenticate" message would have saved me from chasing the wrong thing.

The one that actually mattered for a self improvement loop. I could ingest call logs cleanly, a POST to /observability/v1/observe/ worked every time, but I could not find a documented way to read the scores back programmatically. The observe endpoint is POST only, a GET returns 405, retrieving a call log by id 404'd, and the API reference only documents the POST. Metric scoring is also async, 30 to 60 seconds. So closing the loop on the device, meaning ingest a turn, read its verdict, feed the failure back into learning, was not possible, and I ended up showing the live scores on the dashboard rather than inside the agent. If there were a documented GET for a call log by id, or a webhook that fires when metric evaluation finishes, an agent could react to its own Cekura scores without polling, and a true on-device score, learn, re-score loop would be possible. That is the one change that would unlock self improving agents built on Cekura.

Why it was worth building

This was built fast, for a hackathon, and it was a genuinely fun thing to build. There is something satisfying about making a cheap board on your desk listen to you, do the thing, and visibly get smarter while you watch. The constraints were the fun part. Five watts makes you think hard about what actually needs a giant model. A closed action set makes you design the boundary instead of hand waving about safety. A held out probe in the evaluation makes you prove the agent learned something real instead of trusting a number that went up.

The serious version of this, an assistant that is private, cheap, offline capable, and continuously improving from real evaluation, is a product thesis worth taking seriously. The fact that the prototype fits on a board you can hold in one hand is the part that makes you smile.