Listening Observatory is an experimental system for continuous computational listening.
It connects to live audio streams, analyses their acoustic structure in real time, and translates what it hears into evolving visual forms and textual observations. Rather than treating sound as something to be displayed moment by moment, the system listens across different timescales — from immediate frequency changes to recurring resonances, motifs, and longer-term patterns.
Signal processing provides the observation layer: frequencies, harmonics, spectral energy, entropy, transients, and other measurable characteristics are extracted continuously. A memory layer then compares current activity with what has been heard before, allowing recurring structures to return with a persistent visual identity.
The live visualization is generated in the browser and evolves with the signal. Stable resonances may form persistent structures, transient events may disturb the visual field, and recurring acoustic motifs can reappear as familiar entities. Over time, the visualization becomes not only a representation of the current sound, but also a record of the system's history of listening.
A language model can add a slower narrative layer, turning structured observations into concise textual descriptions. It does not decide what happened in the sound; it gives language to events detected by the analytical system.
Live at https://listen.vrontier.org.
This repository also holds Listening Observatory Workbench (LOW), the research companion where members bring their own streams and decide how they are analysed: see workbench/. Both rest on the same parts: listen-ingest, the one service per machine that connects to stations, usable by any other program too, and listen-engine in listener/, which decodes and analyses for the Workbench. The two websites run side by side under their own subdomains; deploy/README.md shows how they fit together and which ports they use.
Ready-built programs for Linux, macOS and Windows are on the releases page, one release line per part:
| You want to run | Download | Also needs |
|---|---|---|
| the Workbench, for your own research | listen-workbench |
listen-engine and listen-ingest, ffmpeg |
| an engine for the Workbench | listen-engine |
listen-ingest, ffmpeg |
| an Observatory of your own, for streams you choose | listen-observatory |
ffmpeg, PHP; for a server nginx and PHP-FPM |
| one connection per station for programs of your own | listen-ingest |
nothing |
Each archive holds the programs, a README or install guide and, for Linux, systemd units. The Observatory's archive also holds the website and the server templates; its guide is deploy/observatory/INSTALL.md. Releases are made with scripts/release.sh from tags ingest/v…, engine/v…, workbench/v… and observatory/v….
The observatory listens to several sources at once. Each has its own page at /<slug>
with the visualization, the interpretation and (where the source allows it) the audio.
The site's landing page explains each source and how to read the visualization.
Credits and licences for each source are on its page. Audio is never stored in this repository.
Early development. See docs/ for the concept and the
live visualization event & data model.
docs/ concept, event model, visual study
ingest/ listen-ingest, a Go module of its own: the one connection to each station on a machine
listener/ Go module: the Observatory's listener (DSP, event detection, memory, narrator,
WebSocket API) and listen-engine
workbench/ Listening Observatory Workbench, a Go module of its own
site/ website (vanilla PHP front controller, served by NGINX + PHP-FPM):
landing page, contact form, one live page per source (p5.js, site/assets/js/)
deploy/ nginx, PHP-FPM and systemd files for staging and production
scripts/ local development and deployment helpers
- Go ≥ 1.23 (listener), ffmpeg (audio decoding), PHP 8 (site), jq (scripts)
- for a server: Linux with systemd, NGINX and PHP-FPM 8.3
The source catalogue is committed; credentials are not.
| File | Purpose |
|---|---|
site/_config/sources.json |
The streams (local, not committed; copy sources.example.json). Each entry is one source: its page (name, place, time zone, summary, credit, whether audio may be played, whether it is listed on the landing page) and its listener (port, input URL or file, extra flags such as -sample-rate 32000, and optionally mode: continuous or on-demand, overriding the node's default). Unlisted sources are reachable by URL only and marked noindex. rights records the basis for analysing and relaying each source's audio. |
.email (not committed) |
SMTP account for the contact form (EMAIL_ADDRESS, EMAIL_NAME, EMAIL_USER, EMAIL_PASSWORD, SMTP_SERVER, SMTP_PORT). scripts/mail-config.sh turns it into site/_config/mail.php (see mail.example.php); without it the form shows the address instead. |
.llm (not committed) |
Optional interpretation layer: LLM_API_KEY, NARRATOR_URL (any OpenAI-compatible endpoint), NARRATOR_MODEL. Without it the interpretation stays off. |
scripts/dev.sh # every source in sources.json, plus the site
scripts/dev.sh my-stream # or only some
# → http://127.0.0.1:8000/
cd listener && go test ./...
bin/listener -in capture.mp3 -dump > events.jsonl # offline, as fast as possibleEach source runs as its own listener instance (listen-listener@<slug>, systemd,
bound to 127.0.0.1). NGINX routes /<slug>/ws/… and the three API paths the page uses
(api/state/current, api/motifs, api/history/events?type=narrative.update) to it,
and everything else to the PHP front controller. The other API paths stay on 127.0.0.1. The listener is cross-compiled to a
static linux/amd64 binary, so the server needs no Go. scripts/gen-sources.sh
generates the per-source systemd settings and NGINX routes from sources.json.
- With sudo (staging):
scripts/deploy-staging.shbuilds, uploads and installs everything in one go. Host setup and file locations:deploy/README.md. - Without root (production):
scripts/deploy-ionos.shbuilds and stages everything in the deploy account's home, and an admin runs the idempotentdeploy/ionos/install.shthere as root (what it changes:deploy/ionos/INFOS_listen_vrontier.md). Site-only changes need no root:scripts/deploy-ionos.sh --site-only.
Both scripts are idempotent. A source removed from sources.json is stopped on
the next full deploy.
listen-mcp (in listener/cmd/listen-mcp) is a small MCP server that runs next to the
listeners and speaks MCP over stdio, typically through SSH:
{"mcpServers": {"listen": {"command": "ssh", "args": ["<host>", "bin/listen-mcp"]}}}Tools: service_status, list_sources, get_extract (records the next n seconds of
analysis data from a source: features, events, frames, spectrum, never audio),
get_motifs and get_narratives (full reference and client setup: docs/mcp.md). It reads sources.json and queries the listeners
on 127.0.0.1. Its live connections use ?role=tool, so they count as tools rather
than listeners, but they still wake an on-demand source. It needs no root: build it
with GOOS=linux GOARCH=amd64 go build ./cmd/listen-mcp and copy it to ~/bin.
The listener implements the MVP of the event model
(§27): signal.frame and signal.spectrum at 10 Hz, feature.state at 2 Hz,
event.transient, event.resonance and system.status, over GET /ws/live, with
the reconnect snapshot at GET /api/state/current.
Acoustic memory (phase 2, §12–§16): with -memory-dir the listener recognises
recurring structures. Resonance motifs are recurring partials, matched on
frequency, harmonic profile and context. Texture motifs are recurring
overall sound states, compared over 20-second windows. Each motif gets a
persistent id and a visual seed, and the listener emits motif.detected,
motif.returned (after -motif-return of absence, 3 min by default) and
memory.summary (1 h and 24 h). Memory lives in plain files, one directory per
source and input: motifs.json and daily history/*.jsonl, kept for
-history-days. It is served at /api/motifs, /api/motifs/{id},
/api/history/events and /api/history/features.
Interpretation (phase 3, §17): with -narrator-url and -narrator-model (or the
NARRATOR_URL / NARRATOR_MODEL environment variables) and an API key in
LLM_API_KEY, the listener phrases what it measured through any OpenAI-compatible
chat endpoint. It emits narrative.update at most every -narrate-every (90 s), and
only while someone has the source's page open and something has changed. Go builds the evidence
(leading resonances, returns, new motifs, transients, state changes, the last hour)
and the model only phrases it. Text naming a frequency or motif that isn't in the
evidence is retried once, then dropped. Modes: observational (default), minimal,
poetic. For local development the settings go in .llm (see Configuration);
scripts/dev.sh loads it.
Optional in-sync audio: with -audio the listener relays the audio it analyses
on /ws/audio, as fragmented MP4 (AAC and Opus). Every event carries
position_s on the same timeline, so the page shows each event when its sound
is heard. A source offers playback when "audio": true is set in its entry in
sources.json; it is off by default.
Dependencies are pinned and kept in the repository: github.com/coder/websocket
(pure Go, no transitive dependencies) is vendored in listener/vendor/, and p5.js is
vendored in site/assets/vendor/p5/ with checksums in VERSION. Audio captures
in samples/ are never committed.
The DSP system observes.
The memory layer recognizes.
The visualization embodies.
The LLM gives language to what was measured.
Code: MIT. Audio from third-party sources is not part of this repository and remains with its respective owners.