Paste song lyrics. Walk the world they evoke.
Resonance turns text you bring — a song, a short passage — into an explorable first-person 3D world. A contemplative walking simulator where fragments of the text are built into the architecture itself. No goals, no score. The reward is walking through a place that embodies the work.
Asking a language model to emit raw geometry produces garbage you cannot render, cannot validate, and cannot reproduce. Resonance inverts the problem.
The engine ships a fixed vocabulary — a closed set of primitives, layouts, skies, particle types, typography styles and audio beds. The model never invents a mesh. It reads the text, extracts its concrete images and emotional arc, and returns a World Spec: a JSON document that selects and parameterizes that vocabulary.
That single constraint buys everything else:
| Because the model only composes… | …you get |
|---|---|
| Output is a closed-enum JSON document | It validates against a zod schema, so bad output is caught, not rendered |
| Validation failures carry structured errors | The generator retries with the error as feedback, up to 3 times |
| A minimal valid spec always exists | The app never shows a blank screen, even when everything upstream fails |
| Layout is derived from a seeded PRNG | The same spec always builds the same world, on any machine |
| The vocabulary has a known cost | The frame budget is enforced by the schema, not hoped for at runtime |
Two different songs must not produce the same world. That is the bar the prompt is written against.
npm install
npm run devOpen the dev server, paste some text, walk. No API key required — the default generator runs entirely offline and costs nothing.
Requires Node ≥ 20.19 (developed and tested on v24).
flowchart LR
A["Paste text"] --> B["Normalize"]
B --> C{"WorldGenerator"}
C -->|heuristic| D["World Spec"]
C -->|mock| D
C -->|anthropic| D
D --> E["zod validation"]
E -->|invalid| C
E -->|3 failures| F["Minimal fallback spec"]
E -->|valid| G["WorldRenderer"]
F --> G
G --> H["Seeded procedural scene"]
H --> I["Walk it"]
Everything downstream of WorldGenerator is provider-agnostic — the renderer and the telemetry only
ever see a GenerateResult. Swap the generator with the VITE_GENERATOR environment variable:
VITE_GENERATOR |
Generator | What it does | Cost |
|---|---|---|---|
| (unset) | heuristic |
Default. Offline lexical analysis of your text — detects motifs, mood and themes, then assembles a world from recipes. Genuinely content-derived: different texts give different worlds. | Free |
mock |
mock:in-the-end |
A hand-authored four-zone demo world, versioned in the repo. Useful as a fixed target when working on the renderer. | Free |
anthropic |
anthropic:claude-* |
Real Claude, behind a server-side proxy. The richest interpretation of the text. | Metered |
The spec is the whole contract between "understanding the text" and "building a space". It lives in
src/schema/worldSpec.ts, with the closed catalogs in
src/schema/enums.ts.
Every catalog field is a closed enum — sky is one of 8 values, primitive one of 9, mood one of 10.
Anything outside the list is rejected. Cross-field rules are enforced too: zone ids must be unique,
connectsTo must reference a real zone, and every zone.mood must appear in meta.moodArc.
Spec v2 adds the layer that makes worlds recognizable rather than merely atmospheric: imagery
and entities extracted from the text with a sourceRef back to the line they came from, a
dominantMotif per zone, and a motifState (whole → breaking → gone → transcended) so the same
recurring image transforms across the walk and becomes the spine of the story.
The prompt is a versioned artifact (worldspec.v1.md,
worldspec.v2.md). Changing it bumps the version, which is
part of the cache key.
The target is 60fps on a modest integrated GPU (Iris Xe / Vega 8), and the budget is enforced in the schema rather than defended at runtime: at most 6 zones, 12 props per zone, 40 instances per prop.
- Instancing — any prop with
count >= 4collapses into a singleInstancedMeshdraw call. - Fake volumetrics — light shafts are crossed alpha-gradient planes (~0.3–0.8ms each), not a
GodRayspass. Roughly 90% of the effect at 10% of the cost. - No dynamic shadows. Form comes from directional + hemisphere light and procedural contact shadows.
- Selective bloom for free — a 0.9 luminance threshold means only genuinely emissive materials (nearby typography, the hero sun) bloom at all.
- A five-tier degradation ladder driven by an FPS sampler with hysteresis, dropping SMAA → particles → shafts → bloom → the whole composer. The floor tier still walks.
- Two things never degrade: the legibility of the text, and collision.
Determinism comes from src/engine/rng.ts — a seeded mulberry32 PRNG, with
seeds derived hierarchically from zone.id (a string, not an index, so reordering zones doesn't
reshuffle the world). Math.random is banned inside the engine.
Treated as part of the MVP, not as polish. All settings are reachable before you enter the world and from inside it, and they persist.
- Field of view 70–100°, live.
- Motion vignette (on by default) — darkens the edges only while moving, to fight retinal slip.
- Teleport movement as an alternative to smooth locomotion, for motion sickness.
- Reduce particles — full / reduced / none.
- Reduce motion — reads
prefers-reduced-motionat startup and preselects a comfort profile. - Color legibility mode — no affordance is ever encoded by hue alone; text is corrected to high contrast if a fragment's text/background pair falls below 4.5:1.
- Head bob off by default. Crosshair as a fixed anchor. Volume and mute always visible.
- A reading menu (Tab) gives the complete text of the work, grouped by zone, at adjustable size — full access to the piece without moving through 3D at all.
| Key | Action |
|---|---|
| W A S D | Walk |
| Shift | Move slowly |
| Mouse | Look (click to capture) |
| Tab | Reading menu — the full text |
| M | Mute |
| ~ | Debug panel |
| Esc | Release the cursor |
cp .env.example .env
# set ANTHROPIC_API_KEY, and VITE_GENERATOR=anthropic
npm run devThe API key is read only on the server — a Vite dev-server middleware that proxies /api/generate.
It never reaches the browser and never enters the client bundle. If the key is missing, the app falls
back to the offline generator instead of failing.
Cost, latency, token counts and cache hits are tracked per generation and visible in the debug panel
(~). Model pricing lives in one dated, verifiable table in
src/generation/pricing.ts. Responses are cached on
hash(normalizedText + promptVersion + modelId), so re-entering the same world is free.
server/ Dev-server proxy: /api/generate → Claude. The API key lives here and only here.
src/
ingest/ Pasted text → normalized text
generation/ WorldGenerator interface · heuristic/ · mock/ · anthropic/ · prompts/ · cache · pricing
schema/ worldSpec.ts (zod) · enums.ts (closed catalogs) · fallback
engine/ rng · color · layout · zones · primitives/ · typography/ · atmosphere/ · motifs/
WorldRenderer · PerfSampler
player/ First-person controller + AABB collision
audio/ Mood-driven ambient beds (Web Audio)
ui/ Screens, overlays, reading menu, debug panel, accessibility, stores
telemetry/ Cost & latency + render metrics
| Command | What it does |
|---|---|
npm run dev |
Dev server, including the /api/generate proxy |
npm run build |
Typecheck (app + server) then production build |
npm test |
Run the test suite |
npm run test:watch |
Tests in watch mode |
npm run coverage |
Tests with coverage |
npm run typecheck |
TypeScript only |
npm run lint |
ESLint |
npm run format |
Prettier |
128 tests across 28 files, run with Vitest.
The suite deliberately tests computation, not the canvas: schema validation and its cross-field rules, the fallback spec, retry and JSON-extraction behavior, layout math, transform determinism, the primitive registry's coverage of every enum member, color math, cache keys, and cost estimation. WebGL is never rendered in CI — that's verified by hand in a real browser.
Resonance only processes text you provide. It does not download, scrape, or redistribute copyrighted lyrics.
Inside the experience you see short fragments — capped at 90 characters and 12 words, derived from your own input at runtime — never a full work. When your text is too short to draw from, the app falls back to original phrases written for this project. The repository itself contains no copyrighted lyrics, by design and by test.
Phase 1 (MVP) is complete. Paste a text, walk the world it evokes, with cost and latency telemetry.
Known limitations, stated plainly:
- The real Claude path is tested against simulated responses. The parse / retry / fallback logic is covered; an end-to-end call with a live key is not.
- Visual verification is manual. The headless preview sandbox does not size the R3F canvas, so the 3D render is checked in a real browser. The HTML UI, state machine and telemetry are verified.
- Zones are flat. Verticality in the MVP comes from the fog ceiling, not from terrain.
| Phase | Scope |
|---|---|
| 2 | Short books — PDF ingestion, chunking, chapter → zone analysis |
| 3 | Award-grade polish — layered generative audio, procedural crack shader, per-letter scattered_letters assembly, real reflective water, cinematic transitions |
| 4 | Optional third person, saving and sharing worlds, unit economics |
| 5 | LLM benchmarking — several models behind WorldGenerator, compared on cost, latency and quality |
Design decisions and the trade-offs behind them are documented in DESIGN.md and DESIGN-v2.md; the phase plan lives in PLAN.md.
Issues and pull requests are welcome. Before opening a PR:
npm run typecheck && npm test && npm run lintTwo conventions worth knowing:
- Never widen the vocabulary casually. Adding a
primitive,skyormoodmeans touching the enum, the schema, the renderer registry, and the prompt — the registry test will fail until every enum member resolves to a real component. That friction is intentional. Math.randomis banned insrc/engine/. Determinism is a feature; use the seeded RNG.
MIT © nsalloums
{ "version": 2, "meta": { "title": "…", "themes": ["time", "letting_go"], "moodArc": ["determination", "struggle", "catharsis"], "palette": { "primary": "#2b3a67", "secondary": "#4a5d8a", "accent": "#d9a23f", "fog": "#1a1f33", }, "seed": 1337, "imagery": [{ "label": "a clock on the wall", "motif": "machine_of_time", "sourceRef": "…" }], "entities": [{ "name": "the self", "kind": "self", "motif": "monument" }], }, "environment": { "sky": "twilight_overcast", "ground": "fractured_plane", "ambientAudio": "deep_hum", "fog": {}, }, "zones": [ { "id": "zone_start", "name": "The Starting Line", "mood": "determination", "dominantMotif": "monument", "motifState": "whole", "props": [ { "primitive": "monolith", "variant": "tower", "count": 12, "layout": "path", "role": "landmark", }, ], "particles": { "type": "dust", "density": "low" }, "lyricFragments": [{ "text": "…", "style": "floating_glow", "trigger": "proximity" }], "connectsTo": "zone_struggle", }, ], }