dance2music

Music Intelligence Lab: dance video → sound. Pose from MediaPipe, cleaned velocity / acceleration / jerk, Laban Efforts, and simple synths and a pose-driven ii–V–I, compared on one local page. Started Sep 2026 as a tangent from the rope flow project (soniqflow-m5stack-python) for a dance collaboration in the lab.

Code only in git: clips go in videos/, everything generated lands in output/ (both ignored).

The self-contained Dance2Music Mac launcher targets Apple Silicon Mac, macOS 14+, Chrome required. It prepares approximately 4.9 GB of verified assets and opens the local tool in Chrome; later use works offline without Python or build tools. The website enables its installer link only after a signed, notarized release exists. See installation and troubleshooting and release packaging. The commands below remain available for development and the video-analysis workflow.

python dance_pose.py track videos/<clip>.mp4   # pose → output/<clip>/landmarks.npz
python motion.py                                 # kinematics + Efforts for every tracked clip
python harmony.py                                # how often each chord rule fires
python synths.py                                 # render every demo for every clip
python synths.py --serve                         # the page, http://127.0.0.1:5091/
python live.py                                   # live from a camera, http://127.0.0.1:5092/live/
python live.py --offline                         # prepare pose, hands, 3D, and native Magenta for offline use

The live server stays in its terminal; press Ctrl+C to stop it and release port 5092. Connect watch automatically scans for devices whose names start with M5StickC-IMU-, hiding unrelated Bluetooth devices. Select the M5StickC-IMU-<pair>-L/R watch in Chrome's chooser once per watch. Leave Automatically connect paired watches enabled to restore them when opening the page and reconnect after a dropped connection. The launcher enables Chrome's experimental Bluetooth permission persistence feature. Fully quit Chrome once before running the updated python3 live.py so the launch option takes effect. Use the same Chrome profile and page address. On macOS, the launcher opens Chrome directly; fully quitting also helps recover stalled Bluetooth discovery.

file	what
config.py	paths; ROPEFLOW_ROOT points at the rope engine (only dance_pose.py music and sonify.py need it)
dance_pose.py	pose tracking, first rule-based labels, the rope engine on dance
vocabulary.py	her six shapes, discovered by clustering
sonify.py	shape → chord, no clock (SWAM MIDI)
motion.py	cleaned kinematics and Laban Efforts
harmony.py	pose → ii–V–I with voice-led voicings
synths.py	the demos: synthesis, overlay videos, stems, manifest
web/index.html	the comparison page (mixer, chord timeline, traces)
web/methods.html, web/about.html	the math of everything, as a drawer and as a standalone page
web/live/	the live tool: camera picker, causal features, pose chords, Web Audio synth, mixer
live.py	serves web/ on localhost (the camera needs localhost or https)
IDEAS.md	ideas not built yet

Browser Record now downloads one ZIP with the video (or sound only), raw 33-point body pose, detailed hands when enabled, timestamps, and model/session metadata, plus every notification from up to four connected T-Watches. Use Arms/Legs pair assignments in Watches, then connect each watch once. The recording format and Zones v0.2 compatibility notes describe setup and timing. In the updated Python Zones app, Import ZIP… validates and converts the bundle for synchronized review and four-limb MIDI routing.

Choose 3D Models in Scene before Start to scan visible floor and tabletop regions and place up to three shapes. This optional feature targets Chrome on Apple Silicon and live cameras. It uses local WebGPU depth and segmentation models; no inference server, camera movement, or calibration is needed. Music starts after the scan. Show surfaces and Rescan help inspect placement. Stop to change the option. Drag a visible shape with the mouse to reposition it on a detected floor or tabletop; Esc cancels the drag. Rescan and a new session reset the layout. Objects use estimated depth occlusion. Stale depth hides objects; motion suppresses uncertain pixels. Placement and room scale are approximate. With Touch effects on, reach toward a shape with a hand or foot to make it glow, emit sparks and a ripple, and play a note from the current chord. Adjust the objects fader in Mix. Contact is estimated from pose and scene depth; visuals and sounds are included in recordings. The Scene dropdown also offers three Percussions sets: Drum kit (kick/snare/hi-hat), Hand drums (conga/bongo/djembe), and Metallic (cowbell/triangle/cymbal). These place recognizable draggable instruments that respond to any visible body part with their own synthesized sound and effects. Pull your hand or foot back and tap again for repeated hits, even while your body still overlaps the instrument. Resting contact plays once. Use None for no scene objects. All instrument visuals and sounds work offline. See 3D objects, offline setup, and validation.

Choose Floor Piano for a floor-mounted C4–C5 acoustic piano (eight white keys and five black keys). Black keys toward camera is on by default so you can step forward onto them while facing the camera. Toggle it off to put them on the far edge; your choice is saved and can change while playing. Clear the floor during scanning, then step onto keys with visible feet. Holding a key lets its sample decay; stepping off damps it. Lift your foot or jump and land on the same key to replay it. Two feet can play two notes, including repeated two-foot jumps. Drag the whole keyboard to move it; Rescan rebuilds its placement. Listening defaults to Piano only, with Piano + body music available while running. The objects fader controls piano volume without changing other mixer settings. Prepare samples with python live.py --offline --without-magenta; Floor Piano runs entirely in the browser and needs no native helper or C++ compiler. Samples are Salamander Grand Piano by Alexander Holm, licensed CC BY 3.0, transposed from five pinned roots. See Floor Piano details and validation.

Choose Google Magenta for local MRT2 music guided by three editable spatial prompts and the original movement-generated audio. Audio + notes also sends the synth's pitched note activity; Continue recent phrase captures a recent phrase for native continuation. Use the blend control or independent Original and Magenta levels. The Mac app bundles the native helper and prepares both model sizes; the developer CLI builds the helper during preparation. Venue sessions use local assets only. See Google Magenta setup, controls, and verification.

---

Dance → the rope's music engine

A tangent: the performance layer we built for the rope does not actually care that it is a rope. It wants three things — when (a stream of accents), what (a movement name at each accent) and how much (a slow energy) — and a dancer on video can give all three through MediaPipe pose.

python dance_pose.py all videos/<video>.mp4

Stages can be run separately (track, signals, music, overlay); everything lands in output/<video-stem>/.

How the dance becomes the three signals

Pose (track) — MediaPipe Pose per frame, keeping both the image landmarks (where the body is on screen) and the world landmarks (metres, hip-centred, camera-independent). 99 % of frames tracked on the Sep 21 studio clip.

When (signals) — limb speed about the body, in torsos per second, smoothed; its peaks are the accents, the way accelerometer peaks are the rope's whooshes. On that clip: 139 accents in 107 s, median 0.55 s apart. Dance is far less periodic than rope flow (period confidence ≈ 0), so the clock leans on its own averaging rather than on the dancer being metronomic.

What — seven rule-based states, chosen so they can be read and argued with rather than trusted blindly. Shape is tested before transition, because a folded body is something you can see while turning is something the body passes through:

label	the rule
reach	a wrist more than half a torso above the shoulders, body upright
low	hips dropped toward the ankles, or the torso folded over
turn	the shoulder line rotating (world yaw, so it works side-on)
travel	the hips crossing the floor
open	hands far apart or far from the body
close	hands in, still moving
still	none of the above

A state must be the majority of a half-second window to count, which took the clip from 73 label changes a minute to 45.

How much — the same 3-second envelope the rope uses, normalised to its own 90th percentile.

The music

music feeds those into the ordinary PerformEngine through a DanceContext that stands in for a rope take, with a dance chord map (one key, mostly triads: reach → G, open → C, close → Am, turn → Em, travel → F, low → Dm) and a drum pattern per state. Everything downstream is unchanged: the voice-led quartet, the top voice moving only when the chord does, the drum loop on the rope clock.

overlay draws the skeleton, the current label and the energy on the video and muxes the generated audio (the studio's own audio is dropped: two pieces of music at once tells you nothing).

What this is not

The labels are hand-written rules tuned on one dancer in one studio, not a learned model — the rope's kNN bank took a night of guided recording to earn its labels, and this had an afternoon. If the tangent is worth continuing, the honest next step is the same one the rope took: record takes, name the states, bank them, and let the classifier do what rules are standing in for.

Her vocabulary, discovered (vocabulary.py)

python vocabulary.py videos/VIDEO-*.mp4 --k 4 --sub 3

Windows of pose (1.6 s, hop 0.4 s: the length of a dance shape) from all three clips at once → standardise → PCA → UMAP → k-means. Silhouette picks 4, which is the honest answer but a coarse one: 63 % of it is "upright", and for music that is three different things. So the biggest cluster is split again on the arms alone, giving six shapes that can be looked at and named:

shape	what it is	share
0	standing, arms down — rest	31 %
1	both arms overhead — lift	18 %
2	wide lunge, one arm out	17 %
3	standing, one arm up — reach	16 %
4	hands and feet on the floor — ground	13 %
5	deep fold	6 %

The three clips share this vocabulary: overlap 0.77–0.88 (1 = identical use of shapes), while DTW over limb trajectories barely beats a shuffled baseline. Same shapes, same proportions, different order — which is what "the dances are all the same" turns out to mean.

sheet.png shows four examples of each shape, timelines.png when they happen and how much each clip uses them, clusters.json the per-window assignment.

Sonifying from the vocabulary (sonify.py)

python sonify.py videos/VIDEO-....mp4 --watch

No clock and no drums, because the dance has no pulse (period confidence 0.00). Instead:

• the shape sets the harmony — a chord and a register per shape (rest → Am warm, lift → C bright, lunge → F, reach → G, ground → Dm warm, fold → Em warm), voiced by the rope's own QuartetVoicer, so the top voice steps when the shape changes and holds while it lasts;
• the body works the sound, continuously — CC lanes at 15 Hz;
• a change is heard because the harmony moves, not because a bar came round.

For the Ableton side

<stem>__sonify.mid has four tracks, one per section, each with its own CC lanes — drop them on the SWAM tracks as they are:

track	MIDI ch (1-based)	plays
violin	8	top voice of the quartet
viola	9	alto
cello	1	tenor
bass	6	bass

CC	from the body	SWAM
11	hands high + speed	expression (loudness)
1	arms wide	vibrato depth
2	speed alone	bow pressure

The .wav next to it is a General MIDI sketch: CC 11 comes through, CC 1 and 2 do not — those only mean something once it is SWAM.

Motion, not pose: simple synths (motion.py, synths.py)

python motion.py            # cleaned kinematics + Laban Efforts
python synths.py            # 8 ideas × all clips → videos (~80 s)
python synths.py --serve    # the comparison page on :5091

One rule for every idea: moving makes the sound, the feature colours it (loudness always comes from a speed, so stillness is silence). One or two plain sources per idea, and the video draws exactly the joints and signals that drive it.

idea	what drives what
organ	each limb's speed → loudness of its own sine (C3 G3 E4 B4)
siren	each hand's speed → its pitch and loudness
wind	Weight → noise loudness + brightness; hips crossing → pan
plucks	Time: each limb's sudden acceleration plucks its string (Karplus–Strong)
fm	Flow (jerk relative to acceleration) → FM index; speed → loudness
space	Space (hand path straightness, 1 s) → pure fifth vs detuned drift
breath	Shape: body hull growing/shrinking → loudness; hull size → brightness
efforts	wind + plucks + fm + space, stems mutable in the page

Jitter: visibility gate → Hampel glitch removal → gap fill (gaps > 0.3 s get zero confidence) → zero-phase 5 Hz Butterworth → Savitzky–Golay derivatives → per-limb resting-speed floor subtracted in quadrature. Controls reach the audio through a one-pole glide (30–250 ms), so nothing zips. _motion/cleaning.png shows raw vs cleaned speed.

Honest caveats: Weight and Time correlate ~0.94 on these clips (a strong move is a fast-accelerating one), which is why Time is heard as discrete sudden events rather than a second continuous layer. Flow as a ratio still correlates ~0.7 with Time. Everything is 2D image motion in torso lengths; motion toward the camera is invisible. Normalisation is pooled across the three clips, so "strong" means the same thing in each.

ii–V–I from pose (harmony.py, demos twofive, twofive_alt)

Pose picks the chord, the four Efforts play it. First rule that matches wins, otherwise the chord holds; a reading must last 0.35 s to change it:

chord	rule
V (G13)	both hands below mid-thigh
ii (Dm9)	both hands past the same side of both feet
I (Cmaj9)	each hand and foot on its own side of the hips' midline (body frame)

Rootless voicings over a separate bass, chosen so every voice moves by at most a step: Dm9 F A C E → G13 F A B E → Cmaj9 E G B D (G7alt F A♭ B E♭ is a half step from both). Voices glide in pitch (0.25 s, bass 0.12 s). Layers: pad (Space detunes it), FM bass on the root (Flow = harshness), plucks on the limb's chord tone (Time), wind and wind-tuned-to-the-chord (Weight). twofive_alt swaps in G7alt while Weight is high (≥ 0.5 s, hysteresis). Rules fire ~8–12 chord changes/min on the Sep 21 clips, median chord ~5 s. Thresholds are constants at the top of harmony.py.

Every demo now ships its layers as stems; the page mixes them live (volume / mute / solo per layer, remembered by layer name).

Anthony instrument and MIDI

Anthony's ideas 0–2 now have a native Python prototype, separate from live.py:

.venv/Scripts/python.exe -m instrument --demo
.venv/Scripts/python.exe -m instrument --camera 0

Camera mode shows the body skeleton and uses body wrists for the ball even when palms face sideways. Detailed hand gestures select persistent chord qualities; both open palms held 1.5 seconds deliberately return to major. Confirmed fist locking shows ROOT LOCKED. Add --volume 0.5 to halve the louder default audio. Restart after updates; resize/maximize the window for the complete camera view. Native Python root control now starts in interval mode; T switches to the original midpoint ring. Center both wrists to arm, then hold a directional pair to transpose once. Both up/down moves an octave; fist root lock freezes pitch. See the approved interval map. Controls sit above the image. Hand detection follows web/main's far mode: alternating wrist crops and canned gesture labels confirmed for 0.35 seconds. The display shows the confirmed gesture and raw model result separately; joint shape never overrides None.

See the Python quickstart for setup, gestures, headless replay, local exports and checks; decisions records provisional choices for Anthony's review. The implemented instrument has now been translated into JavaScript on explicit request; true 3D remains deferred. Choose Sound & music → Harmony → Mode → Anthony instrument · 300 chords in the browser. Its root/ball controls and help appear below the mode selector; Ableton / MIDI is a disclosure in the same section. The separate Body turns experiment is linked from Movement & mapping → Movement Dictionary studies. For iPhone camera and sound, run python live.py --phone and follow the printed certificate setup URL. See browser and iPhone instructions. Desktop Chrome/Edge/Firefox can also send all chord notes or one root note to Ableton through the Ableton / MIDI section and a local virtual MIDI port. See the desktop MIDI setup. The validation record includes replay, model/device checks, measured latency and the remaining performer/listening review. For short feature instructions, see Feature Tutorials.

For the Anthony website in Safari on this Mac, run .venv/bin/python live.py. The Ableton section lists existing IAC destinations through the optional local MIDI bridge; install requirements/midi.txt, click Connect/refresh MIDI, then choose your bus and sound route. Safari setup.

For one held root note in Ableton on macOS:

.venv/bin/python -m pip install -r requirements/midi.txt
.venv/bin/python -m instrument --camera 0 --midi --mute

Select dance2music - Notes to Ableton as Ableton's MIDI input, channel 1. P pauses/resumes MIDI; exit releases the note. For full sustained chords use:

.venv/bin/python -m instrument --camera 0 --midi-chords --mute

Select dance2music - Chords to Ableton and a polyphonic synth. Chord notes hold while the ball is active; P pauses/resumes, tracking loss and exit release them. See the chord tutorial. See the Ableton setup tutorial for monitoring, the camera-free demo and existing IAC/loopMIDI cables.