SimRealtimeMazurkaBenchmark

A benchmark for simulated real-time (online) audio-to-audio alignment, built on the Chopin Mazurka recordings. Each scenario pairs two recordings of the same piece; a system must align the query to the reference while only ever seeing the query up to the current frame. Alignments are scored against human beat annotations.

The alignment algorithms themselves live in the OnlineAlignment package; this repository is the benchmark harness around them — scenario generation, baselines, evaluation, and the figures.

Quickstart

Everything runs through benchmark.py. The notebooks are thin wrappers around the same commands, for stepping through the pipeline interactively.

conda env create -f environment.yml
conda activate sim-realtime-mazurka

# smoke test: 45 scenarios, no external dependencies
python benchmark.py run --benchmark train_small --systems DTW SOA --jobs 8

The four stages can also be run separately:

python benchmark.py prepare    --benchmark test                    # build scenario dirs
python benchmark.py features   --benchmark test                    # cache chroma features
python benchmark.py experiment --benchmark test --systems DTW SOA --jobs 8
python benchmark.py evaluate   --benchmark test                    # score against annotations

Useful flags: --config <file.json> to override system parameters, --jobs N to run scenarios in parallel, and --resume to keep results already on disk instead of clearing each system directory.

Dataset

benchmark.py expects the Mazurka data at the repository root:

Chopin_Mazurkas/
├── wav_22050_mono/<piece>/<recording>.wav      # 22.05 kHz mono
└── annotations_beat/<piece>/<recording>.beat   # beat annotations

The recordings and annotations come from the Mazurka Project and are not redistributed here. The data is not committed, and neither are the scenario, feature, experiment, or evaluation directories — all of them are regenerated by benchmark.py. Note in particular that scenario directories contain absolute symlinks into Chopin_Mazurkas/, so they must be regenerated after cloning or moving the repository.

Benchmarks

benchmark	pieces	scenarios	purpose
train_small	10 recordings of Op. 17 No. 4	45	quick smoke test
train	all recordings of Op. 17 No. 4	~1.9k	parameter tuning
test	Op. 24 No. 2, Op. 30 No. 2, Op. 68 No. 3	~3.8k	reported results

Each benchmark writes to its own set of directories (scenarios*/, experiments*/, eval*/); see BENCHMARK_CONFIGS at the top of benchmark.py.

Systems

--systems takes the keys below.

key	description	extra requirements
DTW	offline DTW reference (not real-time)	—
SOA	our online alignment	—
SOA_MONOTONIC	as above, predictions forced monotonic	—
OLTW_GLOBAL	our OLTW, unconstrained	—
OLTW_OURS	our OLTW, constrained window	—
OLTW	Dixon's OLTW via PerformanceMatcher.jar	Java, match/PerformanceMatcher.jar
MATCH	Dixon's MATCH via the Vamp plugin	sonic-annotator, match-vamp-plugin
MM_DIXON	MatchMaker OLTW, Dixon variant	matchmaker env
MM_ARZT	MatchMaker OLTW, Arzt variant	matchmaker env

A few keys are written differently in the paper (SOA_MONOTONIC as SOA-Mono, OLTW_GLOBAL as OLTW-Global, MM_DIXON as MM-Dixon); SYSTEM_DISPLAY_NAMES in utils/constants.py handles that at plot time, so directory names stay stable.

Default parameters are in configs/default_systems.json; see configs/README.md for the parameter reference and the tuning sweeps.

MatchMaker baselines

pymatchmaker pins numpy<2, so the MatchMaker systems run out-of-process in their own environment:

conda env create -f environment-matchmaker.yml
export MATCHMAKER_PYTHON=$(conda run -n matchmaker which python)   # optional
python benchmark.py experiment --benchmark test --systems MM_DIXON MM_ARZT

If MATCHMAKER_PYTHON is unset, utils/matchmaker_baseline.py looks for an env named matchmaker alongside the active one.

Notebooks

notebook	purpose
01_DataPrep.ipynb	scenario generation and feature caching (prepare, features)
02_RunExperiment.ipynb	running systems and parameter sweeps (experiment)
03_Evaluate.ipynb	error rate table and error-vs-tolerance figures (evaluate)
04_Analysis.ipynb	per-scenario alignment path and error plots

Notebooks 01-03 shell out to benchmark.py rather than reimplementing it, so the notebook and command-line paths cannot drift. Figures are written to figures/ and the aggregate error rate table to results/.

Layout

benchmark.py            pipeline entry point (prepare / features / experiment / evaluate / run)
eval_tools.py           alignment scoring against beat annotations
utils/
  experiments.py        ExperimentRunner: dispatches a system over the scenarios
  oltw.py               wrapper around PerformanceMatcher.jar
  matchmaker_baseline.py, matchmaker_worker.py
                        out-of-process MatchMaker baselines
  match_features.py     MATCH feature extraction via sonic-annotator
  constants.py          shared defaults and the system display-name map
cfg/                    pickled piece and pair lists per benchmark
configs/                system parameter JSON files
results/                aggregate error rate tables (committed)
figures/                generated figures (committed)