InstruCNN

Clip-level multi-label instrument recognition for music audio. The v1 system uses OpenMIC-2018, a log-mel frontend, and a compact CNN with temporal pooling. It is intentionally plain PyTorch: single GPU, reproducible configs, and a tiny overfit check before full training.

This codebase is written mainly by Codex. The main model architecture, the training process, the data downloading and cleaning, the fintune ideas are provided by me.

Setup

Use Python 3.11 through uv:

uv sync --extra dev

If your PyTorch install needs a CUDA-specific wheel, install the matching torch and torchaudio build for your system before training.

Expected Layout

configs/
data/
  raw/openmic/
  processed/
  splits/
scripts/
src/
outputs/

Dataset Setup

Place the official OpenMIC directory under data/raw/openmic/openmic-2018:

data/raw/openmic/openmic-2018/
  audio/
  class-map.json
  openmic-2018.npz
  partitions/train01.txt
  partitions/test01.txt

Then build the manifest:

uv run python scripts/prepare_openmic.py --openmic-root data/raw/openmic/openmic-2018 --out data/processed/openmic_manifest.csv

The manifest stores labels, label_mask, and soft_labels as JSON lists. soft_labels come from OpenMIC Y_true; labels are binarized with Y_true > 0.5 only where Y_mask is true. Unknown OpenMIC labels are masked out and never treated as negative targets.

If using a Hugging Face style parquet/CSV export with columns track_id, Y_true, Y_mask, and path, keep the official audio/ and partitions/ directories available and pass:

uv run python scripts/prepare_openmic.py \
  --openmic-root data/raw/openmic/openmic-2018 \
  --hf-metadata path/to/openmic_metadata.parquet \
  --out data/processed/openmic_manifest.csv

To download the seungheondoh/openmic-2018 Hugging Face metadata automatically and then build the manifest when local audio is available:

uv run python scripts/download_openmic_hf.py

This downloads metadata to data/raw/openmic/hf/openmic_hf_metadata.parquet. The HF dataset stores relative OGG paths, not the official audio bytes, so data/raw/openmic/openmic-2018/audio/ must still exist for the final manifest step.

Training

uv run python -m src.train --config configs/cnn_baseline.yaml

For a tiny overfit smoke test, set dataset.tiny_subset: 32 and use a short run in the config or via a temporary config copy.

Evaluation

uv run python -m src.eval --config configs/cnn_baseline.yaml --ckpt outputs/cnn_baseline/best.pt

Evaluation reports mAP, macro F1, micro F1, and per-class AP/F1. When threshold tuning is enabled, validation-tuned thresholds are written as JSON beside the checkpoint outputs.

Inference

Short clip or long audio:

uv run python -m src.infer --config configs/infer.yaml --audio path/to/file.wav

Long files are split into overlapping windows and exported as JSON:

[
  {
    "start": 0.0,
    "end": 5.0,
    "labels": {
      "piano": 0.91,
      "drums": 0.88
    }
  }
]

UI Demo

For the live local demo, start the server:

uv run python -m src.web_demo --host 127.0.0.1 --port 7860

Then open http://127.0.0.1:7860. Choose any local audio file, click Analyze Audio, and the backend will slice it into small overlapping clips and run the trained model.

The older static viewer is still available at ui/index.html; it accepts an audio file plus a prediction JSON exported by src.infer.

For long-audio visualization, force long mode and disable adjacent-window merging so the UI has a dense timeline:

uv run python -m src.infer \
  --config configs/infer.yaml \
  --audio path/to/file.wav \
  --output outputs/predictions.json

Set inference.mode: long and inference.merge_adjacent: false in configs/infer.yaml when you want every sliding window displayed separately.

Common Failure Cases

• Missing audio path in the OpenMIC manifest: preparation fails loudly.
• Label vector or mask length mismatch: dataset loading raises an error.
• All labels in a batch are unknown: masked BCE raises an error instead of silently training on invalid data.
• CUDA out of memory: reduce training.batch_size.

Roadmap

• Finish full OpenMIC training and produce a validation report.
• Add IRMAS as a separate sanity-check loader.
• Add PANNs transfer baseline if integration stays simple.
• Add weak sliding-window localization and richer threshold calibration.