Neural Audio Style Transfer between piano and violin using GAN-based Encoder-Decoder architecture based on Disentanglement of Latent Representations of the input in complex domain.
This project implements a deep learning system that can transfer the style characteristics between different musical instruments while preserving the musical content. The system uses a combination of STFT and CQT representations (using both real and imaginary parts), 2 transformer encoders: one for style and one for content, an autoregressive decoder and a discriminator to perform adversarial training using also contrastive and reconstruction losses.
For further details, read the full paper Distentangled Latent Representations for Audio Style Transfer.pdf
- Operating in the full complex domain: Many musical models discard phase information, which is essential for a better reconstruction
- Different encoders for different roles: Definition of content and style encoder and relative constraints and losses to enforce this
- Transformer-based Architecture: Uses self-attention mechanisms for temporal modeling
- Multi-scale Losses: Combines time-domain, frequency-domain, and perceptual losses (reconstruction, contrastive, adversarial, disentanglement)
- Dynamic Sequence Handling: Supports variable-length audio sequences
- Curriculum Learning: Progressive training strategy for stable convergence
- Memory Efficient: Optimized for training with limited GPU memory
- Easy to Scale to more instruments: Architecture designed to be easy to add more instrument classes
The system consists of four main components:
- Style Encoder: Extracts instrument-specific style embeddings
- Content Encoder: Captures musical content independent of instrument
- Autoregressive Decoder: Reconstructs STFT with transferred style
- Discriminator: Ensures proper disentanglement between style and content
Audio-Style-Transfer/
├── README.md
├── requirements.txt
├── configs/
│ └── default.yaml # all hyperparameters (model, optim, losses, training)
├── src/
│ ├── config.py # typed access to the YAML config
│ ├── audio/
│ │ └── transforms.py # STFT/CQT (+inverses), audio I/O, windowing, plots
│ ├── data/
│ │ └── dataset.py # dual-instrument dataset + balanced collate
│ ├── models/
│ │ ├── blocks.py # shared ResBlock, positional encoding, DeepCNN
│ │ ├── style_encoder.py # style encoder (CNN + transformer, CLS pooling)
│ │ ├── content_encoder.py # content encoder (no CLS, full sequence)
│ │ ├── discriminator.py # adversarial discriminator
│ │ ├── decoder_transformer.py # transformer-only decoder (used in training)
│ │ └── decoder_cnn.py # CNN decoder variant
│ ├── training/
│ │ ├── losses.py # InfoNCE, margin, adversarial, HSIC disentanglement
│ │ ├── reconstruction_loss.py # comprehensive STFT reconstruction loss (shared)
│ │ ├── curriculum.py # curriculum loss-weight schedule
│ │ ├── utils.py # weight init, NaN guards, checkpointing
│ │ └── trainer.py # unified AdversarialTrainer (standard + curriculum)
│ └── evaluation/
│ ├── reconstruction.py # reconstruction-task evaluation
│ └── style_transfer.py # style-transfer evaluation
├── scripts/
│ └── train.py # CLI entry point (--config, --curriculum, --decoder)
├── notebooks/
│ ├── training_showcase.ipynb # training run record with loss/reconstruction outputs
│ ├── test_correctness.ipynb # model validation and testing
│ └── style_transfer_inference_test.ipynb # style transfer inference and audio export
├── preprocessing/ # one-off dataset preparation scripts (split, stats)
└── train_set_stats/ # precomputed normalization statistics (.npz)
pip install -r requirements.txt
# point configs/default.yaml at your piano/violin folders, then:
python scripts/train.py # standard training (fixed loss weights)
python scripts/train.py --curriculum # curriculum schedule
python scripts/train.py --decoder cnn # use the CNN decoder variantEvery hyperparameter lives in configs/default.yaml. The
standard and curriculum training regimes share a single AdversarialTrainer
(src/training/trainer.py); they differ only in where the loss weights come
from, plus the class_emb_amplification / apply_warmup settings noted in the config.
Implements the StyleEncoder class using ResNet-like CNN blocks followed by transformer layers. Key features:
- Extracts style embeddings that capture instrument-specific characteristics
- Provides both individual and class-level embeddings (by aggregation through CLS token)
- Uses full STFT + CQT
ContentEncoder for extracting musical content representations. Architecture similar to style encoder.
- No CLS token to focus on temporal content structure
- Outputs sequence-level embeddings for temporal modeling, perfect for the autoregressive decoder
Two decoder variants that reconstruct STFT spectrograms (the transformer-only one is used in training). Features:
- Autoregressive generation during inference with causal masking
- Teacher forcing during training for stable convergence
- CNN encoder-decoder for spatial feature processing
- Dynamic sequence length handling without fixed parameters
- Comprehensive loss functions
Simple MLP discriminator for adversarial training:
- Ensures proper disentanglement between style and content representations
- Designed to correctly classify style/class_emb while performing randomly on content_emb
Comprehensive loss function library including:
- InfoNCE loss: Contrastive learning for style representation
- Margin loss: Class separation in embedding space
- Adversarial losses: Generator/discriminator training
- HSIC-based disentanglement loss: Statistical independence between style and content
Efficient data loading for dual-instrument training:
- Handles STFT/CQT computation with configurable parameters
- Applies instrument-specific normalization statistics
- Creates overlapping windows for temporal modeling
- Ensures balanced batch creation across instrument classes for the contrastive training
Piano-specific normalization statistics:
stft_mean,stft_std: STFT normalization parameters (shape: [2, 513])cqt_mean,cqt_std: CQT normalization parameters (shape: [2, 84])- Computed from piano training data for instrument-specific normalization
Violin-specific normalization statistics:
- Same structure as piano statistics
- Ensures proper normalization for violin characteristics
Combined statistics from both instruments:
- Used as fallback when separate statistics are unavailable
- Provides general normalization for mixed-instrument scenarios
- Maintains compatibility with different data configurations
Implemented also a curriculum learning approach (src/training/curriculum.py):
- Phase 1 (0-20%): Reconstruction-only training
- Phase 2 (20-40%): Add contrastive + margin losses
- Phase 3 (40-60%): Introduce disentanglement
- Phase 4 (60-100%): Gradually add adversarial training This progressive approach ensures stable training and better convergence.
Despite our experiments, the results were not convincing enough, some losses were oscillating too much indicating instability and the model wasn’t reconstructing properly. However we are strongly convinced of the foundations and mathematical proofs behind our choices and we believe that further experiments and refinements could lead to proper learning.
Dataset, statistics and checkpoints: https://drive.google.com/drive/folders/13P5gLzAJPyDuhp-urVOHHelahs5rQehu?usp=sharing
See requirements.txt (PyTorch, torchaudio, librosa, numpy, scipy, scikit-learn, matplotlib, pyyaml, tqdm). Python 3.10+.