usage.md

Usage

Install

pip install torch --index-url https://download.pytorch.org/whl/cpu  # or a GPU build
pip install foleydiff

Quick start

from foleydiff import TextToAudioPipeline, save_wav
from foleydiff.presets import get_preset

pipe = TextToAudioPipeline(get_preset("tiny"))
audio = pipe.generate("rain on a tin roof", seconds=2.0, steps=30, seed=0)
save_wav("rain.wav", audio, pipe.config.mel.sample_rate)

The shipped weights are randomly initialised, so out-of-the-box output is noise-shaped. foleydiff provides the machinery; train the components (or load your own checkpoint) to get meaningful audio.

Choosing a sampler

# Fast & deterministic (good default)
audio = pipe.generate("dog barking", sampler="ddim", steps=50, eta=0.0, seed=1)

# Stochastic DDIM
audio = pipe.generate("dog barking", sampler="ddim", eta=1.0, seed=1)

# Faithful ancestral DDPM (uses the full training schedule)
audio = pipe.generate("dog barking", sampler="ddpm", seed=1)

Classifier-free guidance

Pass guidance_scale > 1 to push samples toward the prompt. The pipeline encodes an empty prompt as the unconditional branch automatically.

audio = pipe.generate("thunderclap", guidance_scale=4.0, steps=50)

Batch generation

batch = pipe.generate(["wind", "footsteps", "engine hum"], seconds=1.5, seed=7)
print(batch.shape)  # (3, samples)

Reproducibility

Passing seed= makes a call fully reproducible — the same generator drives both the diffusion sampler and the vocoder's phase initialisation.

The CLI

foleydiff info --preset small
foleydiff generate "glass shattering" -o sfx.wav --preset tiny --seconds 2 --steps 40 --seed 0

Working with mel-spectrograms directly

import torch
from foleydiff import MelSpectrogram, GriffinLimVocoder
from foleydiff.config import MelConfig

cfg = MelConfig()
mel = MelSpectrogram(cfg)
vocoder = GriffinLimVocoder(cfg)

wav = torch.randn(1, 16000)
spec = mel(wav)                       # (1, 64, frames)
recon = vocoder(spec, length=16000)   # (1, 16000)

See api-reference.md for the full public surface.