Skip to content

Add a portable Stable Audio 3 TFLite backend #111

Description

@brxs

Summary

Add the official Stable Audio 3 TFLite runtime as the portable CPU baseline for Linux and Windows, behind the same application-facing contract as the existing macOS MLX backend.

Dependencies

Architecture

  • Define a runtime-neutral Stable Audio service contract.
  • Keep MLX on supported macOS systems.
  • Select the official TFLite implementation on Linux and Windows as the dependable baseline, including on NVIDIA systems.
  • Keep backend selection and diagnostics explicit.

Scope

Feature parity

Support every Stable Audio feature currently exposed by LSDJ where the pinned TFLite API permits it:

  • text-to-audio music and sound-effects modes;
  • audio-to-audio, continuation, and inpainting;
  • positive and negative prompts;
  • seed, duration, step count, CFG and APG-related controls;
  • Small and Medium model selection;
  • LoRA selection/application;
  • output naming, preview, cancellation, and model/runtime status.

Document any unavoidable backend difference and do not silently ignore a control.

Input/output normalization

  • Normalize accepted user audio internally to the sample rate, channel layout, and WAV representation required by the runtime.
  • Do not require users to install or invoke ffmpeg.
  • Preserve LSDJ's current supported duration range, including up to 380 seconds for the Medium workflow where the upstream model/runtime supports it.
  • Validate outputs before exposing them to the library/player.

Runtime and model management

  • Package or install a pinned TFLite Python runtime without relying on system Python or shell tools.
  • Use Refactor LSDJ's cross-platform application foundation #107's authenticated manifest, checksum, archive-safety, atomic promotion, rollback, and retry behavior.
  • Implement explicit state for runtime/model unavailable, download, verification, install, warm-up, ready, update, failure, and fallback.
  • Disclose storage before download using measured values for the pinned revisions (currently approximately 2.3 GB for Small and 9.5 GB for Medium; update these when pinned assets are selected).
  • Tie technical provenance and cache cleanup to exact model and LoRA revisions.

Real-time protection

  • Run generation outside the audio callback and enforce bounded process/thread/CPU/RAM use.
  • Measure Stable Audio generation while both MRT2 decks are active.
  • Queue, constrain, or pause background generation when necessary to protect live playback.
  • Expose progress, cancellation, worker health, memory use, and failure state.

Tests and documentation

  • Contract tests shared with the MLX adapter where semantics match.
  • Fixture coverage for each mode and control, including audio normalization, LoRA, cancellation, corrupt output, and long-duration validation.
  • Install/update/rollback and path tests on Ubuntu and Windows.
  • Performance characterization on the platform minimum CPU/RAM configurations while both decks are active.
  • User and contributor documentation for backend selection, storage, downloads, cache removal, and diagnostics.

Licensing dependency

All user-facing terms, attribution, gated-access behavior, credential handling, and acknowledgement UX are owned by #108. This issue must provide the revision/provenance metadata that flow consumes.

Acceptance criteria

  • Linux and Windows use the official TFLite backend; supported macOS continues to use MLX.
  • Every LSDJ Stable Audio mode/control either passes parity tests or has a documented, user-visible backend limitation.
  • Input audio is normalized internally with no user-installed media tooling.
  • Small and Medium downloads show measured storage requirements before starting and recover safely from interruption or verification failure.
  • Stable Audio generation cannot block the real-time audio callback and is tested while both decks are active.
  • Supported generation, cancellation, LoRA, and long-duration workflows complete without orphaned workers or corrupt library entries.
  • CI covers contracts, paths, installation, rollback, and fixtures on Ubuntu and Windows.
  • Add third-party model licensing, attribution, and download acknowledgement #108 is complete before this backend is included in a public Linux or Windows release.

Out of scope

  • Replacing the macOS MLX backend.
  • TensorRT or custom CUDA acceleration.
  • Linux/Windows application installers.
  • ARM, AMD, or Intel GPU acceleration.
  • Training or maintaining model forks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions