Skip to content

feat(audio): Add ComfyUI workflows for Stable Audio Open 3.0 & YuE music generationΒ #12

Description

@spelech

🎯 Objective & Overview

Enable text-to-sound-effects (SFX), ambient audio, and full musical song generation by creating ComfyUI workflow presets for Stable Audio Open 3.0 and YuE (乐).


πŸ“‚ Files & Components to Touch

  1. Workflow Presets (Workflows/Audio/):
    • Workflows/Audio/stable_audio_open_sfx.json (Generates up to 47s stereo SFX / music clips)
    • Workflows/Audio/yue_full_song.json (Dual-track lyrics-to-music generation)
  2. Endpoints (Endpoints/WorkflowEndpoints.cs):
    • Add GET /api/audio/workflows listing available audio generation workflow templates.
    • Add POST /api/audio/generate queuing audio workflow execution on ComfyUI.
    • Add GET /api/audio/files listing generated .wav / .flac / .mp3 output files from output_audio/.
  3. UI Player (LocalLLMServerManager.Shared/Views/Controls/EngineStudioTabControl.axaml):
    • Add audio waveform visualizer or embedded HTML5 <audio controls> player for previewing generated tracks.

πŸ”Œ API Contract

POST /api/audio/generate

{
  "workflowId": "stable_audio_open_sfx",
  "prompt": "Cyberpunk atmospheric ambient drone, heavy synthesizer, cinematic low end, 48kHz stereo",
  "negativePrompt": "low quality, harsh distortion",
  "durationSeconds": 30,
  "seed": -1
}

Response: 200 OK

{
  "promptId": "f9e8d7c6-b5a4-...",
  "status": "queued",
  "wsUrl": "ws://127.0.0.1:8188/ws"
}

πŸ› οΈ Step-by-Step Implementation Guide

  1. Create and validate ComfyUI workflow JSONs using ComfyUI-Stable-Audio or native audio nodes.
  2. Implement substitution and queuing in WorkflowEndpoints.cs.
  3. Save generated audio outputs into wwwroot/output_audio/ and serve them statically.
  4. Add audio player controls to the Studio UI.

πŸ§ͺ Verification

  • dotnet build
  • dotnet test --filter Category!=LiveExternal
  • npm run lint & npx tsc --noEmit

Activity

  1. google-labs-jules commented on Aug 23, 2026

    @google-labs-jules
    Contributor

    Jules is on it. When finished, you will see another comment and be able to review a PR.

  2. google-labs-jules commented on Aug 23, 2026

    @google-labs-jules
    Contributor

    Ready for a review! A PR has been created.

  3. spelech commented on Aug 24, 2026

    @spelech
    OwnerAuthor

    Resolved in v3.7.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions