Skip to content

feat(mcp): Expose video and audio generation tools to Model Context Protocol (MCP) server #14

Description

@spelech

🎯 Objective & Overview

Expose new Model Context Protocol (MCP) AI tools in LocalLlmMcpTools.cs so external AI agents and IDEs (Antigravity, Claude Desktop, Open WebUI) can trigger video generation, speech synthesis, and audio generation via JSON-RPC.


📂 Files & Components to Touch

  1. MCP Tools Class (Services/LocalLlmMcpTools.cs):
    • Add [McpServerTool, Description("Generate video from text prompt or image using ComfyUI DiT pipelines (Wan 2.2, LTX-2.5).")]
      public async Task<string> GenerateVideoAsync(string prompt, string workflow = "wan2.2_t2v", int width = 832, int height = 480, int frames = 49)
    • Add [McpServerTool, Description("Synthesize speech audio from text using local Kokoro / AllTalk TTS engine.")]
      public async Task<string> SynthesizeSpeechAsync(string text, string voice = "af_heart", string format = "mp3")
    • Add [McpServerTool, Description("Generate sound effects or ambient musical loops from a prompt.")]
      public async Task<string> GenerateAudioAsync(string prompt, int durationSeconds = 15)
  2. Integration Tests (LocalLLMServerManager.Tests/McpServerIntegrationTests.cs):
    • Add test cases verifying MCP tool registration, parameter validation, and mock execution over SSE/HTTP.

🔌 MCP Tool Signatures

{
  "name": "generate_video",
  "description": "Generate video from text prompt using local ComfyUI",
  "inputSchema": {
    "type": "object",
    "properties": {
      "prompt": { "type": "string" },
      "workflow": { "type": "string", "enum": ["wan2.2_t2v", "ltx2.5_t2v", "hunyuanvideo1.5_t2v"] },
      "frames": { "type": "integer", "default": 49 }
    },
    "required": ["prompt"]
  }
}

🛠️ Step-by-Step Implementation Guide

  1. Inject IWorkflowService / HttpClient into LocalLlmMcpTools.cs.
  2. Implement tool methods with full argument descriptions and error handling.
  3. Return JSON containing output media URLs and status info.
  4. Update MCP integration tests and documentation.

🧪 Verification

  • dotnet test --filter Category!=LiveExternal
  • npm run lint & npx tsc --noEmit

Activity

  1. google-labs-jules commented on Aug 23, 2026

    @google-labs-jules
    Contributor

    Jules is on it. When finished, you will see another comment and be able to review a PR.

  2. google-labs-jules commented on Aug 23, 2026

    @google-labs-jules
    Contributor

    Ready for a review! A PR has been created.

  3. spelech commented on Aug 24, 2026

    @spelech
    OwnerAuthor

    Resolved in v3.7.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions