Skip to content

spec.md

Latest commit

 

History

History
194 lines (153 loc) · 9.91 KB

spec.md

File metadata and controls

194 lines (153 loc) · 9.91 KB
Raw

Markdown

TECHNICAL SPECIFICATION: GOWEBGPU VIDEO RENDERING ENGINE (gowebgpu-engine)

1. System Overview & Core Objective

gowebgpu-engine is a high-performance, headless, server-side video rendering and processing engine written in Go. It combines Go's concurrent stream management with native WebGPU bindings (wgpu-native via CGO) and WGSL (WebGPU Shading Language) shaders to execute GPU-accelerated video effects (Chroma Key, 3D LUT Color Grading, Blurring, Text/Overlay Compositing) without intermediate disk I/O bottlenecks.

The project follows a Hybrid Architecture:

  1. Frontend (Preview Layer): HTML5/JS WebGPU viewer running the exact same .wgsl shaders directly on the client's browser GPU for real-time (60fps) editing and preview.
  2. Backend (Final Export Layer): Go Engine reading raw video streams from FFmpeg pipes, executing GPU shader pipelines in VRAM, and piping raw rendered frames directly back to FFmpeg for high-bitrate MP4/H.264 encoding.

2. Tech Stack & Dependencies

Component Technology / Library Description
Primary Language Go 1.22+ Core system orchestration, worker pools, concurrency, stream I/O.
GPU Native API github.com/rajveermalviya/go-webgpu or wgpu-native C bindings Headless CGO bridge to Vulkan (Linux), Metal (macOS), or DX12 (Windows).
Shading Language WGSL (WebGPU Shading Language) Compute and Render pass pixel manipulation scripts.
Media Decoder/Encoder FFmpeg CLI (os/exec pipes) Decoding MP4 to raw RGBA byte streams and encoding RGBA back to MP4.
Frontend Preview Native WebGPU API (navigator.gpu) HTML5 preview client executing WGSL shaders on client GPU.

3. System Architecture & Zero-Disk Data Flow

+-----------------------------------------------------------------------------------+ | PREVIEW STAGE (Client Web Browser) | | User Adjusts Timeline/Params -> WebGPU JS Canvas -> Instant 60FPS Local Preview | +-----------------------------------------------------------------------------------+ │ (On User Export Action) ▼ (Sends timeline.json) +-----------------------------------------------------------------------------------+ | FINAL EXPORT STAGE (Go Server Backend) | | | | [Input MP4] | | │ | | ▼ | | [FFmpeg Decoder Pipe] (stdout: raw RGBA bytes) | | │ | | ▼ | | [Go Memory Buffer] ──► Upload via queue.WriteTexture() ──► [VRAM Texture] | | │ | | ▼ | | [WGSL Shader Processing] | | │ | | ▼ | | [Go Output Buffer] ◄── Copy via CopyTextureToBuffer() ◄── [Rendered VRAM] | | │ | | ▼ | | [FFmpeg Encoder Pipe] (stdin: raw RGBA bytes) | | │ | | ▼ | | [Final MP4 File] | +-----------------------------------------------------------------------------------+

4. Project Directory Structure

gowebgpu-engine/ ├── cmd/ │ └── main.go # Entry point: CLI args, JSON timeline loader, execution runner ├── config/ │ └── timeline.go # Struct definitions for JSON timeline specs ├── gpu/ │ ├── context.go # Headless WebGPU Instance, Adapter, Device, Queue initialization │ ├── pipeline.go # RenderPass pipeline, BindGroups, Texture allocations │ └── memory.go # Explicit VRAM buffer management & cleanup routines ├── media/ │ ├── decoder.go # FFmpeg process spawn & stdout RGBA pipe reader │ └── encoder.go # FFmpeg process spawn & stdin RGBA pipe writer ├── shaders/ │ ├── chroma_key.wgsl # Green screen removal shader │ ├── color_grade.wgsl # 3D LUT / Exposure / Contrast adjustment shader │ └── blend_overlay.wgsl # Alpha compositing shader for text and images ├── web/ │ └── preview.html # Frontend HTML5/WebGPU preview canvas ├── go.mod └── README.md

5. Timeline JSON Schema (timeline.json)

The engine must accept a structured JSON spec describing tracks, clips, and applied GPU shader effects:

{
  "output": {
    "width": 1920,
    "height": 1080,
    "fps": 30,
    "bitrate": "8M",
    "outputPath": "output.mp4"
  },
  "tracks": [
    {
      "id": "video_track_1",
      "clips": [
        {
          "source": "assets/input.mp4",
          "startTime": 0.0,
          "duration": 10.0,
          "effects": [
            {
              "type": "chroma_key",
              "enabled": true,
              "params": {
                "keyColor": [0.0, 1.0, 0.0],
                "similarity": 0.4,
                "smoothness": 0.1
              }
            },
            {
              "type": "color_grade",
              "enabled": true,
              "params": {
                "exposure": 0.1,
                "contrast": 1.2,
                "saturation": 1.1
              }
            }
          ]
        }
      ]
    }
  ]
}
6. Detailed Implementation Requirements
6.1. Headless WebGPU Context Setup (gpu/context.go)
Initialize a wgpu.Instance without attaching any OS native window handle (Surface-less / Headless mode).

Request an Adapter with PowerPreference_HighPerformance.

Request a Device with necessary queue descriptors.

Implement an error callback logging uncaught GPU validation errors to prevent silent crashes.

6.2. Zero-Disk FFmpeg Reader/Writer (media/decoder.go & media/encoder.go)
Decoder Command:
ffmpeg -loglevel error -i <input.mp4> -f rawvideo -pix_fmt rgba pipe:1

Encoder Command:
ffmpeg -loglevel error -y -f rawvideo -pix_fmt rgba -s <WxH> -r <FPS> -i pipe:0 -c:v libx264 -pix_fmt yuv420p -b:v <bitrate> <output.mp4>

Buffer reading MUST operate in chunk sizes strictly equal to width * height * 4 bytes (RGBA frame size).

6.3. GPU Frame Pipeline & WGSL Shader Execution (gpu/pipeline.go)
Create input GPUTexture (TextureUsage_COPY_DST | TextureUsage_TEXTURE_BINDING).

Write raw RGBA frame chunk from RAM to VRAM using queue.WriteTexture().

Create output GPUTexture (TextureUsage_COPY_SRC | TextureUsage_RENDER_ATTACHMENT).

Execute WGSL Shaders via CommandEncoder and RenderPassEncoder.

Copy processed output texture to a mapped staging GPUBuffer using CopyTextureToBuffer().

Read staging buffer bytes back to Go memory slice and immediately flush into the Encoder Pipe (stdin).

6.4. Memory Management & VRAM Leak Prevention Protocol (gpu/memory.go)
CRITICAL RULE: Go's Garbage Collector DOES NOT manage CGO pointers or WebGPU VRAM allocations.

Every created GPUTexture, GPUTextureView, GPUBuffer, and BindGroup MUST be explicitly destroyed or freed at the end of every single frame render iteration via .Destroy() calls.

Frame buffers MUST be reused across frame loops whenever possible to minimize allocation overhead.

7. Execution Blueprint & Implementation Steps for AI Agent
When instructing your AI Agent (Cursor / Windsurf / Claude Code), execute in 4 sequential phases:

Phase 1: Core Setup & Headless Context Initialization
"Create the project directory structure. Implement config/timeline.go with the timeline JSON structs. Then implement gpu/context.go to successfully initialize a Headless WebGPU instance and device using wgpu-native without creating a GUI window."

Phase 2: FFmpeg Streaming Pipeline
"Implement media/decoder.go and media/encoder.go. Create a test runner in cmd/main.go that decodes an MP4 file frame-by-frame into raw RGBA byte chunks in Go memory, and pipes it directly into the encoder process to create a copy MP4 file without writing any intermediate files to disk."

Phase 3: WebGPU Shader Integration
"Implement shaders/chroma_key.wgsl and gpu/pipeline.go. Intercept the raw RGBA frame chunks from Phase 2, upload them to WebGPU VRAM, execute the chroma key shader pass, copy the output frame back, and write to the encoder. Ensure all WebGPU textures/buffers are explicitly destroyed after each frame."

Phase 4: Web Preview Bridge
"Create web/preview.html with a WebGPU Canvas viewer. Load the exact same shaders/chroma_key.wgsl file and allow real-time parameter tweaking via UI controls, matching the JSON timeline parameter schema."


---

### Lộ trình gợi ý để bạn bắt đầu ngay:

1. **Bước 1**: Copy toàn bộ nội dung trong khung Markdown trên.
2. **Bước 2**: Mở **Cursor IDE** hoặc **Claude Code**, dán bản Spec này vào và gõ câu lệnh: 
   > *"Read this project specification carefully. Let's start with Phase 1: Create the folder structure, go.mod, and implement `gpu/context.go`."*
3. **Bước 3**: Chạy thử nghiệm và từng bước kiểm chứng hệ thống theo đúng lộ trình 4 Phase đã vạch ra trong Spec!