Skip to content

Local AI images and video on this computer's GPU; per-platform captions and scheduling for auto-publish - #2

Merged
norbsss merged 6 commits into
mainfrom
claude/local-ai-models
Nov 4, 2026
Merged

norbsss merged 6 commits into
mainfrom
claude/local-ai-models

Conversation

@norbsss

@norbsss norbsss commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Stacked on #1 (WebUI split). Merge that first; this PR's diff shows only the 6 commits on top of it.

Local AI (this computer)

Two new material sources that run on the local GPU, free and offline:

  • Local AI image: one image per scene, turned into a clip with the existing zoom effect.
  • Local AI video: the image model paints every scene, then a video model animates each image into a 3-second clip. A clip that fails falls back to its image with a zoom.

Settings > AI Image / AI Video > Local AI detects the GPUs and memory, recommends the best model that fits, installs a separate PyTorch runtime that matches the GPU driver, and downloads models with progress. The image server starts on its own when a video needs it.

  • local_ai/: hardware detection with no torch needed (parses both the old CUDA Version and new CUDA UMD Version nvidia-smi headers), a model catalog with fit rules (GPU, CPU offload, CPU), downloads with completion markers, the runtime installer, and the GPU server. The server is OpenAI-compatible for images and adds /v1/videos/generations.
  • Catalog:
    • Images: RealVisXL V5 Lightning (SDXL), DreamShaper 8 (SD 1.5), FLUX.1 schnell (gated).
    • Video: Wan 2.2 TI2V 5B and Wan 2.1 T2V 1.3B, both Apache-2.0.
  • Windows issues found and fixed on real hardware:
    • Model downloads use plain folders, because the Hugging Face cache needs symlinks that Windows refuses without Developer Mode.
    • Wan's 5 GB fp32 shards failed to load with os error 1455 (commit limit) while WSL was running. On first load they are rewritten as bf16 shards of at most 1 GB, and the transformer and T5 load in 4-bit (bitsandbytes NF4).
    • The VAE decode always runs tiled. Untiled, it spilled into shared GPU memory and a clip took 11 minutes instead of 2.
    • The venv's python.exe is a launcher, so stopping the server ends the pid the server reports.
  • Measured on an RTX 4070 Ti SUPER (16 GB): about 6 s per image, and about 2 minutes per 3-second 480p clip (denoise 93 s, decode 32 s).

Auto-publish (Upload-Post)

  • Per-platform captions: TikTok and Instagram each get their own caption with hashtags, sent in tiktok_title and instagram_title. Before, both got one caption and no hashtags.
  • AI-generated label: is_ai_generated is sent for TikTok and Instagram, matching YouTube's existing containsSyntheticMedia. It is on by default and can be turned off, because Instagram's label cannot be removed after publishing.
  • When to publish: right away (the default), the next Upload-Post queue slot, or N hours later. The 202 + job_id response is recorded as a scheduled post instead of being polled for an hour and reported as failed.

Tests

  • New: test_local_ai.py, test_local_ai_service.py, test_upload_post_platform_text.py, test_upload_post_schedule.py, plus a settings AppTest.
  • Full suite on Windows: 3261 passed, 4 failed. All 4 also fail without these changes on this machine: symlink privilege, ffmpeg home-path expansion, a video metadata probe, and one flaky timing test.
  • Real GPU runs: image server; material stage with 3 scenes → 3 zoom clips; video material stage with 2 scenes → 2 Wan clips in 9 minutes including both model loads.
  • Publishing is tested against the documented Upload-Post API with mocked HTTP, not a live account.

🤖 Generated with Claude Code

norbsss and others added 6 commits November 4, 2026 16:14
Adds a "Local AI (this computer)" image source. The WebUI detects the
GPUs and memory, recommends the best model that fits, installs a
separate PyTorch runtime for the detected driver, downloads the model
and starts the image server when a video needs images.

- local_ai/: standard-library hardware detection (nvidia-smi, including
  the newer "CUDA UMD Version" header), a model catalog with fit rules
  (GPU, CPU offload, CPU), downloads with progress and completion
  markers, the runtime installer, and an OpenAI-compatible server that
  hands the GPU back to Ollama between images (from
  docs/local-image-server, now model-agnostic).
- Catalog: RealVisXL V5.0 Lightning (SDXL), DreamShaper 8 (SD 1.5) and
  FLUX.1 schnell (gated). Offloading SDXL ranks above SD 1.5 on the GPU,
  since it costs seconds per image but looks much better.
- Downloads go to plain folders: the Hugging Face cache needs symlinks,
  which Windows refuses without Developer Mode.
- app/services/local_ai.py manages the runtime, downloads and server.
  The venv's python.exe on Windows is a launcher, so stopping the
  server ends the pid the server reports, not only the Popen handle.
- New "local_image" source reuses the image-to-clip pipeline. Requests
  bypass proxies and never send cloud image API keys to localhost, and
  tasks fail in preflight when the runtime or model is missing.
- Settings > AI Image > Local AI: hardware, GPU choice, runtime install,
  model download/use/delete with progress, server start/stop, and a
  Hugging Face token stored as a credential.

Verified on an RTX 4070 Ti SUPER: about 6 s per image once the model is
loaded, and the material stage turned 3 scenes into zoom clips.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s GPU

Adds a "Local AI video (this computer)" source. The local image model
paints every scene, then a video model animates each image into a short
clip; a clip that fails falls back to its image with a slow zoom.

- Catalog: Wan 2.2 TI2V 5B (image-to-video, Apache-2.0) and Wan 2.1 T2V
  1.3B (text-to-video for 8 GB cards). Wan publishes 5 GB float32 shards;
  loading them failed with Windows os error 1455 when WSL held most of
  the commit limit. On first load the weights are rewritten as bfloat16
  shards of at most 1 GB, and the transformer and T5 encoder load in
  4-bit (bitsandbytes NF4), so Wan 2.2 runs fully on a 16 GB card.
- The VAE decode always runs tiled: untiled, it spilled into Windows
  shared GPU memory and a clip took 11 minutes instead of 2.
- The server keeps one image and one video pipeline, unloads the other
  kind before loading a model, and frees memory after a failed load.
  The material flow paints all scenes first, then animates them, so a
  video swaps models once instead of twice per scene.
- The runtime marker records a digest of the requirements, so a runtime
  installed before the video packages (ftfy, imageio, bitsandbytes)
  counts as missing and reinstalls incrementally.
- Settings show image and video models in both AI Image and AI Video
  sections, with widget keys per section.

Measured on an RTX 4070 Ti SUPER: about 2 minutes per 3-second 480p clip
(denoise 93 s, decode 32 s); a 3-scene material stage took 9 minutes
including both model loads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…osts

TikTok and Instagram posts used one caption chosen for the first
platform, and hashtags were dropped because the caption excludes them.

- Generate metadata once per selected platform and send it in
  Upload-Post's tiktok_title and instagram_title fields as the caption
  followed by its hashtags. YouTube keeps its own title, description and
  tags; the generic title remains the fallback.
- Send is_ai_generated for TikTok and Instagram, as YouTube uploads
  already send containsSyntheticMedia. On by default, with a setting to
  turn it off, since Instagram's label cannot be removed after publishing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit generates metadata for TikTok and Instagram
separately; this test still expected a single generation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Streamlit also calls a selectbox's format_func outside the script run,
where tr() cannot read the session language and raises. Precompute the
labels, as the YouTube audience selector already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds "When to publish" to the Auto-Publish settings: right after the
video is ready (the default and previous behaviour), the next free slot
in the Upload-Post queue, or a set number of hours later. Upload-Post
stores scheduled posts and publishes them itself, so this computer can
be off at that time.

- The choice is captured when the post is queued, like the YouTube
  privacy and audience settings, and a delay counts from that moment.
- Uploads send add_to_queue or scheduled_date (UTC, within the 365 days
  Upload-Post accepts).
- A scheduled upload returns 202 with a job_id and no platform results
  yet. It is recorded as a successful, scheduled result instead of being
  polled as a background upload for an hour and reported as failed; a
  202 without a job_id is reported as unconfirmed.
- The roadmap now notes what Upload-Post covers, and defers the
  Publisher protocol until a second publisher needs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@norbsss
norbsss changed the base branch from claude/caveman-feature-planning-8c9d5d to main November 4, 2026 16:35
@norbsss
norbsss merged commit 2d9acf9 into main Nov 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant