SD-CN-Animation is a really cool way to create text-to-video, image-to-video and video-to-video with the checkpoint already loaded in WebUI. Unfortunately, the original extension is no longer maintained and had become unusable on my sd-webui-forge-classic 2.28.1 installation: it still relied on older Gradio component contracts, kept its own generation state instead of following Forge's Generate, Skip and Interrupt lifecycle, expected the former ControlNet integration, and depended on download/import paths that no longer worked.
I did not want such a useful tool to disappear from the Forge workflow, so I migrated it, fixed the older bugs uncovered along the way, and tested the three video modes with a little help from AI. I hope this fork saves other Forge users some time and proves useful to the community.
The original WebUI project is no longer maintained. Its authors recommend the modern ComfyUI port for ComfyUI users. This repository keeps the WebUI workflow alive for Forge Classic.
This branch supports:
- text-to-video, image-to-video and video-to-video with the active Forge checkpoint, including frame-by-frame generation tested with Krea 2;
- Gradio 4 uploads and component updates;
- Forge's shared Generate, Skip and Interrupt lifecycle;
- the integrated Forge ControlNet UI and Dynamic Prompts inside the animation tab, using isolated script runners that cannot corrupt the main txt2img and img2img tabs;
- automatic RAFT and FloweR weight downloads from the maintained ComfyUI mirror, without the old BasicSR/Google Drive dependency;
- partial but valid MP4 output after Interrupt, once at least one frame exists.
Skip keeps the current frame's deterministic fallback and continues with the next frame. Before the first frame of a pure text-to-video job, Skip is disabled because no valid fallback exists yet; Interrupt remains available. A video job waiting for Forge's generation lock does not expose Skip or Interrupt until it actually owns that lock.
Krea 2 can be used as the frame generator without a ControlNet. If ControlNet is enabled, its weights must be compatible with the active model architecture; an SD 1.5 or SDXL ControlNet does not become Krea-compatible through this extension.
The port also fixes historical issues found during testing: non-square optical flow axes, fractional source FPS truncation, RGB histogram matching, missing first frames in V2V MP4 files, stale state between runs, and cleanup after exceptions or interruptions.
The latest maintenance pass tightened that lifecycle further:
- Generate now stays disabled while the video generator returns its final result. It is re-enabled only after RAFT or FloweR, OpenCV handles and per-run buffers have been cleaned, Forge's state has ended, and the shared generation queue lock has been released;
- invalid OpenCV capture and writer objects are released before reporting the error, including failures that happen before those objects can be registered as the active job resources;
- V2V rejects a source that ends before its advertised frame count instead of silently processing zero-filled buffers as black frames;
- large V2V NumPy frame and optical-flow buffers are cleared after success, Interrupt or an exception, so the next run starts without retained job data;
- generated MP4 paths include microseconds, preventing two quick successive jobs from overwriting one another.
The fork includes regression tests under tests/. On the reference Forge
Classic 2.28.1 installation, all 33 tests pass. They cover the generation
lock and Forge state lifecycle, the four button phases, Skip, Interrupt before
the first frame and after partial output, recovery, error cleanup, video
metadata and resource-release regressions.
Real runtime validation was then performed through Forge's actual Gradio callbacks and API paths:
- V2V: 4 input frames -> 4 decoded MP4 frames at 2 FPS;
- V2V Skip after the first frame -> generation continued to a complete 4/4 frame MP4;
- V2V Interrupt after the first frame -> a decodable one-frame partial MP4, all Forge stop flags cleared, then a successful 4/4 recovery run written to a distinct output path;
- I2V: 10 requested frames -> 10 decoded MP4 frames at 4 FPS;
- T2V: 10 requested frames -> 10 decoded MP4 frames at 4 FPS.
These are real Forge runtime results, not a claim of manual browser clicks. A visual browser pass was blocked by the test environment's access-control policy, so button behavior was verified through the real Gradio callbacks/API plus the automated lifecycle tests.
Run the automated checks from the extension directory with Forge's Python:
..\..\venv\Scripts\python.exe -m pytest -q
..\..\venv\Scripts\python.exe -m ruff check scripts tests
..\..\venv\Scripts\python.exe -m compileall -q scripts tests
git diff --checkThis project allows you to automate video stylization task using StableDiffusion and ControlNet. It also allows you to generate completely new videos from text at any resolution and length in contrast to other current text2video methods using any Stable Diffusion model as a backbone, including custom ones. It uses 'RAFT' optical flow estimation algorithm to keep the animation stable and create an occlusion mask that is used to generate the next frame. In text to video mode it relies on 'FloweR' method (work in progress) that predicts optical flow from the previous frames.
sd-cn-animation ui preview
In vid2vid mode do not forget to activate ControlNet model to achieve better results. Without it the resulting video might be quite choppy. Do not put any images in CN as the frames would pass automatically from the video.
Here are CN parameters that seem to give the best results so far:
 |
 |
 |
| Original video | "Jessica Chastain" | "Watercolor painting" |
Examples presented are generated at 1024x576 resolution using the 'realisticVisionV13_v13' model as a base. They were cropped, downsized and compressed for better loading speed. You can see them in their original quality in the 'examples' folder.
 |
 |
 |
| "close up of a flower" | "bonfire near the camp in the mountains at night" | "close up of a diamond laying on the table" |
 |
 |
 |
| "close up of macaroni on the plate" | "close up of golden sphere" | "a tree standing in the winter forest" |
All examples you can see here are originally generated at 512x512 resolution using the 'sd-v1-5-inpainting' model as a base. They were downsized and compressed for better loading speed. You can see them in their original quality in the 'examples' folder. Actual prompts used were stated in the following format: "RAW photo, {subject}, 8k uhd, dslr, soft lighting, high quality, film grain, Fujifilm XT3", only the 'subject' part is described in the table above.
In Forge Classic, open Extensions -> Install from URL and use:
https://github.com/fabiencomte/SD-CN-Animation.git
Leave the local directory name empty, install, then restart Forge. The default
GitHub branch is already forge-classic-2.28.1. A new SD-CN-Animation tab
will appear and generated files will be written below
outputs/sd-cn-animation/.
On first use the extension downloads these small runtime weights outside the Git repository:
models/RAFT/raft-things.pth;models/FloweR/FloweR_0.1.2.pth.
- This branch targets Forge Classic 2.28.1 on Windows with Python 3.13 and Gradio 4.40. Other WebUI and Forge versions are not yet part of the test matrix.
- RAFT and FloweR require PyTorch operations and memory beyond the image model itself. High resolutions and long clips can be expensive.
- The built-in MP4 writer uses OpenCV's
mp4vcodec. FFmpeg is not required for generation, but another encoder may be preferable for final delivery. - macOS has not been validated by this fork.
- Fixed issues #69, #76, #91, #92.
- Fixed an issue in vid2vid mode when an occlusion mask computed from the optical flow may include unnecessary parts (where flow is non-zero).
- Added 'Extra params' in vid2vid mode for more fine-grain controls of the processing pipeline.
- Better default parameters set for vid2vid pipeline.
- In txt2vid mode after the first frame is generated the seed is now automatically set to -1 to prevent blurring issues.
- Added an option to save resulting frames into a folder alongside the video.
- Added ability to export current parameters in a human readable form as a json.
- Interpolation mode in the flow-applying stage is set to ‘nearest’ to reduce overtime image blurring.
- Added ControlNet to txt2vid mode as well as fixing #86 issue, thanks to @mariaWitch
- Fixed a major issue where ControlNet used the wrong input images. Because of this vid2vid results were much worse than they should be.
- Text to video mode now supports video as a guidance for ControlNet. It allows to create much stronger video stylizations.