Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 15 additions & 7 deletions docs/video-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This is the production lane for films about the work: showcase previews, the
LinkedIn and X films, launch reels and GIFs. It is not website motion. The skill
builds pages; this document records how the repository produces and publishes video
with free, locally run tools, and what each tool's licence actually says.
with locally run tools, and where a tool requires a paid licence.

Rules that hold for every stage:

Expand All @@ -18,10 +18,10 @@ Rules that hold for every stage:

| Stage | Tool | Licence | How this repository uses it |
|---|---|---|---|
| Capture a browser run | Playwright recording | Apache-2.0 | `tools/page-proof/record-gallery.mjs` scrolls each example and records it deterministically |
| Capture a browser run | Playwright recording | Apache-2.0 | `tools/page-proof/record-gallery.mjs` records timed scroll passes; this real-time capture is not frame-deterministic |
| Capture a screen or camera | OBS Studio | GPL-2.0 | Manual capture for talking-head or live demo footage |
| Compose titles and cuts | FFmpeg filters | LGPL, GPL with GPL components | `tools/render-linkedin-promo.mjs` composes the LinkedIn film from captures |
| Compose a video in React | Remotion | Free for individuals, for-profit organisations with up to three employees and non-profits; other companies need a company licence | Optional route for templated films. `compile-choreography.mjs` can emit a paused, fixed-time GSAP timeline that a frame-by-frame renderer such as Remotion or HyperFrames seeks per frame; the repository itself has no Remotion dependency |
| Compose a video in React | Remotion | [Free for individuals, for-profit organisations with up to three employees and non-profits; other companies need a company licence](https://github.com/remotion-dev/remotion/blob/main/LICENSE.md) | Optional route for templated films. `compile-choreography.mjs` can emit a paused, fixed-time GSAP timeline that a frame-by-frame renderer such as Remotion or HyperFrames seeks per frame; the repository itself has no Remotion dependency |
| Voiceover | Qwen3-TTS, run locally | Apache-2.0 | Optional; needs a GPU for comfortable speed. Earlier films used the owner's own avatar-service account |
| Cut silences | Auto-Editor | Unlicense | `auto-editor input.mp4 --margin 0.2sec` before a manual pass |
| Clean and mix audio | Audacity | GPL | Noise reduction, levels, music ducking |
Expand All @@ -31,17 +31,25 @@ Rules that hold for every stage:

## Delivery presets

These commands need an FFmpeg build with `libx264`; burned-in captions also need
the `subtitles` filter (`libass`). The landscape preset fits and pads to 1920×1080;
the square preset deliberately crops. Review that crop before publishing.

```sh
# 1080p landscape, platform-safe
ffmpeg -i edit.mov -c:v libx264 -crf 20 -preset slow -pix_fmt yuv420p \
# 1080p landscape, preserve the composition with padding
ffmpeg -i edit.mov \
-vf "scale=1920:1080:force_original_aspect_ratio=decrease:force_divisible_by=2,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,setsar=1" \
-c:v libx264 -crf 20 -preset slow -pix_fmt yuv420p \
-movflags +faststart -c:a aac -b:a 160k out-1080.mp4

# square for feeds
ffmpeg -i edit.mov -vf "scale=1080:1080:force_original_aspect_ratio=increase,crop=1080:1080" \
-c:v libx264 -crf 20 -pix_fmt yuv420p -movflags +faststart -c:a aac out-square.mp4

# burned-in captions
ffmpeg -i out-1080.mp4 -vf subtitles=captions.srt -c:a copy out-1080-captions.mp4
# burned-in captions, retain the delivery encoding and streaming header
ffmpeg -i out-1080.mp4 -vf subtitles=captions.srt \
-c:v libx264 -crf 20 -preset slow -pix_fmt yuv420p -movflags +faststart \
-c:a copy out-1080-captions.mp4

# poster
ffmpeg -i out-1080.mp4 -ss 00:00:03 -frames:v 1 -q:v 2 poster.jpg
Expand Down
41 changes: 28 additions & 13 deletions skills/cinematic-scroll/references/bake-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ plays them under the same rules as any other media mechanism.
| Need | Mechanism | Dependency | Static state |
|---|---|---|---|
| Scroll drives the sequence on desktop | Scrubbed video, as in [product reveal](product-reveal.md) | None; a `<video>` with a poster | Permanent poster |
| Frame-exact seeking, or touch devices that block programmatic seeks | Image sequence drawn on a canvas | None; preloaded frames | Permanent poster |
| Exact frame selection within a measured decode-memory budget | Image sequence drawn on a canvas | None; bounded frame cache | Permanent poster |
| The sequence is atmosphere, not explanation | A short clip that plays once when visible | None | Poster |

Scrubbing needs a fine pointer and motion allowed. Coarse pointers and reduced
Expand All @@ -29,29 +29,34 @@ copy in the document, never inside the frames.

## Interpolate only when it helps

RIFE (`ECCV2022-RIFE`, MIT, Python) doubles the frame rate of a sparse render with
`--exp=1`. Check hard edges and thin lines for ghosting before keeping the result,
RIFE ([ECCV2022-RIFE](https://github.com/hzwer/ECCV2022-RIFE), MIT, Python) doubles
the frame rate of a sparse video with `--exp=1`. Check hard edges and thin lines for ghosting before keeping the result,
and keep the original frames. Interpolation cannot repair a camera path that was
too fast to read; shorten the move instead.

## Encode with FFmpeg

Scrubbing seeks to arbitrary times, so every frame must be independently
decodable. The clip grows, which is why the sequence stays short.
Scrubbing benefits from frequent keyframes; it does not require every frame to be
a keyframe. Start with a short GOP (15 frames at 30 fps below), then compare seek
latency and bytes on target devices. Use `-g 1` only when its larger download buys
a measured improvement. Use the source frame rate in place of 30.
The commands require `libx264`, plus `libvpx-vp9` and `libwebp` for the optional
WebM and WebP outputs; check the installed build with `ffmpeg -encoders`.

```sh
# scrub clip: every frame is a keyframe, no audio, streaming header first
ffmpeg -framerate 30 -i frame_%04d.png -c:v libx264 -g 1 -crf 20 \
# scrub clip: keyframe at most every half-second, no audio, streaming header first
ffmpeg -framerate 30 -i frame_%04d.png -c:v libx264 -g 15 -crf 20 \
-pix_fmt yuv420p -movflags +faststart -an scrub.mp4

# alternative codec for browsers that prefer it
ffmpeg -framerate 30 -i frame_%04d.png -c:v libvpx-vp9 -g 1 -b:v 0 -crf 32 -an scrub.webm
ffmpeg -framerate 30 -i frame_%04d.png -c:v libvpx-vp9 -g 15 -b:v 0 -crf 32 -an scrub.webm

# poster from a readable frame
ffmpeg -i scrub.mp4 -ss 00:00:02.000 -frames:v 1 -q:v 2 poster.jpg

# canvas sequence at a bounded width
ffmpeg -i frame_%04d.png -vf scale=960:-2 -quality 80 seq_%04d.webp
ffmpeg -framerate 30 -i frame_%04d.png -vf "scale='min(960,iw)':-2" \
-c:v libwebp -quality 80 seq_%04d.webp
```

FFmpeg is LGPL, or GPL when built with GPL components. Run it as a tool during
Expand All @@ -65,18 +70,28 @@ production; do not bundle it into the site.
states the clip size in the handoff.
- Decoded memory for canvas sequences: frames multiplied by width, height and four
bytes. Sixty frames at 960 by 540 decode to about 124 MB, which is already heavy
on phones. Fewer frames or a video is usually the right answer.
on phones, before decoder and GPU overhead. Decode near the current frame into a
bounded cache and release evicted bitmaps. Fewer frames or a video is usually the
right answer.
- Network: load the scrub clip after the poster and copy are visible, and only when
the mechanism will actually run.
the mechanism will actually run. Verify byte-range responses (`206` with
`Content-Range`) and seeking on the deployed URL, not just a local file.

## Integrate under the recipe rules

- The poster is in the document before any script runs and stays behind the stage.
- Wait for metadata and one decoded frame before the first seek; seek on
`requestVideoFrameCallback` or a frame callback, never inside the scroll handler.
- Wait for metadata and one decoded frame (`loadeddata`/`canplay`, `readyState >= 2`)
before seeking. Coalesce scroll, resize and restored-position updates into one
`requestAnimationFrame`; keep only the latest requested time while a seek is in
flight and apply it on `seeked`. Clamp to the last frame, below `duration`.
[`requestVideoFrameCallback`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement/requestVideoFrameCallback)
observes decoded frames; it must not be the only driver of seeks in a paused
video, because no new frame arrives until a seek starts.
- Scroll progress maps to time or frame index with one owner, reversible, and holds
still while nearby copy must be read.
- Failure of the clip, the frames or the decoder leaves the poster and all copy.
- Cancel owned callbacks, remove listeners and release media/cache resources when
the mechanism becomes ineligible or unmounts.

## Proof

Expand Down
2 changes: 1 addition & 1 deletion skills/cinematic-scroll/references/recipe-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Keep the host brand, native scrolling, and readable static content.
| Lottie, Rive, animated mascot or icon set inside a story | Editorial story + designer-authored motion in the [stack shelf](stack-shelf.md) | Pause on a readable frame under reduced motion; poster fallback; no decoration-only loops |
| Smooth scroll, momentum, Lenis | The one smoothing layer allowed by the [stack shelf](stack-shelf.md) | Native scroll on touch and under reduced motion |
| Physics, gravity, collisions, bouncing objects | Interaction design + the physics rows of the [stack shelf](stack-shelf.md) | Settled composition without JS or motion; pause off screen |
| Promo film, launch video, voiceover, captions | Not this skill; the repository documents that lane in `docs/video-pipeline.md` | Website motion only here |
| Promo film, launch video, voiceover, captions | Not this skill; see the repository's [video pipeline](https://github.com/MustBeSimo/web-design-studio/blob/main/docs/video-pipeline.md), which is not installed with the skill | Website motion only here |

When the brief or the host project names a library, engine, editor or production
tool, read the [stack shelf](stack-shelf.md) for its job, integration rule, licence
Expand Down
18 changes: 13 additions & 5 deletions skills/cinematic-scroll/references/splat-scene.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,18 @@
# Splat scene

Use Gaussian splats when the subject is a real, captured place or object and the
brief supplies a capture or can produce one. A splat scene is photoreal from the
viewpoints that were captured and falls apart away from them, so the camera path is
the design. Everything in the [real-time 3D](real-time-3d.md) scene contract applies.
brief supplies a capture or can produce one. Inspect its attributes first: ordinary
XYZ/RGB point clouds and mesh PLY files are not Gaussian-splat captures. Keep them
on a suitable point/mesh renderer unless a conversion is explicitly part of the work.
A splat scene is photoreal from the viewpoints that were captured and falls apart
away from them, so the camera path is the design. Everything in the
[real-time 3D](real-time-3d.md) scene contract applies.

## Renderer

Spark (`@sparkjsdev/spark`, MIT, maintained by World Labs) renders splats as a
Three.js object, so the existing renderer, camera and lifecycle stay in charge:
Spark ([`@sparkjsdev/spark`](https://sparkjs.dev/docs/splat-mesh/), MIT, maintained
by World Labs) renders splats as a Three.js object, so the existing renderer,
camera and lifecycle stay in charge:

```js
import { SplatMesh } from "@sparkjsdev/spark";
Expand All @@ -17,6 +21,10 @@ const room = new SplatMesh({ url: "/captures/room.spz" });
scene.add(room);
```

This is attachment only: await `room.initialized` inside the scene's error-handled
loader, and enter `ready` only after a rendered frame. Adding the object does not
prove the capture loaded; retain the poster on rejection or timeout.

Supported inputs include PLY, compressed PLY, SPZ, SPLAT, KSPLAT and SOG. It targets
WebGL2 and its documentation pairs it with recent Three.js releases; pin Spark and
Three.js together and preflight the actual capture in a minimal page before building
Expand Down
Loading
Loading