Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,11 +147,19 @@ no `Math.random()`/`Date` (use `wiggle` with a seed, or pass a `seed` knob).
(video detected by src extension, plays as a clip for its `hold`, audio muted by default
unless a shot sets `volume`) — into a slideshow: crossfades + **seeded Ken Burns**
(pan/zoom) + an optional cinematic grade (vignette + scrim via gradient + blend). Returns
`{ nodes, timeline }` (owns its image/video layers, like `splitText` owns glyphs); stable
addresses `${id}-${i}`, labels `shot-${i}`/`cross-${i}`. Each layer uses the image
node's `fit: "cover"` (crop-to-fill at the image's aspect, renderer-side via
`coverRect` — no pre-cropping, any aspect); the Ken Burns keeps `scale ≥ 1` + bounded
pan so no edge shows. Pure/seeded. Image sources
`{ nodes, timeline }` (owns its image/video layers, like `splitText` owns glyphs). **Each
shot is a SELF-CONTAINED named beat `shot-${i}`** that owns only its own layer's motion
(fade-in ∥ Ken Burns ∥ fade-out); every layer starts at `opacity: 0` and adjacent shots
overlap by the crossfade via a negative `gap` in the `seq` (so `shot-${i}` t0 = the shot's
start, the address a clip `start`/anchored title resolves to). Because no shot references
another, a shot is **structurally editable by overlay and survives regen**: reorder via the
beat `order` patch, drop via `removeTimeline: ["shot-2"]` (its layer just stays invisible),
swap its image via a `nodes.<id>.src` patch — see `docs/guides/regen-contract.md` and
`examples/overlays/montage-restructure.json`. Stable addresses `${id}-${i}`, labels
`shot-${i}`/`cross-${i}`. Each layer uses the image node's `fit: "cover"` (crop-to-fill at
the image's aspect, renderer-side via `coverRect` — no pre-cropping, any aspect); the Ken
Burns keeps `scale ≥ 1` + bounded pan so no edge shows. The montage opens on a fade-up and
closes on a fade-out (symmetric → edit-safe). Pure/seeded. Image sources
don't render in `player`/artifacts → mp4 only. Demo: `examples/scenes/photo-montage.ts`
(CC0 images under `examples/scenes/photo-montage/`, NOT bundled to npm). The photo
analog of motionPreset.
Expand Down
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,30 @@ versions may change them.

## [Unreleased]

## [0.6.40] - 2026-06-20

### Added

#### Structural montage editing — reorder + remove (Spine 2c)

- **`photoMontage`/`videoMontage` rewritten so each shot is a SELF-CONTAINED named beat
`shot-${i}`** that owns only its own layer's motion (fade-in ∥ Ken Burns ∥ fade-out). Every
layer now starts at `opacity: 0`, and adjacent shots overlap by the crossfade via a negative
`gap` in the `seq` — no shot references its neighbour anymore. This makes a shot an
independent unit you can **reorder, drop, or swap by overlay and survive AI regeneration**:
- **Reorder** — the existing beat `order` patch (`timeline.shot-2.order`) re-sorts shots.
- **Remove** — a new generic compose verb **`removeTimeline: ["shot-2"]`** splices a beat/step
out of its parent by label; the `seq` re-accumulates so later shots ripple up and
label-anchored dependents (clip `start`, anchored titles) follow. A dropped shot's layer just
stays invisible (it never fades in).
- **Swap an image** — already a plain `nodes.<id>.src` patch.
- `removeTimeline` is the structural complement of the `timeline` retiming patch (parallel to
`removeNodes`); an unknown label is reported as an orphan, never a silent drop. See
`docs/guides/regen-contract.md` and `examples/overlays/montage-restructure.json`.
- The montage now opens on a fade-up and closes on a fade-out (symmetric → edit-safe). **Note:**
montage timing shifts slightly (shots overlap by the crossfade); the montage has no golden
snapshot, so this is a behaviour change in the generator output only, not a determinism break.

## [0.6.39] - 2026-06-20

### Added
Expand Down
26 changes: 26 additions & 0 deletions docs/guides/regen-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,32 @@ joint `name`s** for any character/device that survives the redesign — overlay
edits (a retimed wave, a nudged limb angle) reference those exact ids. Renaming a
joint orphans the edit, exactly like renaming a hand-authored node id.

## Structural edits (reorder / remove a beat)

Beyond patching props and timing, an overlay can change the **structure** of the
timeline — and those edits survive regen the same way, keyed by stable labels:

- **Reorder** — patch a beat's `order` (the existing `timeline.<beat>.order`
param): `{ "timeline": { "shot-2": { "order": 0 }, "shot-0": { "order": 2 } } }`
re-sorts beats within their parent `seq`. Beats move as whole units, so child
labels and any overlay edits on them ride along.
- **Remove** — `removeTimeline: ["shot-2"]` splices a beat/step out of its parent
by label. The surrounding `seq` re-accumulates, so later steps ripple up and any
label-anchored dependent (a clip `start`, an anchored title) follows. An unknown
label is reported as an orphan, never a silent drop.

These ride the addressable surface a `photoMontage` already exposes — each shot is
the named beat `shot-${i}`, so `removeTimeline: ["shot-3"]` drops a shot (its layer
just stays invisible — it never fades in) and an `order` patch reshuffles the cut,
both without touching the base. Swapping a shot's image is a plain `nodes.<id>.src`
patch. (Caveat: a node that still **anchors** to a removed label — e.g. a video
`start: "shot-3"` — must also be neutralised, by patching its `start` to a number,
or post-compose validation rejects the dangling anchor. Reordering a shot to the
first slot drops its opening fade-up, since the crossfade offset was baked for the
original order — a cosmetic detail, not a break.)

See `examples/overlays/montage-restructure.json`.

## Tooling

Three read-only commands make the address namespace queryable and the contract
Expand Down
13 changes: 13 additions & 0 deletions examples/overlays/montage-restructure.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"reframeOverlay": 1,
"name": "montage-restructure",
"target": "photo-montage",
"removeTimeline": ["shot-4"],
"timeline": {
"shot-1": { "order": 2 },
"shot-2": { "order": 1 }
},
"nodes": {
"shot-0": { "src": "photo-montage/06.jpg" }
}
}
59 changes: 58 additions & 1 deletion packages/core/src/compose.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,18 @@ export interface OverlayDoc {
autoRotate?: boolean;
}
>;
/**
* Remove timeline steps/beats by stable label (or beat name) — splice them out
* of their parent group. The STRUCTURAL complement of a `timeline` retiming
* patch: drop a montage shot (`removeTimeline: ["shot-2"]`), cut a beat, etc.
* The surrounding `seq` re-accumulates, so later steps ripple up and any
* label-anchored dependents follow. An unknown label is an orphan (did the base
* regen drop it?). Reorder is the existing `timeline.<beat>.order` patch; this
* is its delete counterpart. NB a node that still anchors to a removed label
* (e.g. a video `start: "shot-2"`) must also be neutralised — patch its `start`
* to a number — or post-compose validation rejects the dangling anchor.
*/
removeTimeline?: string[];
}

export interface ComposeReport {
Expand All @@ -82,7 +94,8 @@ export interface ComposeReport {
| "remove-node"
| "behavior-set"
| "behavior-remove"
| "add-timeline";
| "add-timeline"
| "remove-timeline";
}[];
orphans: { layer: string; address: string; reason: string }[];
warnings: string[];
Expand Down Expand Up @@ -306,6 +319,50 @@ function applyOverlay(
}
}

// --- removed timeline steps/beats: spliced from their parent by stable label ---
if (overlay.removeTimeline && overlay.removeTimeline.length > 0) {
// walk recording each labelled step / named beat with its parent + the child ref
// (splice by identity, not index, so removing siblings can't shift positions out
// from under each other).
const located = new Map<string, { parent: TimelineIR; child: TimelineIR }>();
const walkParents = (tl: TimelineIR) => {
if (!("children" in tl)) return;
for (const child of tl.children) {
if ("label" in child && child.label !== undefined) located.set(child.label, { parent: tl, child });
if (child.kind === "beat") located.set(child.name, { parent: tl, child });
walkParents(child);
}
};
if (ir.timeline) walkParents(ir.timeline);

const toRemove = new Map<TimelineIR, Set<TimelineIR>>();
let removed = false;
for (const label of overlay.removeTimeline) {
const hit = located.get(label);
if (!hit) {
orphan(
`removeTimeline.${label}`,
`unknown timeline label "${label}" — known labels: ${[...located.keys()].join(", ") || "(none)"}; did the base regeneration drop it?`,
);
continue;
}
let set = toRemove.get(hit.parent);
if (!set) toRemove.set(hit.parent, (set = new Set()));
set.add(hit.child);
applied(`removeTimeline.${label}`, "remove-timeline");
removed = true;
}
for (const [parent, children] of toRemove) {
const p = parent as { children: TimelineIR[] };
p.children = p.children.filter((c) => !children.has(c));
}
// removal shifts later steps (the seq re-accumulates) → re-infer the duration.
if (removed && overlay.scene?.duration === undefined) {
delete ir.duration;
ir.duration = compileScene(ir).duration;
}
}

// --- added nodes: appended at root, painted on top. Duplicate ids are an
// overlay authoring defect and surface via validateScene after composition.
for (const node of overlay.addNodes ?? []) {
Expand Down
81 changes: 57 additions & 24 deletions packages/core/src/montage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,18 @@
* ratio fill the frame (cropped, centered) with no distortion — no pre-cropping. The
* Ken Burns keeps `scale >= 1` with the pan bounded to the scale's slack, so an edge
* is never revealed.
*
* STRUCTURE — each shot is a SELF-CONTAINED named beat `shot-${i}` that owns only its
* own layer's motion (fade-in + Ken Burns + fade-out); every layer starts at
* `opacity: 0`. Adjacent shots overlap by the crossfade duration via a negative `gap`
* in the `seq`, so the outgoing tail and incoming head cross. Because no shot references
* another, a shot can be reordered (beat `order`), dropped (`removeTimeline`), or its
* image swapped (a `src` patch) by an overlay and survive AI regeneration of the base.
* The montage opens on a fade-up and closes on a fade-out (symmetric → edit-safe).
*/

import type { ColorStop, NodeIR, TimelineIR } from "./ir.js";
import { beat, image, par, rect, seq, tween, video } from "./dsl.js";
import { beat, image, rect, seq, tween, video, wait } from "./dsl.js";
import { linearGradient, radialGradient } from "./gradient.js";

export type KenBurns = "in" | "out" | "pan";
Expand All @@ -38,7 +46,7 @@ export interface MontageOpts {
id?: string;
/** Frame size; must match the scene size. Default 1920×1080. */
size?: { width: number; height: number };
/** Seconds each slide is held (incl. its incoming crossfade). Default 3.2. */
/** Seconds each slide is held (its full beat, incl. its fade-in/out). Default 3.2. */
hold?: number;
/** Crossfade seconds between slides. Default 0.6. */
transition?: number;
Expand Down Expand Up @@ -84,21 +92,23 @@ export function photoMontage(images: MontageImage[], opts: MontageOpts = {}): Mo
const hold = Math.max(0.5, opts.hold ?? 3.2);
const zoom = Math.max(1.001, opts.zoom ?? 1.18);
const grade = opts.grade !== false;
const T = opts.transition ?? 0.6;
const rand = makeRng((opts.seed ?? 0) + 1);

const slides = images.map(norm);
const cx = W / 2;
const cy = H / 2;

const nodes: NodeIR[] = [];
const shots: TimelineIR[] = [];

slides.forEach((slide, i) => {
const nid = `${id}-${i}`;
const n = slides.length;

// First pass — seeded framing + per-slide hold. Drawn in slide order so the RNG
// sequence is fixed → deterministic. Holds are precomputed because each crossfade is
// capped by BOTH neighbours' holds (needs the next slide's hold while building this one).
interface Frame {
slideHold: number;
kA: number; kB: number; xA: number; xB: number; yA: number; yB: number;
}
const frames: Frame[] = slides.map((slide) => {
const slideHold = Math.max(0.5, slide.hold ?? hold);
const transition = Math.min(opts.transition ?? 0.6, slideHold * 0.9);

// Seeded framing (draw in a fixed order → deterministic).
const kind: KenBurns = slide.ken ?? (["in", "out", "pan"] as const)[Math.floor(rand() * 3)] ?? "in";
const angle = rand() * Math.PI * 2;
const panFrac = 0.4 + rand() * 0.35; // 0.40..0.75 of the available slack
Expand All @@ -123,8 +133,26 @@ export function photoMontage(images: MontageImage[], opts: MontageOpts = {}): Mo
yA = cy + dy * (kA - 1) * (H / 2) * panFrac;
yB = cy + dy * (kB - 1) * (H / 2) * panFrac;
}
return { slideHold, kA, kB, xA, xB, yA, yB };
});

// Crossfade durations. The boundary fade between shot i-1 and i (`tb`) is capped by
// both neighbours so neither hold is overwhelmed; it is BOTH shot i's fade-in and shot
// i-1's fade-out, so the overlap regions line up exactly. The opening (shot 0 fade-in)
// and closing (last shot fade-out) are capped by their own hold.
const cap = (h: number) => Math.min(T, h * 0.9);
const tb = (i: number) => Math.min(T, frames[i - 1]!.slideHold * 0.9, frames[i]!.slideHold * 0.9);
const fadeIn = (i: number) => (i === 0 ? cap(frames[0]!.slideHold) : tb(i));
const fadeOut = (i: number) => (i === n - 1 ? cap(frames[i]!.slideHold) : tb(i + 1));

const box = { id: nid, src: slide.src, x: xA, y: yA, width: W, height: H, anchor: "center" as const, fit: "cover" as const, scale: kA, opacity: i === 0 ? 1 : 0 };
const nodes: NodeIR[] = [];
const shots: TimelineIR[] = [];

frames.forEach((fr, i) => {
const nid = `${id}-${i}`;
const slide = slides[i]!;
// every layer starts hidden — each shot fades itself in/out, so any shot can be first
const box = { id: nid, src: slide.src, x: fr.xA, y: fr.yA, width: W, height: H, anchor: "center" as const, fit: "cover" as const, scale: fr.kA, opacity: 0 };
nodes.push(
isVideoSrc(slide.src)
// anchor the clip's playback start to its shot label, so it ripples when the
Expand All @@ -133,19 +161,24 @@ export function photoMontage(images: MontageImage[], opts: MontageOpts = {}): Mo
: image(box),
);

const ken = tween(
nid,
{ scale: kB, x: xB, y: yB },
{ duration: slideHold, ease: "easeInOutQuad", label: `shot-${i}` },
const inDur = fadeIn(i);
const outDur = fadeOut(i);
// a self-contained shot: fade-in ∥ Ken Burns ∥ (hold then fade-out). The beat NAME is
// the stable `shot-${i}` address (its t0 = the shot's start — same semantic the Ken
// Burns label used to carry; carrying it on the beat avoids a duplicate label).
const shot = beat(
`shot-${i}`,
{ nodes: [nid], parallel: true, ...(i > 0 && { gap: -inDur }) },
[
tween(nid, { opacity: 1 }, { duration: inDur, ease: "linear" }),
tween(nid, { scale: fr.kB, x: fr.xB, y: fr.yB }, { duration: fr.slideHold, ease: "easeInOutQuad" }),
seq(
wait(fr.slideHold - outDur),
// label the crossfade INTO the next shot `cross-${i+1}` (no label on the closing fade)
tween(nid, { opacity: 0 }, { duration: outDur, ease: "linear", ...(i < n - 1 && { label: `cross-${i + 1}` }) }),
),
],
);
const shot =
i === 0
? par(ken)
: par(
ken,
tween(`${id}-${i - 1}`, { opacity: 0 }, { duration: transition, ease: "linear", label: `cross-${i}` }),
tween(nid, { opacity: 1 }, { duration: transition, ease: "linear" }),
);
shots.push(shot);
});

Expand Down
47 changes: 47 additions & 0 deletions packages/core/test/beat.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,53 @@ describe("beats", () => {
expect(c.labelTimes.get("La")?.t0).toBe(0.5);
});

it("removeTimeline splices a beat out of its seq and later steps ripple up", () => {
const base = scene({
id: "demo",
size,
nodes: [card()],
timeline: seq(
beat("a", {}, [tween("card", { opacity: 1 }, { duration: 0.5, label: "La" })]),
beat("b", {}, [tween("card", { opacity: 0.5 }, { duration: 0.5, label: "Lb" })]),
beat("c", {}, [tween("card", { opacity: 0.2 }, { duration: 0.5, label: "Lc" })]),
),
});
expect(compileScene(base).labelTimes.get("Lc")?.t0).toBe(1.0);

const { ir, report } = composeScene(base, {
reframeOverlay: 1,
name: "drop",
target: "demo",
removeTimeline: ["b"],
});
const c = compileScene(ir);
expect(report.orphans).toHaveLength(0);
expect(report.applied.some((a) => a.action === "remove-timeline" && a.address === "removeTimeline.b")).toBe(true);
expect(c.beatTimes.has("b")).toBe(false); // gone
expect(c.labelTimes.has("Lb")).toBe(false); // its child went with it
expect(c.labelTimes.get("La")?.t0).toBe(0); // a unchanged
expect(c.labelTimes.get("Lc")?.t0).toBe(0.5); // c rippled up by b's 0.5s
expect(c.duration).toBeCloseTo(1.0, 6); // two 0.5s beats remain
});

it("removeTimeline orphans an unknown label and leaves the timeline intact", () => {
const base = scene({
id: "demo",
size,
nodes: [card()],
timeline: seq(beat("a", {}, [tween("card", { opacity: 1 }, { duration: 0.5, label: "La" })])),
});
const { ir, report } = composeScene(base, {
reframeOverlay: 1,
name: "x",
target: "demo",
removeTimeline: ["nope"],
});
expect(report.orphans).toHaveLength(1);
expect(report.orphans[0]!.address).toBe("removeTimeline.nope");
expect(compileScene(ir).labelTimes.get("La")?.t0).toBe(0); // untouched
});

it("scale stretches the interior proportionally", () => {
const c = compileScene(
scene({
Expand Down
Loading
Loading