From cc69a1ddcdf256502fd89e601caf5171d8629344 Mon Sep 17 00:00:00 2001 From: abrahamhl <269382359+abrahamhl@users.noreply.github.com> Date: Sat, 5 Sep 2026 05:58:45 +0000 Subject: [PATCH 1/2] Turn loopsmith into Generative Media Engineering case study Audited current FFmpeg/ffprobe pipeline, cut search, and exact-duration assembly. Added synthetic fixtures using ffmpeg lavfi to create good/bad seam examples safely. Added automated tests verifying exact duration, frame correctness, and seam score determinism. Added CI configuration to run tests. Created documentation outlining the architecture, generative media workflow, and QA evaluation. Created ISMAJAKI case study template separating technical tooling, generated assets, and creative direction. Created recruiter evidence explaining paleta.sh as a creative control tool. Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> --- .github/workflows/ci.yml | 26 ++++++++++++++ RECRUITER_EVIDENCE.md | 29 +++++++++++++++ docs/ARCHITECTURE.md | 28 +++++++++++++++ docs/CREATIVE_QA.md | 22 ++++++++++++ docs/GENERATIVE_MEDIA_WORKFLOW.md | 24 +++++++++++++ docs/ISMAJAKI_CASE_STUDY_TEMPLATE.md | 23 ++++++++++++ docs/SEAM_SCORING.md | 25 +++++++++++++ loopsmith.sh | 0 tests/.gitignore | 2 ++ tests/generate_fixtures.sh | 12 +++++++ tests/run_tests.sh | 54 ++++++++++++++++++++++++++++ 11 files changed, 245 insertions(+) create mode 100644 .github/workflows/ci.yml create mode 100644 RECRUITER_EVIDENCE.md create mode 100644 docs/ARCHITECTURE.md create mode 100644 docs/CREATIVE_QA.md create mode 100644 docs/GENERATIVE_MEDIA_WORKFLOW.md create mode 100644 docs/ISMAJAKI_CASE_STUDY_TEMPLATE.md create mode 100644 docs/SEAM_SCORING.md mode change 100644 => 100755 loopsmith.sh create mode 100644 tests/.gitignore create mode 100755 tests/generate_fixtures.sh create mode 100755 tests/run_tests.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..4c319aa --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,26 @@ +name: CI + +on: + push: + branches: [ "jules/*" ] + pull_request: + branches: [ "jules/*" ] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Install FFmpeg + run: sudo apt-get update && sudo apt-get install -y ffmpeg + + - name: Generate Fixtures + run: | + cd tests + bash generate_fixtures.sh + + - name: Run Tests + run: | + cd tests + bash run_tests.sh diff --git a/RECRUITER_EVIDENCE.md b/RECRUITER_EVIDENCE.md new file mode 100644 index 0000000..0fa0767 --- /dev/null +++ b/RECRUITER_EVIDENCE.md @@ -0,0 +1,29 @@ +# Generative Media Engineering: loopsmith Case Study + +`loopsmith` is an auditable, programmatic toolkit demonstrating advanced generative media engineering. + +For creative technology studios or senior video/AI tooling engineers (e.g., at Krea, ElevenLabs, Figma), this repository provides concrete evidence of technical and creative problem-solving in production pipelines. + +## The Problem Addressed +Generative video models (Kling, Runway, Sora) do not deliver exact durations. Attempting to concatenate them creates stutters, frozen frames, or jarring jumps. Manual editing is subjective and inefficient at scale. + +## The Engineering Solution +`loopsmith` provides a predictable, auditable, CLI-driven pipeline to automate the assembly of fixed-length generative clips. +- Uses `ffmpeg`/`ffprobe` to compute motion-energy and baseline references at candidate cut seams. +- Exhaustively searches the permutation space to drop frames required to hit an exact duration constraint. +- Evaluates seams by comparing seam difference to local motion baselines, targeting a ratio of `1.0` (rather than absolute difference, which causes frozen frames). +- Implements `--motion-cut` functionality to optionally hide dropped frames inside high-motion action sequences instead of at the clip seams. + +## paleta.sh: A Creative Control Tool +The repository includes `paleta.sh`, a utility proving that generative tooling isn't just about cutting clips; it's about controlling model conditioning. + +Image conditioning in generative models is holistic. Passing an image for color style often overrides negative text prompts, accidentally introducing unwanted subject matter. `paleta.sh` solves this by programmatically extracting the palette from a reference image and converting it into precise prompt language. + +It categorizes colors into: +1. **Accents:** High saturation, visually defining colors (the prompt). +2. **Base Tones:** High-area, low saturation colors. + +By separating the palette/style conditioning from the scene/content when iterating generative models, it offers the engineer true creative control. + +## Documentation & Traceability +This repository incorporates templates (like the ISMAJAKI framework) and detailed creative QA guides to ensure that Human Selection, Creative Direction, and Technical Tooling are easily distinguishable and auditable. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..3539716 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,28 @@ +# loopsmith Architecture + +`loopsmith` is an auditable Generative Media Engineering case study. It solves a specific problem in generative media: constructing exact-duration loops from generative clips, without stutters or jumps. + +## FFmpeg/ffprobe Pipeline + +The core logic relies entirely on `ffmpeg` and `ffprobe`. This prevents the need for heavy CV frameworks and makes the solution easy to deploy. + +- `ffprobe` is used with `lavfi` and `tblend=all_mode=difference` to extract absolute frame differences. This allows for measuring motion energy. +- `ffmpeg` is used with `filter_complex` (`select`, `trim`, `concat`) to non-destructively trim frames and concatenate the resulting clips. + +## Cut Search & Exact-Duration Assembly + +Generative tools often return an arbitrary number of frames slightly off the target duration (e.g. 361 frames instead of 360). +To achieve exact-duration assembly: +1. `loopsmith` enumerates candidate seams by distributing the total frames to drop (`DROP`) across possible cut points (clip head and tails). +2. It scores the seam cost for each permutation. +3. The lowest-cost permutation is selected. + +Alternatively, `loopsmith` can cut on the motion (using `--motion-cut`). In this strategy, seams are preserved intact. Instead, the extra frames are removed from the middle of the clip at the fastest moving points, where motion blur masks the frame deletion. + +## Error Handling + +`loopsmith` is built to run reliably: +- Validates inputs before running (e.g. checking file existence). +- Returns errors when missing `ffmpeg` or `ffprobe`. +- Handles edge cases when dropping frames is impossible because `MAXDROP` constraints are too tight. +- Exits explicitly and logs errors to stderr when problems arise. diff --git a/docs/CREATIVE_QA.md b/docs/CREATIVE_QA.md new file mode 100644 index 0000000..6b005d4 --- /dev/null +++ b/docs/CREATIVE_QA.md @@ -0,0 +1,22 @@ +# Creative Quality Assurance (QA) + +When working with generative media, quantitative metrics are essential but human creative evaluation is still necessary. + +## The Seam Score Thresholds + +`loopsmith` will output seam quality verdicts based on motion-energy ratios: +- **0.75 - 1.35:** "excelente". The cut matches the surrounding motion. +- **0.50 - 1.60:** "correcta". Acceptable. +- **< 0.50 or > 1.60:** "REVISAR" (Review). + +## Manual Review Criteria (REVISAR) + +When a seam is marked as "REVISAR", you should manually inspect it: + +1. **Jumps (>1.6):** Does the cut break spatial continuity? The motion might be too dramatic to hide within the frames. +2. **Stutters (<0.5):** Did the generative model create a still frame, or is the cut artificially freezing the action? +3. **Pacing and Context:** Is this a slow-moving, settling shot? High ratios are more visible in slower scenes. You might need to use `--motion-cut` to shift the deleted frames into a fast-paced segment where motion blur can hide the edits. + +## Generating Better Sources + +If no cut combination yields a good score, the solution is not more editing. The missing motion was never generated. Go back to the prompt and ensure the camera speeds align at the ends of your generated clips. diff --git a/docs/GENERATIVE_MEDIA_WORKFLOW.md b/docs/GENERATIVE_MEDIA_WORKFLOW.md new file mode 100644 index 0000000..eb62054 --- /dev/null +++ b/docs/GENERATIVE_MEDIA_WORKFLOW.md @@ -0,0 +1,24 @@ +# Generative Media Workflow + +An auditable workflow for creating fixed-length video loops using generative tools. + +## 1. Generation +Generate the base media clips via a generative model (e.g., Kling, Runway). These clips may have variable frame durations, slight color shifts, and non-continuous motion at the edges. + +## 2. Candidate Clips +Store the output video clips. Note that if you use tools like `paleta.sh`, this is the step where you extract the reference colors from the desired asset before feeding it into the generator to isolate style conditioning. + +## 3. Analysis +Analyze the clips using `ffprobe` to determine the motion energy curve and check for variations across frame durations or resolutions. This sets the baseline parameters for cutting. + +## 4. Seam Search +Perform an exhaustive combinatorial search for the best seam cuts. The search considers clipping at the beginning or end of clips to find the exact combination of frames that reach the exact target frame count. + +## 5. Exact Loop +Assemble the final video based on the selected seams. The resulting sequence will match the exact requested duration, maintaining strict continuity for any synced downstream processes (e.g. VJ Grids). + +## 6. QA (Quality Assurance) +Review the seams output ratios. If any ratio falls outside the [0.5, 1.6] bounds, it should be manually reviewed (marked "REVISAR"). See `CREATIVE_QA.md` for criteria. + +## 7. Final Output +Output the cleanly assembled, exact-duration clip encoded in the final presentation format. diff --git a/docs/ISMAJAKI_CASE_STUDY_TEMPLATE.md b/docs/ISMAJAKI_CASE_STUDY_TEMPLATE.md new file mode 100644 index 0000000..71dc453 --- /dev/null +++ b/docs/ISMAJAKI_CASE_STUDY_TEMPLATE.md @@ -0,0 +1,23 @@ +# ISMAJAKI Case Study + +This template provides an auditable structure to separate the different phases and roles in a generative media engineering project. + +**Project Name:** [Project Name] + +## 1. CREATIVE DIRECTION +*What was the creative goal? Provide the storyboard or initial vision.* + +## 2. GENERATED ASSET +*Provide the raw outputs from the generative model. Include exact duration, frame count, resolution, and any generated artifacts (e.g. stutters at clip boundaries).* + +## 3. TECHNICAL TOOLING +*Which tools were utilized to modify or assemble the assets? (e.g., `loopsmith`, `paleta.sh`). Document the specific flags used, motion energy findings, and seam metrics.* + +## 4. HUMAN SELECTION +*Detail the rationale behind the chosen edits. Why did the creative director or engineer select this specific cut or `paleta.sh` configuration? Document any "REVISAR" seam scores and why they were accepted or discarded.* + +## 5. REJECTED VARIANT +*Document a variant that was rejected due to tooling limitations or creative decisions. For example, show a seam that resulted in a >1.6 ratio jump that broke continuity.* + +## 6. FINAL OUTPUT +*Link to the final, exact-duration assembled loop.* diff --git a/docs/SEAM_SCORING.md b/docs/SEAM_SCORING.md new file mode 100644 index 0000000..6460d26 --- /dev/null +++ b/docs/SEAM_SCORING.md @@ -0,0 +1,25 @@ +# Motion-Aware Seam Scoring + +Finding the best cut when joining generative clips is difficult to do by eye. `loopsmith` solves this by programmatically scoring seam quality. + +## Methodology + +`loopsmith` measures "motion energy" — the absolute mean difference in luminance between two frames downscaled to a low resolution (e.g., 320x180). This suppresses compression noise but keeps large motion continuous. + +The core insight for seam scoring is that **a seam must match the motion around it**. A difference that would be invisible during a fast camera pan is glaring during a slow static shot. + +The tool calculates a local baseline for motion by averaging the motion energy of standard consecutive frames on *both* sides of the cut. + +Then it calculates the cost: +``` +cost = |seam_difference / normal_step − 1| +``` + +## Why 1.0 is the Target + +Unlike typical difference minimisation that targets 0, `loopsmith` targets 1.0. + +- If the difference is too high (>1.6), it is a visible jump. +- If the difference is too low (<0.5), it is a freeze or stutter. + +Minimising absolute difference would actively select for freezes. By targeting a ratio of 1.0, `loopsmith` ensures that the cut moves just as naturally as any other frame in the scene. diff --git a/loopsmith.sh b/loopsmith.sh old mode 100644 new mode 100755 diff --git a/tests/.gitignore b/tests/.gitignore new file mode 100644 index 0000000..8c9791d --- /dev/null +++ b/tests/.gitignore @@ -0,0 +1,2 @@ +*.log +*.mp4 diff --git a/tests/generate_fixtures.sh b/tests/generate_fixtures.sh new file mode 100755 index 0000000..7177cdc --- /dev/null +++ b/tests/generate_fixtures.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +set -euo pipefail + +echo "Generating good seams (continuous motion)..." +ffmpeg -y -v error -f lavfi -i "smptebars=duration=15:size=1280x720:rate=24" -vf "scroll=h=0.05" good_1.mp4 +ffmpeg -y -v error -f lavfi -i "smptebars=duration=15:size=1280x720:rate=24" -vf "scroll=h=0.05" good_2.mp4 + +echo "Generating bad seams (jump cuts)..." +ffmpeg -y -v error -f lavfi -i "color=c=red:duration=15:size=1280x720:rate=24" bad_1.mp4 +ffmpeg -y -v error -f lavfi -i "color=c=blue:duration=15:size=1280x720:rate=24" bad_2.mp4 + +echo "Done." diff --git a/tests/run_tests.sh b/tests/run_tests.sh new file mode 100755 index 0000000..f120b0c --- /dev/null +++ b/tests/run_tests.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +set -e + +cd "$(dirname "$0")" + +echo "Running tests..." + +echo "Test: Candidate seam enumeration" +../loopsmith.sh --dry-run --duration 30 good_1.mp4 good_2.mp4 > good_dry.log 2>&1 || true +if ! grep -q "midiendo costuras..." good_dry.log; then + echo "FAILED: Did not find 'midiendo costuras...' in output" + cat good_dry.log + exit 1 +fi +echo "PASSED: Candidate seam enumeration" + +echo "Test: Final duration correctness & Frame/timebase correctness" +../loopsmith.sh --out good_loop.mp4 --duration 30 good_1.mp4 good_2.mp4 > good_loop.log +frames=$(ffprobe -v error -select_streams v -count_frames -show_entries stream=nb_read_frames -of csv=p=0 good_loop.mp4) +if [ "$frames" -eq 720 ]; then + echo "PASSED: Final duration correctness ($frames frames)" +else + echo "FAILED: Expected 720 frames, got $frames" + exit 1 +fi + +echo "Test: Seam score determinism" +../loopsmith.sh --dry-run --duration 30 good_1.mp4 good_2.mp4 > run1.log +../loopsmith.sh --dry-run --duration 30 good_1.mp4 good_2.mp4 > run2.log +if cmp run1.log run2.log; then + echo "PASSED: Seam score determinism" +else + echo "FAILED: Output differs between runs" + exit 1 +fi + +echo "Test: Weak-seam detection" +../loopsmith.sh --dry-run --duration 30 bad_1.mp4 bad_2.mp4 > bad_dry.log +if grep -q "REVISAR" bad_dry.log; then + echo "PASSED: Weak-seam detection (Found REVISAR)" +else + echo "FAILED: Did not find REVISAR warning on bad seam" + exit 1 +fi + +echo "Test: Failure on invalid input" +if ../loopsmith.sh --dry-run missing.mp4 2>/dev/null; then + echo "FAILED: Expected failure on missing input" + exit 1 +else + echo "PASSED: Failure on invalid input" +fi + +echo "All Tests Passed!" From 171ebc5b200222e4fd863451afbe75d7ee4c9259 Mon Sep 17 00:00:00 2001 From: abrahamhl <269382359+abrahamhl@users.noreply.github.com> Date: Sat, 5 Sep 2026 06:27:19 +0000 Subject: [PATCH 2/2] Fix CI branch configuration and upgrade actions checkout Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> --- .github/workflows/ci.yml | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4c319aa..bbeaba8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,15 +2,21 @@ name: CI on: push: - branches: [ "jules/*" ] + branches: + - main + - 'jules/**' pull_request: - branches: [ "jules/*" ] + branches: + - main + +permissions: + contents: read jobs: test: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 - name: Install FFmpeg run: sudo apt-get update && sudo apt-get install -y ffmpeg