feat(ai-restyle): v2 background replacement (stacked on #35)#1
Open
vansteenbergenmatisse wants to merge 21 commits into
Open
feat(ai-restyle): v2 background replacement (stacked on #35)#1vansteenbergenmatisse wants to merge 21 commits into
vansteenbergenmatisse wants to merge 21 commits into
Conversation
Pivots AI Restyle from relight+v2v to a personalized background replacement product: one-time selfie onboarding generates 5 Gemini backgrounds; per-video uses fal.ai matting + camera-blur composite. Supersedes 2026-05-20-ai-restyle-design.md once PR mutonby#35 (v1) merges. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
12-task plan derived from the bg-replace design spec. Covers backend modules (profile store, Gemini bg-gen, clean-bg detect, fal matting, composite, pipeline rewrite, route changes, v1 deletes), frontend (profile store, Settings section, Wizard reshape), and the OpenAPI/ROADMAP/CLAUDE.md regen. Prerequisite: PR mutonby#35 (v1) merged into main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…test Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…fie) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex security review flagged two blockers on Task 6 routes: - POST /select had no X-Gemini-Key check; other mutating routes do. - POST /profile read the full selfie into RAM before size-checking; swap to the streaming chunk pattern that start_restyle already uses. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…c auth intent - Move mid-file `import re` and `from fastapi.responses import FileResponse` to module-level imports (Fix 1) - Extract shared `_run_background_generation` module-scope helper replacing the near-duplicate `_run_generation` / `_run_regeneration` nested closures; remove unused `folder` local in `regenerate_route` (Fix 2) - Add comment on `select_background_route` explaining why X-Gemini-Key is required even though the key is not used downstream (Fix 3) - Remove unused `from unittest.mock import patch` in test_profile_routes.py (Fix 4) - Expand `serve_profile_file` docstring to document the unauthenticated-by-design rationale; update OpenAPI baseline snapshot accordingly (Fix 5) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop background_prompt + lighting_prompt form fields and MAX_PROMPT_LEN constant; replace with profile_id. Add profile existence + selection guard (404/400) before any disk I/O. Wire call site to v2 pipeline signature (profile_id + fal_key). Update 2 existing tests, add 3 new contract tests, regenerate OpenAPI snapshot (291 passing). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
frame_relight, video_restyle, and frame_extract are fully retired. No remaining importers outside their own test files (verified by grep). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
AI Restyle v1 (relight + fal.ai v2v) marked retired 2026-05-21; v2 (background replacement, fal matting + composite) marked shipped with full pipeline description. OpenAPI snapshot already current (5/5 contract tests pass). CLAUDE.md MODULE-MAP regenerated via update_claude_md.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move PROFILES_ROOT out of OUTPUT_DIR so /videos static mount can't serve selfies / generated bgs. - Unify OUTPUT_DIR + UPLOAD_DIR to env-aware in main.py (matches the store + pipeline). - Drop broken original_url from restyle result (file lives in UPLOAD_DIR, not OUTPUT_DIR; frontend uses its own blob URL anyway). - regenerate now wipes stale bg-*.png so generated_count + selected_idx never reference dead files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New backend/tests/e2e/test_bg_replace_smoke.py drives the full route chain in one test (POST /profile → GET /profile poll → POST /select → POST /api/restyle → GET /api/restyle poll). Mocks Gemini bg generation + fal matting + the heavy FFmpeg ops; everything else (routes, profile_store, BackgroundTasks, state machine) runs for real. Three tests: - happy: full chain reaches completed status with bg_verdict=clean + four canonical milestone logs present. - rejects-on-no-selection: skipping /select must 400 the restyle POST. - regenerate-resets-state: regenerate clears stale selected_idx AND replenishes to 5 fresh backgrounds. Catches contract drift that per-module mocks miss (response shapes, header propagation, profile_id flow, state transitions). 276 unit/api + 3 new e2e + 1 pre-existing e2e all green in docker. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Browser-smoke uncovered: 1. BackgroundProfileSection.jsx polling only continued on generation_status === 'generating'. Right after upload the status is briefly 'pending' (before the bg-gen background task fires mark_generation_status). If the first poll lands during that window, polling stops AND no UI block matches 'pending' — the section renders only its header (blank to the user) until a manual reload. Now treats 'pending' the same as 'generating': keeps polling, shows the "Generating 5 backgrounds…" spinner. 2. Upload.jsx step-1 helper text still said "We'll relight the lighting and replace the background" — a v1 leftover. v2 doesn't relight; rewritten to describe matte + composite. 3. AIRestyle/index.jsx top-of-file docstring still mentioned "relighting … Nano-Banana + fal.ai v2v". Same pivot leftover; rewritten to "background-replace via fal.ai matting + composite". Also gitignored .tmp-screenshots/ (used by browser-smoke captures). Verified via chrome-devtools MCP: real Gemini bg-gen → 5 thumbnails render → pick bg-2 → backend confirms selected_idx=2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pivots AI Restyle from v1 (Nano-Banana relight + fal.ai v2v, mutonby PR mutonby#35) to v2: personalized background replacement with one-time selfie onboarding.
profile_idstored in browser localStorage; files inbackend/profiles/<id>/.gblur sigma=14(realistic shallow-DOF feel) → mux audio.Stacking
This PR targets
feat/ai-restyle(mutonby PR mutonby#35) on the fork. After mutonby#35 merges to mutonby/main, this branch can be rebased onto main and a fresh PR opened against mutonby:main. Diff shown here is v2-only.Backend changes
New modules:
backend/app/profile/store.py— per-profile JSON+files store with atomic writesbackend/app/ml/profile_backgrounds.py— Gemini 5-background generatorbackend/app/ml/bg_detect.py— OpenCV clean-background heuristic (warn-only)backend/app/ml/video_matte.py— fal.ai matting wrapper with SSRF guardbackend/app/video/composite.py— FFmpeg matte-over-blurred-bg compositeModified:
backend/app/restyle/pipeline.py— full rewrite (6-step v2 flow)backend/app/routes/ai_restyle.py— 5 new onboarding routes + POST /api/restyle now takes `profile_id`backend/app/main.py— `OUTPUT_DIR` + `UPLOAD_DIR` env-awareDeleted (v1 retired): `frame_relight.py`, `video_restyle.py`, `frame_extract.py` + their tests.
Frontend changes
New: `state/profileStore.js`, `Settings/sections/BackgroundProfileSection.jsx`, `AIRestyle/steps/Precheck.jsx`
Modified: `App.jsx` (route swap), `Wizard.jsx` (Precheck step), `Review.jsx` (bg_verdict warning), `ApiKeysSection.jsx` (fal.ai copy)
Deleted: `aiRestylePresets.js`, `AIRestylePresetsSection.jsx`, `Configure.jsx`
Security baseline (per global CLAUDE.md rule)
Final holistic review flagged + fixed two CRITICALs:
All LLM-CALL + STATE-MUTATING surfaces require `X-Gemini-Key`. Static profile-file serve uses regex allowlist + UUID format check + realpath traversal guard. 10MB selfie cap + 250MB video cap streamed with mid-stream 413 (no full-body buffer).
Test plan
Known follow-ups
🤖 Generated with Claude Code