Generate Test Framework

[horner]

Make a test URL package to install load example project on device for testing.

I want to be able to click a button and pull down 2-3 projects that we know to be simple, average, and complex. Average being 10-15 clips, Complex being like 80.

[horner]

Goal

One button (dev + internal builds) that downloads a known-good example project onto a
device and lands it in the drafts list — no camera, no Photos library, no 55 MB of dev-only
media in the app bundle.

Three tiers: simple, average (10–15 clips), complex (~80 clips).

---

1. Format — reuse .pulse, no new format

We already ship a self-contained draft bundle: src/features/draft-transfer/
(manifest.ts / pack.ts / unpack.ts) — a STORE-level ZIP with manifest.json +
media/*.mp4, imported today via the document picker (useDraftTransfer.importDrafts).

A test pack is just a .pulse file fetched over HTTPS instead of picked from Files.
That inherits, for free:

• manifest validation (isPulseManifest), zip-bomb cap (MAX_BUNDLE_BYTES), per-draft
  rollback, fresh draft ids on every import (re-importable, never clobbers a local draft),
  thumbnail regeneration.
• local paths minted from draftId/segIndex in file-store.writeOriginalBytes, never from
  manifest strings → no path traversal from a hostile pack. (Verified today; keep it that way.)

Media reuse is producer-side only, zero format change. unpack.ts looks up
archive[seg.original] and writes a fresh copy per segment, so several segments may point at
one archive entry — the same trick src/dev/seed.ts uses for the speed drafts today.

---

2. The packs — hard 3:00 cap, short clips

Every pack's total timeline stays under 3 minutes, so clip length is the knob, not byte
size. Clip count is what makes a tier hard.

Measured from the current corpus (ffprobe): recorder-match speed/portrait-h264 is
0.82 Mbps; the mixed-format set averages ~1.7 Mbps. (BBB at CRF encodes ~6× cheaper
than real capture, which is ~10 Mbps for 1080p30 H.264.)

pack	clips	clip len	total	distinct files	download	on device
simple	3	6 s	0:18	3 (all distinct)	~2 MB	~2 MB
average	15	2–6 s (avg 4 s)	1:00	15 (all distinct)	~13 MB	~13 MB
complex	80	1–3 s (avg 2 s)	2:40	50	~21 MB	~34 MB

Why 50 distinct files for complex (was 12)

Maximum reuse made the pack cheap but made it a worse test:

• Ordering bugs become visible. 50 different BBB scenes means a wrong order, a dropped
  segment, or an off-by-one trim is obvious on sight. 80 copies of one clip hides all of it.
• Cache bugs surface. Thumbnail paths, expo-image cover URIs and expo-video sources are
  all keyed by path/content; a corpus of near-identical files can make a stale-cache bug look
  correct. This is exactly the class of bug the revision-stamped editedSegmentRelPath exists
  to prevent.
• The merge does real work. A heavily-deduped pack collapses to a couple of signatures and
  takes the uniform passthrough fast path — i.e. it tests the easy branch at scale. 50
  distinct files across the format matrix keeps selective-conform genuinely exercised.
• It's still cheap. ~21 MB download, ~34 MB on device. The dedup was solving a problem we
  no longer have once clips are 1–3 s.

Proposed composition for complex: 50 distinct files = 30 recorder-match portrait 1080p30
h264 (different scenes, 1–3 s each) + 20 outliers spread across hevc / 60 fps / 4K /
landscape / square / rot270 / no-audio / HDR 10-bit / VFR / NTSC. The remaining 30 of the 80
segments are repeats drawn from the recorder-match pool — keeps a dominant format so the merge
still picks a sensible conform target, and still exercises the repeat path.

Fourth pack: upload-weight — recorder-exact

First, what does a Pulse project actually store? (verified in code, not raw)

	value	source
resolution	1080p pinned (CommonResolutions.FHD_16_9)	use-recorder.ts
frame rate	30 fps pinned ({ fps: 30 } constraint)	app/recorder.tsx
video bitrate	5 Mbps (targetBitRate: 5_000_000)	use-recorder.ts
codec	VisionCamera default → HEVC on modern devices (so iPhone 17)	use-recorder.ts
container	true MP4 (fileType: 'mp4'), portrait via 90° rotation matrix	use-recorder.ts
audio	AAC from the mic (no audio track at all when muted / on a call)	use-recorder.ts
imports	normalized to ≤1920 long edge, ≤30 fps, 8-bit SDR, AAC, re-encoded at 5 Mbps; anything under 8 Mbps passes through untouched	import-normalization.ts
export	passthrough join when signatures match — zero re-encode, so record-time bitrate IS upload bitrate	use-recorder.ts comment

So the stored data is already compressed and bounded — nothing raw, nothing 4K, nothing 60 fps.
That makes the worst case computable rather than guessed:

project	bytes on device ≈ bytes uploaded
3:00 at the recorder's 5 Mbps + 128k AAC	~115 MB
3:00 of imports riding the 8 Mbps passthrough ceiling	~180 MB (absolute worst case)

⚠️ Contradiction to resolve first

@morepriyam:

The codebase disagrees with itself about the recorder's codec:

• use-recorder.ts: "the codec stays on VisionCamera's default (HEVC on modern devices)"
• import-normalization.ts: "Recordings are pinned (1080p30 H.264 MP4, 5 Mbps)"
• import-pipeline.e2e.test.ts: "A recorder-signature clip (1920x1080 30 fps H.264 5 Mbps)"
• assets/dev/: portrait-1080p-30fps-h264.mp4 is labelled "EXACT recorder match"

If an iPhone 17 actually records HEVC, then our "recorder match" fixture is the wrong codec, and
every merge-fast-path test built on it has been testing the wrong signature. Task zero: record
one clip on an iPhone 17, ffprobe it, and make the fixtures match reality. Everything below
assumes the probe result; the pack profile is one variable in the script either way.

The pack

pack	clips	clip len	total	profile	on device
upload-weight	36	5 s	3:00	1080p30, 5 Mbps, AAC 128k 48 kHz stereo, MP4, rot 90 — byte-for-byte the recorder's own output	~115 MB

This is the largest project a user can actually make under the 3:00 cap. Because every
segment carries an identical signature it also exercises the passthrough merge fast path, so the
merged output is ~115 MB too — that's the real number pulsevault has to swallow for #123 /
resume / slow-network work.

Generate it from the live-action master (Tears of Steel): grain is what makes 5 Mbps look like
5 Mbps instead of compressing away to nothing.

---

2b. Source content & clip design

Two masters — and one of them talks

master	character	role
Tears of Steel (Blender, CC-BY 3.0) — primary	live action + VFX, real sensor grain, 4K, real English dialogue	realistic bitrate and encode/conform cost, and the speech source for the transcription tests
Big Buck Bunny (CC-BY 3.0) — current, keep	flat-shaded cartoon, huge clean gradients, essentially no dialogue	the cheap-encode end of the ladder, and a natural "no speech" case for caption edge tests

BBB alone is why our fixtures sit at 0.8–1.7 Mbps while the recorder stores 5 Mbps — a 3–6×
gap. A live-action master closes it, gives upload-weight realistic bytes for free (grain won't
compress away), and hands us a transcription fixture we can't fake with a cartoon.

Optionally a third for variety of palette/motion (Cosmos Laundromat, Sintel, Spring — all CC-BY).
Attribution goes in assets/dev/README.md, as BBB's already does.

Never pure test patterns

Synthetic testsrc/smptehdbars content encodes absurdly cheaply and would make every perf
number meaningless. So: real footage underneath, synthetic HUD on top.

The HUD — burn identity into every clip

Correction to a stale assumption: the fixture scripts and assets/dev/README.md say "this
ffmpeg has no freetype", which is why clips carry a sweeping bar instead of a timecode. Not
true anymore — the current Homebrew build is --enable-libfreetype and drawtext works
(verified). We can burn real text now.

Drawn in display orientation, before the transpose:

element	position	what it catches
huge clip index (07)	top-center, thumbnail-legible	ordering, drops, duplicates — at a glance and by OCR
timecode + frame number (%{pts:hms} / %{n})	bottom-left	frame-accurate trim boundaries; off-by-one at joins
format label (1080p30 h264)	bottom-right	tells you which clip caused a conform/normalization bug
per-clip hue border (index → hue)	full frame edge	ordering visible in the timeline thumbnail strip without reading anything
asymmetric corner L-mark	one corner only	rotation / flip / mirror bugs become unmistakable (symmetric content hides them)
9:16 safe-area rect	centered	crop, letterbox and scale errors
green first frame / red last frame	1 frame each	join integrity — a missing or duplicated frame at a boundary is visible and probe-able

Keep the existing sweeping progress bar too; it reads instantly at 1–3 s.

Audio — make the soundtrack an assertion

1. Real dialogue from Tears of Steel. Cut every speech-carrying clip so it lands on a known
   line, and store that line in the answer key. The merged transcript then becomes a
   machine-checkable statement of segment order — and it's the first deterministic input the
   whisper/caption path has ever had (today captions are tested against cartoon noise with no
   expected output at all).

   The cut points come from ToS's published subtitle file: parse the .srt, pick cues of the
   right length, and the cue text is the expected transcript for that clip — the answer key is
   generated, never hand-authored. (If the official subs turn out to be unusable, transcribe the
   master once with whisper on a desktop and commit that as the key instead.)

   Real dialogue beats synthesized speech here: it comes with score, SFX, accents and overlapping
   audio, which is what whisper actually has to survive. Comparison must therefore be fuzzy —
   normalize case/punctuation and assert word error rate under a threshold, plus cue timing
   within a tolerance — not string equality.

2. A 50 ms sine blip in each clip's lead-in, pitched per clip (300 + idx*25 Hz), placed
   before the dialogue starts so it can't corrupt the transcript. Enough for
   silencedetect/astats to locate every join in the merged file → measured A/V drift and
   boundary timing vs. the manifest.

3. One silent clip (BBB has long dialogue-free stretches) and one mono 44.1 kHz clip per
   pack so the no-audio and resample branches stay covered.

   No TTS, no macOS say — pack generation stays pure ffmpeg and runs anywhere.

Shuffle everything that isn't sort_order

Pretend the clips were captured out of order and dragged into place afterwards — which is what
users actually do. Three independent scrambles, all from one fixed seed so packs stay
reproducible:

1. Scene order ≠ clip index. Clip 07 is cut from late in the master, 08 from early. The
   burned-in index is then the only truth about intended order — nobody (and no reviewer) can
   "fix" an ordering bug by following the story, and a merged output that looks narratively
   smooth is still provably wrong.
2. Archive/filename order ≠ timeline order. Manifest entries are emitted shuffled
   (media/d0-s47.mp4 sitting at array position 3) with explicit order fields carrying the
   real timeline positions. unpack.ts already prefers seg.order over array index and
   segmentsForDraft already does orderBy(asc(segments.order)) — this pack makes sure it
   stays that way, and catches anything that quietly sorts by insertion, filename or path.
3. Timestamps ≠ order. ZIP entry mtimes are written in yet another shuffled order (fflate
   takes a per-entry mtime), so any code that reaches for file time as a proxy for sequence
   breaks loudly instead of accidentally working.

Note the schema has no per-segment createdAt — timestamps live on the draft — so scramble #3
covers on-disk/archive time only. That's still where the accident would happen.

The pack ships an expected.json sidecar — the true index sequence, per-clip durations,
cumulative offsets, and each clip's expected dialogue line — that the verify script asserts the
OCR and transcript read-backs against. The scrambling is useless without a machine-readable
answer key.

Generation

scripts/make-test-packs.sh cuts the palette from the two masters at the same encoder settings
as the existing generators, just shorter: 1/2/3/4/6 s cuts × the format matrix (portrait h264 /
hevc / 60 / 4K, landscape 1080 / 4K, plus the wild-import variants), each from a different
scene, each with its own HUD, and — for the ToS cuts — aligned to a subtitle cue so the clip
carries a known line. Deterministic and regenerable → timings are comparable across devices,
builds and releases.

---

3. Hosting

GitHub Release assets on mieweb/pulse, under a dedicated non-app tag fixtures-v1.

https://github.com/mieweb/pulse/releases/download/fixtures-v1/index.json
https://github.com/mieweb/pulse/releases/download/fixtures-v1/pulse-simple.pulse
https://github.com/mieweb/pulse/releases/download/fixtures-v1/pulse-average.pulse
https://github.com/mieweb/pulse/releases/download/fixtures-v1/pulse-complex.pulse

Why: free, CDN-backed, immutable per tag, no Git-LFS bandwidth billing, no new infra — and it
keeps ~50 MB of media out of the repo and out of the app bundle. Not pulsevault (that's the
system under test — don't make the fixture host and the SUT the same box). Not S3/R2 unless we
later need per-tester auth.

index.json is the catalog the app reads:

{
  "format": "pulse-test-catalog", "version": 1, "generatedAt": 1750000000000,
  "packs": [{
    "id": "complex", "tier": "complex", "title": "Complex — 80 clips",
    "description": "80 segments, 2:40, mixed formats. ~34 MB on device.",
    "url": "https://github.com/.../pulse-complex.pulse",
    "bytes": 21000000, "sha256": "…", "clipCount": 80, "durationMs": 160000,
    "bundleVersion": 1              // must be <= the PULSE_VERSION the app understands
  }]
}

Publishing: make-test-packs.sh builds the packs + index.json; a manual-dispatch GitHub
Action runs it, validates each pack (unzip + isPulseManifest in Node), and uploads to the
fixtures-vN release. Packs are never committed — a new corpus is a new tag (fixtures-v2),
and old builds keep working because their catalog URL is pinned to the tag they shipped with.

---

4. In-app

New src/features/test-packs/:

• catalog.ts — pure fetch + validate of index.json (scheme/host/schema checks). Unit-testable.
• download.ts — File.downloadFileAsync into cache with progress + cancel, size cap, sha256
  verify, then hand the local uri to the existing importPulseFile.
• use-test-packs.ts — idle → fetching-catalog → downloading(pct) → importing(n/N) → done.
• UI: replace the five-button dev row (src/dev/dev-seed-row.tsx) with a "Test packs" entry
  opening a sheet that lists the catalog with size / clip count / duration and a per-row install
  button, plus the existing clear. Importing 80 clips runs 80 native thumbnail generations —
  needs real progress, not a spinner, and must survive backgrounding.

Availability. Not __DEV__-only — the point is testing real/TestFlight builds. Gate on
EXPO_PUBLIC_TEST_PACKS=1 (set for internal/preview channels), __DEV__, or a hidden unlock
(7 taps on the version row). Off in App Store production; the module stays behind the same
deferred require() pattern used today so it never lands in the production bundle.

Deep link / QR. Extend the versioned parser in src/features/upload/deep-link.ts
(pulsecam://?v=1&…) with a second action:

pulsecam://testpack?v=1&pack=complex             # names a CATALOG ENTRY, not a URL
pulsecam://testpack?v=1&url=https://…&sha256=…   # dev / private-host builds only

The link carries an id, not an arbitrary URL, so a link can't point the app at an
attacker-chosen host. QR codes for the packs go in assets/dev/README.md — a tester scans and
installs, no Xcode, no Files app. Reuses isPrivateDevOrigin for the local-server case.

---

5. Security constraints (non-negotiable)

• HTTPS only; host allowlist (github.com, objects.githubusercontent.com) outside dev builds.
• sha256 from the catalog verified before unzip, hashed off the JS thread — mirror the
  reasoning in upload-manager.md5Checksum (never file.bytes() on a large file).
• Declared-size cap before download; MAX_BUNDLE_BYTES during unzip (already there); hard
  clip-count cap (~200) so a bad pack can't fill the device.
• Explicit confirm dialog with name + size before anything is written.
• Media only: manifest-referenced entries must end in .mp4; ignore all other archive entries.

---

6. How we use it in testing

1. Manual QA matrix — each tier × {import, timeline scroll/reorder, trim, merge/export,
   transcription, upload + kill/resume, save to Photos}. One tester command to set up.

2. Automated verification of the merged output — the HUD makes the export self-checking.
   Pull the merged file off the device (or off pulsevault) and run scripts/verify-merge.sh:

   probe	asserts
   ffmpeg -vf "fps=1,ocr,metadata=mode=print"	the burned-in clip indices appear in the expected order, for the expected durations
   whisper transcript vs. expected.json (fuzzy, WER threshold)	segment order and caption timings — clip 07's dialogue line falls where OCR says clip 07 is
   silencedetect / astats on the tone blips	every join is where the manifest says, A/V drift within tolerance
   ffprobe on the output	resolution / fps / codec / rotation of the conform result

   Verified this works end to end before proposing it: burned CLIP 07 with drawtext, read it
   back with the ocr filter at 95% confidence. That turns "did the 80-clip merge come out
   right?" from an eyeball question into a script.

3. Perf baselines — per tier: import time, merge time, peak memory, output size, upload wall
   time. Table in docs/, regenerated per release and diffed. Deterministic packs are what make
   the numbers comparable — this is what makes Segment bar: simplify gesture surface, keep scroll position on delete, stable per-clip numbers #126/Segments record silent audio after in-recorder preview — expo-video flips AVAudioSession to .playback #124-class regressions visible.

4. Unit tests (jest, pure) — catalog schema validation, host/scheme rejection, sha256
   mismatch, oversize rejection, deep-link action parsing, manifest reuse → N segment rows.
   Same shape as deep-link.test.ts / import-normalization.test.ts.

5. CI pack validation — the publish workflow unzips + validates before upload, so a
   malformed pack never reaches a device.

6. E2E bootstrap — a deep-link install is the deterministic setup step for any future
   Maestro/Detox flow, and gives Pulse App on Mac IOS #128 (Pulse on Mac/iOS) a one-tap way to get real content.

---

7. Phasing

• P1 — scripts/make-test-packs.sh + catalog schema + publish workflow + fixtures-v1.
• P2 — src/features/test-packs/ download → verify → importPulseFile, progress UI, gating.
• P3 — pulsecam://testpack action + QR codes + docs + first perf-baseline table.
• P4 — retire the bundled dev fixtures (assets/dev/**.mp4, ~55 MB in LFS and in every dev
  build) and the + seed/+s2/+s3/+s4 buttons, re-expressing s2/s3/s4 as hosted packs. Ties into
  docs/app-size-research.md.

---

Decisions

1. Hosting is public. ✅ GitHub Release assets, world-downloadable. BBB / Tears of Steel are
   CC-BY, attribution lives in assets/dev/README.md.

2. upload-weight is recorder-exact and ships in P1, not deferred — see §2. Blocked on
   probing a real iPhone 17 clip to settle the H.264-vs-HEVC contradiction.

3. .pulse manifest stays at v1 — no per-entry hashes. (The question was: the bundle's
   manifest.json records each clip's path and duration but no checksum, so a single corrupted
   clip inside an otherwise-valid pack imports silently as a broken segment. Adding a per-entry
   sha256 would let unpack.ts name the bad clip. Not worth it: the whole pack is already
   sha256-verified from the catalog before it's opened, ZIP carries a CRC32 per entry that
   fflate checks anyway, and a format bump costs us cross-version compatibility with drafts
   shared between devices. Revisit only if we see real-world partial corruption.)

4. Tears of Steel is in, as the primary master. ✅ Live action supplies both the realistic
   bitrate/grain and the real dialogue for the transcription answer key. No TTS — pack
   generation stays pure ffmpeg (no macOS say, runs on any CI box). Cost: another ~700 MB LFS
   object in fixtures/, excluded from normal clones like BBB, so clone cost is unchanged.