Skip to content

Render native stickers and bounded animated previews - #162

Merged
nixfred merged 4 commits into
nixfred:mainfrom
marcus:upstream-stickers
Nov 10, 2026
Merged

nixfred merged 4 commits into
nixfred:mainfrom
marcus:upstream-stickers

Conversation

@marcus

@marcus marcus commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Renders native stickers separately from photos: transparent standalone media, additive placements on parent bubbles, and a compact strip for reaction/unplaceable stickers. Bounded metadata exposes only selected fields; unknown media keeps a labeled fallback.

PNG, GIF and WebP preserve transparency/animation. HEIC PNG stills are bounded on the Mac before transfer. APNG and recognized HEICS/MOV sequences can produce lossless WebP previews; both MOV probe and decode explicitly select the demuxer and enforce decoder pixel limits. Output frame ceilings independently enforce the decoded-pixel budget when container counts lie. Failed previews are cached; unsupported animation retains a still or explicit original Open. Ordinary attachment cache behavior and legacy type-1000 unread/toast behavior are preserved. Qt WebP remains optional, including discovery through QT_PLUGIN_PATH.

Based on current upstream main, including #158/#161, preserving its UI, controls and manifest. Save documentation distinguishes original media from converted PNG/HEICS stills. Details are in docs/STICKERS.md.

Validation: full Bun and synthetic Mac Python suites pass after rebase, plus 31 focused sticker tests and 6 real-codec/Qt rendering checks. Independent thresholds, valid oversized plists, concat refusal, failed conversion caching and ordinary 6 MB GIF cache hits are covered. Rendered synthetic previews inspected; no real conversations opened or messages sent. GitHub CI passes on the updated head.

@nixfred

nixfred commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Thank you, Marcus. This is careful work: bounded plists, parameterised SQL, a 0600 cache, extensions that follow the checked MIME, and a cache that counts its derived files. I rendered the branch in a clean Omarchy VM with your demo fixtures, and the transparent star, the corner placement and the reaction strip all look right.

It cannot merge as it stands, because of one blocking problem and a handful of smaller ones. An independent review found them, and I reproduced the first one myself.

Blocking

  1. stickerSequencePreview in fetch.ts runs ffprobe and ffmpeg on bytes a stranger chose, without telling either which format to expect, so ffmpeg guesses from the content. A two-line file, ffconcat version 1.0 then file 123-orig-secret.png, delivered as a video, came out as a PNG of a different file in the cache directory (identical to it). -protocol_whitelist file,pipe does not stop that, and it needs no click because video stickers up to 5 MB are now auto-fetched. Please add -f mov before -i on both calls (checked: it rejects that payload and still reads a real MOV) and -max_pixels 4194304 (checked: it is enforced).

Should fix

  1. The pixel and frame budget trusts the frame count the file reports about itself. A 64-frame 1024 by 1024 APNG with acTL patched to 2 converted to 64 frames, 67.1M pixels, over the stated budget. The real limit is only -frames:v 100 plus the 30 s timeout. A failed conversion is also never remembered, so every thread open runs ffmpeg again. docs/STICKERS.md promises 64 million decoded pixels; that is not what is enforced.
  2. The new cache-hit size gate (fetch.ts line 591) applies to every attachment, not only stickers. A 6 MB GIF that Open already cached is a cache hit on main and fails the 5 MB auto-fetch cap on this branch, costing an ssh pull. Please scope the gate to sticker.
  3. --png goes out without --max-dim, so a large HEIC sticker comes back full size over ssh and is then thrown away by the 2048 px gate on every open. --max-dim 2048 fixes both ends (sips never upscales).
  4. Legacy type-1000 rows are now tapback: true (bridge/mac/imsg line 1356), so collector.ts stops counting and toasting them. On main they counted. If that is intended, please say so in the docs and add a test; deleting the == 1000 clause leaves every test green today.
  5. Several documented limits have no test that fails when they change: HEICS count <= 100, seconds <= 8, the 64M budget, the 2048 px checks (the validSize test passes only because the mask-size mismatch rejects first), and the 64 KB plist cap. A one-line change to each leaves the suite green.
  6. scripts/blip-setup: a missing Qt WebP plugin is now a hard stop while ffmpeg stays optional. A warning would match how the rest of the script treats optional pieces, and the check should also honour QT_PLUGIN_PATH, not only the four fixed directories.

Smaller notes: save-attachment.ts now accepts -png-, which for HEICS is a single-frame still rather than the original, so the "Save keeps the original" wording in CLAUDE.md needs a line about it. Tapback.kind declares "classic" and "sticker", which nothing emits.

Order. I expect to land #158 and #161 first, and both are clean. This PR then needs a small rebase: the sentReceipt lines in dedupeSelfEcho and one comment in BlipView.qml. I resolved both in a scratch merge and all 840 tests plus the Qt checks pass on that stack, so I can send you the resolved hunks if that helps. #163, #159 and #160 conflict with this one and will follow it in that order.


Generated by Claude Code

nixfred added a commit that referenced this pull request Nov 10, 2026
…ll 4x faster (CHANGELOG, CHANGE.log, roster, weekly notes)

WHAT: CHANGELOG entries for #158 (Delivered until Read) and #161 (composer emoji picker) with the real end-to-end numbers for the poll speed-up (shallow 0.77 s to 0.2 s, deep 2.1 s to 1.0 s on gus); CHANGE.log entries for 401b99d, #158 and #161; Marcus Vorwaller added to the no-X table in CONTRIBUTORS-X.md (nothing on his GitHub profile or social accounts, his site marcus.pictures lists none, checked 10 Nov); WEEKLY-NOTES W46 section for 10 Nov with Marcus's own words about the rebases and the reasons #159 #160 #162 #163 were sent back.
WHY: the PR-run checklist records every merge the day it lands, with the contributor's real name and a verified X handle or a visible "none found" plus asking Fred the same day.
CONTEXT: independent review of #159 #160 #162 #163 (read-only, nothing sent to the Mac) found: #162 ffmpeg guessed the format from sender bytes (reproduced: a two-line ffconcat file delivered as a video rendered a different cached picture), #163 Qt plays bytes the safety check never saw and playback stops on any list rebuild, #160 three failure paths after Return report "nothing sent" and the recipient is never checked before Return (both confirmed against the code), #159 the shipped macOS 27 path has no test. Comments with the specifics are on each PR. Deployed 10 Nov to the gateway Mac (read_state.py and imsg, backups in ~/.blip/backups-20261110), gus and vic, all healthy, no log errors.
TAGS: docs, changelog, change-log, weekly-notes, roster, marcus, pr-run, review

LR-T: pr-run, contributor-pr, bookkeeping
LR-D: blip, docs
LR-K: pr-158-161-merged, pr-159-160-162-163-sent-back, marcus-no-x
LR-R: 401b99d, ea64c98, 0d302b1

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXN1v67b9f9VHGe3FhaQuD

marcus commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the review and rebased onto current main (including #158/#161).

  • Both sequence probe/decode pin -f mov before -i and set -max_pixels 4194304; a real ffconcat payload regression is refused.
  • APNG and sequence output frame ceilings derive from dimensions and the 64M budget, independently of advertised frame count. Failed APNG conversions cache the still; failed video previews have a one-hour negative cache.
  • The cache-hit auto-fetch size gate applies only to stickers. PNG conversion and HEICS fallback request --max-dim 2048 on the Mac.
  • Restored main's type-1000 unread/toast behavior rather than changing it incidentally.
  • WebP is optional with a warning; discovery includes QT_PLUGIN_PATH. Save docs now distinguish bounded PNG/HEICS stills from original sequences; removed unused Tapback kinds.
  • Added independent threshold tests for count, seconds, pixel budget, edge size and valid oversized plists, plus ordinary 6 MB GIF cache-hit coverage. Mock output follows the actual codec argv so changing a limit fails the test.

Validation: full Bun suite passed after rebase, full synthetic Mac Python discovery passed (344 tests), and the real-codec/Qt sticker harness passed (6 Qt checks). Additional focused edge/cache tests pass. No real conversations were opened or messages sent. CI is running on the pushed head. I'll follow your #162 → #163 → #159 → #160 landing order and handle subsequent rebases.

Preserve sticker associations and activity timestamps, render known placements and use a compact parent strip for unknown placements and sticker reactions. Validate bounded raster decodes, retain alpha and animate supported previews without changing the surrounding theme. Add synthetic metadata, codec, Qt rendering and demo coverage.
Pin MOV demuxing and decoder pixel limits, cap output frames by the pixel budget, and cache failed previews. Bound PNG conversion on the Mac, retain legacy unread behavior, and make WebP optional. Add regression coverage with synthetic media only.
Document the 2048-pixel Mac conversion and one-hour negative video cache without claiming full-size originals.
Derive mocked codec output from actual argv so threshold mutations fail, and cover the 2048 edge and cached non-sticker GIF above the auto-fetch cap.
nixfred added a commit that referenced this pull request Nov 10, 2026
…100, reactions are searched for from the window's oldest message onward

WHAT: enrich() read EVERY reaction in chat.db (a full scan of the message table: 28,413 reaction and sticker rows out of 282,650 messages) for every --rich call, i.e. every conversation open and every push reload. It now asks only for rows stamped no earlier than the window's oldest message minus 90 days (TAPBACK_SKEW_NS), which SQLite answers from message_idx_date. The rows are sorted by ROWID in Python so the reactions on a message keep exactly the order the old scan produced (the date index would return date order). skew_ns=None is the old query, kept as the test oracle and as the fallback when a window has no usable date. New bridge/mac/test_tapback_window.py and one CI step.
WHY: measured on the real 530 MB gateway database, 5 repeats: the scan costs 93 ms (old) against 2 ms (a week-old window), 6 ms (3 months) or 24 ms (a year); profiling one conversation load put 100 ms of a ~400 ms one-shot run in enrich(), and on the persistent channel, where the cluster map and contact index are cached, it was the largest cost left. The margin exists because clocks disagree: on the real database 3 of 28,305 reactions are stamped BEFORE their target, the worst by 0.83 days, so a plain "reactions are newer than the message" bound would have dropped real ones. 90 days is 100 times the worst observed.
VERIFIED: 90 real windows (the newest 40/150/500/3200 messages and 86 per-conversation windows, 4,440 rows, 421 reactions) give byte-identical enrich() output with and without the bound, compared by hash on the gateway, nothing printed; typical window 103 ms to 7 ms, the 3200-row window 126 ms to 32 ms. 1,200 seeded random databases (every guid shape, 60-day-early to 200-day-late reactions, stickers, removals, two senders, newest-N and random windows) agree with the unbounded query. 10 mutations (>= to >, min to max, margin 1 day, no sort, zero and negative dates, bool as a date, None not honoured, no-date window still bounded, date hidden from the planner, floor ignored) each fail a test. A plan test fails if the planner stops using message_idx_date: an ORDER BY in SQL did exactly that in the synthetic DB, which is why the sort is in Python. All 28 Python suites and 862 bun tests pass.
LIMIT (documented, tested): a reaction stamped more than 90 days before the message it reacts to is not read; the oracle still reads it.
CONTEXT: built on the #162 head because #162 widens this same WHERE clause (type 1000 stickers); Fred approved the literal diff on 10 Nov. Door classifier one-way (bridge/mac/imsg, bridge test, ci.yml).
TAGS: enrich, tapback, reactions, message_idx_date, conversation-load, latency, bridge, oracle-fuzz, mutation-tested

LR-T: latency, reactions, sqlite-index
LR-D: blip, bridge-mac
LR-K: tapback-window-bound, message_idx_date, TAPBACK_SKEW_NS, enrich-oracle
LR-R: 401b99d

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXN1v67b9f9VHGe3FhaQuD
@nixfred
nixfred merged commit aee3394 into nixfred:main Nov 10, 2026
1 check passed
nixfred added a commit that referenced this pull request Nov 10, 2026
… (CHANGELOG, CHANGE.log, roster, weekly notes, CLAUDE.md)

WHAT: CHANGELOG and CHANGE.log entries for #162 (native stickers) and the reaction-scan speed-up (9b4f8d9) with the real numbers; CONTRIBUTORS-X.md moves Marcus Vorwaller to 3 merged (#158, #161, #162), still no X handle found and asked Fred; WEEKLY-NOTES W46 gets the #162 and speed-up bullets plus the note that #159, #160 and #163 got a second independent review of their rebased heads; CLAUDE.md's measured-latency paragraph gains the lesson from the reaction scan (the 90-day margin is real data: 3 of 28,305 reactions are stamped before their target, the worst by 0.83 days; an ORDER BY in SQL brings the full scan back; the date index reorders reactions unless sorted by ROWID; do not probe the guid index by shape because _strip_assoc is lenient on purpose).
WHY: the PR-run checklist records every merge the day it lands, and the next agent that "optimises" enrich() must find the traps written down instead of rediscovering them.
CONTEXT: Fred approved the #162 merge and the speed-up from the literal diffs on 10 Nov. Merge commit aee3394, speed-up 9b4f8d9.
TAGS: docs, changelog, change-log, weekly-notes, roster, claude-md, pr-run, stickers, latency

LR-T: pr-run, bookkeeping, latency
LR-D: blip, docs
LR-K: pr-162-merged, tapback-window-bound, marcus-no-x
LR-R: aee3394, 9b4f8d9

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXN1v67b9f9VHGe3FhaQuD
@nixfred

nixfred commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Merged as aee3394 (a merge commit, so your commits stay in main's history). Thank you, Marcus.

I want to be specific about why this one landed fast. You took six review points and answered each with a change and a test, not an explanation. I did not take the reply on trust: I re-ran the exploit I had reproduced on the first head (a two-line ffconcat text file delivered as a video) against your new head, and it is refused on all three decode paths now, with nothing left in the cache directory. Stickers.mjs is byte-identical to a fresh build of stickers.ts, 862 bun, 21 Python and 6 Qt tests pass, and against the real gateway database the new enrich SQL ran clean on 90 windows and the chat list is identical apart from your sticker key.

Two follow-ups, neither blocks anything:

  1. ffmpeg now decodes sender-supplied video stickers on the user's machine without a click. Your bounds are good (8 streams, 2048 px, the pixel budget, 30 s, external track references off by default), but a decoder bug is a decoder bug. If you are willing, run ffprobe and ffmpeg in a sandbox (bubblewrap with no network and a read-only filesystem apart from the one temp file). A separate small PR is fine.
  2. Cosmetic: scripts/blip-setup has a stray space before the first loop's done.

For your rebases:

nixfred added a commit that referenced this pull request Nov 11, 2026
Marcus Vorwaller's reactions work for macOS 27.2: a tapback on any message, not
only the newest, through the Tapback Picker that "Tapback Details..." opens on
that bubble, and an exact custom emoji (type 2006/3006, confirmed against
associated_message_emoji in chat.db) through public native AX APIs and one
frame-stable, hit-tested click on the glyph. The searchable ReactionPicker.qml
is the UI; TapbackActions.mjs is regenerated from tapback-actions.ts.

Conflicts resolved here: .github/workflows/ci.yml keeps both new steps (#163's
audio suites and #159's reaction-picker suite); the "kind": "emoji" tapback
line in bridge/mac/imsg was already on main from #162, so the duplicate hunk is
dropped.

Proven on the real macOS 27.2 gateway, self-thread only, with Fred's word:
classic Heart on a 10 Oct message, custom emoji reaction (type 2006, exact
emoji confirmed, kind "emoji" on the read path), a second reaction on an
already-reacted bubble, and removal back to none. The first custom-emoji try
refused with "moved" while the Mac was in use and pressed nothing.
Checked on 5fc0454 merged onto ef9a1ab: bun test 877 pass, 7 Qt/Python suites,
92 bridge tests, qmllint clean, TapbackActions.mjs rebuilds identical,
gitleaks clean, door one-way (ci.yml, bridge/mac/imsg-react) shown to Fred.

LR-T: blip, pr-159, tapbacks, macos-27, custom-emoji, ax, merge
LR-D: blip, bridge, qml
LR-K: picker-route-any-message, exact-emoji-click-glyph, reacted-label-pattern
LR-R: 33aab7b, ef9a1ab
LR-L: law-4, law-10, law-11

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nixfred added a commit that referenced this pull request Nov 11, 2026
WHAT: the same-turn bookkeeping for the #159 merge (6a4fbce, 56bc294).
  - CHANGELOG.md: a proper Unreleased entry; the one-line note Marcus's rebase
    left above the "## Unreleased" heading is removed (it was orphaned there).
  - CHANGE.log: #159 entry with the live 27.2 proof and what Larry added.
  - CONTRIBUTORS-X.md: Marcus 5 merged (#158 #161 #162 #163 #159), #160 under
    review, X handle still none found (rechecked 2026-11-11, asked Fred).
  - WEEKLY-NOTES.md: W46 Wed 11 Nov, #159 bullet and the two Blip-side findings.
NOT staged on purpose: bridge/mac/findmy_city.py and test_findmy_city.py are
  someone else's uncommitted work in this tree.

LR-T: blip, pr-159, changelog, contributors, weekly-notes
LR-D: blip, docs
LR-K: pr-run-bookkeeping
LR-R: 6a4fbce, 56bc294

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants