Skip to content

Play voice messages inline with real waveforms and seeking - #163

Merged
nixfred merged 5 commits into
nixfred:mainfrom
marcus:upstream-inline-audio
Nov 11, 2026
Merged

nixfred merged 5 commits into
nixfred:mainfrom
marcus:upstream-inline-audio

Conversation

@marcus

@marcus marcus commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Audio attachments gain inline play/pause, a measured waveform, remaining time, seeking and keyboard controls. Exactly one plays. The view owns playback across message-model rebuilds; leaving the conversation or closing it cancels playback and delayed play intent. Decoding starts on a play click.

A recognized, explicitly selected demuxer feeds bounded FFmpeg decoding. Qt receives only a private, generated 24 kHz mono PCM WAV, never the received container. The content-hash WAV cache also supplies the waveform and counts toward cache eviction. Inline normalization is limited to ten minutes. QtMultimedia loads optionally; unavailable normalization/tools retain explicit-demuxer external mpv playback or Save. Original attachment files remain available for Save.

Based on current upstream main, including #158 and #161, and preserves its controls, theme and manifest. Validation: 829 Bun tests, 323 synthetic Mac Python tests, 7 muted real-WAV/Qt playback checks (including delegate rebuild), and 13 play-intent/session checks pass. Rendered media controls inspected; GitHub CI passes. No real conversations opened or live messages sent. Live-bar resource measurements remain unverified. Limits are in docs/AUDIO.md.

Synthetic inline audio players

@nixfred

nixfred commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Thank you, Marcus. Inline players with a measured waveform, seeking and exactly one voice playing at a time are a real step up from opening mpv, and the in-memory analysis with no PCM sidecar is the right shape. The review of this branch found three blocking problems, all about what happens after the checks pass, plus some smaller ones.

Blocking

  1. The demuxer gate does not cover what actually plays. -f <format> and -protocol_whitelist file,pipe constrain only the ffprobe and ffmpeg calls. Qt's MediaPlayer then opens the same cached bytes with automatic format detection inside quickshell, so the comment "only recognized audio containers reach the media decoder" holds only for the analysis. A file can pass the exact ffprobe argv in audio-metadata.ts as MP3 while Qt's own probe picks a different demuxer. On the ffmpeg here that second probe failed to open and no other file was touched, so it holds by luck of a version, not by construction. The robust shape is the one rewrapCaf already uses: have ffmpeg write a verified Ogg/Opus or WAV into the 0700 cache from the gated demuxer, and let Qt play only that.
  2. A missing QtMultimedia stops the whole view. AudioMessage.qml imports QtMultimedia and is instantiated inside every attachment chip, gated only by visible: chipRow.isAudio. quickshell and Omarchy do not depend on Qt multimedia (on this machine qt6-multimedia is required by other apps, not by them), so a plugin update on a machine that has not re-run blip-setup would stop BlipView.qml loading. A Loader with active: chipRow.isAudio and the old chip plus Save as the fallback fixes that, and also stops building a MediaPlayer, AudioOutput, Slider and Canvas for every image chip.
  3. Playback stops whenever the conversation changes, including by itself. bubbleRepeater is model: root.inThread ? root.bubbles : [], and root.bubbles = list is reassigned on every reload whose JSON differs (and on your own send). That destroys every delegate, and Component.onDestruction in the chip calls stopAudio(), so an incoming message, a read receipt or a tapback cuts off the voice message you are listening to and loses the paused position. The player needs an owner that survives a model rebuild (the view, with the delegate only drawing its state), or the Repeater must not rebuild for a change that does not touch audio.

Should fix

  1. Audio now joins auto-fetch, so opening a thread runs rewrapCaf, ffprobe and a full ffmpeg decode for every voice message without a click, and MediaPlayer decodes in the shell process. mpv used to fail on its own; a decoder crash now takes the bar down. Worth measuring in the live shell, and worth deferring analysis until the bubble is on screen.
  2. Without ffprobe nothing plays at all (a missing binary gives playable:false), mpv is gone with no external-open fallback, and audioDemuxer has no AMR, AIFF or WebM, so those can only be saved. blip-setup still calls ffmpeg and ffprobe optional.
  3. Every play click pays for the analysis again: toggleAudio always enqueues a fetch and the cache-hit path re-runs ffprobe plus a full decode (about 0.7 s for a 10 minute Opus file before sound starts). The analysis can be cached by file.
  4. focusedAudio !== null makes editorActive true, so the host key catcher stands down while the player has focus. Please check Esc and arrow navigation from there.

Tests that do not fail when the safeguard goes. Each of these one-line changes left the suite green: the decoded-duration check set to if (false), removing "-t", String(AUDIO_ANALYSIS_SECONDS), removing "-nostdin", removing Component.onDestruction: player.stop(), and removing the delegate-destruction root.stopAudio(). The audio test in ui.test.ts only greps the source for strings. Dropping the belongsHere chat check and O_NOFOLLOW were both caught.

Order. #162 conflicts with this branch in the attachment chip and fetch.ts, and goes first, so this one will need a rebase after it. #158 and #161 land before either.


Generated by Claude Code

nixfred added a commit that referenced this pull request Nov 10, 2026
…ll 4x faster (CHANGELOG, CHANGE.log, roster, weekly notes)

WHAT: CHANGELOG entries for #158 (Delivered until Read) and #161 (composer emoji picker) with the real end-to-end numbers for the poll speed-up (shallow 0.77 s to 0.2 s, deep 2.1 s to 1.0 s on gus); CHANGE.log entries for 401b99d, #158 and #161; Marcus Vorwaller added to the no-X table in CONTRIBUTORS-X.md (nothing on his GitHub profile or social accounts, his site marcus.pictures lists none, checked 10 Nov); WEEKLY-NOTES W46 section for 10 Nov with Marcus's own words about the rebases and the reasons #159 #160 #162 #163 were sent back.
WHY: the PR-run checklist records every merge the day it lands, with the contributor's real name and a verified X handle or a visible "none found" plus asking Fred the same day.
CONTEXT: independent review of #159 #160 #162 #163 (read-only, nothing sent to the Mac) found: #162 ffmpeg guessed the format from sender bytes (reproduced: a two-line ffconcat file delivered as a video rendered a different cached picture), #163 Qt plays bytes the safety check never saw and playback stops on any list rebuild, #160 three failure paths after Return report "nothing sent" and the recipient is never checked before Return (both confirmed against the code), #159 the shipped macOS 27 path has no test. Comments with the specifics are on each PR. Deployed 10 Nov to the gateway Mac (read_state.py and imsg, backups in ~/.blip/backups-20261110), gus and vic, all healthy, no log errors.
TAGS: docs, changelog, change-log, weekly-notes, roster, marcus, pr-run, review

LR-T: pr-run, contributor-pr, bookkeeping
LR-D: blip, docs
LR-K: pr-158-161-merged, pr-159-160-162-163-sent-back, marcus-no-x
LR-R: 401b99d, ea64c98, 0d302b1

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXN1v67b9f9VHGe3FhaQuD
@marcus
marcus force-pushed the upstream-inline-audio branch from 6eb5fce to bc6af69 Compare November 10, 2026 16:55

marcus commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the review and rebased onto current main.

Qt now plays only a generated, private 24 kHz mono PCM WAV, built from bounded decoding through the explicit input demuxer. It never receives the original container. The normalized WAV doubles as a content-hash analysis cache, participates in LRU accounting, and preserves source originals for Save. Recordings exceeding the ten-minute normalization budget use the external fallback rather than a truncated inline timeline.

The view owns the optional multimedia Loader independently of message delegates. Attachment cards are lazy-loaded without a QtMultimedia import, and missing multimedia falls back to an ordinary chip plus external Open/Save. Model rebuilds preserve the active player's position; closing/leaving stops it. Audio is no longer auto-fetched/analyzed on thread open: a play click requests it. Added AMR, AIFF and WebM recognition; missing tools or unavailable normalization can use external mpv with the explicit recognized demuxer. Setup treats multimedia as optional and mentions mpv.

Tests now assert the decode-duration check and actual -t/-nostdin arguments, normalized output/cache reuse, and rejection of lying durations. The Qt playback harness tests pause/resume, seek/keyboard, completion, and destruction/recreation of rendering delegates while the owner continues playing. Esc returns focus from the card; timeline arrows seek. The session harness executes the shipping surface-close handler and checks late fetches/conversation ownership.

Validation: 829 Bun tests, 323 synthetic Mac Python tests, 7 real WAV/Qt playback checks and 13 session checks passed; rendered controls inspected. No real conversations were opened. CI is running. I have not measured decoder behavior inside a live bar; click-only normalization and the optional loader reduce that exposure, while the live-shell measurement remains unclaimed. I'll rebase over #162 when it lands, per your order.

@nixfred

nixfred commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Three small things I left out of the review above, so you can take them in the same pass:

  • fetch.ts line 405: the ok: true, online: true, ... line lost its indentation inside the cache-hit return.
  • mime.startsWith("audio/") is case sensitive, while the MIME_EXT lookup lowercases the MIME first (fetch.ts line 103). A bridge that reports Audio/... would be cached as audio but never analysed. String(mime).toLowerCase() in both places keeps them in step.
  • The reviewer also noted that the audio branch under fetchJobAction === "open" looks dead now that openAttachment routes audio to toggleAudio first. I did not trace that one myself, so please check it before deleting anything.

And blip-setup prints pacman -S ffmpeg even when only ffprobe is reported missing. Harmless on Arch, but worth a precise message.


Generated by Claude Code

@marcus
marcus force-pushed the upstream-inline-audio branch from bc6af69 to d968ff6 Compare November 10, 2026 18:02
nixfred added a commit that referenced this pull request Nov 10, 2026
… (CHANGELOG, CHANGE.log, roster, weekly notes, CLAUDE.md)

WHAT: CHANGELOG and CHANGE.log entries for #162 (native stickers) and the reaction-scan speed-up (9b4f8d9) with the real numbers; CONTRIBUTORS-X.md moves Marcus Vorwaller to 3 merged (#158, #161, #162), still no X handle found and asked Fred; WEEKLY-NOTES W46 gets the #162 and speed-up bullets plus the note that #159, #160 and #163 got a second independent review of their rebased heads; CLAUDE.md's measured-latency paragraph gains the lesson from the reaction scan (the 90-day margin is real data: 3 of 28,305 reactions are stamped before their target, the worst by 0.83 days; an ORDER BY in SQL brings the full scan back; the date index reorders reactions unless sorted by ROWID; do not probe the guid index by shape because _strip_assoc is lenient on purpose).
WHY: the PR-run checklist records every merge the day it lands, and the next agent that "optimises" enrich() must find the traps written down instead of rediscovering them.
CONTEXT: Fred approved the #162 merge and the speed-up from the literal diffs on 10 Nov. Merge commit aee3394, speed-up 9b4f8d9.
TAGS: docs, changelog, change-log, weekly-notes, roster, claude-md, pr-run, stickers, latency

LR-T: pr-run, bookkeeping, latency
LR-D: blip, docs
LR-K: pr-162-merged, tapback-window-bound, marcus-no-x
LR-R: aee3394, 9b4f8d9

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXN1v67b9f9VHGe3FhaQuD
@nixfred

nixfred commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Second-round review of d968ff6, thank you for the fast turnaround. An independent reviewer went through the new head read-only (nothing sent to a Mac), and I re-ran the blocker myself. Three of the four original points are fixed in code; one is not.

Fixed and confirmed

  • A missing QtMultimedia no longer takes the view down: only AudioPlayback.qml imports it and the view loads it by URL; with the module stubbed out the outer file still loads and falls back to chips.
  • Playback survives a list rebuild: the player belongs to the view, checked with the shipping AudioPlayback.qml across a metrics rebuild and a same-file refetch.
  • The four small nits from my follow-up are all fixed (indentation, lowercase MIME match, the "open" audio branch, the blip-setup message). Both mpv commands put -- before the path; no message text or sender string reaches a command line; the WAV is 0600 and moved into place atomically.

Blocker: Qt still plays bytes the sender controls
When the source is already 24 kHz mono 16-bit, the conversion leaves the audio untouched and Blip's 44-byte WAV header goes directly in front of the sender's bytes. Qt opens that file without naming a format, so FFmpeg guesses the demuxer from the first 2048 bytes, and anything that scores above WAV's 99 wins.

  • I wrapped an MPEG-TS file (MPEG-2 video plus MP2 audio) as the data chunk of a genuine 24 kHz WAV. The real analyzeAudio returns playable: true and a playbackUrl. ffprobe on that cached file with no format named reports mpegts with a mpeg2video stream and an mp2 stream; forced as -f wav it reports pcm_s16le. That is exactly the difference between what the file claims and what Qt opens.
  • The reviewer also got Qt's MediaPlayer (offscreen) to log Input #0, mpegts, hasVideo true, and to play 504 ms of the smuggled stream, so any codec MPEG-TS can carry reaches the decoders inside the bar after one click. A second crafted opening makes FFmpeg pick the act demuxer, which refuses the sample rate and so fails safe, but only because of the rate Blip happens to write.
  • Same family as the ffmpeg format guess on Render native stickers and bounded animated previews #162, which you closed with -f mov before -i.
  • Suggested fix, verified by the reviewer on both hostile payloads: write an 18-byte fmt chunk plus a JUNK chunk so the audio data starts at offset 4096, so FFmpeg's first look sees only Blip's bytes; rename the cache file (for example .pcm2-) because the cache-hit path reads samples from offset 44; add a CI test that wraps an MPEG-TS payload and asserts ffprobe reports wav.

Should-fix (from the independent review; I did not re-run these)

  1. Long and some ordinary recordings can no longer play at all: audio-metadata.ts returns no externalDemuxer at lines 74, 77, 82 and 85, so the mpv fallback is never offered. Repros: a 605 s Ogg/Opus file and a 20 s VBR MP3 without a Xing header; main played both through mpv. This contradicts your reply and docs/AUDIO.md.
  2. The tests would not catch the first-round blockers returning. These mutations all left bun test and both audio UI tests green: stopping playback again in the card's onDestruction; dropping the Loader.Error term from the card loader; loading AudioPlayback as a type instead of by URL; the cache-hit path returning the original file's URL; dropping the owner check on the cached WAV.
  3. The external mpv cannot be stopped: it is started with Quickshell.execDetached (BlipView.qml 1138 and 2054), so closing the panel or switching conversations leaves it talking, which is the "kept talking after I clicked away" bug from 26 Sep, back on the fallback paths.

Nits: without QtMultimedia the first Play click does nothing visible; a file name of about 66 to 75 CJK characters makes the temporary WAV name too long; fetch.ts:404 scans the whole cache on every cache hit, images included; "Loading…" never shows while a play request is queued; "one voice at a time" holds per view, not across the panel and the window; the demo edits (LAKE TRAIL text, fake-imsg passthrough, re-escaped quotes in fixtures.json) ride along unrelated.

Order is unchanged: #163 is next after #162, which is now merged. You will hit conflicts in BlipView.qml, fetch.ts and the fixtures on rebase; I am glad to send you my resolved hunks if that helps. bun test 839 pass, the audio Qt tests pass, and GitHub CI is green on your head.

@marcus
marcus force-pushed the upstream-inline-audio branch from d968ff6 to 6458ce0 Compare November 10, 2026 22:47
nixfred added a commit that referenced this pull request Nov 11, 2026
… 11 Nov, #163 merged

WHAT: two tests and the same-turn bookkeeping for the #163 merge.
  - audio-waveform.test.ts: a cache hit must hand Qt the WAV Blip generated
    (a .pcm2- file, same URL as the first analysis), never the received file,
    and must run neither ffprobe nor ffmpeg.
  - ui.test.ts: the mpv fallback keeps its demuxer named
    (--demuxer=lavf --demuxer-lavf-format=<format>) and "--" before the path.
WHY: mutation check on #163's head, 7 guard mutations: 4 failed a test, these
  2 (cache hit returning the original file's URL; mpv format no longer forced)
  plus the cache owner check left everything green. Both now fail a test
  (verified: M5, M7 and dropping "--" each give 1 fail). The owner check needs
  a file owned by another uid, so root; left as is.
DOCS: CHANGELOG.md, CHANGE.log, CONTRIBUTORS-X.md (Marcus 4 merged, X still
  none found, rechecked 11-11), WEEKLY-NOTES.md W46 Wed 11 Nov.
VERIFIED: bun test 872 pass.

LR-T: blip, pr-163, audio, tests, mutation-testing, changelog, contributors
LR-D: blip, tests, docs
LR-K: cache-hit-serves-generated-wav, mpv-forced-demuxer-test
LR-R: e2dd817
LR-L: law-10, law-11, law-14

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXN1v67b9f9VHGe3FhaQuD
@nixfred
nixfred merged commit e2dd817 into nixfred:main Nov 11, 2026
1 check passed
@nixfred

nixfred commented Nov 11, 2026

Copy link
Copy Markdown
Owner

Merged as e2dd817. Thank you, Marcus: three rounds in a day, and every point answered in code, not in words. Voice messages playing inside Blip with a real waveform is a big one.

What I checked on 0a20ad5 before merging:

  • The blocker is closed. I rebuilt my round-2 attack (an MPEG-TS stream with MPEG-2 video and MP2 audio as the data chunk of a genuine 24 kHz WAV) and ran it through the real analyzeAudio. The sender's bytes still pass through bit-exact (the TS sync byte sits at 4096), and ffprobe with no format named reads Blip's cached file as wav / pcm_s16le, even with a 5 MB probe. The fmt + JUNK layout is the right fix.
  • The mpv fallback is back for long and VBR recordings with the demuxer forced, and it is a Process that stopAudio() kills, so the 26 Sep "kept talking" bug stays fixed.
  • GitHub CI is green, bun test 871 pass, the three audio Qt suites pass (7 + 3 + 15), AudioWaveform.mjs rebuilds byte-identical, and gitleaks is clean.

One small follow-up I did myself (dee1ac1): of 7 guard mutations, 4 now fail your tests. Two of the survivors are cheap to cover, so I added tests: a cache hit must hand Qt the generated .pcm2- WAV (never the received file) and run nothing, and the mpv command keeps --demuxer-lavf-format and -- before the path. The third survivor, dropping the cache owner check, needs a file owned by another uid, so I left it.

Nits from round 2 that are still open, none blocking, take them or leave them: without QtMultimedia the first Play click does nothing visible; "Loading…" never shows while a play request is queued; "one voice at a time" holds per view, not across the panel and the window; the mpv command lost --no-terminal --really-quiet from the old one.

Next in the queue are #159 and then #160. Both need a live check on the real 27.2 gateway (self-thread only) once Fred gives the word.


Generated by Claude Code

nixfred added a commit that referenced this pull request Nov 11, 2026
Marcus Vorwaller's reactions work for macOS 27.2: a tapback on any message, not
only the newest, through the Tapback Picker that "Tapback Details..." opens on
that bubble, and an exact custom emoji (type 2006/3006, confirmed against
associated_message_emoji in chat.db) through public native AX APIs and one
frame-stable, hit-tested click on the glyph. The searchable ReactionPicker.qml
is the UI; TapbackActions.mjs is regenerated from tapback-actions.ts.

Conflicts resolved here: .github/workflows/ci.yml keeps both new steps (#163's
audio suites and #159's reaction-picker suite); the "kind": "emoji" tapback
line in bridge/mac/imsg was already on main from #162, so the duplicate hunk is
dropped.

Proven on the real macOS 27.2 gateway, self-thread only, with Fred's word:
classic Heart on a 10 Oct message, custom emoji reaction (type 2006, exact
emoji confirmed, kind "emoji" on the read path), a second reaction on an
already-reacted bubble, and removal back to none. The first custom-emoji try
refused with "moved" while the Mac was in use and pressed nothing.
Checked on 5fc0454 merged onto ef9a1ab: bun test 877 pass, 7 Qt/Python suites,
92 bridge tests, qmllint clean, TapbackActions.mjs rebuilds identical,
gitleaks clean, door one-way (ci.yml, bridge/mac/imsg-react) shown to Fred.

LR-T: blip, pr-159, tapbacks, macos-27, custom-emoji, ax, merge
LR-D: blip, bridge, qml
LR-K: picker-route-any-message, exact-emoji-click-glyph, reacted-label-pattern
LR-R: 33aab7b, ef9a1ab
LR-L: law-4, law-10, law-11

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nixfred added a commit that referenced this pull request Nov 11, 2026
WHAT: the same-turn bookkeeping for the #159 merge (6a4fbce, 56bc294).
  - CHANGELOG.md: a proper Unreleased entry; the one-line note Marcus's rebase
    left above the "## Unreleased" heading is removed (it was orphaned there).
  - CHANGE.log: #159 entry with the live 27.2 proof and what Larry added.
  - CONTRIBUTORS-X.md: Marcus 5 merged (#158 #161 #162 #163 #159), #160 under
    review, X handle still none found (rechecked 2026-11-11, asked Fred).
  - WEEKLY-NOTES.md: W46 Wed 11 Nov, #159 bullet and the two Blip-side findings.
NOT staged on purpose: bridge/mac/findmy_city.py and test_findmy_city.py are
  someone else's uncommitted work in this tree.

LR-T: blip, pr-159, changelog, contributors, weekly-notes
LR-D: blip, docs
LR-K: pr-run-bookkeeping
LR-R: 6a4fbce, 56bc294

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nixfred

nixfred commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Hi Marcus, one question from Fred, who runs the Blip updates. Each week he posts an update on X that names everyone whose work landed. Four of your pull requests are in (#158, #161, #162, #163) and more are on the way, and he wants to credit you properly.

Do you have an X handle you would like him to tag? If you would rather not be tagged, or would like your name or GitHub login written a particular way, say so here and that is what we will do. No handle is on your GitHub profile or on marcus.pictures, so without an answer you appear as "Marcus Vorwaller (github.com/marcus)".

Thanks again for the quality of the work.


Generated by Claude Code

@marcus

marcus commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

haha tell Fred I say thanks, totally unnecessary but appreciated. he can tag my GitHub handle and if he wants, point to haplab.com.

nixfred added a commit that referenced this pull request Nov 12, 2026
…GitHub handle and optionally link haplab.com

WHAT: CONTRIBUTORS-X.md roster row, WEEKLY-NOTES.md "handles Fred still needs
  to answer" entry, and the PEOPLE list at the top of CHANGE.log now record
  Marcus's own answer: no X account to tag; write "Marcus Vorwaller
  (github.com/marcus)" and, if wanted, point to haplab.com.
WHY: Fred asked 2026-11-11 for him to be asked about credit on the Blip
  update posts. Asked on #163; he replied 2026-11-12 03:12Z ("he can tag my
  GitHub handle and if he wants, point to haplab.com"). Never guess an X handle
  for him.
NOTE: older CHANGE.log entries that say "X: none found, asked Fred" are
  history and stay as written.

LR-T: blip, contributors, marcus, credit, roster
LR-D: blip, docs
LR-K: marcus-no-x-credit-by-github-handle

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BXN1v67b9f9VHGe3FhaQuD
@nixfred

nixfred commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Thanks Marcus, noted: Fred will credit you as Marcus Vorwaller (github.com/marcus) and can point to haplab.com. It is in the roster now, so it sticks for every update post. And he is glad to say it back: thanks for the work.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants