Skip to content

fix(chat): show tool-result images as cached previews - #223

Merged
kahme247 merged 85 commits into
kahme247:mainfrom
andrebrait:fix/deferred-tool-images
Oct 8, 2026
Merged

kahme247 merged 85 commits into
kahme247:mainfrom
andrebrait:fix/deferred-tool-images

Conversation

@andrebrait

@andrebrait andrebrait commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Tool-result images (agent screenshots, read of an image file) were invisible after opening or reloading a session. Every history load uses ?deferMedia=1, which replaced each tool-result image with [1 tool result image omitted from initial history payload …, ~57 bytes], and nothing ever loaded the image back. The byte count was also wrong: it measured the blob:sha256: reference, not the image. The deferral came from 41d2f27.

Tool-result images now reach the browser as links, both in history and in live output:

  • Each image becomes {type: "image", mimeType, url: "/api/media/<sha256>"}. The hash is omp's own blob address (sha256 of the bytes), so history entries map directly. Live images are hashed and copied to omp-web's data folder (media/) until omp writes its blob.
  • GET /api/media/<sha256> serves the full image. ?thumb=1 serves a 480px WebP preview, generated once with sharp, EXIF-rotated, and cached on disk.
  • An expanded tool row loads only the previews (1–7 KB each in the smoke test, against 30 KB to several MB for the originals). Clicking a preview opens the full image in the lightbox.
  • No base64 for tool-result images reaches the browser any more, including live tool_execution_*, message_*, turn_end and agent_end frames and the reconnect snapshot. Many large screenshots therefore no longer fill browser memory or slow the chat down. User and custom-message images are unchanged.

Safety

  • The route accepts only ^[a-f0-9]{64}$ hashes and serves only PNG, JPEG, GIF and WebP, detected by magic bytes (everything else gets 415, with nosniff). It sits behind the existing password guard.
  • omp writes blobs in place, so the route serves omp-web's copy until omp's blob has the same size. A truncated image is never cached as immutable.
  • A daily sweep, run on the next media write, deletes copies omp now holds and any file older than 30 days. Previews regenerate on demand.
  • A missing blob shows the existing [image unavailable: blob … not found] text instead of a broken image.
  • sharp is added as a direct dependency. It was already installed as Next's optional dependency, and Next keeps it external. If it fails to load, the route serves the full image instead of a preview.

Verification

  • tsc --noEmit, lint (no new warnings), npm test: 1,223 pass / 0 fail. New and updated tests:
    • lib/media-route.test.mjs: full image and preview size, preview reuse, EXIF orientation, non-image 415, a real decoy image outside the store that stays unreachable, partial blob, sweep, and every rewritten event type.
    • lib/session-reader.test.mjs: deferred history returns URLs and keeps user images inline. A missing blob gives the placeholder text.
    • lib/rpc-manager.test.mjs: live frames and the reconnect snapshot carry the URL, and no base64 reaches a listener.
  • Smoke test with a dev server and headless Chromium on an isolated agent dir: an expanded tool row showed five 480px previews, including a legacy inline image, and the lightbox opened the full 1024×640 image.
  • Live run with real omp 18.6.2 and gemini-3.7-flash reading a WebP: the SSE stream carried /api/media/<hash> and no base64.
  • One condensed adversarial review round, then Codex and CodeRabbit on the review companion ([companion] fix(chat): show tool-result images as cached previews andrebrait/ompweb#83). Every finding is fixed in follow-up commits, and Codex's last review of the final head found nothing. Fixes: incomplete-blob handling on the live path, a 404 retry after concurrent removal, a 16 MiB cap on the hash memo, no-cache when the full image stands in for a preview, rewriting subagent_event payloads, and unique temp names. CodeRabbit reviewed the first two commits; its follow-up review was rate-limited.

andrebrait and others added 30 commits October 3, 2026 12:54
The pruning that bounds the job store deletes a `gone` tombstone with no
client-visible signal, so a browser that adopted another browser's recording
could get a 404 and raise an error toast for a job that had in fact been
delivered elsewhere. Only the browser that recorded (or retried) a job holds
the audio to fall back on, so only it may report it as a failure; a follower
that is locked out of the transcript is standing down by design.

The attached-messages toast also blamed the agent for a rule that has nothing
to do with it: `sendQueued` refuses attachments whether or not a run is
active, and `followUp` is dispatched immediately when the agent is idle. The
rule and its explanation are now one predicate, `queueAllowsAttachments`, used
by both `sendQueued` and the dictation branch, so they cannot drift apart.
Stop withdrew queued messages one by one from this client's queue snapshot
before aborting. A steer missing from that snapshot (promoted after the last
queue_update) or claimed by the run but not yet recorded was never withdrawn,
and omp drained it into a new turn right after the abort: the user pressed
Stop and the steer ran anyway.

Stop now sends abort_and_restore_queue, omp's Esc in one step (clearQueue
forInterrupt, then abort), and restores the returned texts to the draft. omp
without the command falls back to the per-entry withdrawal.
Review of the atomic Stop:
- Capture the run fence at the click; a failed request is retried once and
  only while that run is current, so it can never abort a newer run.
- Fall back to per-entry withdrawal only on "Unknown command" (omp without
  abort_and_restore_queue); any other failure retries the same command, and
  warns (new localized queueRestoreUncertain) when nothing came back.
- Narrow the try to the request; drop an image-only "[Image]" label.
- Tests: retry restores, lost response warns, no retry into a newer run,
  overlapping Stops share one request; tidy the server abort branch.
Prepare ompweb for publisher-default Auto effort in can1357/oh-my-pi#14114. Persist the choice through native config, preserve provider settings, and show the control and search result only when Reasoning is auto.
…in browser code

Older browsers lack these APIs: Settings crashed on open, session catch-up stalled, and transcription polling failed. Use AbortController timers and plain promises, keeping the same timeouts and cancellation. Fixes kahme247#196.
Reuse the workspace refresh control and success checkmark. Bypass the server usage cache only for manual refreshes while retaining the existing five-minute automatic polling schedule.
Keep background polling silent, preserve refresh queries on older browsers, and ensure manual refresh is not satisfied by a concurrent background fetch. Cover persisted caching and hook behavior with regression checks.
A manual refresh must attempt cache invalidation even when the automatic fetch it waited behind rejects. Extend the CLI-backed regression to cover this failure path.
The reasoning trigger shows the terminal status line's level glyph instead
of the level name (chevron hidden in narrow toolbars), so the model name
keeps the room on phones. The menu pairs each glyph with its name.

Slow was wired only for saved sessions, so a new session never showed it
even after Fast had spawned omp. Offer it in new sessions too, and refresh
live state after a new-session model change so support follows the model.
Without live omp state (new or idle session), predict omp's slowModeTarget
from the catalog: /api/models marks supportsSlowMode by provider and
reports providers.anthropic.slowMode, read fresh outside the models cache.
Claude shows the shared setting (global scope); flex starts off (session).
Live get_state wins as soon as omp runs, and clicking the toggle spawns
omp like Fast does.
…t code

Older mobile browsers lack these APIs, so calling them crashed the whole
tree into the unexpected-error screen. Use browser-safe helpers.

Fixes kahme247#196

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011RZdyVoycH4WVVDWJNctDR
… again

Handle the session-persistence notice: re-read identity, keep the run
state, re-key the registry (old id stays as an alias) and reuse a live
wrapper that already reports the requested session file.

Fixes kahme247#202

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011RZdyVoycH4WVVDWJNctDR
- auto-name asks omp (generate_title, else /rename) and only falls back to
  the derived title when omp cannot (Fixes kahme247#203)
- provider detail lists the accounts omp reports for the provider (Refs kahme247#197)
- tests and AGENTS.md notes

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011RZdyVoycH4WVVDWJNctDR
fix(queue): take the queue back on Stop through omp's atomic Esc
feat: add a Slow toggle and provider-neutral usage-limit badge
…urce

feat(settings): add Auto thinking source selector
…efresh

feat(usage): add manual provider usage refresh
…-compatibility

fix(client): avoid AbortSignal.timeout/any and Promise.withResolvers
Take main's kahme247#199 client fix instead of the duplicate abort-signal helper.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011RZdyVoycH4WVVDWJNctDR
…edes-297gxo

Support omp session moves and native title generation
Show the branch below the workspace name, keep the activity indicator beside the name, and preserve space for rename and branch controls.
Use OMP live resolution instead of duplicating discovery.
Keep the native notice preference and never resume stopped sessions for
inspection.
Add an icon-only dismiss button that hides the current report until its diagnostics change, and shorten the disable action to "Turn this off" so the notice text keeps its width on phones.
claude and others added 8 commits October 5, 2026 14:19
# Conflicts:
#	CHANGELOG.md
# Conflicts:
#	components/AppShell.tsx
#	components/ChatInput.tsx
#	components/SessionSidebar.tsx
# Conflicts:
#	README.md
#	hooks/useAgentSession.ts
History loads dropped tool-result images for a text placeholder with a wrong
byte count, and nothing loaded them back. Send them as /api/media/<sha256>
URLs instead, in history and in live frames, so no base64 reaches the
browser. The route serves the image from omp's blob store (or omp-web's
copy until omp writes it) and a 480px WebP preview generated once with
sharp; the lightbox opens the full image.
Serve omp-web's copy until omp's in-place blob write is complete, sweep
copies and previews daily, keep the base64 memo to 4 entries, rotate
previews by EXIF, show the unavailable-image text for missing blobs, and
serve buffers without copying. Tests cover traversal with a real decoy,
partial blobs, preview reuse, EXIF and every rewritten event type.
Write omp-web's copy unless omp's blob already has the image's full size,
return 404 when a file disappears between lookup and read, and use unique
temp names for concurrent writes.
The base64 memo now holds at most 16 MiB of keys. A full image served for
?thumb=1 when no preview can be made is sent with no-cache, so a later
WebP preview replaces it.
The memo now skips only decoding and hashing; a copy removed by the sweep
is written again when the same image shows up.
andrebrait added a commit to andrebrait/ompweb that referenced this pull request Oct 7, 2026
Squashed from origin/fix/deferred-tool-images (c5ed579) onto upstream/main.
andrebrait added a commit to andrebrait/ompweb that referenced this pull request Oct 7, 2026
Squashed from origin/fix/deferred-tool-images (c5ed579) onto upstream/main.
andrebrait added a commit to andrebrait/ompweb that referenced this pull request Oct 7, 2026
Squashed from origin/fix/deferred-tool-images (c5ed579) onto upstream/main.
andrebrait added a commit to andrebrait/ompweb that referenced this pull request Oct 7, 2026
Squashed from origin/fix/deferred-tool-images (c5ed579) onto upstream/main.
andrebrait added a commit to andrebrait/ompweb that referenced this pull request Oct 7, 2026
Squashed from origin/fix/deferred-tool-images (c5ed579) onto upstream/main.
andrebrait added a commit to andrebrait/ompweb that referenced this pull request Oct 7, 2026
Squashed from origin/fix/deferred-tool-images (c5ed579) onto upstream/main.
andrebrait added a commit to andrebrait/ompweb that referenced this pull request Oct 7, 2026
Squashed from origin/fix/deferred-tool-images (c5ed579) onto upstream/main.
andrebrait added a commit to andrebrait/ompweb that referenced this pull request Oct 7, 2026
Squashed from origin/fix/deferred-tool-images (c5ed579) onto upstream/main.
andrebrait added a commit to andrebrait/ompweb that referenced this pull request Oct 7, 2026
Squashed from origin/fix/deferred-tool-images (c5ed579) onto upstream/main.
andrebrait added a commit to andrebrait/ompweb that referenced this pull request Oct 7, 2026
Squashed from origin/fix/deferred-tool-images (c5ed579) onto upstream/main.

@kahme247 kahme247 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — approve (SHIP, all three)

Verified by independent review with attack probes and joint-tree merges.

Tool-result images (#223). [hash] validated by ^[a-f0-9]{64}$ (traversal/uppercase/suffix all 404 by test); MIME from magic bytes only (SVG/HTML rejected); nosniff; 16MB memo cap + 30-day sweep. Only new dep is sharp, lock in sync. Non-blocking follow-ups noted: disk-quota cap, sharp as optionalDependency for exotic platforms.

Lightbox (#226). Zoom math sound (fit returns 1 on zero-size; steps clamp to [min(fit,10%), 800%]); native <dialog> focus handling; locales consistent; inline reset SVG replaced with lucide Maximize.

Markdown links (#224). href="" trap closed (rejected schemes arrive as undefined → <span>); anchors stay, externals get target=_blank rel=noopener noreferrer, unopenable locals become plain text; no server-side fetch of attacker URLs (no SSRF). Joint trees compose cleanly.

@kahme247
kahme247 force-pushed the main branch 2 times, most recently from 3f1dbea to 42b3c29 Compare October 8, 2026 13:30
@kahme247
kahme247 merged commit 14e8186 into kahme247:main Oct 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants