diff --git a/AGENTS.md b/AGENTS.md index f3d5dc4d..fc958a07 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -257,12 +257,14 @@ client-side — every client viewing the session must show the same queue. One sequence (`queueSeqRef`) orders every source: a get_state snapshot takes a number when requested and applies only if no newer snapshot or `queue_update` was applied (HTTP and SSE can reorder). Edit/Delete use -`remove_queued_message` (act only on `removed: true`), Steer uses +`remove_queued_message` (act only on `removed: true`; newer omp also returns +the message's `images`, which Edit restores), Steer uses `promote_queued_message`; the chip changes when omp's next snapshot arrives. `handleAbort` coalesces overlapping Stops, then sends `abort_and_restore_queue`: omp's Esc (`clearQueue({ forInterrupt: true })`, then abort) in one step, -returning the withdrawn user messages, which go to the session draft via -`recoverDraftText`. It covers what a client snapshot cannot: a steer promoted +returning the withdrawn user messages, whose texts and images go to the +session draft via `recoverDraft`. omp labels an image-only message `[Image]`; +that label is never restored as text. It covers what a client snapshot cannot: a steer promoted after the last `queue_update`, and live-steered input the run claimed but never recorded (omp would otherwise requeue it and drain it into a new turn right after the abort). Never reimplement this client-side. A failed request is @@ -303,9 +305,9 @@ during the wait. running `btw_record` comes first. - `/btw ` and `/btw` are client builtins (`handleBuiltinSlashCommand` case `"btw"`). `ChatInput.sendSideQuestion` routes them there from both the - idle and the streaming submit path, *before* the attachment gate: never sent - as a prompt, never queued, and refused with a toast (draft and attachments - kept) while attachments are attached. Asking starts the wrapper + idle and the streaming submit path, *before* attachments are folded in: + never sent as a prompt, never queued, and refused with a toast (draft and + attachments kept) while attachments are attached. Asking starts the wrapper (`get_state`) and attaches SSE first when it is not open, so no early delta is lost; a second ask while one is starting is ignored. - The `btw` response, history snapshots and frames race (HTTP vs SSE): merge diff --git a/CHANGELOG.md b/CHANGELOG.md index 99cf728e..00404973 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,13 +25,14 @@ All notable changes to **omp-web** (`@kahme247/ompweb`) are documented in this f - Show all of an agent's ask-tool questions in one panel, with checkboxes for multi-select, radio buttons for single-select (recommended option marked and preselected), and an **Other** free-text answer per question, submitted together. Requires an omp that supports `set_ask_dialog`; older omp keeps the one-question-at-a-time dialog. - Make `agent://` subagent handles in chat messages clickable. Bare handles, inline code containing only a handle, and Markdown links open that subagent's result dialog; `agent://Parent/Child` opens the nested `Parent.Child` subagent when the roster knows it. A `read` of an `agent://` handle opens the same dialog from its tool row. - Show the full model name, including its provider (for example `anthropic/claude-sonnet-5-5`), and the reasoning effort in the subagent result dialog, for running subagents and for finished ones after a reload. Finished background subagents take them from their saved transcript, so they also appear in the composer's subagent chips. +- Attach images and text files while the agent is running, and send them with a steer or a queued follow-up. Attaching through **+**, paste, or drag and drop now works during a run; with only attachments in the composer, **Stop** becomes **Queue**. An image-only message shows as `[Image]` in the queue. A steer or follow-up that omp refuses returns to the composer with its images, and so does a queued message taken back by **Edit** or **Stop**. Getting images back from **Edit** and **Stop** requires an omp with `abort_and_restore_queue` and image-returning `remove_queued_message` ([can1357/oh-my-pi#14179](https://github.com/can1357/oh-my-pi/pull/14179)); older omp returns only the text. ### Fixes & Improvements - Fix settings crashes, stalled session catch-up, and failed voice transcription polling in browsers without `AbortSignal.timeout`, `AbortSignal.any`, or `Promise.withResolvers`, including development-mode clients. Client requests now use `AbortController` timers and ordinary promises while retaining timeouts and cancellation (#196). - The **Agent environment variables** editor in Settings → System & Updates now sits below its title and description and follows the page width, instead of sitting beside them and extending past the right edge. The text box also grows with its contents. - Show queued steers and follow-ups on every device viewing a session. The queue panel now shows omp's own queue (omp 18.4.4 or later) instead of a copy kept by the tab that sent them, so it also stays correct across reloads. -- **Stop** no longer loses a queued steer or lets the agent run it anyway: as with Esc in omp's terminal, every message still waiting in the queue moves back into the composer, except attached images, which are dropped. With omp's `abort_and_restore_queue` (newer omp), omp takes the queue back and aborts in one step, which also covers a steer you just promoted that this page's queue had not caught up with, and one the model already picked up through live steering (for example OpenAI models with `providers.openaiLiveSteering`). If that request fails, Stop retries it once and warns when the messages may not have come back. Older omp (18.4.4 or later) falls back to removing each message the page lists before the abort; there, a steer the page does not list or the model already took still runs after Stop. +- **Stop** no longer loses a queued steer or lets the agent run it anyway: as with Esc in omp's terminal, every message still waiting in the queue moves back into the composer. With omp's `abort_and_restore_queue` (newer omp), omp takes the queue back and aborts in one step, which also covers a steer you just promoted that this page's queue had not caught up with, and one the model already picked up through live steering (for example OpenAI models with `providers.openaiLiveSteering`). If that request fails, Stop retries it once and warns when the messages may not have come back. Older omp (18.4.4 or later) falls back to removing each message the page lists before the abort; there, a steer the page does not list or the model already took still runs after Stop, and attached images are dropped. - Long voice transcriptions no longer fail with `Unexpected token '<', " { const oldKey = `edit-old-${navigation}`; @@ -118,13 +121,14 @@ for (const navigation of ["draft-key change", "unmount"]) { } fireEvent.change(screen.getByRole("textbox"), { target: { value: "new session draft" } }); await act(async () => { - release(true); + release(QUEUED_IMAGES); await cancellation; }); assert.equal(screen.getByRole("textbox").value, "new session draft"); assert.equal(getDraft(newKey).value, "new session draft"); assert.equal(getDraft(oldKey).value, "queued question\n\ndraft updated while waiting"); assert.deepEqual(getDraft(oldKey).files, originalFiles); + assert.deepEqual(getDraft(oldKey).images, QUEUED_IMAGES, "the queued message's images come back with its text"); if (navigation === "unmount") { view.unmount(); @@ -167,13 +171,14 @@ for (const navigation of ["original composer", "same-key remount"]) { await act(async () => { fireEvent.change(screen.getByRole("textbox"), { target: { value: "typed while waiting" } }); ref.current.insertText("and queued insertion"); - release(true); + release(QUEUED_IMAGES); await cancellation; }); const recovered = "queued question\n\ntyped while waiting and queued insertion"; assert.equal(screen.getByRole("textbox").value, recovered); assert.equal(getDraft(draftKey).value, recovered); assert.deepEqual(getDraft(draftKey).files, originalFiles); + assert.deepEqual(getDraft(draftKey).images, QUEUED_IMAGES, "the queued message's images come back with its text"); fireEvent.change(screen.getByRole("textbox"), { target: { value: `${recovered}\nnext user edit` } }); assert.equal(screen.getByRole("textbox").value, `${recovered}\nnext user edit`); diff --git a/components/ChatInput.queue-attachments.test.mjs b/components/ChatInput.queue-attachments.test.mjs index 40ce4752..53b4d5a6 100644 --- a/components/ChatInput.queue-attachments.test.mjs +++ b/components/ChatInput.queue-attachments.test.mjs @@ -1,21 +1,139 @@ -import test from "node:test"; +import "../tests/setup-dom.mjs"; import assert from "node:assert/strict"; +import test, { afterEach, beforeEach } from "node:test"; +import React, { act } from "react"; +import { cleanup, fireEvent, render, screen, waitFor } from "@testing-library/react/pure.js"; +import userEvent from "@testing-library/user-event"; import { createJiti } from "jiti"; -// ChatInput is TSX; parse it as such. `tsconfigPaths` resolves the "@/…" alias. const jiti = createJiti(import.meta.url, { jsx: { runtime: "automatic" }, tsconfigPaths: true }); -const { queueAllowsAttachments } = await jiti.import("../components/ChatInput.tsx"); +const { ChatInput } = await jiti.import("./ChatInput.tsx"); +const { clearDraft, getDraft } = await jiti.import("@/lib/draft-store"); -test("a queued message is text-only, run or not", () => { - assert.equal(queueAllowsAttachments(0, 0), true); - assert.equal(queueAllowsAttachments(1, 0), false, "an image blocks queueing"); - assert.equal(queueAllowsAttachments(0, 1), false, "a pasted text file blocks queueing"); - assert.equal(queueAllowsAttachments(3, 2), false); +const KEY = "queue-attachments"; +// 1x1 transparent PNG. +const PNG = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII="; + +// Same stand-in as ChatInput.clipboard-image.test.mjs: jsdom's FileReader +// brand-checks against its own realm's Blob; the composer only needs +// readAsDataURL -> `result`. +const previousFileReader = globalThis.FileReader; +class TestFileReader { + result = null; + onload = null; + onerror = null; + readAsDataURL(blob) { + blob.arrayBuffer().then( + (buffer) => { + this.result = `data:${blob.type};base64,${Buffer.from(buffer).toString("base64")}`; + this.onload?.(); + }, + (error) => this.onerror?.(error), + ); + } +} + +beforeEach(() => { + window.matchMedia = () => ({ matches: false, addEventListener() {}, removeEventListener() {} }); + globalThis.FileReader = TestFileReader; + URL.createObjectURL ??= () => "blob:test"; + URL.revokeObjectURL ??= () => {}; +}); +afterEach(() => { + cleanup(); + clearDraft(KEY); + localStorage.clear(); + delete window.matchMedia; + globalThis.FileReader = previousFileReader; +}); + +/** A composer mid-run with an image and a text file attached through the picker path. */ +async function renderRunningWithAttachments({ queued = true } = {}) { + const calls = []; + // Images are the last argument of every queue callback, which resolves + // false when omp refused the message. + const record = (name) => async (message, ...rest) => { calls.push({ name, message, images: rest.at(-1) }); return queued; }; + const ref = React.createRef(); + render(React.createElement(ChatInput, { + ref, + draftKey: KEY, + isStreaming: true, + onSend: record("onSend"), + onAbort: record("onAbort"), + onSteer: record("onSteer"), + onFollowUp: record("onFollowUp"), + onPromptWithStreamingBehavior: record("onPromptWithStreamingBehavior"), + })); + await act(async () => { + ref.current.addFiles([ + new File([Buffer.from(PNG, "base64")], "dot.png", { type: "image/png" }), + new File(["hello"], "notes.txt", { type: "text/plain" }), + ]); + }); + await waitFor(() => { + const draft = getDraft(KEY); + assert.equal(draft?.images.length, 1, "attaching is allowed while the agent runs"); + assert.equal(draft?.files.length, 1); + }); + return calls; +} + +function assertQueuedWithAttachments(call, text) { + assert.ok(call.message.startsWith(text), "typed text leads the message"); + assert.match(call.message, /Attached file: notes\.txt\n```[^\n]*\nhello\n```/, "text file is folded in"); + assert.deepEqual(call.images?.map((image) => [image.data, image.mimeType]), [[PNG, "image/png"]]); +} + +test("Enter during a run steers with the attached image and text file, then clears them", async () => { + const user = userEvent.setup(); + const calls = await renderRunningWithAttachments(); + await user.type(screen.getByRole("textbox"), "look at this"); + await user.keyboard("{Enter}"); + await waitFor(() => assert.equal(calls.length, 1)); + assert.equal(calls[0].name, "onSteer", "default submit-during-run behavior steers"); + assertQueuedWithAttachments(calls[0], "look at this"); + await waitFor(() => assert.equal(getDraft(KEY), null)); +}); + +test("attachments alone turn Stop into Queue, which queues them as a follow-up", async () => { + const calls = await renderRunningWithAttachments(); + fireEvent.click(screen.getByRole("button", { name: /queue/i })); + await waitFor(() => assert.equal(calls.length, 1)); + assert.equal(calls[0].name, "onFollowUp"); + assertQueuedWithAttachments(calls[0], "Attached file:"); +}); + +test("a queued web slash command is expanded and keeps the attachments", async () => { + const { expandWebSlashCommand } = await jiti.import("@/lib/web-slash-commands"); + window.HTMLElement.prototype.scrollIntoView = () => {}; + const user = userEvent.setup(); + const calls = await renderRunningWithAttachments(); + await user.type(screen.getByRole("textbox"), "/goal ship it"); + await user.keyboard("{Enter}"); + await waitFor(() => assert.equal(calls.length, 1)); + delete window.HTMLElement.prototype.scrollIntoView; + assert.equal(calls[0].name, "onPromptWithStreamingBehavior"); + assertQueuedWithAttachments(calls[0], expandWebSlashCommand("/goal ship it").prompt); +}); + +test("a refused follow-up goes back whole to its draft, ahead of what was typed meanwhile", async () => { + let refuse; + const calls = await renderRunningWithAttachments({ queued: new Promise((resolve) => { refuse = () => resolve(false); }) }); + fireEvent.click(screen.getByRole("button", { name: /queue/i })); + await waitFor(() => assert.equal(calls.length, 1)); + fireEvent.change(screen.getByRole("textbox"), { target: { value: "typed meanwhile" } }); + await act(async () => { refuse(); }); + await waitFor(() => assert.deepEqual(getDraft(KEY)?.images, [{ data: PNG, mimeType: "image/png" }])); + assert.equal(getDraft(KEY)?.value, `${calls[0].message}\n\ntyped meanwhile`, "the text, file contents included, comes back too"); + assert.equal(screen.getByRole("textbox").value, `${calls[0].message}\n\ntyped meanwhile`); }); -test("the rule has no run-state input, so it cannot report the agent's state as the reason", () => { - // Regression guard for the shipped copy: the toast once said attachments - // "cannot be queued while the agent is running", which is narrower than the - // rule it explains (omp refuses attachments regardless of `isStreaming`). - assert.equal(queueAllowsAttachments.length, 2, "only the two attachment counts decide"); +test("recovered images past the attachment cap all stay in the composer", async () => { + const { MAX_ATTACHED_IMAGES } = await jiti.import("@/lib/image-attachments"); + const { recoverDraft } = await jiti.import("@/lib/draft-store"); + await renderRunningWithAttachments(); + const recovered = Array.from({ length: MAX_ATTACHED_IMAGES }, () => ({ data: PNG, mimeType: "image/png" })); + await act(async () => { recoverDraft(KEY, { text: "", images: recovered }); }); + await waitFor(() => assert.equal(getDraft(KEY)?.images.length, MAX_ATTACHED_IMAGES + 1)); + assert.equal(document.querySelectorAll("img").length >= MAX_ATTACHED_IMAGES + 1, true, "every image has a preview to remove"); }); diff --git a/components/ChatInput.tsx b/components/ChatInput.tsx index 8b99e757..bdb00793 100644 --- a/components/ChatInput.tsx +++ b/components/ChatInput.tsx @@ -14,7 +14,7 @@ import type { SttAfter } from "@/lib/stt"; import { formatCompactNumber, formatPercent } from "@/lib/format"; import { ContextDetailPanel } from "./ComposerPanels"; import { RecordingDeck } from "./RecordingDeck"; -import { clearDraft, getDraft, mergeRecoveredText, recoverDraftText, setDraft, subscribeDraftRecovery } from "@/lib/draft-store"; +import { clearDraft, getDraft, mergeRecoveredText, recoverDraft, setDraft, subscribeDraftRecovery, type ChatDraftImage } from "@/lib/draft-store"; import { expandWebSlashCommand } from "@/lib/web-slash-commands"; import type { AttachedImage, AttachedTextFile } from "./ChatInput-draft-attachments"; import { @@ -140,9 +140,10 @@ interface Props { onPredictWord?: PredictWord; onPredictWordFeedback?: PredictWordFeedback; onAbort: () => void; - onSteer?: (message: string, images?: AttachedImage[]) => void; - onFollowUp?: (message: string, images?: AttachedImage[]) => void; - onPromptWithStreamingBehavior?: (message: string, behavior: "steer" | "followUp", images?: AttachedImage[]) => void; + /** Steer/follow-up callbacks resolve false when omp refused the message. */ + onSteer?: (message: string, images?: AttachedImage[]) => Promise; + onFollowUp?: (message: string, images?: AttachedImage[]) => Promise; + onPromptWithStreamingBehavior?: (message: string, behavior: "steer" | "followUp", images?: AttachedImage[]) => Promise; isStreaming: boolean; model?: { provider: string; modelId: string } | null; isAutoModelSelection?: boolean; @@ -193,8 +194,9 @@ interface Props { modelCapacity?: { contextWindow?: number; maxTokens?: number } | null; /** Generation speed shown in the context ring popover. */ generationSpeed?: GenerationSpeedInfo | null; - /** Cancel one pending message in omp before removing it from the queue panel. */ - onRemoveQueuedMessage?: (text: string, queue: keyof QueuedMessages) => Promise; + /** Cancel one pending message in omp before removing it from the queue + * panel; resolves to its images once omp confirms, else false. */ + onRemoveQueuedMessage?: (text: string, queue: keyof QueuedMessages) => Promise; /** Promote the first matching native follow-up into steering. */ onPromoteQueuedToSteer?: (text: string) => void | Promise; slashCommands?: SlashCommandInfo[]; @@ -303,15 +305,6 @@ function menuDropStyle(placement: MenuPlacement, maxHeight: number | null): Reac }; } - -/** A queued message is text-only: omp refuses attachments in a steer or a - * follow-up, whether or not a run is active. One predicate decides both - * whether `sendQueued` may queue and what a refused dictation tells the user, - * so the rule and its explanation cannot drift apart. */ -export function queueAllowsAttachments(attachedImages: number, attachedTextFiles: number): boolean { - return attachedImages === 0 && attachedTextFiles === 0; -} - export const ChatInput = memo(forwardRef(function ChatInput({ onSend, onPredictWord, onPredictWordFeedback, onAbort, onSteer, onFollowUp, isStreaming, model, isAutoModelSelection, modelNames, modelList, modelError, modelsLoading, onModelChange, fastModeEnabled, fastModeActive, usageLimit, fastModeSupported, onFastModeChange, slowModeSupported, slowModeEnabled, slowModeScope, onSlowModeChange, onAbortCompaction, isCompacting, compactResult, @@ -636,15 +629,11 @@ export const ChatInput = memo(forwardRef(function ChatIn }, []); const processFiles = useCallback((files: File[]) => { - if (isStreaming) { - setAttachError(t("chatInput.attachmentsDisabled")); - return; - } const imageFiles = files.filter((file) => file.type.startsWith("image/")); const otherFiles = files.filter((file) => !file.type.startsWith("image/")); void processImageFiles(imageFiles); void processTextFiles(otherFiles); - }, [isStreaming, processImageFiles, processTextFiles]); + }, [processImageFiles, processTextFiles]); const processFilesRef = useRef(processFiles); processFilesRef.current = processFiles; @@ -783,6 +772,10 @@ export const ChatInput = memo(forwardRef(function ChatIn const { text } = recovery; // Merge with pending edits rather than replacing them with a store snapshot. setValue((current) => mergeRecoveredText(current, recovery)); + const images = recovery.images ?? []; + if (images.length) { + setAttachedImages((prev) => [...prev, ...draftImagesToAttachedImages(images)]); + } setAtQuery(null); setHistoryMenuOpen(false); requestAnimationFrame(() => { @@ -902,13 +895,7 @@ export const ChatInput = memo(forwardRef(function ChatIn const finalText = base + sep + text; insertTextAtCursor(text); if (after && onFollowUp) { - // Queued messages are text-only: with attachments in the composer the - // queue would refuse it, so keep it here and say why. - if (!queueAllowsAttachments(attachedImagesRef.current.length, attachedTextFilesRef.current.length)) { - toast.info(t("chatInput.dictationKeptWithAttachments")); - } else { - sendQueued(after === "send" ? "followup" : after, finalText); - } + sendQueued(after === "send" ? "followup" : after, finalText); } else if (after && !isStreaming) { void handleSend(finalText); } else { @@ -1262,7 +1249,18 @@ export const ChatInput = memo(forwardRef(function ChatIn const msg = raw.trim(); if (!msg && !attachedImagesRef.current.length && !attachedTextFilesRef.current.length) return; if (sendSideQuestion(msg, overrideText)) return; - if (!queueAllowsAttachments(attachedImagesRef.current.length, attachedTextFilesRef.current.length)) return; + const images = attachedImagesRef.current.length ? attachedImagesRef.current : undefined; + // The queue callbacks resolve false when omp refused the message: it goes + // back, text and images together, to the draft it was sent from, even if + // the user has typed or switched sessions since. + const key = draftKeyRef.current; + const keptImages = images?.map(imageToDraftImage); + const recoverOnFailure = (queued: Promise, text: string) => { + void Promise.resolve(queued).then((ok) => { + if (ok === false && key) recoverDraft(key, { text, images: keptImages }); + }); + }; + const files = attachedTextFilesRef.current; onAudioUnlock?.(); const streamingBehavior = mode === "steer" ? "steer" : "followUp"; if (msg.startsWith("/") && onPromptWithStreamingBehavior) { @@ -1279,8 +1277,9 @@ export const ChatInput = memo(forwardRef(function ChatIn // own ACP handlers can run them. const expansion = expandWebSlashCommand(msg); if (expansion.kind === "expand") { - if (rejectsOversizedPrompt(expansion.prompt, attachedImagesRef.current)) return; - onPromptWithStreamingBehavior(expansion.prompt, streamingBehavior, attachedImagesRef.current.length ? attachedImagesRef.current : undefined); + const prompt = composeMessageWithTextAttachments(expansion.prompt, files); + if (rejectsOversizedPrompt(prompt, attachedImagesRef.current)) return; + recoverOnFailure(onPromptWithStreamingBehavior(prompt, streamingBehavior, images), prompt); clearInput(); return; } @@ -1291,29 +1290,29 @@ export const ChatInput = memo(forwardRef(function ChatIn })); return; } - if (rejectsOversizedPrompt(msg, attachedImagesRef.current)) return; - onPromptWithStreamingBehavior(msg, streamingBehavior, attachedImagesRef.current.length ? attachedImagesRef.current : undefined); + const prompt = composeMessageWithTextAttachments(msg, files); + if (rejectsOversizedPrompt(prompt, attachedImagesRef.current)) return; + recoverOnFailure(onPromptWithStreamingBehavior(prompt, streamingBehavior, images), prompt); clearInput(); return; } - if (rejectsOversizedPrompt(msg, attachedImagesRef.current)) return; + const composedMessage = composeMessageWithTextAttachments(msg, files); + if (rejectsOversizedPrompt(composedMessage, attachedImagesRef.current)) return; if (mode === "steer" && onSteer) { - onSteer(msg, attachedImagesRef.current.length ? attachedImagesRef.current : undefined); + recoverOnFailure(onSteer(composedMessage, images), composedMessage); } else if (mode === "followup" && onFollowUp) { - onFollowUp(msg, attachedImagesRef.current.length ? attachedImagesRef.current : undefined); + recoverOnFailure(onFollowUp(composedMessage, images), composedMessage); } clearInput(); }, [value, onPromptWithStreamingBehavior, onSteer, onFollowUp, clearInput, onAudioUnlock, t, advisorEnabled, rejectsOversizedPrompt, sendSideQuestion]); - // A typed, text-only message during a run is a queued follow-up — and so is - // a dictation in progress: the primary button must take the same state it + // A typed message or attachment during a run is a queued follow-up — and so + // is a dictation in progress: the primary button must take the same state it // would have if the composer already held text. Keep Stop as the action // while the composer is empty and nothing is being recorded. const dictationCapturing = isRecording || isPaused || isReviewing; const primaryActionQueuesMessage = isStreaming - && (Boolean(value.trim()) || dictationCapturing) - && attachedImages.length === 0 - && attachedTextFiles.length === 0 + && (Boolean(value.trim()) || attachedImages.length > 0 || attachedTextFiles.length > 0 || dictationCapturing) && Boolean(onFollowUp); // ── Queued follow-up bar ──────────────────────────────────────────────── @@ -1351,7 +1350,8 @@ export const ChatInput = memo(forwardRef(function ChatIn setQueuedDeleteTarget(null); if (!removed || action !== "edit") return; // Recover through the store even if a new composer now owns this key. - recoverDraftText(key, entry.text); + // omp labels an image-only message "[Image]": not text when it has images. + recoverDraft(key, { text: entry.text === "[Image]" && removed.length > 0 ? "" : entry.text, images: removed }); } catch (error) { setQueuedDeleteTarget(null); toast.error(error instanceof Error ? error.message : String(error)); @@ -1780,9 +1780,8 @@ export const ChatInput = memo(forwardRef(function ChatIn const attachmentMenuItemStyle: React.CSSProperties = { display: "flex", alignItems: "center", gap: 8, width: "100%", padding: "7px 10px", border: 0, borderRadius: 5, - background: "transparent", color: isStreaming ? "var(--text-dim)" : "var(--text-muted)", - cursor: isStreaming ? "not-allowed" : "pointer", fontSize: 12, textAlign: "left", - opacity: isStreaming ? 0.5 : 1, + background: "transparent", color: "var(--text-muted)", + cursor: "pointer", fontSize: 12, textAlign: "left", }; return ( @@ -1825,7 +1824,6 @@ export const ChatInput = memo(forwardRef(function ChatIn // only hide files the app can attach (code, config, logs, ...). accept="*/*" multiple - disabled={isStreaming} style={{ display: "none" }} onChange={(e) => { const files = Array.from(e.target.files ?? []); @@ -2744,7 +2742,6 @@ export const ChatInput = memo(forwardRef(function ChatIn role="menuitem" type="button" onClick={() => { setPlusMenuOpen(false); fileInputRef.current?.click(); }} - disabled={isStreaming} title={t("chatInput.attachFile")} style={attachmentMenuItemStyle} > @@ -2756,7 +2753,6 @@ export const ChatInput = memo(forwardRef(function ChatIn role="menuitem" type="button" onClick={() => void pasteClipboardImage()} - disabled={isStreaming} title={t("chatInput.pasteImage")} style={attachmentMenuItemStyle} > diff --git a/components/ChatWindow.tsx b/components/ChatWindow.tsx index 6257d52d..67c87323 100644 --- a/components/ChatWindow.tsx +++ b/components/ChatWindow.tsx @@ -837,9 +837,8 @@ export function ChatWindow({ session, newSessionCwd, newSessionWorkspace, toolCa const onDrop = useCallback((files: File[]) => { - if (sessionBusy) return; chatInputRef?.current?.addFiles(files); - }, [sessionBusy, chatInputRef]); + }, [chatInputRef]); const { isDragOver, handleDragEnter, handleDragOver, handleDragLeave, handleDrop } = useDragDrop(onDrop); @@ -1143,7 +1142,7 @@ export function ChatWindow({ session, newSessionCwd, newSessionWorkspace, toolCa onDragLeave={handleDragLeave} onDrop={handleDrop} > - {isDragOver && !sessionBusy && ( + {isDragOver && (
{[0, 0.8, 1.6].map((delay) => ( diff --git a/hooks/useAgentSession.rpc.test.mjs b/hooks/useAgentSession.rpc.test.mjs index b2ff37c8..fdd6a7e6 100644 --- a/hooks/useAgentSession.rpc.test.mjs +++ b/hooks/useAgentSession.rpc.test.mjs @@ -363,7 +363,12 @@ test("the queue panel shows omp's snapshot on load and follows queue_update, not test("queued cancellation reports omp's answer and leaves the chip to the snapshot", async (t) => { for (const outcome of [ - { name: "removed", response: { value: { success: true, data: { removed: true } } }, result: true, notices: [] }, + { + name: "removed", + response: { value: { success: true, data: { removed: true, images: [{ type: "image", data: "AAAA", mimeType: "image/png" }, { bogus: true }] } } }, + result: [{ data: "AAAA", mimeType: "image/png" }], + notices: [], + }, { name: "not pending", response: { value: { success: true, data: { removed: false } } }, result: false, notices: ["warning"] }, { name: "unsupported", response: { status: 400, value: { error: "Unknown RPC command: remove_queued_message" } }, result: false, notices: ["error"] }, ]) { @@ -390,7 +395,7 @@ test("queued cancellation reports omp's answer and leaves the chip to the snapsh ]); await act(async () => { release(outcome.response); - assert.equal(await first, outcome.result); + assert.deepEqual(await first, outcome.result); }); assert.deepEqual(w.latest.queuedMessages, { steering: [], followUp: ["target", "target"] }); assert.deepEqual(w.latest.notices.map((n) => n.type), outcome.notices); @@ -581,7 +586,7 @@ test("a cancellation omp confirms after unmount still reports success so Edit ca await act(async () => { cancellation = w.latest.removeQueuedMessage("target", "followUp"); }); w.unmount(); release({ value: { success: true, data: { removed: true } } }); - assert.equal(await cancellation, true); + assert.deepEqual(await cancellation, [], "an omp that returns no images still confirms the removal"); }); test("a Stop whose run ended during the withdrawal does not abort the next prompt", async () => { @@ -3190,8 +3195,11 @@ test("Stop takes queued input back through omp in one step, including a steer th primeSession("abort-atomic", [userMsg("u0", "loaded question")]); const { w } = await startRun("abort-atomic", "hello agent"); world.abortRestoreQueue = { - steering: [{ text: "steer the snapshot missed" }], - followUp: [{ text: "later follow-up", images: [{ type: "image", data: "x", mimeType: "image/png" }] }], + steering: [{ text: "steer the snapshot missed" }, { text: "[Image]" }], + followUp: [ + { text: "later follow-up", images: [{ type: "image", data: "x", mimeType: "image/png" }] }, + { text: "[Image]", images: [{ type: "image", data: "y", mimeType: "image/webp" }] }, + ], }; await act(async () => { await w.latest.handleAbort(); }); @@ -3200,7 +3208,8 @@ test("Stop takes queued input back through omp in one step, including a steer th assert.equal(commands.filter((type) => type === "abort_and_restore_queue").length, 1); assert.equal(commands.includes("abort"), false, "omp's own abort already stopped the run"); assert.equal(commands.includes("remove_queued_message"), false); - assert.equal(getDraft("abort-atomic")?.value, "steer the snapshot missed\n\nlater follow-up"); + assert.equal(getDraft("abort-atomic")?.value, "steer the snapshot missed\n\n[Image]\n\nlater follow-up", "only an image-only message's label is dropped"); + assert.deepEqual(getDraft("abort-atomic")?.images, [{ data: "x", mimeType: "image/png" }, { data: "y", mimeType: "image/webp" }]); assert.deepEqual(w.latest.notices, []); clearDraft("abort-atomic"); }); diff --git a/hooks/useAgentSession.ts b/hooks/useAgentSession.ts index 4ede381a..7d68fdfa 100644 --- a/hooks/useAgentSession.ts +++ b/hooks/useAgentSession.ts @@ -16,7 +16,7 @@ import { normalizeToolCalls } from "@/lib/normalize"; import { hasVisibleAssistantContent } from "@/lib/assistant-response"; import type { ThinkingModelMeta } from "@/lib/thinking-levels"; import { sendAgentCommand, setSessionAdvisorSpawn } from "@/lib/agent-client"; -import { recoverDraftText, setDraft } from "@/lib/draft-store"; +import { recoverDraft, setDraft, toDraftImages, type ChatDraftImage } from "@/lib/draft-store"; import { formatExitedSessionNotice, translate } from "@/lib/i18n"; import { usePrefersReducedMotion } from "@/hooks/usePrefersReducedMotion"; import { toastBtwError, useBtw } from "@/hooks/useBtw"; @@ -1837,8 +1837,10 @@ export function useAgentSession(opts: UseAgentSessionOptions) { /** Cancel one pending message in omp. The chip itself leaves through omp's - * queue snapshot; resolves true only when omp confirms the removal. */ - const removeQueuedMessage = useCallback(async (text: string, queue: keyof QueuedMessages): Promise => { + * queue snapshot; resolves to the removed message's images (empty when it + * had none or omp does not return them) only when omp confirms the + * removal, else false. */ + const removeQueuedMessage = useCallback(async (text: string, queue: keyof QueuedMessages): Promise => { const sid = sessionIdRef.current; if (!hookAliveRef.current || !sid || !text) return false; if (!queuedMessagesRef.current[queue].includes(text)) { @@ -1849,10 +1851,10 @@ export function useAgentSession(opts: UseAgentSessionOptions) { const removal = { sessionId: sid }; queuedRemovalRef.current = removal; try { - const result = await sendAgentCommand<{ removed: boolean }>(sid, { + const result = await sendAgentCommand<{ removed: boolean; images?: unknown }>(sid, { type: "remove_queued_message", message: text, queue, }); - if (result?.removed === true) return true; + if (result?.removed === true) return toDraftImages(result.images); if (hookAliveRef.current && sessionIdRef.current === sid) { addNotice({ type: "warning", message: translate("agentSession.queuedRemovalUnavailable") }); } @@ -2830,10 +2832,11 @@ export function useAgentSession(opts: UseAgentSessionOptions) { // aborts, in one step: it also catches a steer this client's queue // snapshot does not list yet, or one the run already claimed but never // recorded, which a removal by text cannot reach and which omp would - // otherwise run as a new turn right after the abort. The texts return to - // this session's draft (attached images are dropped). Known window: the - // texts exist only in the response until the abort finishes, so a reload - // during a slow abort loses them. + // otherwise run as a new turn right after the abort. The texts and images + // return to this session's draft (omp drops the images from a response + // over its transport limit). Known window: they exist only in the + // response until the abort finishes, so a reload during a slow abort + // loses them. type RestoredQueue = { steering?: Array<{ text?: unknown; images?: unknown[] }>; followUp?: Array<{ text?: unknown; images?: unknown[] }> }; let restoredQueue: RestoredQueue | null | undefined; let unsupported = false; @@ -2852,13 +2855,15 @@ export function useAgentSession(opts: UseAgentSessionOptions) { if (!unsupported) { const steering = Array.isArray(restoredQueue?.steering) ? restoredQueue.steering : []; const followUp = Array.isArray(restoredQueue?.followUp) ? restoredQueue.followUp : []; - const texts = [...steering, ...followUp] - // An image-only message comes back as omp's "[Image]" chip label; the - // images themselves are not restored, so neither is the label. + const entries = [...steering, ...followUp]; + const texts = entries + // An image-only message comes back as omp's "[Image]" chip label, + // which is not text to put back; a literal "[Image]" without images is. .filter((entry) => !(entry.text === "[Image]" && Array.isArray(entry.images) && entry.images.length > 0)) .map((entry) => entry.text) .filter((text): text is string => typeof text === "string" && text.length > 0); - if (texts.length > 0) recoverDraftText(sid, texts.join("\n\n")); + const images = entries.flatMap((entry) => toDraftImages(entry.images)); + if (texts.length > 0 || images.length > 0) recoverDraft(sid, { text: texts.join("\n\n"), images }); // A failed attempt may have withdrawn messages whose texts were in the // lost response; say so unless the retry brought texts back. if (failures > 0 && texts.length === 0 && hookAliveRef.current && sessionIdRef.current === sid) { @@ -2887,7 +2892,7 @@ export function useAgentSession(opts: UseAgentSessionOptions) { if (fresh.length === 0) return; entries.forEach((_, i) => { if (removed[i]) restored[i] = true; }); const block = entries.filter((_, i) => restored[i]).map((entry) => entry.text).join("\n\n"); - recoverDraftText(sid, block, recoveredBlock ? { lead: recoveredBlock, fallback: fresh.join("\n\n") } : undefined); + recoverDraft(sid, { text: block, replace: recoveredBlock ? { lead: recoveredBlock, fallback: fresh.join("\n\n") } : undefined }); recoveredBlock = block; }; const removals = Promise.all(entries.map(async (entry, i) => { @@ -3439,7 +3444,7 @@ export function useAgentSession(opts: UseAgentSessionOptions) { const toPiImages = (images?: AttachedImage[]) => images?.map((img) => ({ type: "image" as const, data: img.data, mimeType: img.mimeType })); const handleSteer = useCallback(async (message: string, images?: AttachedImage[]) => { const sid = sessionIdRef.current; - if (!sid) return; + if (!sid) return false; const piImages = toPiImages(images); try { await sendAgentCommand(sid, { @@ -3447,12 +3452,13 @@ export function useAgentSession(opts: UseAgentSessionOptions) { message, ...(piImages?.length ? { images: piImages } : {}), }); + return true; } catch (e) { console.error("Failed to steer:", e); addNotice({ type: "error", message: e instanceof Error ? e.message : String(e) }); - opts.chatInputRef?.current?.insertIfEmpty(message); + return false; } - }, [addNotice, opts.chatInputRef]); + }, [addNotice]); const handlePromptWithStreamingBehavior = useCallback(async ( message: string, @@ -3460,7 +3466,7 @@ export function useAgentSession(opts: UseAgentSessionOptions) { images?: AttachedImage[], ) => { const sid = sessionIdRef.current; - if (!sid) return; + if (!sid) return false; const piImages = toPiImages(images); try { await sendAgentCommand(sid, { @@ -3469,16 +3475,17 @@ export function useAgentSession(opts: UseAgentSessionOptions) { streamingBehavior: behavior, ...(piImages?.length ? { images: piImages } : {}), }); + return true; } catch (e) { console.error("Failed to queue prompt:", e); addNotice({ type: "error", message: e instanceof Error ? e.message : String(e) }); - opts.chatInputRef?.current?.insertIfEmpty(message); + return false; } - }, [addNotice, opts.chatInputRef]); + }, [addNotice]); const handleFollowUp = useCallback(async (message: string, images?: AttachedImage[]) => { const sid = sessionIdRef.current; - if (!sid) return; + if (!sid) return false; const piImages = toPiImages(images); try { await sendAgentCommand(sid, { @@ -3486,12 +3493,13 @@ export function useAgentSession(opts: UseAgentSessionOptions) { message, ...(piImages?.length ? { images: piImages } : {}), }); + return true; } catch (e) { console.error("Failed to follow up:", e); addNotice({ type: "error", message: e instanceof Error ? e.message : String(e) }); - opts.chatInputRef?.current?.insertIfEmpty(message); + return false; } - }, [addNotice, opts.chatInputRef]); + }, [addNotice]); const handleAbortCompaction = useCallback(async () => { const sid = sessionIdRef.current; diff --git a/lib/draft-store.ts b/lib/draft-store.ts index 7bb97917..3fe1416f 100644 --- a/lib/draft-store.ts +++ b/lib/draft-store.ts @@ -16,15 +16,29 @@ export interface ChatDraft { files: ChatDraftFile[]; } -/** Text to put back in a composer. `replace` merges a later recovery with an - * earlier one in order: while the draft still starts with `lead` (the earlier - * block), it becomes `text`; otherwise only `fallback` is prepended. */ +/** Content to put back in a composer. `replace` merges a later recovery with + * an earlier one in order: while the draft still starts with `lead` (the + * earlier block), it becomes `text`; otherwise only `fallback` is prepended. + * `images` are appended to the draft's attachments. */ export interface DraftRecovery { text: string; replace?: { lead: string; fallback: string }; + images?: ChatDraftImage[]; +} + +/** The `{ data, mimeType }` images in an omp RPC payload; anything else is dropped. */ +export function toDraftImages(value: unknown): ChatDraftImage[] { + if (!Array.isArray(value)) return []; + return value.flatMap((image: unknown) => ( + image && typeof image === "object" && "data" in image && "mimeType" in image + && typeof image.data === "string" && typeof image.mimeType === "string" + ? [{ data: image.data, mimeType: image.mimeType }] + : [] + )); } export function mergeRecoveredText(current: string, { text, replace }: DraftRecovery): string { + if (!text) return current; if (replace && current.startsWith(replace.lead)) return text + current.slice(replace.lead.length); const lead = replace ? replace.fallback : text; return current ? `${lead}\n\n${current}` : lead; @@ -99,11 +113,14 @@ export function getDraft(key: string): ChatDraft | null { return draft ? cloneDraft(draft) : null; } -export function recoverDraftText(key: string, text: string, replace?: DraftRecovery["replace"]): void { - const recovery = { text, replace }; +export function recoverDraft(key: string, recovery: DraftRecovery): void { if (key) { const draft = getDraft(key) ?? { value: "", images: [], files: [] }; - setDraft(key, { ...draft, value: mergeRecoveredText(draft.value, recovery) }); + setDraft(key, { + ...draft, + value: mergeRecoveredText(draft.value, recovery), + images: [...draft.images, ...(recovery.images ?? [])], + }); } // Publish the recovery intent separately: ordinary persistence must not // reapply it, and the mounted composer may have React updates still queued. diff --git a/lib/i18n/locales/en.json b/lib/i18n/locales/en.json index c9f91a4e..6c93a5ce 100644 --- a/lib/i18n/locales/en.json +++ b/lib/i18n/locales/en.json @@ -306,7 +306,6 @@ "chatInput.attachmentFilesSkipped": "{count} file(s) skipped.", "chatInput.attachmentBinarySkipped": "{count} file(s) skipped: binary or non-text files cannot be attached.", "chatInput.attachmentTextReadFailed": "One or more files could not be read. Try a different file.", - "chatInput.attachmentsDisabled": "Attachments are disabled while the agent is running.", "chatInput.abortRetry": "Abort retry", "chatInput.abortRetryTitle": "Stop automatic retry and leave the failed turn as-is", "chatInput.removeImage": "Remove image", @@ -323,7 +322,6 @@ "chatInput.dictationPermissionDenied": "Microphone access denied", "chatInput.dictationTimedOut": "Transcription timed out", "chatInput.dictationSuccess": "Transcribed", - "chatInput.dictationKeptWithAttachments": "Transcribed into the composer: a message with attachments can't be queued or steered", "chatInput.dictationRecording": "Recording", "chatInput.pauseDictation": "Pause recording", "chatInput.resumeDictation": "Resume recording", @@ -428,7 +426,6 @@ "chatInput.groupPrompts": "Prompts", "chatInput.groupSkills": "Skills", "chatInput.dormant": "dormant", - "chatInput.imagesCannotQueue": "Image attachments cannot be queued while the agent is running", "chatInput.indexTruncated": "index truncated", "chatInput.inputHistory": "Input history", "chatInput.kindFollowUp": "follow-up", diff --git a/lib/i18n/locales/ja.json b/lib/i18n/locales/ja.json index 04e93326..f09e4c4c 100644 --- a/lib/i18n/locales/ja.json +++ b/lib/i18n/locales/ja.json @@ -293,7 +293,6 @@ "chatInput.attachmentFilesSkipped": "{count} 件のファイルをスキップしました。", "chatInput.attachmentBinarySkipped": "{count} 件のファイルをスキップしました：バイナリまたは非テキストファイルは添付できません。", "chatInput.attachmentTextReadFailed": "ファイルを読み取れませんでした。別のファイルを試してください。", - "chatInput.attachmentsDisabled": "エージェントの実行中はファイルを添付できません。", "chatInput.abortRetry": "再試行を中止", "chatInput.abortRetryTitle": "自動再試行を停止し、失敗したターンをそのまま保持します", "chatInput.removeImage": "画像を削除", @@ -310,7 +309,6 @@ "chatInput.dictationPermissionDenied": "マイクへのアクセスが拒否されました", "chatInput.dictationTimedOut": "文字起こしがタイムアウトしました", "chatInput.dictationSuccess": "文字起こしが完了しました", - "chatInput.dictationKeptWithAttachments": "入力欄に文字起こししました。添付ファイル付きのメッセージはキュー追加も割り込み送信もできません", "chatInput.dictationRecording": "録音中", "chatInput.pauseDictation": "録音を一時停止", "chatInput.resumeDictation": "録音を再開", @@ -415,7 +413,6 @@ "chatInput.groupPrompts": "プロンプト", "chatInput.groupSkills": "スキル", "chatInput.dormant": "休止中", - "chatInput.imagesCannotQueue": "エージェントの実行中は画像添付をキューに入れられません", "chatInput.indexTruncated": "インデックスは一部省略", "chatInput.inputHistory": "入力履歴", "chatInput.kindFollowUp": "フォローアップ", diff --git a/lib/i18n/locales/zh-CN.json b/lib/i18n/locales/zh-CN.json index 2bd433e5..bf3c69e6 100644 --- a/lib/i18n/locales/zh-CN.json +++ b/lib/i18n/locales/zh-CN.json @@ -293,7 +293,6 @@ "chatInput.attachmentFilesSkipped": "已跳过 {count} 个文件。", "chatInput.attachmentBinarySkipped": "已跳过 {count} 个文件：二进制或非文本文件无法附加。", "chatInput.attachmentTextReadFailed": "无法读取一个或多个文件，请尝试其他文件。", - "chatInput.attachmentsDisabled": "智能体运行时无法附加文件。", "chatInput.abortRetry": "中止重试", "chatInput.abortRetryTitle": "停止自动重试并保留失败回合", "chatInput.removeImage": "移除图片", @@ -310,7 +309,6 @@ "chatInput.dictationPermissionDenied": "麦克风访问被拒绝", "chatInput.dictationTimedOut": "转写超时", "chatInput.dictationSuccess": "转写完成", - "chatInput.dictationKeptWithAttachments": "已转写到输入框：带附件的消息无法加入队列，也无法作为打断发送", "chatInput.dictationRecording": "录音中", "chatInput.pauseDictation": "暂停录音", "chatInput.resumeDictation": "恢复录音", @@ -415,7 +413,6 @@ "chatInput.groupPrompts": "提示词", "chatInput.groupSkills": "技能", "chatInput.dormant": "休眠", - "chatInput.imagesCannotQueue": "智能体运行时无法排队图片附件", "chatInput.indexTruncated": "索引已截断", "chatInput.inputHistory": "输入历史", "chatInput.kindFollowUp": "追加消息",