You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[feature] Add inspect_image tool for visual review of generated images #2283
After generate_image completes, the agent receives only workspace paths for the generated artifacts. It cannot visually review the image because:
read_file rejects binary content;
the server is headless, so desktop capture has no X11 display;
browser tools do not load local file:// images;
OCR can verify literal text but cannot detect layout defects, overlaps, clipping, alignment, spacing, or visual hierarchy.
Multimodal conversation models can inspect images, but the generated artifact is not currently fed back as multimodal input.
Proposal
Add an optional tool plugin, image-inspector, exposing:
inspect_image(path, focus=None)
The tool should:
resolve an image only inside the Octop workspace;
accept PNG, JPEG, and WebP files up to a bounded size;
return the same multimodal [image_block, text_block] shape already used by browser_use screenshots;
be invoked manually when the agent needs visual QA;
update the generate_image tool guidance so deliverable images are visually checked when needed.
Initial version should not fetch arbitrary URLs. That can remain a separate extension after SSRF and content-size handling are defined.
Why a plugin
A standalone tool plugin can be installed on the current release without modifying Octop core, and it can also be maintained independently if the upstream project does not accept it. The same directory can later be merged into the official repository.
Acceptance criteria
A multimodal model can inspect a generated workspace image and report layout or visual defects.
Missing files, directories, non-images, oversized files, and paths outside the workspace fail with clear errors.
Text-only models remain unaffected when the plugin is disabled or not enabled for the agent.
Installing or removing the plugin does not modify the Octop package.
Environment
Octop: v1.0.2
Model capability: multimodal model with text and image inputs
During live validation on Octop v1.0.2, returning image blocks directly from a plugin tool result was converted to text before the next model call. The final implementation therefore invokes the Agent's resolved auto multimodal model from inside inspect_image, sends the workspace image as a Base64 image_url, and returns a textual visual review to the outer Agent turn.
The plugin remains independently installable and does not modify Octop core. Tests cover workspace boundaries, supported image formats, size limits, missing files, directories, and the multimodal model request. No credentials, server addresses, or user images are included.
Follow-up validation uncovered a parent-stream isolation issue in the first implementation. Because inspect_image invoked the Agent's auto model from inside the tool call, LangGraph exposed that nested model's tokens through the parent message stream. IM channels could therefore send the internal visual-review text before the final answer, and history could persist it as an extra assistant message.
Fixed in #2286 by invoking the dedicated model through an isolated call configuration:
inherited run callbacks are disabled;
the nested call is marked with lc_source=image_inspector;
the visual review remains available only as the inspect_image tool result.
Added a regression test that installs a parent chat-model callback and verifies that it receives no model-start event from the nested review request.
Validation now includes 28 passed for the image-inspector and plugin-system tests, full Ruff checks, and mypy across 609 source files. No user images, chat content, credentials, server addresses, or other account data are included.
Activity
henryZhouLikeStudy commentedon Oct 21, 2026
Implemented in #2286.
During live validation on Octop
v1.0.2, returning image blocks directly from a plugin tool result was converted to text before the next model call. The final implementation therefore invokes the Agent's resolvedautomultimodal model from insideinspect_image, sends the workspace image as a Base64image_url, and returns a textual visual review to the outer Agent turn.The plugin remains independently installable and does not modify Octop core. Tests cover workspace boundaries, supported image formats, size limits, missing files, directories, and the multimodal model request. No credentials, server addresses, or user images are included.
henryZhouLikeStudy commentedon Oct 23, 2026
Follow-up validation uncovered a parent-stream isolation issue in the first implementation. Because
inspect_imageinvoked the Agent'sautomodel from inside the tool call, LangGraph exposed that nested model's tokens through the parent message stream. IM channels could therefore send the internal visual-review text before the final answer, and history could persist it as an extra assistant message.Fixed in #2286 by invoking the dedicated model through an isolated call configuration:
lc_source=image_inspector;inspect_imagetool result.Added a regression test that installs a parent chat-model callback and verifies that it receives no model-start event from the nested review request.
Validation now includes
28 passedfor the image-inspector and plugin-system tests, full Ruff checks, andmypyacross 609 source files. No user images, chat content, credentials, server addresses, or other account data are included.henryZhouLikeStudy commentedon Oct 29, 2026
为什么需要修改
image-inspector最初把插件进程当成普通本地文件访问者,只尝试从$OCTOP_HOME/workspaces读取路径。这只覆盖了 local backend 和默认数据目录,无法覆盖实际部署中的两种关键情况:OCTOP_HOME或OCTOP_DATA_DIR,不一定是~/.octop。send_file_to_user等通道交付后,主机侧副本才出现在.harness-materialize/。因此 Agent 返回的localPath可能对应虚拟工作区路径,而服务器进程不能直接按该路径读取文件。这会导致一个本应存在的图片在
inspect_image中被误报为image file not found。图片可见性和边界校验不能只假设一种宿主目录结构,否则图片质检能力在 Docker sandbox、自定义数据目录以及agents//workspaces/两套工作区布局中不可用。修改内容
PR: #2286
验证提交:
613a8b7a636cbb409d90fff51e4c7dcd7e6ec2d6OCTOP_HOME、OCTOP_DATA_DIR和默认~/.octop推导数据根;agents/或workspaces/下的文件;agents/<id>/或workspaces/<id>/;<workspace>/.harness-materialize/;验证结果
31 passed.harness-materialize/analysis/...放置文件,同时向工具传入不同的虚拟workspaces/<id>/analysis/...路径本记录不包含用户图片、聊天内容、账号标识、服务器地址、API Key 或 token。