Skip to content

[feature] Add inspect_image tool for visual review of generated images #2283

Open

Description

@henryZhouLikeStudy

Summary

After generate_image completes, the agent receives only workspace paths for the generated artifacts. It cannot visually review the image because:

  • read_file rejects binary content;
  • the server is headless, so desktop capture has no X11 display;
  • browser tools do not load local file:// images;
  • OCR can verify literal text but cannot detect layout defects, overlaps, clipping, alignment, spacing, or visual hierarchy.

Multimodal conversation models can inspect images, but the generated artifact is not currently fed back as multimodal input.

Proposal

Add an optional tool plugin, image-inspector, exposing:

inspect_image(path, focus=None)

The tool should:

  • resolve an image only inside the Octop workspace;
  • accept PNG, JPEG, and WebP files up to a bounded size;
  • return the same multimodal [image_block, text_block] shape already used by browser_use screenshots;
  • be invoked manually when the agent needs visual QA;
  • update the generate_image tool guidance so deliverable images are visually checked when needed.

Initial version should not fetch arbitrary URLs. That can remain a separate extension after SSRF and content-size handling are defined.

Why a plugin

A standalone tool plugin can be installed on the current release without modifying Octop core, and it can also be maintained independently if the upstream project does not accept it. The same directory can later be merged into the official repository.

Acceptance criteria

  • A multimodal model can inspect a generated workspace image and report layout or visual defects.
  • Missing files, directories, non-images, oversized files, and paths outside the workspace fail with clear errors.
  • Text-only models remain unaffected when the plugin is disabled or not enabled for the agent.
  • Installing or removing the plugin does not modify the Octop package.

Environment

  • Octop: v1.0.2
  • Model capability: multimodal model with text and image inputs

Activity

  1. henryZhouLikeStudy commented on Oct 21, 2026

    @henryZhouLikeStudy
    Author

    Implemented in #2286.

    During live validation on Octop v1.0.2, returning image blocks directly from a plugin tool result was converted to text before the next model call. The final implementation therefore invokes the Agent's resolved auto multimodal model from inside inspect_image, sends the workspace image as a Base64 image_url, and returns a textual visual review to the outer Agent turn.

    The plugin remains independently installable and does not modify Octop core. Tests cover workspace boundaries, supported image formats, size limits, missing files, directories, and the multimodal model request. No credentials, server addresses, or user images are included.

  2. henryZhouLikeStudy commented on Oct 23, 2026

    @henryZhouLikeStudy
    Author

    Follow-up validation uncovered a parent-stream isolation issue in the first implementation. Because inspect_image invoked the Agent's auto model from inside the tool call, LangGraph exposed that nested model's tokens through the parent message stream. IM channels could therefore send the internal visual-review text before the final answer, and history could persist it as an extra assistant message.

    Fixed in #2286 by invoking the dedicated model through an isolated call configuration:

    • inherited run callbacks are disabled;
    • the nested call is marked with lc_source=image_inspector;
    • the visual review remains available only as the inspect_image tool result.

    Added a regression test that installs a parent chat-model callback and verifies that it receives no model-start event from the nested review request.

    Validation now includes 28 passed for the image-inspector and plugin-system tests, full Ruff checks, and mypy across 609 source files. No user images, chat content, credentials, server addresses, or other account data are included.

  3. henryZhouLikeStudy commented on Oct 29, 2026

    @henryZhouLikeStudy
    Author

    为什么需要修改

    image-inspector 最初把插件进程当成普通本地文件访问者,只尝试从 $OCTOP_HOME/workspaces 读取路径。这只覆盖了 local backend 和默认数据目录,无法覆盖实际部署中的两种关键情况:

    1. 自托管部署可能通过挂载卷使用非默认数据目录,实际根目录来自 OCTOP_HOME 或 OCTOP_DATA_DIR,不一定是 ~/.octop。
    2. Docker sandbox 中生成的文件只存在于容器文件系统。工具通过 send_file_to_user 等通道交付后,主机侧副本才出现在 .harness-materialize/。因此 Agent 返回的 localPath 可能对应虚拟工作区路径,而服务器进程不能直接按该路径读取文件。

    这会导致一个本应存在的图片在 inspect_image 中被误报为 image file not found。图片可见性和边界校验不能只假设一种宿主目录结构,否则图片质检能力在 Docker sandbox、自定义数据目录以及 agents//workspaces/ 两套工作区布局中不可用。

    修改内容

    PR: #2286

    验证提交:613a8b7a636cbb409d90fff51e4c7dcd7e6ec2d6

    • 从 OCTOP_HOME、OCTOP_DATA_DIR 和默认 ~/.octop 推导数据根;
    • 只允许访问 Octop agents/ 或 workspaces/ 下的文件;
    • 将 Agent 可见的绝对路径重映射到主机侧 agents/<id>/ 或 workspaces/<id>/;
    • 对 Docker sandbox 文件回退到 <workspace>/.harness-materialize/;
    • 支持唯一的 Agent 工作区相对路径,歧义时拒绝猜测;
    • 文件仍不可用时返回可操作的 materialize 提示。

    验证结果

    • 插件及路径回归测试:31 passed
    • Ruff、格式检查和 mypy 通过
    • 在自托管 Octop 生产实例完成热部署,Octop 重启后 4 秒恢复健康,5 个 Agent 正常运行,启动后无 warning
    • 使用一张不含用户数据的合成测试图,只在主机侧 .harness-materialize/analysis/... 放置文件,同时向工具传入不同的虚拟 workspaces/<id>/analysis/... 路径
    • 修复后的解析器成功找到 materialize 文件,线上真实多模态模型准确识别出右侧卡片、圆角、边框和文字被画布裁切

    本记录不包含用户图片、聊天内容、账号标识、服务器地址、API Key 或 token。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions