Skip to content

Feature: add a named image proxy for text-only main models #1728

Description

@Aaronontheweb

Goal

Let a text-only main model use a named vision model to create a durable image description.

Netclaw must preserve the original image. A later vision-capable main model must receive the original image again.

Configuration

Add an optional named assignment:

{
  "Models": {
    "Definitions": {
      "main-text": {
        "Provider": "primary-provider",
        "ModelId": "text-model"
      },
      "vision-small": {
        "Provider": "vision-provider",
        "ModelId": "vision-model",
        "InputModalities": ["Text", "Image"],
        "OutputModalities": ["Text"]
      }
    },
    "Roles": {
      "Main": "main-text"
    },
    "Proxies": {
      "Image": "vision-small"
    }
  }
}

Models.Proxies.Image references an existing named definition.

Startup must fail when the reference is unknown. Startup must also fail when the model lacks image input or text output.

Runtime foundation

Add the smallest reusable slice of #648.

  • Resolve named model definitions at runtime.
  • Cache each composed client pipeline.
  • Expose the effective capabilities with each client.
  • Keep the current role-based API as an adapter.
  • Do not add subagent model selection in this issue.

This registry can later support explicit subagent model selection.

Image proxy behavior

  • Use the proxy only when the selected main model lacks image input.
  • Send one image and a fixed OCR-aware description prompt.
  • Do not send the complete session history.
  • Do not expose tools to the proxy.
  • Persist the proxy result before the main model call.
  • Record the source media path, proxy definition, model ID, prompt version, and timestamp.
  • Insert the derived description as untrusted user content.
  • Include the session-relative image path.
  • Neutralize proxy-output delimiters before prompt insertion.
  • Reuse the durable result after session recovery.
  • Create missing results lazily for historical images.
  • Stop the turn when proxy analysis fails or returns empty text.
  • Do not silently omit the image.

When the main model supports images, Netclaw sends the original image and omits the proxy description.

Operator surfaces

Support:

netclaw model set image-proxy <provider> <model-id>
netclaw model clear image-proxy
netclaw model list

Add the same optional assignment to the interactive model manager.

The model command must reuse existing named definitions and preserve their metadata.

Acceptance criteria

  • A text-only main model receives a durable image description.
  • The proxy runs once for each missing image analysis.
  • Session recovery reuses the saved description.
  • Historical images receive lazy proxy analysis.
  • A later vision-capable main model receives the original image.
  • Invalid proxy references fail before persistence or startup.
  • Proxy failure does not call the main model.
  • CLI and TUI tests cover assignment, removal, and validation.
  • Native TUI smoke coverage proves the interactive path.

Non-goals

Related issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    context-pipelineLLM context assembly: prompt layers, dynamic injection, memory recall, temporal groundingenhancementNew feature or requestprovidersProvider integrations and capability detection across OpenAI-compatible backends.sessionsLLM session actor, turn lifecycle, pipelinestuiTerminal UI (Termina) issues

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions