LLM-vision-skill

A Claude Code skill that wraps the Zhipu GLM-4V vision model. When you paste an image, it auto-invokes and converts the image into text the main model can process — scene description + verbatim text transcription + objective image parameters + UI structure.

English · 中文

The main model (glm-5.2, via local gateway) does not accept visual input; this skill bridges the gap by "translating" images into text.

---

What it does

Claude Code's main model glm-5.2 has no multimodal ability — it just replies "I can't process images" when given one. This skill owns the image → text step:

user pastes image → hook detects → glm-vision called → GLM-4V returns text → main model continues

Vision calls go through the Zhipu open platform API (GLM-4V family, generous free tier), separate from the main model token.

---

Features

Feature	Flag	Description
Image understanding	default	Scene summary + verbatim transcription + key details
Objective image params	default output	Size/format/dominant color/brightness/EXIF (via Pillow, no model cost)
Red-box focus	on by default	Auto-detect red annotation boxes, feed bbox coords to focus the model (OpenCV)
Set-of-Mark numbering	on by default	Draw green [1][2] labels on boxes for precise reference
Structured layout	--layout	Output a UI element tree (type|text|position|style) — for UI/code restoration
High-res slicing	--slice	Tile large images, send multiple sub-images (LLaVA-UHD idea) for small text
Multi-image	multiple paths	Compare several images in one call

---

Install

1. Copy files into the Claude Code skills directory

cp -r skills/glm-vision ~/.claude/skills/
cp hooks/glm-vision-trigger.js ~/.claude/hooks/   # optional, recommended

2. Configure the Zhipu API key

Get a key at https://open.bigmodel.cn → console → API Keys → create.

# Option A: env var
echo 'export ZHIPUAI_API_KEY=YOUR_KEY' >> ~/.bashrc

# Option B: skill-local .env (doesn't pollute global env)
echo 'ZHIPUAI_API_KEY=YOUR_KEY' > ~/.claude/skills/glm-vision/scripts/.env

3. Register the hook (optional, for paste auto-trigger)

Add to ~/.claude/settings.json under hooks.UserPromptSubmit:

{
  "hooks": [
    {
      "type": "command",
      "command": "node \"/path/to/.claude/hooks/glm-vision-trigger.js\"",
      "timeout": 5
    }
  ]
}

The hook fires when ~/.claude/image-cache/ has a new image within 90s, or the prompt contains trigger words (look at image / OCR / read image / screenshot).

4. Dependencies

Python 3.10+; packages below are recommended (auto-degrades if missing):

pip install zhipuai pillow opencv-python numpy requests

---

Usage

Paste auto-trigger (recommended)

Ctrl+V a screenshot in Claude Code, or say "look at this image" — the hook injects the instruction, the main model calls the script and continues.

CLI

SCRIPT=~/.claude/skills/glm-vision/scripts/glm_vision.py

python $SCRIPT <image>                          # basic
python $SCRIPT <image> --prompt "transcribe all text"
python $SCRIPT <image> --layout                 # structured UI element tree
python $SCRIPT <image> --slice --max-tiles 6    # tile large image for small text
python $SCRIPT a.png b.png --prompt "compare"  # multi-image
python $SCRIPT <image> --meta-only             # params only, no model call
python $SCRIPT <image> --json                   # structured JSON

Inputs: local path / http(s) URL / data: base64 URI.

Output structure

Default text output has three sections:

## Image parameters (Pillow)        ← size/format/colors/brightness/EXIF
## Red-box detection (OpenCV)       ← bbox coords, only if boxes found
## Vision model understanding       ← GLM-4V description

--json fields: text / vision_text / image_meta / red_boxes / model / usage.

---

Model selection

Model	Notes	max_tokens
glm-4v-flash (default)	Free tier, fast, enough for daily use	1024
glm-4v-plus-9b	Stronger	4096
glm-4v-plus	Strongest; requires enabling a resource pack (else 429)	8192

Switch with --model glm-4v-plus-9b. max_tokens is auto-clamped per model to avoid 400 errors.

---

Directory

LLM-vision-skill/
├── README.md
├── README.zh-CN.md
├── .gitignore
├── skills/glm-vision/
│   ├── SKILL.md                    # skill trigger contract
│   └── scripts/glm_vision.py       # core script
└── hooks/
    └── glm-vision-trigger.js        # paste auto-trigger hook

After install, the live paths are ~/.claude/skills/glm-vision/ and ~/.claude/hooks/. This repo is the source copy.

---

Exit codes

Code	Meaning	Action
0	success	read stdout
2	missing API key	configure key (Install step 2)
3	bad image input	check path
4	model/network error	see stderr

---

Design constraints

• Hard network timeout: all API calls timeout at 60s; failures exit cleanly, never hang the session
• Graceful degradation: zhipuai SDK fails → requests HTTP; missing cv2/Pillow → skip that feature
• Red-box detection: HSV red threshold + morphology + fill-ratio filter (distinguishes hollow box vs solid red block)
• URL images: not downloaded; red-box/SoM detection skipped

---

On "1:1 webpage restoration"

True 1:1 is impossible via the vision path (information-theoretic limit: rasterization is a lossy one-way function; multiple source files render to identical pixels, so inversion has no unique solution). This skill provides high visual similarity restoration (--layout outputs a structured element tree) — the ceiling is "visually similar," not "identical." For true 1:1, bypass vision and grab the source directly (SingleFile / HTTrack). Per Design2Code (arXiv:2403.03163) benchmark findings.

---

Related research

• Design2Code (arXiv:2403.03163) — element recall + layout fidelity bottleneck
• ScreenCoder (arXiv:2507.22827) — bbox grounding + layout tree
• LLaVA-UHD (arXiv:2403.11703) — variable slicing + token compression
• Set-of-Mark (arXiv:2310.11441) — region numbering

---

License

MIT