Zero-dependency AI agent skill for controlling native WinUI 3 desktop apps. Drive any Windows App SDK application through the UI Automation tree with plain PowerShell — no screenshots, no OCR, no installs.
中文说明见 README.zh-CN.md。
When an AI agent needs to operate a Windows desktop app, the default playbook is screenshot → OCR/guess coordinates → simulate mouse. For WinUI 3 apps there is a better path: every control is already exposed in the Windows UI Automation tree, with stable identifiers and live coordinates.
| Screenshot + OCR | This skill (UIA tree) | |
|---|---|---|
| Locating | read pixels → guess coordinates | AutomationId / class / title → live coordinates |
| DPI & resolution changes | fragile (coordinates go stale) | robust (re-read per step) |
| Window moved / resized | fragile | robust (rectangles re-resolved) |
| Non-Latin text | OCR misreads | exact text from the tree |
| Failure mode | silently clicks the wrong thing | pattern not found → stops |
| Installs | vision stack, extra tools | none |
Zero dependencies means exactly that: Windows PowerShell 5.1+ (or PowerShell
7), the .NET UIAutomation assemblies that ship with Windows, and user32.dll
P/Invoke. Nothing to install — which is the point for locked-down machines,
ephemeral CI runners, and agent sandboxes where you cannot add tooling.
- Window discovery — find WinUI 3 top-level windows by class
(
WinUIDesktopWin32WindowClass) and exact title, across multiple apps. - Accessibility-tree inspection — dump control types, names, AutomationIds, supported patterns before acting.
- Pattern-driven actions — invoke buttons (
InvokePattern), select tabs/items (SelectionItemPattern), read/write values where supported. - Real mouse input — for canvas/owner-drawn controls with no usable
pattern, via
user32with fresh bounding rectangles. - DPI-aware screenshot verification — capture the target window as objective evidence of what happened.
- Launch verification —
MainWindowHandle -ne 0+ expectedMainWindowTitle; a spawned process alone is not a launch.
It also encodes the discipline that makes this reliable in agent hands:
match by AutomationId first, re-observe the tree after every action, never
reuse coordinates, fail closed on ambiguity.
Copy this folder into your agent's skills directory:
| Harness | Path |
|---|---|
| Claude Code | ~/.claude/skills/ |
| Codex | ~/.codex/skills/ |
| ZCode | ~/.zcode/skills/ |
Any .agents-compatible harness |
~/.agents/skills/ |
Or use the PowerShell recipes standalone — every code block in SKILL.md runs by itself on Windows.
- microsoft/winappCli — the
official
winappCLI covers similar ground (winapp uiinspect/invoke/ click/…) and is excellent if you can install it. This skill is the zero-install counterpart, plus the operational knowledge (re-observation discipline, version-brittleness notes) that turns raw UIA access into dependable agent behavior. Ifwinappis present, prefer it; this skill is why you are not blocked when it isn't. - Vision-based CUA frameworks (trycua/cua etc.) — different layer: they drive pixels for arbitrary apps. This skill drives the accessibility tree for WinUI 3 specifically: deterministic, no vision model, no coordinate guessing.
Developed and verified against a real WinUI 3 app across a 14-version jump — including what survived (window class, patterns, dialog AutomationIds) and what broke (tab labels, view-model type names leaking into UIA names). See references/verified-apps.md. Contributions of new verified-app notes are welcome.
UIA-pattern operations are in-app actions, not OS-level mutations. Real-mouse input is reserved for controls with no usable pattern and always targets a freshly re-read rectangle. The skill instructs agents to stay at navigation level and read-back verification unless the user explicitly requests destructive actions.