diff --git a/README.md b/README.md index 651c640..5635fc6 100644 --- a/README.md +++ b/README.md @@ -218,6 +218,8 @@ flowchart LR | Connect Codex, Claude Code, Claude Desktop, VS Code or Cursor | [Connecting agents](docs/CONNECTING-AGENTS.md) | | Share the desktop between agents | [Multiple agents and leases](docs/MULTI-AGENT.md) | | Help an agent discover and choose Anode | [Agent guide](docs/FOR-AGENTS.md) | +| Decide whether Anode fits, or another tool does | [When to use which](docs/COMPARISON.md): Windows-MCP, Cua Driver, Desktop Commander, Windows 365 for Agents | +| Follow a worked example | [Test a desktop app with Claude Code](docs/GUIDE-DESKTOP-APP.md), [fill in a web form](docs/GUIDE-WEB-FORM.md), [check an Android app](docs/GUIDE-ANDROID.md) | | Run apps, control the viewer or use a gamepad | [Usage guide](docs/USAGE.md) | | Look up a CLI command or option | [Command reference](docs/USAGE.md#command-reference) | | Test native apps and browsers | [Development and testing](docs/DEVELOPMENT-TESTING.md) | @@ -235,6 +237,8 @@ flowchart LR ## Limits to know - One seat per Windows session: agents and CLI clients share it and take turns. +- Not for Windows Home or macOS, or when you need a security sandbox: + [when to use which](docs/COMPARISON.md) says what to use instead. - Files, accounts, network ports and application singletons are shared. Steam games run in the seat using the Steam client on your desktop, which stays there, without the overlay or Steam Input; other single-instance apps may already belong to your main session. Use separate browser profiles for diff --git a/assets/guide-android-dark-theme.png b/assets/guide-android-dark-theme.png new file mode 100644 index 0000000..699505b Binary files /dev/null and b/assets/guide-android-dark-theme.png differ diff --git a/assets/guide-web-form.gif b/assets/guide-web-form.gif new file mode 100644 index 0000000..0bd2eee Binary files /dev/null and b/assets/guide-web-form.gif differ diff --git a/docs/ANDROID.md b/docs/ANDROID.md index 43982b4..e691c10 100644 --- a/docs/ANDROID.md +++ b/docs/ANDROID.md @@ -13,7 +13,8 @@ Android Studio opened at its Welcome screen on its own profile; and Chrome opene Anode uses the SDK Android Studio installed: `ANDROID_HOME`, then `ANDROID_SDK_ROOT`, then `%LOCALAPPDATA%\Android\Sdk`, and the AVDs in `ANDROID_AVD_HOME` or `%USERPROFILE%\.android\avd`. The emulator needs a hypervisor: the Windows Hypervisor Platform or the Android Emulator hypervisor -driver, as it does on your desktop. `anode android` shows what Anode finds, and starts nothing. +driver, as it does on your desktop. `anode android` shows what Anode finds, and starts nothing, so it +needs Anode running (`anode start --hidden`); otherwise it says "Anode is not running". ## Emulators diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md new file mode 100644 index 0000000..87abc73 --- /dev/null +++ b/docs/COMPARISON.md @@ -0,0 +1,141 @@ +# When to use Anode, and when to use something else + +Anode gives an agent a second Windows session on your own PC: a desktop with its own screen, pointer +and keyboard focus, so the agent works there while you keep using yours. Other tools solve nearby +problems in other ways, and some of them suit a job better. This page says which, as fairly as we +can. + +Facts about the other tools come from their own pages and repositories, **read on 8 October 2026**. +They change often, so follow the links before you decide. Corrections are welcome as an +[issue](https://github.com/skulitom/Anode/issues/new?template=bug_report.yml). + +## The short answer + +| You want… | Use | +| --- | --- | +| An agent that drives Windows apps and headed browsers while you keep working on the same PC, with nothing moving under your mouse | **Anode** | +| The same without a second session, on macOS or Linux as well, and you can accept that some targets refuse background input | [Cua Driver](#cua-driver) | +| An agent that drives your own desktop while you watch, on any Windows edition, Home included | [Windows-MCP](#windows-mcp) | +| Files, terminal commands and processes, with no GUI at all | [Desktop Commander](#desktop-commander) | +| A whole Windows PC for the agent that isn't yours: managed in the cloud, signed in with your organization's identities | [Windows 365 for Agents](#windows-365-for-agents) | +| A security boundary for untrusted code | A virtual machine or sandbox. None of the local tools here is one, Anode included. | + +## Side by side + +| | Anode | Windows-MCP | Cua Driver | Desktop Commander | Windows 365 for Agents | +| --- | --- | --- | --- | --- | --- | +| Where the agent works | A second Windows session for your account, on your PC | Your own session | Your own session | Your own session (files and shell only) | A Cloud PC in Microsoft's cloud | +| Your pointer and focus | Untouched: the seat has its own | Moved by the agent; you take control back by moving the mouse | Untouched in its default background mode; its foreground mode briefly brings the target forward | Not used | Not on your PC | +| GUI apps | Yes | Yes | Yes | No | Yes | +| Screenshots and accessibility tree | Yes (seat only) | Yes | Yes | No | Yes (accessibility tree of the foreground window) | +| Browsers | Headed Chrome or Edge in the seat, on a profile you sign in to once | DOM mode for Chrome, Edge and Firefox | Chrome and Edge through the DevTools protocol | No | DOM tools on Edge only | +| Android emulators | Yes, booted in the seat | Not mentioned | Not mentioned | No | Not mentioned | +| Windows editions | 10/11 Pro, Enterprise, Education, Server; not Home | Windows 7 to 11 in its README | Windows 10/11, plus macOS and Linux | Windows, macOS, Linux | Cloud PCs; nothing to install locally | +| Price | Free, MIT | Free, MIT | Free, MIT (cloud machines billed separately) | Free locally, MIT; a hosted remote tier | $0.40 an hour in the US, plus $5 a month for an always-available PC | + +## Anode's difference, and its limits + +**The difference.** The seat is a Windows *child session*: Windows' own second session for your +account, with its own desktop, focus and screen ([how it works](ARCHITECTURE.md)). An agent can +click, type into any app, use Chromium pages that ignore background messages, run games with a +virtual controller and boot Android emulators there, without bringing anything to the front of your +desktop. Tools that work inside your session have to share your focus and pointer, or avoid them +target by target; the seat has its own. + +**The limits.** Be sure these fit before you choose it: +- Windows 10/11 **Pro, Enterprise or Education**, or Windows Server, on x64. Windows Home has no + child sessions. `anode setup` needs administrator rights once + ([what it changes](SECURITY.md#what-anode-setup-changes)). +- One seat per Windows session, shared by your agents in turn ([leases](MULTI-AGENT.md)). +- The seat runs as you and shares your files, accounts and network. **Desktop isolation, not a + security sandbox** ([security](SECURITY.md)). +- Single-instance apps may already belong to your main session; use separate browser profiles. +- Builds are unsigned, so SmartScreen may warn. Anode is young and has few users so far. + +## Windows-MCP + +[CursorTouch/Windows-MCP](https://github.com/CursorTouch/Windows-MCP) is a Python MCP server with +about 8,200 GitHub stars, installed with `uvx windows-mcp serve`. It has `Screenshot` and `Snapshot` +(UI Automation) tools, and a DOM mode for Chrome, Edge and Firefox +([README](https://github.com/CursorTouch/Windows-MCP)). + +- It works in your own session and moves the real pointer. Its README says moving the physical + mouse "returns control to the user until 10 seconds of inactivity", and calls that takeover + "best-effort, not a security boundary" + ([README, ControlStatus](https://github.com/CursorTouch/Windows-MCP)). +- Its security notes say it is not sandboxed and suggest Windows Sandbox or a VM for isolation + ([SECURITY.md](https://github.com/CursorTouch/Windows-MCP/blob/main/SECURITY.md)). +- Anonymous telemetry is on by default; `ANONYMIZED_TELEMETRY=false` turns it off + ([README, Telemetry](https://github.com/CursorTouch/Windows-MCP)). + +**Choose it** when you watch the agent work on your own desktop, need Windows Home or older +Windows, or want a mature project with many users. **Choose Anode** when you want to keep using +your mouse and keyboard during the agent's work. + +## Cua Driver + +[Cua Driver](https://cua.ai/docs/cua-driver/quickstart) is part of +[trycua/cua](https://github.com/trycua/cua) (about 28,900 stars across the project), with a CLI, +an MCP server and SDKs for macOS, Windows and Linux. Its driver releases, for every platform, are +marked pre-release on GitHub ([releases](https://github.com/trycua/cua/releases)). + +- It runs in your own interactive session. Its default **background** mode never moves your pointer + or raises the target: it uses UI Automation, pen or touch injection on a visible window, and + window messages + ([Windows notes](https://github.com/trycua/cua/blob/main/libs/cua-driver/rust/Skills/cua-driver/WINDOWS.md)). +- On its native input route, when the target is covered by another window or the event is one it + knows gets dropped (Chromium mouse events and key combinations, GTK buttons, LibreOffice shortcuts, + some terminal and WPF text), it returns `background_unavailable` and recommends its **foreground** + mode, which brings the target forward briefly and then restores it (same page). +- Chrome and Edge pages are driven through the DevTools protocol instead, where it reports validated + background delivery on Windows + ([browser notes](https://github.com/trycua/cua/blob/main/libs/cua-driver/rust/Skills/cua-driver/BROWSER.md), + [platform support](https://cua.ai/docs/cua-driver/concepts/platform-support)). Its pages don't + mention Android emulators. + +**Choose it** when you need macOS or Linux too, are on Windows Home, or your targets accept +background input. **Choose Anode** when the agent must reach covered windows, Electron and other +Chromium windows without DevTools, games or emulators without anything coming to the front of your +desktop. + +## Desktop Commander + +[Desktop Commander](https://github.com/wonderwhy-er/DesktopCommanderMCP) (about 9,960 stars) gives +an agent files, search, editing, terminal commands and processes. It has no screenshot, click or +accessibility tools, so it doesn't drive GUI apps. The local server is free; a hosted remote tier +is free up to 10,000 tool calls a month or $20 a month +([pricing](https://desktopcommander.app/pricing)). + +**Choose it** for file and shell work across operating systems. It and Anode do different jobs and +work well side by side: Anode's own `seat_exec` is for builds and servers that a desktop test needs, +not a general shell. + +## Windows 365 for Agents + +[Windows 365 for Agents](https://learn.microsoft.com/en-us/windows-365/agents/introduction-windows-365-for-agents) +gives agents Cloud PCs that they check out and back in, managed with Microsoft Entra and Intune. It +has been generally available since the week of 1 June 2026 +([what's new](https://learn.microsoft.com/en-us/windows-365/agents/whats-new)). + +- Agents reach it through an MCP server: they check a Cloud PC out with a tool call, then use mouse + and keyboard, screenshots, the accessibility tree of the foreground window, an allow-listed shell, + and DOM-level browser tools that work on Edge only + ([tool reference](https://learn.microsoft.com/en-us/microsoft-copilot-studio/mcp-windows-365-agents)). +- In the US it costs $0.40 an hour, rounded up to the next full hour; an always-available Cloud PC + adds $5 a month on top ([pricing](https://learn.microsoft.com/en-us/windows-365/agents/pricing-paygo-always-available)). + It is billed through an Azure subscription ([billing](https://learn.microsoft.com/en-us/windows-365/agents/billing-w365a)). + +**Choose it** for organizations that want agents on managed machines away from people's PCs, or +many machines at once. **Choose Anode** to test what is installed on your own PC (your builds, your +emulators, your signed-in tools) for free, with nothing to provision. + +## Coming from Microsoft + +Microsoft announced process and session isolation for agents in June 2026, as an early preview that +starts with non-interactive sessions +([Windows Developer Blog](https://blogs.windows.com/windowsdeveloper/2026/06/02/windows-platform-security-for-ai-agents/)). +Its [MXC SDK](https://github.com/microsoft/mxc) reached 1.0 on 7 October 2026; its session backend +has no display inside the session and captures only standard output and error +([design notes](https://github.com/microsoft/mxc/blob/main/docs/development/architecture/backends/isolation-session/oneshot.md)). +That makes it a sandbox for commands, not a desktop for GUI work, for now. We will update this page +as it changes. diff --git a/docs/GUIDE-ANDROID.md b/docs/GUIDE-ANDROID.md new file mode 100644 index 0000000..b853392 --- /dev/null +++ b/docs/GUIDE-ANDROID.md @@ -0,0 +1,95 @@ +# Guide: check an Android app on an emulator in the seat + +Before a release, you want an agent to install your APK on an emulator, open it, look at a few +screens in light and dark mode, and check the log for crashes. Anode boots the emulator inside the +seat, so its window never appears on your screen, and its Android tools act only on that emulator, +never on your own emulators or a phone on USB. + +![The same Android settings screen in light mode, then in dark mode after the agent switched it, captured from an emulator in the seat](https://raw.githubusercontent.com/skulitom/Anode/main/assets/guide-android-dark-theme.png) + +Captured with Anode 0.11.0 from a Pixel 9 Pro emulator (API 36) in the seat: the device's own +pixels, before and after `cmd uimode night yes`. It booted in 6.7 s. + +## Before you start + +You need Android Studio's SDK and at least one AVD, as on your desktop. Check what Anode finds. The +check starts nothing, so it needs Anode running: if it says "Anode is not running", start it with +`anode start --hidden` and run it again. + +```powershell +anode android | Out-Host +``` + +Note the AVD name, such as `Pixel_9_Pro`. Emulators are heavy: run one at a time, and only while the +seat is free. + +## Ask for the check + +> Use Anode to boot Pixel_9_Pro in the seat, install C:\work\app-release.apk, open +> com.example.quiz, and send me screenshots of the first screen in light and dark mode. Then check +> logcat for crashes and stop the emulator. + +Or run the guided prompt `/mcp__anode__android_test` and name your APK. + +## What the agent does + +```json +{"tool": "android_status", "arguments": {}} +{"tool": "seat_lease", "arguments": {"action": "acquire"}} +{"tool": "android_emulator", "arguments": {"action": "start", "avd": "Pixel_9_Pro"}} +{"tool": "android_emulator", "arguments": {"action": "adb", "serial": "emulator-5554", "args": ["install", "-r", "C:\\work\\app-release.apk"]}} +{"tool": "android_emulator", "arguments": {"action": "adb", "serial": "emulator-5554", "args": ["shell", "monkey", "-p", "com.example.quiz", "1"]}} +{"tool": "android_emulator", "arguments": {"action": "screenshot", "serial": "emulator-5554", "maxWidth": 540}} +{"tool": "android_emulator", "arguments": {"action": "adb", "serial": "emulator-5554", "args": ["shell", "cmd", "uimode", "night", "yes"]}} +{"tool": "android_emulator", "arguments": {"action": "screenshot", "serial": "emulator-5554", "maxWidth": 540}} +{"tool": "android_emulator", "arguments": {"action": "adb", "serial": "emulator-5554", "args": ["logcat", "-d", "-s", "AndroidRuntime:E"]}} +{"tool": "android_emulator", "arguments": {"action": "stop", "serial": "emulator-5554"}} +{"tool": "seat_lease", "arguments": {"action": "release"}} +``` + +- `start` returns the serial to use, such as `emulator-5554`, and `booted: true` once Android is up. + It starts **read-only**: the AVD keeps nothing from the test, and it can run in the seat while the + same AVD is open on your desktop. +- `screenshot` returns the device's own pixels. To tap something in it, scale the image's + coordinates back to the device (`x * sourceWidth / width`) and use + `["shell", "input", "tap", "X", "Y"]`. +- An empty `logcat -d -s AndroidRuntime:E` means no app hit an uncaught Java or Kotlin exception + since boot. Native crashes log elsewhere (`logcat -d -s DEBUG:F` shows their tombstones). + +## The same from a script + +```powershell +$env:ANODE_AGENT_ID = 'android-check' +$lease = anode lease acquire | Out-String | ConvertFrom-Json +if ($LASTEXITCODE -ne 0) { throw 'Desktop acquisition failed.' } +$env:ANODE_LEASE_TOKEN = $lease.leaseToken +anode android start Pixel_9_Pro | Out-Host +anode android adb emulator-5554 -- install -r C:\work\app-release.apk | Out-Host +anode android adb emulator-5554 -- shell monkey -p com.example.quiz 1 | Out-Host +anode android shot emulator-5554 light.png | Out-Host +anode android adb emulator-5554 -- shell cmd uimode night yes | Out-Host +anode android shot emulator-5554 dark.png | Out-Host +anode android adb emulator-5554 -- logcat -d -s AndroidRuntime:E | Out-Host +anode android stop emulator-5554 | Out-Host +anode lease release | Out-Host +``` + +Use the serial `android start` prints; `android shot` saves the screen at the device's resolution. + +## When it doesn't work + +- **`booted: false`.** The boot outlasted the wait, and the emulator keeps booting; check + `anode android status --json` (or `android_status`) until the seat's emulator shows + `"booted": true`. The plain-text summary doesn't show it. A cold boot takes longer. +- **adb keeps saying `device offline`.** The device itself is failing. After a quick boot, stop it + and start again with `coldBoot` (`--cold`). If it still never comes online, the AVD may be damaged, + which only you can repair or recreate; trying another AVD tells you quickly + ([details](ANDROID.md#when-an-emulator-does-not-finish-booting)). +- **The screen stays black.** Start again with `gpu: "software"` (`--gpu software`) + ([troubleshooting](TROUBLESHOOTING.md#an-android-emulator-will-not-start-or-stays-black)). +- **"runs outside the seat".** That serial is an emulator on your desktop, which Anode never drives; + start one in the seat and use its serial. +- **The emulator keeps running after the test.** A lease can end between two steps, so emulators + outlive it; stop yours when you're done. + +More: [Android apps and web consoles](ANDROID.md). diff --git a/docs/GUIDE-DESKTOP-APP.md b/docs/GUIDE-DESKTOP-APP.md new file mode 100644 index 0000000..65cdaf2 --- /dev/null +++ b/docs/GUIDE-DESKTOP-APP.md @@ -0,0 +1,128 @@ +# Guide: test your desktop app with Claude Code, without losing your mouse + +You are building a Windows app. You want Claude Code to build it, click through it and tell you what +broke, while you keep typing in your editor. Anode runs the app in the seat, a second Windows +session with its own screen, pointer and focus, so nothing the agent does lands on your desktop. + +![An agent clicking through Calculator and ticking off a task in a sample web app, inside Anode's background seat](https://raw.githubusercontent.com/skulitom/Anode/main/assets/demo.gif) + +Recorded inside the seat with Anode 0.11.0, from real MCP tool calls. Idle time between calls is +cut, and the nine clicks, which took 0.4 s, are shown four times slower. It is the same loop as below, +on Calculator instead of your app. + +## Once per machine + +In PowerShell, after [installing Anode](INSTALL.md): + +```powershell +anode setup | Out-Host # one administrator prompt +anode configure claude | Out-Host # registers Anode and the anode-desktop skill with Claude Code +``` + +Restart Claude Code and check that `/mcp` lists `anode`. (Using the Claude Code plugin instead? See +[Connecting agents](CONNECTING-AGENTS.md#1-claude-code).) + +## Ask for the test + +Name the app, the steps and what counts as a pass. For example: + +> Build C:\src\Invoices with `dotnet build`. Then use Anode to open +> bin\Debug\net10.0-windows\Invoices.exe in the background seat, add an invoice for "Sample Ltd" +> with one line of 120.00, check that the total reads 120.00, and show me a screenshot. Keep the +> viewer hidden and close the app when you're done. + +Or run the guided prompt `/mcp__anode__desktop_test` and name your app. + +## What the agent does + +The agent builds with its ordinary shell (`dotnet build C:\src\Invoices`), since a build needs no +desktop. These are the MCP calls that follow, in order. Each one is a tool call with the arguments +shown; IDs such as `w_…`, `s_…` and `e12` come from the previous reply, and the control IDs are +examples. + +```json +{"tool": "seat_lease", "arguments": {"action": "acquire"}} +{"tool": "seat_capabilities", "arguments": {}} +{"tool": "seat_run", "arguments": {"path": "C:\\src\\Invoices\\bin\\Debug\\net10.0-windows\\Invoices.exe"}} +{"tool": "seat_windows", "arguments": {"query": "Invoices"}} +{"tool": "seat_observe", "arguments": {"windowId": "w_RETURNED_ID"}} +{"tool": "seat_element", "arguments": {"snapshotId": "s_RETURNED_ID", "elementId": "e12", "action": "set_value", "value": "Sample Ltd"}} +{"tool": "seat_observe", "arguments": {"windowId": "w_RETURNED_ID"}} +{"tool": "seat_element", "arguments": {"snapshotId": "s_NEW_ID", "elementId": "e14", "action": "set_value", "value": "120.00"}} +{"tool": "seat_observe", "arguments": {"windowId": "w_RETURNED_ID"}} +{"tool": "seat_element", "arguments": {"snapshotId": "s_NEW_ID", "elementId": "e15", "action": "invoke"}} +{"tool": "seat_wait", "arguments": {"windowId": "w_RETURNED_ID", "automationId": "TotalLabel", "textContains": "120.00", "waitMs": 10000}} +{"tool": "seat_screenshot", "arguments": {"maxWidth": 1000}} +{"tool": "seat_window", "arguments": {"windowId": "w_RETURNED_ID", "action": "close"}} +{"tool": "seat_lease", "arguments": {"action": "release", "cancelJobs": true}} +``` + +- **Accessible controls first.** `seat_observe` returns the window's controls with the actions each + one supports; `seat_element` acts on one of them. An observation is used up by an action, so the + agent observes again before the next one. +- **Waits, not sleeps.** `seat_wait` returns as soon as the total shows 120.00, or reports that it + didn't within the time given. +- **Pixels when there is no control.** For a custom-drawn canvas or a game, the agent takes a + `seat_screenshot` and uses `seat_click` with coordinates from it. `seat_click` takes the seat's + full-size pixels, so a point read from a screenshot scaled with `maxWidth` is scaled back first + (`x * sourceWidth / width`, the same for `y`). +- **Jobs the test itself needs.** A local server, or a long build the desktop test depends on, can run + in the seat with `seat_exec`; read its output with `seat_job` until the reply says `finished: true` + ([command jobs](DEVELOPMENT-TESTING.md#command-jobs)). + +## The same from a script + +Every step has a CLI command, so a test you like can become a script. It needs an agent ID and the +lease token, as in [desktop leases](MULTI-AGENT.md): + +```powershell +dotnet build C:\src\Invoices +if ($LASTEXITCODE -ne 0) { throw 'Build failed.' } +$env:ANODE_AGENT_ID = 'invoices-test' +$lease = anode lease acquire | Out-String | ConvertFrom-Json +if ($LASTEXITCODE -ne 0) { throw 'Desktop acquisition failed.' } +$env:ANODE_LEASE_TOKEN = $lease.leaseToken +anode run C:\src\Invoices\bin\Debug\net10.0-windows\Invoices.exe | Out-Host +anode windows --query Invoices | Out-Host +anode inspect w_RETURNED_ID --html invoices.html | Out-Host +anode element s_RETURNED_ID e12 set_value --value "Sample Ltd" | Out-Host +anode inspect w_RETURNED_ID | Out-Host +anode element s_NEW_ID e14 set_value --value 120.00 | Out-Host +anode inspect w_RETURNED_ID | Out-Host +anode element s_NEW_ID e15 invoke | Out-Host +anode wait w_RETURNED_ID --automation-id TotalLabel --text 120.00 | Out-Host +anode shot invoices.png | Out-Host +anode window w_RETURNED_ID close | Out-Host +anode lease release --cancel-jobs | Out-Host +``` + +Build before `lease acquire`: the lease ends 120 seconds after the last desktop command, and a build +isn't one, so a longer build would leave `anode run` failing with `lease_expired`. For a script with +long steps between desktop commands, `anode lease acquire --ttl 600` keeps it longer. + +Each `element` needs the snapshot ID from the `inspect` just before it. `inspect --html` writes a +report you can search, with the screenshot and each control's bounds, which helps when you write the +test. `wait` exits 3 when the control never matched; check `$LASTEXITCODE` after each command. + +## Why your mouse stays yours + +- The app runs in the seat's session, and input goes there: your pointer, focus and keyboard are + not used. +- A program in the seat that moves its own cursor, as games do, can't move your real pointer: + Anode gates that call ([details](TROUBLESHOOTING.md#my-real-pointer-jumps-while-something-runs-in-the-seat)). +- The viewer stays hidden unless you run `anode show`. **Ctrl+Alt+Shift+K** stops the seat at any + time. + +The seat runs as you and shares your files, so a test that writes files writes your files. Point +the app at a scratch folder, and see [security](SECURITY.md) for what the seat does not isolate. + +## When it doesn't work + +- **No controls in the tree.** Some apps draw everything themselves; use screenshots and + `seat_click`. WinForms and WPF expose rich trees; [desktop tools](DESKTOP-TOOLS.md) covers the + actions. +- **Clicks or typing have no effect.** Run `anode capabilities`: it reports known input blockers + ([capture works, but input does not](TROUBLESHOOTING.md#capture-works-but-focus-and-input-do-not)). +- **The app opened on your desktop instead.** Single-instance apps hand a second start to the copy + already running; close yours, or test a build with its own settings. +- More: [development and testing](DEVELOPMENT-TESTING.md), [troubleshooting](TROUBLESHOOTING.md). diff --git a/docs/GUIDE-WEB-FORM.md b/docs/GUIDE-WEB-FORM.md new file mode 100644 index 0000000..b0c95fe --- /dev/null +++ b/docs/GUIDE-WEB-FORM.md @@ -0,0 +1,116 @@ +# Guide: fill in a web form in the seat's browser + +An agent can fill in and submit a web form in a real, headed Chrome or Edge window inside the seat, +while your own browser stays untouched. Use it to test a form you built, or for a routine task on a +site you have signed in to for agents. + +![An agent filling in a sample sign-up form field by field and submitting it, inside Anode's seat](https://raw.githubusercontent.com/skulitom/Anode/main/assets/guide-web-form.gif) + +Recorded inside the seat with Anode 0.11.0, from real MCP tool calls, on a local sample page in +a Chrome window with a temporary profile. Each frame is one step; the time between steps is cut. + +## Pick the browser profile + +| You are testing… | Open it with | Profile | +| --- | --- | --- | +| Your own page or app, on localhost or a test server | `seat_run` with Chrome and its own `--user-data-dir` | A temporary folder you delete afterwards: no accounts, nothing left behind | +| A site the agent should use as you, such as a console | `seat_browser` | Anode's seat profile, which you sign in to once | + +**A signed-in site: sign in first, yourself.** Do it once, before any agent uses the profile, from +your own terminal; it opens the seat's profile on your desktop, and you close it when you're done: + +```powershell +anode browser --sign-in https://example.com/account +``` + +Agents never sign in or answer a 2-step check: when a site asks, they stop and ask you. Anyone holding +the desktop lease can use what that profile is signed in to, so sign in only to what agents should +act on ([details](ANDROID.md#web-consoles-in-a-signed-in-browser)). + +Then, before the agent opens either browser, it acquires the desktop lease. This starts the hidden +seat if Anode is stopped; keep the lease until the form is finished and its window is closed. + +```json +{"tool": "seat_lease", "arguments": {"action": "acquire"}} +``` + +**Your own page.** Start Chrome in the seat on a folder of its own (here an `--app` window, which +has no address bar): + +```json +{"tool": "seat_run", "arguments": {"path": "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe", "args": ["--user-data-dir=C:\\temp\\form-test-profile", "--no-first-run", "--no-default-browser-check", "--force-renderer-accessibility", "--app=http://localhost:5173/signup"]}} +``` + +**A signed-in site.** The agent opens pages on the seat profile with `seat_browser` +(`{"url": "https://example.com/account"}`). + +## Fill in the form + +```json +{"tool": "seat_windows", "arguments": {"query": "chrome"}} +{"tool": "seat_observe", "arguments": {"windowId": "w_RETURNED_ID", "maxDepth": 16, "includeScreenshot": false}} +{"tool": "seat_observe", "arguments": {"windowId": "w_RETURNED_ID", "maxDepth": 16, "includeScreenshot": false}} +{"tool": "seat_element", "arguments": {"snapshotId": "s_RETURNED_ID", "elementId": "e25", "action": "set_value", "value": "Sam Example"}} +{"tool": "seat_observe", "arguments": {"windowId": "w_RETURNED_ID", "maxDepth": 16, "includeScreenshot": false}} +{"tool": "seat_element", "arguments": {"snapshotId": "s_NEW_ID", "elementId": "e27", "action": "set_value", "value": "sam@example.com"}} +{"tool": "seat_observe", "arguments": {"windowId": "w_RETURNED_ID", "maxDepth": 16, "includeScreenshot": false}} +{"tool": "seat_element", "arguments": {"snapshotId": "s_NEW_ID", "elementId": "e29", "action": "expand"}} +{"tool": "seat_screenshot", "arguments": {"maxWidth": 1000}} +{"tool": "seat_click", "arguments": {"x": 246, "y": 448}} +{"tool": "seat_wait", "arguments": {"windowId": "w_RETURNED_ID", "automationId": "plan", "textContains": "Team", "waitMs": 5000}} +{"tool": "seat_element", "arguments": {"snapshotId": "s_FROM_WAIT", "elementId": "e30", "action": "toggle"}} +{"tool": "seat_observe", "arguments": {"windowId": "w_RETURNED_ID", "maxDepth": 16, "includeScreenshot": false}} +{"tool": "seat_element", "arguments": {"snapshotId": "s_NEW_ID", "elementId": "e31", "action": "invoke"}} +{"tool": "seat_wait", "arguments": {"windowId": "w_RETURNED_ID", "textContains": "Account created", "waitMs": 10000}} +{"tool": "seat_window", "arguments": {"windowId": "w_RETURNED_ID", "action": "close"}} +{"tool": "seat_lease", "arguments": {"action": "release"}} +``` + +- **Observe twice at first.** Chrome turns its accessibility tree on when something first asks, so the + first observation of a new window may have no page in it. For a window you start yourself, + `--force-renderer-accessibility` asks Chrome to build the tree from the start, as Anode's own + browser test does. Page controls sit deep in the tree, so pass `maxDepth` 16. +- **A full browser window.** In a window with tabs and toolbars, such as the one `seat_browser` + opens, the browser's own controls can use up the observation's element budget before the page's + appear. Raise `maxElements`, or work from `seat_screenshot` and `seat_click`. +- **Fields by name.** Text fields appear as `Edit` with the field's label as their name and its HTML + `id` as `automationId`; checkboxes offer `toggle`, buttons `invoke`. `seat_wait` returns a fresh + observation when it matches, so its snapshot can be used for the next action. That observation is + its own (depth 20, up to 500 controls), so take the next element ID from the wait's reply, as the + `toggle` on `e30` above does, not from an earlier observation. +- **Drop-down lists (`