pi-computer-use

pi-computer-use lets AI agents use desktop apps on macOS and Windows.

An agent can look at an app window, understand the buttons and text inside it, and perform actions like clicking, typing, scrolling, and waiting for something to change. This is useful when the agent needs to work with a normal desktop app instead of an API, a terminal command, or a file.

New to computer use? Start with: Wait, what exactly is Computer Use?

What this package does

This is a Pi extension. After installation, Pi agents get tools for:

• finding open apps and windows
• observing what is visible in a window
• searching the visible interface for text, buttons, and controls
• inspecting parts of the interface in more detail
• clicking, typing, scrolling, and pressing UI controls
• waiting for UI changes

In short: it gives an agent a controlled way to operate desktop software.

What this package is not

pi-computer-use is not a replacement for app APIs or MCP servers. If an app has a reliable direct integration, use that first.

Computer use is most helpful when the only available interface is the app on screen.

Install

pi install git:github.com/injaneity/pi-computer-use@v0.4.1

Start Pi and complete the platform setup flow.

On macOS, grant permissions to:

/Applications/pi-computer-use.app

Required macOS permissions:

• Accessibility
• Screen Recording, shown as Screen and System Audio Recording on newer macOS versions

The macOS setup flow registers the helper first, so it should already appear in both Settings panes. Enable the toggles and choose Recheck.

On Windows, use an interactive desktop session. Windows support uses the platform accessibility APIs and does not use the macOS helper app or TCC permission flow.

Use /computer-use inside Pi to show the active configuration and where it came from.

Main tools

• find_roots
• observe_ui
• search_ui
• expand_ui
• inspect_ui
• act_ui
• read_text
• wait_for

See docs/usage.md for the full tool reference.

Renamed from the 0.4.x names find, observe, and act so the generic names no longer collide with tools from other extensions (see issue #19).

Documentation

• Usage
• Architecture
• Configuration
• Development
• Troubleshooting
• Contributing

Development status

The current architecture is centered on observe_ui and act_ui: first inspect the current UI state, then ask the helper to perform one grounded action transaction. Older direct tools such as screenshot, click, set_text, and computer_actions are no longer part of the public extension surface.

Behavioral benchmarking should use cubench-runner with ./bin/pi-cubench-client.mjs; node scripts/cubench.mjs is only the local measurement harness.

License

MIT