Screenshot backend: direct uiautomator2 ATX HTTP

[Coldaine]

Goal

Implement the _capture_u2() backend in adb_vision/screenshot.py.

uiautomator2's ATX agent captures screenshots via Android's UiDevice.takeScreenshot() API served over HTTP on port 7912. Unlike adb shell screencap, this uses the Android graphics stack directly, bypassing the Linux framebuffer — which is critical because adb shell screencap returns blank images on MEmu/VirtualBox.

Interface

async def _capture_u2(*, adb_run: AdbRunFn, serial: str, adb_exe: str) -> str:
    """Return base64-encoded PNG screenshot via uiautomator2 ATX HTTP agent."""

Implementation Plan

1. Check if ATX agent is running: GET http://localhost:7912/info (after port forward)

   • If not: push and start the ATX agent APKs

2. Initialize ATX agent (if needed):

   • Push ATX app + test APKs to device
   • APKs are at: alas_wrapped/bin/uiautomator2/ (check exact paths)
   • Start via: adb shell am instrument -w -r -e ... or the simpler adb shell /data/local/tmp/atx-agent server -d

3. Forward port: adb_run("forward", "tcp:7912", "tcp:7912", timeout=5.0)

4. Take screenshot: GET http://localhost:7912/screenshot/0?compress=true

   • Returns JPEG by default. Request PNG: GET http://localhost:7912/screenshot/0?format=png
   • Or get JPEG and convert to PNG with Pillow

5. Return base64-encoded PNG

Key Reference

The ALAS codebase already has ATX initialization code. Key patterns to borrow from (but NOT import):

• alas_wrapped/module/device/connection.py — ATX agent setup
• alas_wrapped/module/device/method/uiautomator_2.py — screenshot via u2

The standalone approach should NOT use the uiautomator2 Python library. Instead, talk to the ATX HTTP API directly — it's just REST calls.

Testing (TDD)

• Mock test: Mock adb_run for port forward. Mock HTTP GET /screenshot/0 returning fake PNG bytes. Verify base64 result.
• Agent-not-running test: Mock HTTP returning connection refused → should attempt to start agent, or raise clear error.
• Port-forward test: Verify adb forward tcp:7912 tcp:7912 is called.

Tests must pass without a device — all ADB and HTTP calls mocked.

Files to Modify

• adb_vision/screenshot.py — replace the _capture_u2 stub
• New adb_vision/test_u2.py — tests

Acceptance Criteria

• _capture_u2() returns valid base64 PNG when ATX agent is running
• All existing tests still pass
• New tests cover: happy path, agent not running, HTTP failure
• No ALAS imports — talk to ATX HTTP API directly
• No uiautomator2 Python library dependency

[google-labs-jules]

Jules is on it. When finished, you will see another comment and be able to review a PR.

[google-labs-jules]

Ready for a review! A PR has been created.