Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 67 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,63 @@
# Android Device Perception Skill

A read-only Codex skill that samples an authorized Android device through ADB and turns discrete observations into a structured `SituationSnapshot`.
A portable, read-only Android observation skill for AI coding agents.

It turns discrete ADB observations into a structured `SituationSnapshot`, so an agent can understand the current device state before it decides what to do. No APK, daemon, MCP server, or model configuration is required.

Use it when an agent needs to answer questions such as:

- Is the target app actually in the foreground?
- Is a system dialog, notification shade, or lock screen covering it?
- Did the page change while the agent was thinking?
- Did an operation produce an observable device-side result?

The skill works with agents that support Agent Skills and can run local shell tools. It deliberately observes rather than controls the device.

## Why a perception skill?

A screenshot shows pixels. Raw ADB commands expose fragments of system state. This skill combines the relevant evidence into one timestamped contract and teaches the agent how to reason about freshness, conflicts, partial failures, and sensitive data.

| Approach | Structured UI | System state | Stale-state detection | Changes the device | Setup |
| --- | --- | --- | --- | --- | --- |
| Screenshot only | No | No | No | No | Low |
| Ad hoc ADB commands | Partial | Partial | No | Depends on commands | Low |
| Full mobile automation server | Yes | Usually | Tool-specific | Usually yes | Medium to high |
| Android Device Perception | Yes | Yes | Page and state fingerprints | No by default | Low |

This makes the skill useful as a safe observation layer before a separate automation tool, test harness, or human performs an action.

## Example

A compact snapshot can reveal that an app is still resumed while a system overlay owns focus:

```json
{
"captured_at": "2026-08-04T08:30:12+00:00",
"foreground": {
"package": "com.example.form",
"activity": "com.example.form.MainActivity"
},
"focus": {
"package": "com.android.systemui",
"name": "NotificationShade",
"window_type": 2040
},
"screen": {
"screen_on": true,
"interactive": true,
"keyguard_showing": false
},
"ui": {
"semantic_node_count": 12,
"focused_resource_id": null
},
"page_fingerprint": "...",
"state_fingerprint": "...",
"errors": []
}
```

It combines foreground activity, focused window, screen state, a compact UIAutomator semantic tree, optional screenshots, and stable page/state fingerprints. The skill is intended for debugging UI state, understanding why an app appears unresponsive, and verifying device-side outcomes without assuming continuous access to the device.
An agent can then report that the app did not necessarily freeze: the notification shade is covering it. The conclusion comes from structured evidence, without tapping, dismissing, launching, or changing anything.

## Features

Expand All @@ -16,6 +71,16 @@ It combines foreground activity, focused window, screen state, a compact UIAutom
- Partial-success diagnostics with per-source timing
- Python standard library only

## Good fits

- Debugging a connected emulator or authorized test device with an AI coding agent
- Investigating why an Android app appears unresponsive or hidden
- Capturing before/after evidence around a manual or separately authorized action
- Providing a read-only perception boundary for mobile or OS-agent prototypes
- Collecting reproducible UI-state evidence without granting device-control permissions

If the task is to tap, type, swipe, install, launch, or continuously automate a workflow, pair this skill with a dedicated execution tool. Those actions are intentionally outside this skill's default boundary.

## Requirements

- Python 3.9 or newer
Expand Down
Loading