Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .github/workflows/deploy-website.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Deploy Website to GitHub Pages

on:
push:
branches: [master, website]
paths:
- 'website/**'
- '.github/workflows/deploy-website.yml'
workflow_dispatch:

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: pages-website
cancel-in-progress: true

jobs:
build:
runs-on: ubuntu-latest
defaults:
run:
working-directory: website
steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
cache-dependency-path: website/package-lock.json

- name: Install dependencies
run: npm ci

- name: Build (static export)
env:
NEXT_PUBLIC_BASE_PATH: /agenTest
run: npm run build

- name: Add .nojekyll
run: touch out/.nojekyll

- uses: actions/configure-pages@v5

- uses: actions/upload-pages-artifact@v3
with:
path: website/out

deploy:
needs: build
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
16 changes: 8 additions & 8 deletions docs/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ This is the common case for apps built with code-generation tools (Cursor, Bolt,
```json
// Response excerpt
{
"uiTree": "screen 1440x2960 com.expo.app #a1b2c3\n \"Welcome to Ira\"\n @f1 input \"you@example.com\"\n @f2 input password\n @b1 btn \"Sign in\"\n \"or\"\n @b2 btn \"Sign up\"",
"uiTree": "screen 1440x2960 com.expo.app #a1b2c3\n \"Welcome\"\n @f1 input \"you@example.com\"\n @f2 input password\n @b1 btn \"Sign in\"\n \"or\"\n @b2 btn \"Sign up\"",
"screenFingerprint": "a1b2c3",
"framework": "react_native",
"frameworkSync": ["hermes"]
Expand Down Expand Up @@ -284,9 +284,9 @@ The header has screen size, package, and fingerprint. Every operable gets a `@re
On a chat screen with no testIDs and only icon buttons:

```
screen 1440x2960 ai.rumik.ira.twa #f9e8d7
screen 1440x2960 com.example.chatapp #f9e8d7
@b1 btn "ArrowLeft" ← Lucide component
"Ira"
"Chat"
@b2 btn "Phone"
@b3 btn "DotsVertical"
@g1 @ref scroll
Expand Down Expand Up @@ -325,7 +325,7 @@ When you have a React Native app running in a dev build with Metro on port 8081,

```json
{
"packageName": "ai.rumik.ira.twa"
"packageName": "com.example.chatapp"
}
```

Expand All @@ -334,23 +334,23 @@ Response (lean, no diagnostics — success path):
```json
{
"deviceId": "emulator-5554",
"packageName": "ai.rumik.ira.twa",
"packageName": "com.example.chatapp",
"backend": "grpc",
"helperInstalled": true,
"framework": "react_native",
"frameworkSync": ["hermes"],
"uiTree": {
"role": "container",
"bounds": "[0,0][1280,2784]",
"id": "ai.rumik.ira.twa:id/action_bar_root",
"id": "com.example.chatapp:id/action_bar_root",
"children": [/* ... */]
}
}
```

`frameworkSync: ["hermes"]` confirms that:
1. Helper detection first returned `native` (RN Fabric flattens views, so no `ReactRootView` classes in the a11y tree, and on API 34+ `/proc/<pid>/maps` is SELinux-blocked from shell UID).
2. The Metro `/json/list` backstop fetched the debug target list, found two Hermes pages with `appId === "ai.rumik.ira.twa"`, and overrode the framework to `react_native`.
2. The Metro `/json/list` backstop fetched the debug target list, found two Hermes pages with `appId === "com.example.chatapp"`, and overrode the framework to `react_native`.
3. `FrameworkSync.attach()` then opened the CDP WebSocket to the first page, ran `Runtime.enable`, and is now alive in the session.

### Step 2: Run a flow and watch the per-step durations
Expand All @@ -370,7 +370,7 @@ Each heavy action (tap, swipe, double_tap, long_press) now composes **three** si
- Hermes CDP `Runtime.evaluate("Promise.resolve(1)")` round-trip (~20-80ms; proves the JS event loop is responsive)
- Loading-indicator visibility check

On the `ai.rumik.ira.twa` build this typically resolves in ~1.3-1.5s per heavy action on an API 36 emulator — 2-3× faster than the pre-Phase-3.5 polling path.
On the `com.example.chatapp` build this typically resolves in ~1.3-1.5s per heavy action on an API 36 emulator — 2-3× faster than the pre-Phase-3.5 polling path.

### What happens in release builds

Expand Down
14 changes: 7 additions & 7 deletions src/__tests__/compact-tree.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -104,12 +104,12 @@ describe('hoistClickableLabels', () => {
// We assert on the COMPACT OUTPUT (what the LLM sees) rather than the
// raw hoisted map because the serializer also does transparent
// collapse on unlabeled wrapper-of-interactives: the outer/inner
// "phone row" wrappers in the Ira fixture intentionally don't get
// "phone row" wrappers in the chat fixture intentionally don't get
// their own refs (their children are addressable directly), so they
// won't appear in the hoisted map either. What matters is that the
// LLM ends up with an addressable, labeled entry for every real
// affordance on the screen.
const tree = loadTree('ira-login-rn.xml');
const tree = loadTree('chat-login-rn.xml');
const compact = serializeTreeCompact(tree);

// Every labeled CTA must appear once — and only once — in the final
Expand Down Expand Up @@ -226,8 +226,8 @@ describe('byte-size regression', () => {
expect(compact.text.length).toBeLessThan(400);
});

it('Ira RN login (zero-a11y) compact format is < 700 bytes', () => {
const tree = loadTree('ira-login-rn.xml');
it('chat RN login (zero-a11y) compact format is < 700 bytes', () => {
const tree = loadTree('chat-login-rn.xml');
const compact = serializeTreeCompact(tree);
expect(compact.text.length).toBeLessThan(700);
});
Expand All @@ -245,14 +245,14 @@ describe('byte-size regression', () => {
expect(ratio).toBeGreaterThanOrEqual(3);
});

it('compact format beats JSON tree by at least 4x on Ira RN fixture (the target user case)', () => {
const tree = loadTree('ira-login-rn.xml');
it('compact format beats JSON tree by at least 4x on chat RN fixture (the target user case)', () => {
const tree = loadTree('chat-login-rn.xml');
const compact = serializeTreeCompact(tree);
const json = JSON.stringify(serializeTreeForLlm(tree), null, 2);
const ratio = json.length / compact.text.length;
// eslint-disable-next-line no-console
console.log(
`[compact-tree] ira: JSON=${json.length}b compact=${compact.text.length}b ratio=${ratio.toFixed(2)}x`,
`[compact-tree] chat: JSON=${json.length}b compact=${compact.text.length}b ratio=${ratio.toFixed(2)}x`,
);
expect(ratio).toBeGreaterThanOrEqual(4);
});
Expand Down
14 changes: 7 additions & 7 deletions src/__tests__/fiber-merger.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ describe('mergeFiberLabels — Stage B (containment)', () => {
});

it('picks the SVG icon (tight container) over a wrapper RCTView (broader)', () => {
// This is the Ira profile screen "back button" case. The a11y tree
// This is the chat profile screen "back button" case. The a11y tree
// has one clickable at [60,192,132,264] (72×72). The fiber tree has:
// - a Pressable RCTView at the same bounds with component=ProfilePage
// - an SVG inside with component=ArrowLeft
Expand Down Expand Up @@ -454,15 +454,15 @@ describe('mergeFiberLabels — generic component filtering', () => {
});

// ---------------------------------------------------------------------------
// Real-world fixture — Ira app profile screen
// Real-world fixture — chat app profile screen
// ---------------------------------------------------------------------------

describe('mergeFiberLabels — Ira RN profile fixture', () => {
describe('mergeFiberLabels — chat RN profile fixture', () => {
it('produces labels for unlabeled interactives using real fiber data', () => {
const tree = loadTree('ira-login-rn.xml');
const fibers = loadFibers('ira-profile-fibers.json');
const tree = loadTree('chat-login-rn.xml');
const fibers = loadFibers('chat-profile-fibers.json');

// The Ira fixture has no bounds on fibers (we synthesized it without
// The chat fixture has no bounds on fibers (we synthesized it without
// measurements). So Stage A is what we exercise here: testID and
// accessibilityLabel matching. Density=0 disables Stage B.
const labels = mergeFiberLabels(tree, fibers, 0);
Expand All @@ -474,7 +474,7 @@ describe('mergeFiberLabels — Ira RN profile fixture', () => {
// for enrichment.
//
// The real win shows up on a screen where the a11y tree has
// UNLABELED clickables (like the Ira chat screen's icon buttons).
// UNLABELED clickables (like the chat screen's icon buttons).
// The fiber data we captured includes Camera/Photo/HangoutIcon
// entries that WOULD label those if the a11y fixture contained
// them without descriptions. The fiber-merger unit here just
Expand Down
63 changes: 63 additions & 0 deletions src/__tests__/fixtures/chat-login-rn.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Real-world fixture from a Bolt/Lovable-style React Native app:
com.example.chatapp — a zero-testID chat app. Captured live from the emulator
via agentest_get_ui_tree on 2026-04-10.

Critical properties for the target user:
- Empty resource-ids on every node (RN doesn't auto-populate testID)
- The "sign up" button is a clickable container wrapping a TextView
"sign up" — the classic TouchableOpacity > Text pattern that
hoisting + duplicate-suppression must handle correctly
- The outer clickable wrapper has an AGGREGATED a11y description that
concatenates all descendant text — the 80-char threshold in
ownLabelOf() drops this so transparent-collapse can fire
- The +91 country code is a nested clickable container wrapping its
own TextView, inside two unlabeled wrapper clickables — the wrapper
hell the transparent-collapse rule exists to prevent
- The phone EditText is clickable AND editable, inside the same
wrapper chain as the +91 tap

This fixture is the acceptance test for the Bolt/v0/Lovable target
user. Every real affordance must end up as an addressable, labeled
ref in the compact output without duplicate text lines and without
nested wrapper hell.
-->
<hierarchy rotation="0">
<node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1280,2784]">
<node index="0" text="" resource-id="" class="com.facebook.react.views.view.ReactViewGroup" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1280,2784]">
<node index="0" text="" resource-id="" class="com.facebook.react.views.scroll.ReactScrollView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="true" long-clickable="false" password="false" selected="false" bounds="[0,0][1280,2784]">
<node index="0" text="" resource-id="" class="com.facebook.react.views.view.ReactViewGroup" package="com.example.chatapp" content-desc="meet bot, the friend who is always present, good conversations, 100% private, read our privacy policy and terms here., or" checkable="false" checked="false" clickable="true" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1280,2784]">
<node index="0" text="" resource-id="" class="com.facebook.react.views.image.ReactImageView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1280,2784]" />
<node index="1" text="" resource-id="" class="com.facebook.react.views.view.ReactViewGroup" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[99,1126][1182,2643]">
<node index="0" text="meet bot" resource-id="" class="com.facebook.react.views.text.ReactTextView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[99,1126][1056,1314]" />
<node index="1" text="the friend who is always present" resource-id="" class="com.facebook.react.views.text.ReactTextView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[99,1326][1056,1512]" />
<node index="2" text="good conversations, 100% private" resource-id="" class="com.facebook.react.views.text.ReactTextView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[99,1584][1056,1652]" />
<node index="3" text="read our privacy policy and terms here." resource-id="" class="com.facebook.react.views.text.ReactTextView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[99,1724][1056,1775]" />
<node index="4" text="" resource-id="" class="com.facebook.react.views.view.ReactViewGroup" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[99,1871][1182,2039]">
<node index="0" text="" resource-id="" class="com.facebook.react.views.view.ReactViewGroup" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[99,1871][1182,2039]">
<node index="0" text="" resource-id="" class="com.facebook.react.views.view.ReactViewGroup" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[102,1874][1179,2036]">
<node index="0" text="" resource-id="" class="com.facebook.react.views.view.ReactViewGroup" package="com.example.chatapp" content-desc="+91" checkable="false" checked="false" clickable="true" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[150,1922][227,1988]">
<node index="0" text="+91" resource-id="" class="com.facebook.react.views.text.ReactTextView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[150,1922][227,1988]" />
</node>
<node index="1" text="enter number here" resource-id="" class="com.facebook.react.views.textinput.ReactEditText" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="true" password="false" selected="false" bounds="[302,1913][1131,1997]" />
</node>
</node>
</node>
<node index="5" text="" resource-id="" class="com.facebook.react.views.view.ReactViewGroup" package="com.example.chatapp" content-desc="sign up" checkable="false" checked="false" clickable="true" enabled="false" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[99,2075][1182,2243]">
<node index="0" text="sign up" resource-id="" class="com.facebook.react.views.text.ReactTextView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[548,2117][734,2201]" />
</node>
<node index="6" text="or" resource-id="" class="com.facebook.react.views.text.ReactTextView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[622,2291][660,2332]" />
<node index="7" text="" resource-id="" class="com.facebook.react.views.view.ReactViewGroup" package="com.example.chatapp" content-desc="sign up with google" checkable="false" checked="false" clickable="true" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[99,2380][1182,2548]">
<node index="0" text="" resource-id="" class="com.facebook.react.views.image.ReactImageView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[337,2428][409,2500]" />
<node index="1" text="sign up with google" resource-id="" class="com.facebook.react.views.text.ReactTextView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[445,2422][945,2506]" />
</node>
<node index="8" text="" resource-id="" class="com.facebook.react.views.view.ReactViewGroup" package="com.example.chatapp" content-desc="need help?" checkable="false" checked="false" clickable="true" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[556,2596][725,2643]">
<node index="0" text="need help?" resource-id="" class="com.facebook.react.views.text.ReactTextView" package="com.example.chatapp" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[556,2596][725,2643]" />
</node>
</node>
</node>
</node>
</node>
</node>
</hierarchy>
Loading
Loading