Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
BIN := bin/mobium
VERSION := $(shell cat VERSION 2>/dev/null || echo dev)

.PHONY: all build dist mcpb test fmt fmt-check vet lint clients java crosscompile api api-check flags flags-check quickstart license-check docs-check ci clean
.PHONY: all build dist mcpb test fmt fmt-check vet lint clients java crosscompile api api-check flags flags-check quickstart license-check docs-check ci clean reader

all: build test

Expand All @@ -10,6 +10,22 @@ build:
go build -ldflags "-X main.version=$(VERSION)" -o $(BIN) ./cmd/mobium
@echo "built $(BIN)"

# reader rebuilds internal/mobiumdriver/reader/reader.dex, the screen reader-
# safe hierarchy reader, from Main.java. Only for changing the reader: the
# dex is checked in and pinned by checksum, so a build of Mobium needs neither
# a JDK nor the Android SDK. Needs javac, d8 (9.3.16 builds the pinned bytes)
# and an android.jar, found under ANDROID_HOME.
READER := internal/mobiumdriver/reader
ANDROID_JAR ?= $(lastword $(sort $(wildcard $(ANDROID_HOME)/platforms/android-*/android.jar)))

reader:
@test -n "$(ANDROID_JAR)" || { echo "no android.jar — set ANDROID_HOME or ANDROID_JAR"; exit 1; }
@tmp=$$(mktemp -d) && \
javac --release 8 -Xlint:-options -nowarn -cp "$(ANDROID_JAR)" -d $$tmp $(READER)/Main.java && \
d8 --release --min-api 24 --output $$tmp $$tmp/dev/mobium/reader/*.class && \
cp $$tmp/classes.dex $(READER)/reader.dex && rm -rf $$tmp && \
echo "built $(READER)/reader.dex — set reader.SHA256 to:" && shasum -a 256 $(READER)/reader.dex

# The Go client is a separate module, so ./... does not reach it and it has to
# be named. Forgetting that would mean it never runs in CI.
MODULES := . clients/go
Expand Down
35 changes: 34 additions & 1 deletion docs/CHALLENGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ the source.

The pattern across them is the reason the document exists: **almost nothing
here was found by reading code or by a test written from imagination.** Of
207 defects, 171 were found only by running against a real device. The other
208 defects, 172 were found only by running against a real device. The other
thirty-six — 4, 5, 6, 14, 23, 24, 32, 35, 36, 39, 44, 50, 53, 54, 57, 66, 89,
99, 100, 121, 123, 129, 130, 133, 134, 139, 140, 141, 142, 144, 150, 158, 162,
164, 165 and 172 — came from reading code, the compiler, a test, a linter,
Expand Down Expand Up @@ -4623,6 +4623,39 @@ app with no second install. Mobium does not answer the dialog itself: which
answer to give is the phone owner's, and "Always send" changes a setting for
every app.

### 208. Every read silenced the device's screen reader

**Found by:** reading Prime Video on a Fire TV, which publishes nothing but
one surface to `map`, to see whether it publishes more while Amazon's screen
reader, VoiceView, is on. Every read of the screen — `uiautomator dump` and
the UiAutomator2 server alike — unbound VoiceView for as long as it ran:
`dumpsys accessibility` listed it before the read and not during it, and
touch exploration went off. Android's test-automation connection suppresses
every other accessibility service unless it is opened with
`FLAG_DONT_SUPPRESS_ACCESSIBILITY_SERVICES`, and neither opens it so: the dump
command has no option for it, and the UiAutomator2 server accepted
`disableSuppressAccessibilityService`, as a setting and as a capability, and
went on unbinding VoiceView — the setting did not even read back. So anybody
who relies on TalkBack or VoiceView lost it for each read, without a word,
and an app that publishes its contents only to a screen reader stopped
publishing them before the read could see them. The same held on a Pixel 8
Pro with TalkBack.

The dump backend now reads through a reader of Mobium's own when any
accessibility service is enabled: a small dex, pinned by checksum, pushed into
a folder of its own in `/data/local/tmp`, run with `app_process`, and deleted
with the runtime's compiled copy of it after every read. It writes what
`uiautomator dump` writes, over a connection opened with that flag. On the
Fire TV (Android 9) VoiceView stayed bound through every read, and Prime Video
gave 14 nodes, 3 of them labeled, where the dump gave 12 and none; on the
Pixel 8 Pro (Android 17) `uiautomator dump` unbound TalkBack in both reads and
the reader in neither, the same 13 nodes either way. The first build died on
Android 17 within 150 ms: newer accessibility code builds a Handler on the
main looper, which a bare `app_process` does not have, and Android 9 had never
asked for one. With no service enabled, reads go through `uiautomator dump`
as before. The default driver, UiAutomator2, still silences a screen reader
(ROADMAP).

## Findings that were not defects

Worth recording because each one closed off an approach that looked obvious.
Expand Down
2 changes: 1 addition & 1 deletion docs/RELEASE-CHECKLIST.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
[CI workflow](../.github/workflows/ci.yml) runs on every push — formatting,
vet, both modules' tests, the four non-Go clients' own tests, and six
cross-compile targets. This file is the rest, and the rest is where most
serious defects in this project have come from: 171 of 207 were found only by
serious defects in this project have come from: 172 of 208 were found only by
running against a real device.

Work through it before tagging a release, on three substrates: an Android
Expand Down
9 changes: 9 additions & 0 deletions docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,15 @@ this is what is not.
now where the Never button is, and every later login of a package gets
Never (six of six). `alert` read it, refused accept and dismiss, and a rule
`--when "Save password" --press "Not now"` answered it in the way of a tap.
- **Reading without silencing a screen reader, on UiAutomator2 too.** The
dump backend reads through a reader of Mobium's own whenever an
accessibility service is enabled, and TalkBack and VoiceView stay running
through it, measured on a Pixel 8 Pro and a Fire TV (CHALLENGES 208). The
UiAutomator2 server, the default driver, still unbinds them for as long as
a session is open: it accepted `disableSuppressAccessibilityService` and
went on unbinding. Left: the same for UiAutomator2 — its connection opened
with the flag, or its reads done by Mobium's reader — and saying so when a
session opens on a device with a screen reader on, until then.
- **Windows.** Everything that needs no device passes on a GitHub-hosted
Windows runner, every run: both modules' tests, the named-pipe daemon
transport's acceptance tests five times over, and the built `mobium.exe` —
Expand Down
23 changes: 22 additions & 1 deletion docs/SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,25 @@ One more, rarely: if debugging is on and still nothing appears, pull down the
notification shade, tap the USB notification and change the mode from charging
to **File transfer**. On Android 12+ this is usually unnecessary.

### With a screen reader on

Android's test-automation connection silences every other accessibility
service while it is open, unless it is opened not to. So on a device with
TalkBack — or VoiceView, on a Fire TV — on, a read of the screen takes the
screen reader away for as long as it runs, and an app that publishes its
contents only to a screen reader stops publishing them (CHALLENGES 208).

`--driver uiautomator` reads through a reader of Mobium's own whenever an
accessibility service is enabled, and that one leaves the screen reader
running. The default driver, UiAutomator2, does not yet: with it, the screen
reader is silent while a session is open and comes back when the session
ends. Nothing in the device's settings changes either way.

`MOBIUM_DUMP_READER=mobium` uses Mobium's reader for every read, and
`=uiautomator` uses `uiautomator dump` for every read. **It is read by the
daemon**, so it takes effect only if it is set when the daemon starts — set
on a later command, it does nothing. Run `mobium daemon stop` first.

### Over Wi-Fi

Android 11+. The phone and the Mac must be on the same network.
Expand Down Expand Up @@ -750,7 +769,9 @@ session — is [SHUTDOWN.md](SHUTDOWN.md).
Nothing is left in the device's filesystem. The dump backend writes a
hierarchy file to `/data/local/tmp` and deletes it after every read — that
file is a serialization of whatever is on screen, so on a real phone it is
your data, and it is not left behind. Verify with:
your data, and it is not left behind. With a screen reader on, it pushes its
own reader to `/data/local/tmp/mobium-reader` instead, and deletes the folder,
with the copy Android compiles into it, after every read. Verify with:

```sh
adb -s <serial> shell ls /data/local/tmp
Expand Down
83 changes: 83 additions & 0 deletions internal/mobiumdriver/android.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,12 @@ import (
"context"
"fmt"
"github.com/mobiumdev/mobium/internal/mobiumerr"
"os"
"strings"
"time"

"github.com/mobiumdev/mobium/internal/device"
"github.com/mobiumdev/mobium/internal/mobiumdriver/reader"
"github.com/mobiumdev/mobium/internal/uitree"
)

Expand Down Expand Up @@ -76,6 +78,13 @@ func (a *Android) dump(ctx context.Context) ([]byte, error) {
}

func (a *Android) dumpOnce(ctx context.Context) ([]byte, error) {
use, err := a.useReader(ctx)
if err != nil {
return nil, err
}
if use {
return a.readOnce(ctx)
}
out, diag, err := a.adb.ShellDiagnostics(ctx, "uiautomator", "dump", dumpPath)
if err != nil {
return nil, err
Expand Down Expand Up @@ -123,6 +132,80 @@ func (a *Android) dumpOnce(ctx context.Context) ([]byte, error) {
return xml, nil
}

// useReader says whether a read goes through Mobium's own reader rather than
// `uiautomator dump`. uiautomator connects in a way that suppresses every
// other accessibility service while it reads — a screen reader goes quiet,
// and an app that publishes its contents only to one stops publishing — so
// the reader is used whenever an accessibility service is enabled
// (CHALLENGES 208).
// MOBIUM_DUMP_READER=mobium or =uiautomator chooses one regardless.
func (a *Android) useReader(ctx context.Context) (bool, error) {
switch v := os.Getenv("MOBIUM_DUMP_READER"); v {
case "mobium":
return true, nil
case "uiautomator":
return false, nil
case "":
default:
return false, mobiumerr.New(mobiumerr.InvalidArgument,
"MOBIUM_DUMP_READER is %q — it takes mobium or uiautomator, or unset to decide by the device", v)
}
out, err := a.adb.Shell(ctx, "settings get secure accessibility_enabled; settings get secure enabled_accessibility_services")
if err != nil {
return false, err
}
lines := strings.Split(strings.TrimSpace(string(out)), "\n")
if len(lines) < 2 {
return false, nil
}
services := strings.TrimSpace(lines[1])
return strings.TrimSpace(lines[0]) == "1" && services != "" && services != "null", nil
}

// readOnce reads the screen with Mobium's own reader: pushed into a folder of
// its own, run with app_process, and the folder deleted after, with the
// runtime's compiled copy of it inside — whether or not the read worked.
func (a *Android) readOnce(ctx context.Context) ([]byte, error) {
dex, err := reader.Dex()
if err != nil {
return nil, err
}
f, err := os.CreateTemp("", "mobium-reader-*.dex")
if err != nil {
return nil, mobiumerr.New(mobiumerr.Internal, "stage the reader: %w", err)
}
defer os.Remove(f.Name())
if _, err := f.Write(dex); err != nil {
f.Close()
return nil, mobiumerr.New(mobiumerr.Internal, "stage the reader: %w", err)
}
if err := f.Close(); err != nil {
return nil, mobiumerr.New(mobiumerr.Internal, "stage the reader: %w", err)
}

defer func() {
cctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer cancel()
_, _ = a.adb.Shell(cctx, "rm", "-rf", reader.Dir)
}()
if _, err := a.adb.Shell(ctx, "mkdir", "-p", reader.Dir); err != nil {
return nil, err
}
remote := reader.Dir + "/reader.dex"
if _, err := a.adb.Run(ctx, "push", f.Name(), remote); err != nil {
return nil, err
}
out, diag, err := a.adb.ShellDiagnostics(ctx, "CLASSPATH="+remote, "app_process", "/system/bin", reader.Class)
if err != nil {
return nil, err
}
xml, err := reader.Hierarchy(out)
if err != nil && len(bytes.TrimSpace(diag)) > 0 {
return nil, fmt.Errorf("%w (stderr: %s)", err, firstLine(strings.TrimSpace(string(diag))))
}
return xml, err
}

// Screenshot captures the framebuffer as PNG.
func (a *Android) Screenshot(ctx context.Context) ([]byte, error) {
png, err := a.adb.ExecOut(ctx, "screencap", "-p")
Expand Down
93 changes: 93 additions & 0 deletions internal/mobiumdriver/dumpreader_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
package mobiumdriver

import (
"context"
"strings"
"testing"

"github.com/mobiumdev/mobium/internal/mobiumdriver/reader"
"github.com/mobiumdev/mobium/internal/uitree"
)

// readerADB is an adb on a device whose accessibility settings are given, and
// whose app_process prints what the reader would: a vendor line first, then
// the hierarchy between the markers (or the error line, when fail is set).
func readerADB(t *testing.T, a11yEnabled, services string, fail bool) (*Android, string) {
t.Helper()
readerOut := `echo "Init wrapper sys mutex successful. Pid:1029"; echo MOBIUM-READER-BEGIN; echo '` + miniDump + `'; echo MOBIUM-READER-END`
if fail {
readerOut = `echo "MOBIUM-READER-ERROR: IllegalStateException: no active window from UiAutomation in 10s"`
}
adb, log := fakeADB(t, `case "$*" in
*"settings get secure accessibility_enabled"*) echo `+a11yEnabled+`; echo "`+services+`" ;;
*app_process*) `+readerOut+` ;;
*"uiautomator dump"*) echo "UI hierchary dumped to: /data/local/tmp/mobium-dump.xml" ;;
*"cat /data/local/tmp/mobium-dump.xml"*) echo '`+miniDump+`' ;;
esac`)
return NewAndroid(adb), log
}

// With a screen reader enabled, the read goes through Mobium's reader, and the
// tree is the same one uiautomator's dump would have given.
func TestSnapshotUsesTheReaderWhenAScreenReaderIsOn(t *testing.T) {
t.Setenv("MOBIUM_DUMP_READER", "")
a, log := readerADB(t, "1", "com.amazon.logan/com.amazon.logan.LoganAccessibilityService", false)
tree, err := a.Snapshot(context.Background())
if err != nil {
t.Fatal(err)
}
found := false
tree.Walk(func(n *uitree.Node) bool {
found = found || n.Text == "Hello"
return true
})
if !found {
t.Fatal("the reader's hierarchy did not parse into the tree")
}
got := calls(t, log)
for _, want := range []string{"push", reader.Dir + "/reader.dex", "app_process /system/bin " + reader.Class, "rm -rf " + reader.Dir} {
if !strings.Contains(got, want) {
t.Errorf("no %q in the calls:\n%s", want, got)
}
}
if strings.Contains(got, "uiautomator dump") {
t.Errorf("uiautomator dump ran, which silences the screen reader:\n%s", got)
}
}

// The positive control: with no accessibility service on, nothing changes.
func TestSnapshotUsesUiautomatorWithNoServiceOn(t *testing.T) {
t.Setenv("MOBIUM_DUMP_READER", "")
for _, c := range []struct{ enabled, services string }{{"0", "null"}, {"1", "null"}, {"1", ""}, {"0", "x/y"}} {
a, log := readerADB(t, c.enabled, c.services, false)
if _, err := a.Snapshot(context.Background()); err != nil {
t.Fatal(err)
}
got := calls(t, log)
if strings.Contains(got, "app_process") || !strings.Contains(got, "uiautomator dump") {
t.Errorf("enabled=%s services=%q: want uiautomator dump, got:\n%s", c.enabled, c.services, got)
}
}
}

// A failed read still deletes what it put on the device, and says why.
func TestReaderCleansUpAfterAFailedRead(t *testing.T) {
t.Setenv("MOBIUM_DUMP_READER", "mobium")
a, log := readerADB(t, "0", "null", true)
_, err := a.Snapshot(context.Background())
if err == nil || !strings.Contains(err.Error(), "no active window") {
t.Fatalf("want the reader's reason, got %v", err)
}
got := calls(t, log)
if strings.Count(got, "app_process") != snapshotRetries || strings.Count(got, "rm -rf "+reader.Dir) != snapshotRetries {
t.Errorf("want %d reads each followed by a cleanup:\n%s", snapshotRetries, got)
}
}

func TestReaderOverrideIsChecked(t *testing.T) {
t.Setenv("MOBIUM_DUMP_READER", "talkback")
a, _ := readerADB(t, "0", "null", false)
if _, err := a.Snapshot(context.Background()); err == nil || !strings.Contains(err.Error(), "MOBIUM_DUMP_READER") {
t.Fatalf("an unknown value was accepted: %v", err)
}
}
Loading
Loading