diff --git a/domains/testing/skills/e2e-testing/references/running-tests-locally.md b/domains/testing/skills/e2e-testing/references/running-tests-locally.md new file mode 100644 index 00000000..5885eac5 --- /dev/null +++ b/domains/testing/skills/e2e-testing/references/running-tests-locally.md @@ -0,0 +1,245 @@ +# Running E2E Tests Locally — MetaMask Mobile + +> **Audience:** AI agents and engineers validating E2E changes in `metamask-mobile`. +> **Goal:** Run the same **main-e2e release** builds CI uses — not debug/Expo dev-launcher binaries. + +## Test runners at a glance + +| Runner | Spec location | Config | When to use | +| ------ | ------------- | ------ | ----------- | +| **Appium smoke** (Playwright) | `tests/smoke-appium/` | `tests/playwright.smoke-appium.config.ts` | Appium framework/POM changes, smoke parity with CI Appium jobs | +| **Detox smoke** | `tests/smoke/` | `.detoxrc.js` | Detox-only smoke specs, legacy local workflow | +| **Detox regression** | `tests/regression/` | `.detoxrc.js` | Full regression suites | + +**Prefer iOS Appium smoke locally** when validating Appium-related PRs: no physical device, matches CI Appium smoke, and the main-e2e iOS `.app` runs on Apple Silicon simulators. + +For writing new Detox specs, see the `e2e-test` skill (`references/running-tests.md`). + +--- + +## Agent execution standard (Appium smoke on iOS) + +Follow these steps **in order**. Do not skip the build check. + +### Step 0 — Prerequisites + +```bash +# From metamask-mobile repo root +cp .e2e.env.example .e2e.env # if missing; fill MM_TEST_ACCOUNT_SRP etc. +yarn install +``` + +Ensure Xcode simulators are available (`xcrun simctl list devices`). + +### Step 1 — Resolve the correct build + +Appium smoke requires a **main-e2e release** build with `HAS_TEST_OVERRIDES=true` baked in at compile time. That activates `ReadOnlyNetworkStore` and fixture loading from `/state.json`. + +| Build | Artifact name (CI) | Local path (after download) | Appium smoke? | +| ----- | ------------------ | --------------------------- | ------------- | +| **main-e2e iOS** (use this) | `main-e2e-MetaMask.app` | `build/ci-main-e2e/MetaMask.app` | ✅ Yes | +| **main-e2e Android** | `main-e2e-release.apk` | `build/ci-main-e2e/app-prod-release.apk` | ✅ Yes (see ABI note below) | +| Detox debug iOS | `Debug-iphonesimulator/MetaMask.app` | local Xcode output | ❌ Wrong runner/config | +| Detox debug Android (`yarn test:e2e:android:debug:build`) | `app-prod-debug.apk` | Expo **Development Build** — stops on Connect-to-Metro | ❌ No | +| `main-e2e-release-androidTest.apk` | instrumentation APK | Detox only | ❌ No | + +**Do not** use `yarn test:e2e:ios:debug:build` or `yarn test:e2e:android:debug:build` for Appium smoke. + +### Step 2 — Download CI build (preferred over local compile) + +Download from a successful **`build` workflow** on `main` (or the PR branch if it produced artifacts). Use `gh`: + +```bash +mkdir -p build/ci-main-e2e + +# Find a recent successful build run on the target branch +gh run list --repo MetaMask/metamask-mobile --workflow build --branch main --limit 5 + +# Download iOS main-e2e app (replace RUN_ID) +gh run download RUN_ID --repo MetaMask/metamask-mobile \ + -n main-e2e-MetaMask.app -D build/ci-main-e2e + +# Optional: Android release APK (same run) +gh run download RUN_ID --repo MetaMask/metamask-mobile \ + -n main-e2e-release.apk -D build/ci-main-e2e +mv -f build/ci-main-e2e/main-e2e-release.apk build/ci-main-e2e/app-prod-release.apk +``` + +**iOS artifact layout:** `gh run download` may extract loose files instead of a `.app` bundle. Reassemble if needed: + +```bash +mkdir -p build/ci-main-e2e/MetaMask.app +# Move Info.plist, MetaMask binary, Frameworks/, etc. into MetaMask.app/ +chmod +x build/ci-main-e2e/MetaMask.app/MetaMask +``` + +**Restore execute permissions** (CI does this automatically): + +```bash +chmod +x build/ci-main-e2e/MetaMask.app/MetaMask +find build/ci-main-e2e/MetaMask.app/Frameworks -type f \( -name '*.dylib' -o -path '*.framework/*' \) -exec chmod +x {} \; 2>/dev/null || true +``` + +Verify before running: + +```bash +test -x build/ci-main-e2e/MetaMask.app/MetaMask && echo "✅ iOS build ready" || echo "❌ iOS build missing or not executable" +``` + +### Step 3 — Override `.e2e.env` prebuilt paths + +`tests/playwright.smoke-appium.config.ts` resolves the app path in this order: + +1. `IOS_APP_PATH` / `ANDROID_APK_PATH` (explicit — **agents should set these**) +2. `PREBUILT_IOS_APP_PATH` / `PREBUILT_ANDROID_APK_PATH` from `.e2e.env` +3. Default: `build/ci-main-e2e/MetaMask.app` / `build/ci-main-e2e/app-prod-release.apk` + +`.e2e.env` often points `PREBUILT_*` at **debug** builds. Always pass `IOS_APP_PATH` when using CI artifacts so debug paths do not win. + +### Step 4 — Boot simulator and prepare runner (iOS) + +CI runs `node scripts/e2e/prepare-ios-appium-runner.mjs` before tests. Locally: + +```bash +# Boot iPhone 16 Pro (or your chosen sim) +xcrun simctl boot "iPhone 16 Pro" 2>/dev/null || true +open -a Simulator + +# Prepare WDA + install app (exports UDID to use below) +IOS_APP_PATH=build/ci-main-e2e/MetaMask.app \ +IOS_SIMULATOR_NAME="iPhone 16 Pro" \ +node scripts/e2e/prepare-ios-appium-runner.mjs + +# Capture booted UDID +export IOS_SIMULATOR_UDID=$(xcrun simctl list devices booted -j | node -e " + const d=JSON.parse(require('fs').readFileSync(0,'utf8')); + const booted=Object.values(d.devices).flat().find(x=>x.state==='Booted'); + console.log(booted?.udid||''); +") +echo "IOS_SIMULATOR_UDID=$IOS_SIMULATOR_UDID" +``` + +First run may take several minutes while WebDriverAgent builds. + +### Step 5 — Run Appium smoke tests + +```bash +# Full iOS smoke suite +IOS_APP_PATH=build/ci-main-e2e/MetaMask.app \ +IOS_SIMULATOR_UDID="$IOS_SIMULATOR_UDID" \ +yarn appium-smoke:ios + +# Single spec +IOS_APP_PATH=build/ci-main-e2e/MetaMask.app \ +IOS_SIMULATOR_UDID="$IOS_SIMULATOR_UDID" \ +yarn appium-smoke:ios --grep "Secret Recovery Phrase Reveal" \ + tests/smoke-appium/accounts/reveal-secret-recovery-phrase.spec.ts + +# Filter by smoke tag (matches describe title via tags.js) +IOS_APP_PATH=build/ci-main-e2e/MetaMask.app \ +IOS_SIMULATOR_UDID="$IOS_SIMULATOR_UDID" \ +yarn appium-smoke:ios --grep SmokePerps +``` + +Package scripts (from `package.json`): + +```bash +yarn appium-smoke:ios # --project ios-smoke +yarn appium-smoke:android # --project android-smoke +``` + +### Step 6 — Lint before/after code changes + +```bash +yarn lint tests/smoke-appium/ tests/page-objects/ --fix +yarn lint:tsc +``` + +### Step 7 — Iterate + +``` +Change code → lint/tsc → run targeted Appium spec → read HTML report → fix → repeat +``` + +Reports: `test-reports/appium-smoke-report/` (HTML), `test-reports/appium-smoke-junit.xml`. + +--- + +## Android Appium smoke (local caveats) + +```bash +IOS_APP_PATH=... # not needed +ANDROID_APK_PATH=build/ci-main-e2e/app-prod-release.apk \ +ANDROID_AVD_NAME=Pixel_5_Pro_API_34 \ +yarn appium-smoke:android --grep SmokeAccounts +``` + +**ABI mismatch on Apple Silicon Macs:** CI Android main-e2e APKs are built for **x86_64** emulators. arm64-v8a local emulators fail with `INSTALL_FAILED_NO_MATCHING_ABIS`. Options: + +1. **Use iOS** for local Appium validation (recommended on Mac). +2. Build locally: `BUILD_CONFIG_NAME=main-e2e HAS_TEST_OVERRIDES=true METAMASK_ENVIRONMENT=e2e yarn build:android:main:e2e` (produces arm64-compatible APK). +3. Use an x86_64 AVD only on Intel hosts or CI runners. + +--- + +## Building main-e2e locally (when CI artifact unavailable) + +Local compile is slower (~20–30+ min) but produces the correct build type: + +```bash +# iOS simulator release e2e +HAS_TEST_OVERRIDES=true METAMASK_ENVIRONMENT=e2e \ + CONFIGURATION=Release yarn build:ios:main:e2e +# Output: ios/build/Build/Products/Release-iphonesimulator/MetaMask.app + +# Android release e2e (arm64 on Mac) +BUILD_CONFIG_NAME=main-e2e HAS_TEST_OVERRIDES=true METAMASK_ENVIRONMENT=e2e \ + yarn build:android:main:e2e +# Output: android/app/build/outputs/apk/prod/release/app-prod-release.apk +``` + +Then set `IOS_APP_PATH` or `ANDROID_APK_PATH` to those paths. + +**Warn the user** before starting a local native build — do not kick off long builds silently. + +--- + +## Detox local runs (smoke / regression) + +Detox uses **debug** simulator/emulator builds from `.detoxrc.js` / `.e2e.env` `PREBUILT_*` paths. + +```bash +# Verify debug build exists +ls ios/build/Build/Products/Debug-iphonesimulator/MetaMask.app 2>/dev/null \ + || echo "Run: yarn test:e2e:ios:debug:build" + +# Run one Detox spec (iOS) +IS_TEST='true' NODE_OPTIONS='--experimental-vm-modules' \ + detox test -c ios.sim.main \ + --testPathPattern="tests/smoke/perps/perps-position-stop-loss.spec.ts" +``` + +See `e2e-test` skill → `references/running-tests.md` for full Detox commands and debugging. + +--- + +## Common failures + +| Failure | Likely cause | Fix | +| ------- | ------------ | --- | +| Expo dev launcher / Connect to Metro | Debug APK used for Appium | Use `main-e2e-release.apk` or CI artifact | +| `INSTALL_FAILED_NO_MATCHING_ABIS` (Android) | CI x86_64 APK on arm64 emulator | Use iOS locally or build arm64 main-e2e | +| `IOS_APP_PATH does not exist` | Artifact not downloaded or wrong bundle layout | Re-download; ensure `MetaMask.app/` directory | +| WDA / `ECONNREFUSED 127.0.0.1:8100` | WDA not ready | Run `prepare-ios-appium-runner.mjs`; retry (framework has terminate retry) | +| Fixture/state not loading | Build without `HAS_TEST_OVERRIDES` | Use main-e2e release build only | +| Wrong app launched | `.e2e.env` `PREBUILT_*` overrides defaults | Set `IOS_APP_PATH` explicitly | + +--- + +## Agent checklist (before claiming tests pass) + +- [ ] Used **main-e2e** release build (CI download or local e2e build) — not debug +- [ ] Set `IOS_APP_PATH` (and `IOS_SIMULATOR_UDID` after boot) for iOS Appium runs +- [ ] Ran the **specific spec or tag** relevant to the PR change +- [ ] `yarn lint` / `yarn lint:tsc` clean on touched files +- [ ] Captured pass/fail from command output (do not assume green without running) diff --git a/domains/testing/skills/e2e-testing/repos/metamask-mobile.md b/domains/testing/skills/e2e-testing/repos/metamask-mobile.md index ad388d10..197aa637 100644 --- a/domains/testing/skills/e2e-testing/repos/metamask-mobile.md +++ b/domains/testing/skills/e2e-testing/repos/metamask-mobile.md @@ -5,6 +5,25 @@ parent: e2e-testing # MetaMask Mobile E2E Testing Guidelines +## Running tests locally (agents — read first) + +When asked to **run**, **validate**, or **debug** E2E tests, open +[references/running-tests-locally.md](references/running-tests-locally.md) and follow the **Agent execution standard**. + +Quick rules: + +1. **Appium smoke** (`tests/smoke-appium/`, `yarn appium-smoke:ios`) — use the **main-e2e release** build, same as CI Appium jobs. Download artifact `main-e2e-MetaMask.app` from a successful `build` workflow run; local path `build/ci-main-e2e/MetaMask.app`. Always set `IOS_APP_PATH` so `.e2e.env` debug `PREBUILT_*` paths do not override. +2. **Do not** use Detox debug builds (`yarn test:e2e:*:debug:build`) for Appium smoke — Android debug opens the Expo dev launcher; iOS debug is the wrong config for `playwright.smoke-appium.config.ts`. +3. **Prefer iOS** for local Appium validation on Mac (CI Android APK is x86_64-only). +4. Boot sim → `node scripts/e2e/prepare-ios-appium-runner.mjs` → run with `IOS_SIMULATOR_UDID`. +5. For **writing** new Detox specs, use the `e2e-test` skill instead. + +| Task | Open | +| ---- | ---- | +| Run Appium smoke / validate PR locally | [running-tests-locally.md](references/running-tests-locally.md) | +| Write new Detox spec, POM, mocks | `e2e-test` skill | +| Cross-framework POM (`resolve`, Appium vs Detox) | Sections below + `docs/testing/e2e-testing.md` in repo | + ## Core Principles 1. **Test Coverage is Critical**: Higher coverage creates more confidence and helps identify bugs effectively. diff --git a/domains/testing/skills/e2e-testing/skill.md b/domains/testing/skills/e2e-testing/skill.md index c2cd6c5a..7c59da6f 100644 --- a/domains/testing/skills/e2e-testing/skill.md +++ b/domains/testing/skills/e2e-testing/skill.md @@ -1,4 +1,8 @@ --- name: e2e-testing -description: E2E testing guidelines +description: >- + MetaMask Mobile E2E testing guidelines — POM patterns, fixtures, flakiness + handling, and local execution. For Appium smoke runs use main-e2e CI release + builds (main-e2e-MetaMask.app on iOS); see references/running-tests-locally.md. + For writing new Detox specs see the e2e-test skill. ---