Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/last-read-grace.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'e2e': patch
---

`locator.waitFor`, `expect` on a locator, and `toHaveScreenshot` give their last read extra time past the timeout, up to their slowest read so far and within the test timeout, so a screen that shows up during a slow last read still passes.
5 changes: 5 additions & 0 deletions .changeset/mobile-reads-budget.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@e2e-dev/mobile': patch
---

Snapshots and screenshots now run within the timeout of the call that takes them (an action's `timeout`, an `expect` timeout), so a device that stops answering fails the step instead of holding it until the test timeout. Requires `e2e` 0.22.0 or later.
10 changes: 8 additions & 2 deletions docs/mobile.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -716,8 +716,14 @@ message and hint, never `LAUNCH_TIMEOUT`, `ACTION_FAILED`, or
`back` gets its `timeout` (default `actionTimeout`). If the device doesn't
answer in time, the step fails with `ACTION_FAILED` and isn't retried: the
input may have landed. A `longPress` that holds at least as long as the
time left is refused before it's sent; raise its `timeout`. Finding the
element isn't cut off, so a missing one still fails as `LOCATOR_NOT_FOUND`.
time left is refused before it's sent; raise its `timeout`. Snapshots and
screenshots run within the timeout of the call that takes them, so a device
that stops answering fails the step with `ACTION_FAILED` instead of holding
it until the test timeout. `locator.waitFor`, `expect`, and
`toHaveScreenshot` give their last snapshot extra time, up to their slowest
one so far and within the test timeout, so a screen that shows up during it
still passes. An element that never shows up fails as
`LOCATOR_NOT_FOUND`, or `ASSERTION_FAILED` for `expect`.

<CodeGroup>
```bash npm
Expand Down
11 changes: 8 additions & 3 deletions docs/reference/engine.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,11 @@ export interface OperationContext {
* abort at `timeoutMs`: bound each call by that budget yourself.
*/
readonly signal: AbortSignal;
/** Remaining operation budget when the call starts. */
/**
* Remaining operation budget when the call starts. A waiting call's last
* locate or read can get more than its deadline leaves: as long as the
* slowest one it has seen.
*/
readonly timeoutMs: number;
/** The run id. */
readonly runId: string;
Expand Down Expand Up @@ -1417,8 +1421,9 @@ export const RETRYABLE_ENGINE_ERROR_CODES: ReadonlySet<EngineErrorCode> = new Se

#### Timeouts at a deadline

An `OPERATION_TIMEOUT` on a locate or read that started with under 100 ms of
a waiting call's deadline left, after one that completed, ends the wait
An `OPERATION_TIMEOUT` on a locate or read that started with under 100 ms, or
under its slowest completed locate or read, of a waiting call's deadline left,
after one that completed, ends the wait
instead:

| Waiting call | Fails with |
Expand Down
6 changes: 5 additions & 1 deletion packages/e2e/src/engine/contract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,11 @@ export interface OperationContext {
* abort at `timeoutMs`: bound each call by that budget yourself.
*/
readonly signal: AbortSignal;
/** Remaining operation budget when the call starts. */
/**
* Remaining operation budget when the call starts. A waiting call's last
* locate or read can get more than its deadline leaves: as long as the
* slowest one it has seen.
*/
readonly timeoutMs: number;
/** The run id. */
readonly runId: string;
Expand Down
17 changes: 9 additions & 8 deletions packages/e2e/src/expect/async.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import {
type TextPattern,
} from '../internal/text.ts';
import { isValueControl } from '../internal/roles.ts';
import { Deadline, pollCondition } from '../internal/time.ts';
import { pollReads, type ReadWindow } from '../internal/time.ts';
import { SampleHistory } from './samples.ts';
import { attributeOf, denySecureRead, isNodeVisible } from '../locator/engine.ts';
import { describeExpression } from '../locator/expression.ts';
Expand Down Expand Up @@ -173,12 +173,13 @@ class AsyncExpectationImpl implements AsyncExpectation {
let lastSample: Sample = { count: 0, node: null, nodes: [] };
const samples = new SampleHistory('expect', (text) => engine.redact(text));
await samples.record(engine, () =>
pollCondition({
pollReads({
deadline,
reads: engine.reads(deadline),
signal: engine.signal,
negated: this.negated,
evaluate: async () => {
const sample = await this.sample(spec, deadline);
evaluate: async (reads) => {
Comment thread
okwasniewski marked this conversation as resolved.
Resolved
const sample = await this.sample(spec, reads);
lastSample = sample;
if (spec.readsWithheld === true) {
denySecureRead(sample.node === null ? sample.nodes : [sample.node], this.label);
Expand Down Expand Up @@ -221,18 +222,18 @@ class AsyncExpectationImpl implements AsyncExpectation {
return spec.evaluableNode?.(sample.node) ?? true;
}

private async sample(spec: MatcherSpec, deadline: Deadline): Promise<Sample> {
private async sample(spec: MatcherSpec, reads: ReadWindow): Promise<Sample> {
const { engine } = this.internals.context;
if (spec.wholeSet === 'read') {
const nodes = await engine.readAllNow(this.internals.expression, deadline);
const nodes = await engine.readAllNow(this.internals.expression, reads);
return { count: nodes.length, node: null, nodes };
}
if (spec.wholeSet === true) {
const refs = await engine.resolveNow(this.internals.expression, deadline);
const refs = await engine.resolveNow(this.internals.expression, reads);
return { count: refs.length, node: null, nodes: [] };
}
const missingFrame = spec.evaluableWithoutNode === true ? 'empty' : 'wait';
const { node, count } = await engine.tryRead(this.internals.expression, deadline, missingFrame);
const { node, count } = await engine.tryRead(this.internals.expression, reads, missingFrame);
return { count, node, nodes: [] };
}

Expand Down
34 changes: 13 additions & 21 deletions packages/e2e/src/expect/screenshot-capture.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import type { LocatorExpression } from '../engine/surface.ts';
import { ConfigurationError, isE2EError } from '../internal/errors.ts';
import { clipBox, cropImage, decodePng, fillBoxes, resampleImage, type ImageBox, type RgbaImage } from '../internal/image.ts';
import { isOperationTimeout, type Deadline } from '../internal/time.ts';
import type { ReadWindow } from '../internal/time.ts';
import { isNodeVisible } from '../locator/engine.ts';
import { describeExpression } from '../locator/expression.ts';
import type { ScreenContext } from '../locator/screen.ts';
Expand All @@ -29,44 +29,36 @@ export type Capture = { readonly image: RgbaImage } | { readonly missing: string
/**
* Captures for one call. The subject is scrolled into view once, the first
* time it is found; a subject that is not there yet is a capture still
* missing, so the call's poll keeps waiting and fails as an assertion. So is
* a capture the deadline cut short: the call then fails on what the
* captures before it showed.
* missing, so the call's poll keeps waiting and fails as an assertion. A
* capture the deadline cut short throws, and the poll judges it as it judges
* any read it cut off.
*/
export function screenshotCapturer(context: ScreenContext, request: CaptureRequest, deadline: Deadline): () => Promise<Capture> {
const { engine } = context;
const state = { scrolled: !engine.session.actions.has('scrollIntoView') };
return async () => {
try {
return await captureOnce(context, request, deadline, state);
} catch (cause) {
if (deadline.expired() && isOperationTimeout(cause)) return { missing: 'the last screenshot ran past the timeout' };
throw cause;
}
};
export function screenshotCapturer(context: ScreenContext, request: CaptureRequest, reads: ReadWindow): () => Promise<Capture> {
const state = { scrolled: !context.engine.session.actions.has('scrollIntoView') };
return () => captureOnce(context, request, reads, state);
}

/** One capture; `state.scrolled` records that the subject was scrolled into view. */
async function captureOnce(context: ScreenContext, request: CaptureRequest, deadline: Deadline, state: { scrolled: boolean }): Promise<Capture> {
async function captureOnce(context: ScreenContext, request: CaptureRequest, reads: ReadWindow, state: { scrolled: boolean }): Promise<Capture> {
const { engine } = context;
const shown = request.subject === undefined ? 'screen' : describeExpression(request.subject);
let box: ImageBox | undefined;
if (request.subject !== undefined) {
let { node } = await engine.tryRead(request.subject, deadline);
let { node } = await engine.tryRead(request.subject, reads);
if (node !== null && !state.scrolled) {
try {
await engine.performUntil(request.subject, { kind: 'scrollIntoView' }, deadline);
await engine.performUntil(request.subject, { kind: 'scrollIntoView' }, reads.deadline);
} catch (cause) {
if (!isE2EError(cause) || cause.code !== 'LOCATOR_NOT_FOUND') throw cause;
}
state.scrolled = true;
({ node } = await engine.tryRead(request.subject, deadline));
({ node } = await engine.tryRead(request.subject, reads));
}
if (node === null) return { missing: `no node matches ${shown}` };
if (!isNodeVisible(node) || node.rect === undefined) return { missing: `${shown} is not visible` };
box = node.rect;
}
const observation = await engine.session.observe(engine.operationWithin(deadline), { pixels: true, comparable: true });
const observation = await reads.read((timeoutMs) => engine.session.observe(engine.readOperation(timeoutMs), { pixels: true, comparable: true }));
if (observation.pixels === undefined) return { missing: 'the engine returned no screenshot' };
if (!pixelsProvenMasked(observation.redaction)) {
throw new ConfigurationError('POLICY_DENIED', `${request.api} could not prove every secure field on screen was masked, so the screenshot is not kept`);
Expand All @@ -85,7 +77,7 @@ async function captureOnce(context: ScreenContext, request: CaptureRequest, dead
if (request.masks.length === 0) return { image };
const boxes: ImageBox[] = [];
for (const mask of request.masks) {
for (const node of await engine.readAllNow(mask, deadline)) {
for (const node of await engine.readAllNow(mask, reads)) {
if (isNodeVisible(node) && node.rect !== undefined) boxes.push({ ...node.rect, x: node.rect.x - origin.x, y: node.rect.y - origin.y });
}
}
Expand Down
18 changes: 11 additions & 7 deletions packages/e2e/src/expect/screenshot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import type { LocatorExpression } from '../engine/surface.ts';
import { writeFileAtomic } from '../internal/atomic-write.ts';
import { TestError } from '../internal/errors.ts';
import { compareImages, decodePng, encodePng, type ImageComparison, type RgbaImage } from '../internal/image.ts';
import { pollCondition, type Deadline } from '../internal/time.ts';
import { pollReads, type Deadline, type ReadWindow } from '../internal/time.ts';
import { describeExpression } from '../locator/expression.ts';
import type { ScreenContext } from '../locator/screen.ts';
import type { ScreenshotContext } from '../run/screenshots.ts';
Expand All @@ -32,6 +32,7 @@ interface Run {
readonly api: string;
readonly call: ScreenshotCall;
readonly stored: Stored;
readonly reads: ReadWindow;
readonly capture: () => Promise<Capture>;
}

Expand Down Expand Up @@ -75,8 +76,9 @@ export async function expectScreenshot(
const label = `${subject === undefined ? 'screen' : describeExpression(subject)} vs ${call.stem}.png`;
await context.steps.run('assertion', api, label, async () => {
const deadline = context.engine.deadline(call.timeout ?? context.engine.assertionTimeout);
const capture = screenshotCapturer(context, { api, subject, masks: call.masks, maskColor: call.maskColor }, deadline);
const run: Run = { context, store, api, call, stored, capture };
const reads = context.engine.reads(deadline);
const capture = screenshotCapturer(context, { api, subject, masks: call.masks, maskColor: call.maskColor }, reads);
const run: Run = { context, store, api, call, stored, reads, capture };
const expected = existsSync(file) ? readStored(api, stored) : undefined;
if (expected !== undefined && (negated || !store.update) && store.written.has(file)) {
throw failure(api, `${stored.shown} was written earlier in this run and is not reviewed yet: look at it, commit it, and run again`, stored, 'unreviewed stored screenshot');
Expand Down Expand Up @@ -107,8 +109,9 @@ async function matchOrSettle(run: Run, expected: RgbaImage | undefined, deadline
let unsettled: readonly [RgbaImage, RgbaImage] | undefined;
let settled: RgbaImage | undefined;
let captures = 0;
await pollCondition({
await pollReads({
deadline,
reads: run.reads,
signal: run.context.engine.signal,
negated: false,
evaluate: async () => {
Expand All @@ -134,7 +137,7 @@ async function matchOrSettle(run: Run, expected: RgbaImage | undefined, deadline
return false;
},
onTimeout: (cause) => {
if (latest === undefined) return failure(run.api, missing ?? 'no screenshot was taken', run.stored, 'no screenshot', cause);
if (latest === undefined) return failure(run.api, missing ?? (cause === undefined ? 'no screenshot was taken' : 'the last screenshot ran past the timeout'), run.stored, 'no screenshot', cause);
if (!settles && expected !== undefined && latest.comparison !== undefined && latest.comparison.kind !== 'match') {
return mismatch(run, expected, latest.image, latest.comparison, cause);
}
Expand Down Expand Up @@ -181,8 +184,9 @@ async function expectDifferent(run: Run, expected: RgbaImage | undefined, deadli
}
let compared = false;
let missing: string | undefined;
await pollCondition({
await pollReads({
deadline,
reads: run.reads,
signal: run.context.engine.signal,
negated: true,
evaluate: async () => {
Expand All @@ -197,7 +201,7 @@ async function expectDifferent(run: Run, expected: RgbaImage | undefined, deadli
onTimeout: (cause) =>
compared
? failure(run.api, `the screen still matches ${run.stored.shown}`, run.stored, 'matching', cause)
: failure(run.api, missing ?? 'no screenshot was taken', run.stored, 'no screenshot', cause),
: failure(run.api, missing ?? (cause === undefined ? 'no screenshot was taken' : 'the last screenshot ran past the timeout'), run.stored, 'no screenshot', cause),
});
}

Expand Down
Loading
Loading