Skip to content
Draft
2 changes: 1 addition & 1 deletion .changeset/tame-dogs-listen.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@

Keep healthy engine sessions open on leased devices across worker replacement. After a failed test, a retired worker could drop the device claim. A competing session could then take the device and make later tests fail with "Device is already in use".

Forced worker shutdown and unconfirmed agent-device recording stops still close the engine session. At run end, the engine closes its sessions and the provider releases its device leases.
Forced worker shutdown and unconfirmed agent-device recording stops still close the engine session. A later attempt also resets a leftover native screen recording by closing that session, dropping its device claim even on a leased slot. At run end, the engine closes its sessions and the provider releases its device leases.

Before another attempt on the same worker, retry a provider recording whose stop failed. Wait for that stop to finish before starting the attempt, and keep its original artifact directory. The worker also retries cleanup at attempt end and disposal while it owns the recording handle. Providers remain responsible for any external recording cleanup when releasing a lease at run end. Closing an agent-device session cannot guarantee that a cloud recording stops.
1 change: 1 addition & 0 deletions docs/reference/mobile.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -709,6 +709,7 @@ await device.openApp('com.example.app'); // launches fresh, on its first screen
```

- A screen query before the next `openApp` is `APP_NOT_OPEN` too.
- If a previous attempt left a native screen recording unfinished, the next attempt resets it by closing the agent-device session. On a leased device, that also drops the session's device claim, even when worker reuse was otherwise permitted. The provider lease itself is released at run end.

#### `clearKeychain`

Expand Down
18 changes: 17 additions & 1 deletion packages/e2e/src/run/scheduler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,11 @@ class SchedulerWorker {
this.killAfter(graceMs);
}

/** Revokes a retired worker's reuse permission without restarting its kill deadline. */
revokeReuse(): void {
if (this.runner.alive) this.runner.send({ type: 'shutdown' });
}

/**
* Asks for immediate engine disposal and force-kills once the budget is
* spent, replacing the interrupt grace a busy worker was given.
Expand Down Expand Up @@ -499,6 +504,12 @@ class Scheduler {
}
}

/**
* The forced interrupt: no worker gets to finish its unit any more. Each
* busy one disposes its engine now and is killed after the force budget,
* however long the unit's own grace still had to run. A live retired
* worker also receives it to revoke resource reuse during disposal.
*/
private broadcastForce(): void {
if (!this.options.forceSignal.aborted || this.forceBroadcast) return;
this.forceBroadcast = true;
Expand Down Expand Up @@ -701,7 +712,12 @@ class Scheduler {
* it first, and the others only ever retire idle workers.
*/
private retire(worker: SchedulerWorker, preserveResources: boolean): void {
if (worker.state === 'retired' && preserveResources) return;
if (worker.state === 'retired') {
// A final interrupt can withdraw reuse permission while the earlier
// replacement retirement is still disposing its engine.
if (!preserveResources) worker.revokeReuse();
return;
}
worker.state = 'retired';
worker.shutdown(SHUTDOWN_GRACE_MS, preserveResources);
}
Expand Down
1 change: 1 addition & 0 deletions packages/e2e/src/run/worker/protocol.ts
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ export interface InterruptMessage {

export interface ShutdownMessage {
readonly type: 'shutdown';
/** Permission to retain run-owned resources for a replacement. Missing means final cleanup. */
readonly preserveResources?: true;
}

Expand Down
4 changes: 4 additions & 0 deletions packages/mobile/src/bindings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@ export interface SlotBinding {
readonly client?: DeviceClientConfig | undefined;
/** App a provider already installed from `appPath`, so the worker installs nothing and opens this. */
readonly installedApp?: string | undefined;
/**
* The slot's provider lease id. The engine uses it to retain healthy
* sessions across worker replacement and to find the lease for `record`.
*/
readonly leaseId?: string | undefined;
}

Expand Down
7 changes: 7 additions & 0 deletions packages/mobile/src/pool.ts
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,13 @@ export class DevicePool {
return { workers: bindings.length, env: { [bindingsVariable(info.targetName)]: encodeBindings(bindings) } };
}

/**
* Closes every engine session `warm` opened for the target and releases
* what the source acquired. This is the normal run-end close for leased
* slots and a fallback for local sessions or a failed `prepare`. Session
* closes and source release run concurrently under the hook's cleanup
* budget, so a hung close does not delay lease release.
*/
async finish(info: EngineFinishInfo): Promise<void> {
const clients = this.warmed.get(info.targetName) ?? [];
this.warmed.delete(info.targetName);
Expand Down
7 changes: 6 additions & 1 deletion packages/mobile/src/surface.ts
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,7 @@ function tellHarness(call: () => void): void {
export class AgentDeviceSurface {
private client: AgentDeviceClient | undefined;
private attempt: Attempt | undefined;
/** Retained across attempt cleanup until a stop or native session reset is confirmed. */
private recording: Recording | undefined;
/** The attempt's `EngineAttemptContext.screen`, which locate captures feed; undefined between attempts. */
private screenSink: ((snapshot: EngineSnapshot) => void) | undefined;
Expand Down Expand Up @@ -704,8 +705,12 @@ export class AgentDeviceSurface {
}).catch(() => undefined);
};
const reuse = context.reuseSignal;
const canKeepSession =
this.device?.leaseId !== undefined && recording?.kind !== 'screen' && reuse !== undefined && !reuse.aborted;
// Retain only a leased, healthy session while a replacement is allowed.
// Native recording reset and final/forced cleanup must close it.
reuse?.addEventListener('abort', close, { once: true });
if (this.device?.leaseId === undefined || recording?.kind === 'screen' || reuse === undefined || reuse.aborted) close();
if (!canKeepSession) close();
try {
const stopping = recording?.kind === 'lease' ? this.stopLeaseRecording(recording, context.signal).catch(() => undefined) : undefined;
await withinCleanupBudget((async () => {
Expand Down
2 changes: 2 additions & 0 deletions packages/testmu/src/provider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,8 @@ export function testmu(options: TestmuOptions): DeviceProvider {
leaseBackend,
leaseProvider: PROVIDER,
ttlMs: LEASE_TTL_MS,
// Retain the provider lease when closeApp or cleanup closes the engine session.
// The provider releases the lease at run end.
retainOnClose: true,
...selectors,
},
Expand Down
Loading