diff --git a/.changeset/mcp-engine-session.md b/.changeset/mcp-engine-session.md new file mode 100644 index 0000000000..a29d80f41d --- /dev/null +++ b/.changeset/mcp-engine-session.md @@ -0,0 +1,7 @@ +--- +"e2e": patch +"@e2e-dev/web": patch +"@e2e-dev/mobile": patch +--- + +Open an MCP session in a directory with no config: `open_session {engine: "@e2e-dev/web", app: {url}}` names the engine to drive and `app` the app under test. `engine` is a package name, or the factory call a config would write as `{package, export, options}`, where an option can be another such call: `{package: "@e2e-dev/mobile", options: {platform: "ios", device: {package: "@e2e-dev/limrun", export: "limrun"}}}`. Each package is found in the project, beside e2e, or in `~/.e2e/engines`; when it is in none, `ENGINE_NOT_INSTALLED` names the install command. `@e2e-dev/web` and `@e2e-dev/mobile` export their factory as the default export for this, and `mobile()` called without a platform now says which platforms it takes. diff --git a/AGENTS.md b/AGENTS.md index 0d3f3b061c..126243ba98 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,7 +61,11 @@ suites that consume the built packages the way a user would. `src/agent/` the agent (the `act` executor socket plus the judgment methods), `src/mcp/` the `e2e mcp` server (a live session that rides the `act` socket with a queue executor so every MCP call is a harness - action), `src/oauth/` subscription sign-in (the `e2e login`, `logout`, + action; in a directory with no config, `open_session {engine}` names the + engine as a package, whose default export is called, or as a factory + call, a package with an export and options that may hold further such + calls, which `mcp/default-engine.ts` resolves in the project, beside e2e, + or in `~/.e2e/engines`, never by a specifier written in core), `src/oauth/` subscription sign-in (the `e2e login`, `logout`, and `models` commands and the `e2e/oauth/*` model constructors; each constructor subpath is the place its `@ai-sdk/*` optional peers are imported, and the Copilot constructor loads `@ai-sdk/openai` lazily, only diff --git a/docs/mcp-server.mdx b/docs/mcp-server.mdx index a858fad367..c981c6dba6 100644 --- a/docs/mcp-server.mdx +++ b/docs/mcp-server.mdx @@ -75,6 +75,19 @@ A session starts the target's app command, if it has one, and names it with its URL and port. The config reads the server's environment, not the agent's shell. +## Try it without a config + +In a directory with no e2e config file, the agent names the engine when it +opens a session: + +```txt +open_session {engine: "@e2e-dev/web", app: {url: "http://localhost:3000"}} +``` + +`run_tests` still needs a config. For engine options, device providers, and +installing an engine, see +[Without a config](/reference/mcp#without-a-config). + ## Ask the agent - "Add a Remember me checkbox to the login form. Check it in the app with e2e." diff --git a/docs/reference/errors.mdx b/docs/reference/errors.mdx index 398eacbbbb..e18d749035 100644 --- a/docs/reference/errors.mdx +++ b/docs/reference/errors.mdx @@ -140,7 +140,8 @@ These appear in terminal output and in the report as plain string codes. | `NO_SESSION` | `e2e mcp`: `call` or `tools` before `open_session` or after the session ended (the message names why), or `call`, `tools`, or `close_session` naming a `session` id that is not open or is closing | configuration | 2 | | `SESSION_REQUIRED` | `e2e mcp`: `call`, `tools`, or `close_session` without a `session` while several sessions are open. The message lists them | configuration | 2 | | `ENGINE_IN_USE` | `e2e mcp`: `open_session` on a target whose engine instance another open session drives, because the config gets it from a package every session shares. Create the engine in the config, or in a file it imports by path | configuration | 2 | -| `CONFIG_IN_USE` | `e2e mcp`: `open_session` on a config other than the one the open sessions use. Credentials and secrets resolve process-wide, so open on that config or close those sessions first | configuration | 2 | +| `ENGINE_NOT_INSTALLED` | `e2e mcp`: `open_session {engine}` named a package that is not installed in the project, beside e2e, or in `~/.e2e/engines`. The message names the install command | configuration | 2 | +| `CONFIG_IN_USE` | `e2e mcp`: `open_session` on a config other than the one the open sessions use, or on a config while sessions opened with `engine` are open. Credentials and secrets resolve process-wide, so open on that config or close those sessions first | configuration | 2 | | `SESSION_OPEN` | `e2e mcp`: `open_session` while every session slot (`--max-sessions`, default 4) holds an open, opening, or closing session, or while the server shuts down. Close one first | configuration | 2 | | `UNKNOWN_TOPIC` | `e2e mcp`: an `e2e://guide/{topic}` resource named a topic the skill does not have. The message lists them | configuration | 2 | | `UNKNOWN_TOOL` | `e2e mcp`: `call` or `tools` named a tool the session's catalog does not have. The message lists the catalog. An action verb the engine does not declare is `UNSUPPORTED_CAPABILITY` instead | configuration | 2 | diff --git a/docs/reference/mcp.mdx b/docs/reference/mcp.mdx index 4addb3c298..587f22ee63 100644 --- a/docs/reference/mcp.mdx +++ b/docs/reference/mcp.mdx @@ -119,6 +119,8 @@ open_session {target: "web", headed: true} | `config` | string (path) | no | The config for this session, absolute or relative to the server's working directory (`cwd` in the client's MCP config). Otherwise `--config`, else discovered like `e2e run`. | | `headed` | boolean | no | `true` shows this session's browser or simulator, `false` hides it. Without it the session follows `--headed`. | | `test` | string | no | One test to run first, named like an `e2e run` positional. The session stops where the test stopped. See [Running a test](#running-a-test). | +| `engine` | string or object | no | The session's engine in a directory with no config file, as a package name or `{package, export, options}`. See [Without a config](#without-a-config). | +| `app` | object | no | The app under test for an `engine` session, as `url`, `bundleId`, or `appPath`. | #### `tools` @@ -534,6 +536,70 @@ Start the server in the project directory. Config discovery works like `e2e run`. For another config, see [`--config`](#config) and [`open_session`](#open_session). +## Without a config + +In a directory with no config file, `open_session` needs an `engine`. The +session has one target on that engine. Every other setting is at its +default, so the session has no credentials, secrets, or project tools. + +```txt +open_session {engine: "@e2e-dev/web", app: {url: "http://localhost:3000"}} +open_session {engine: {package: "@e2e-dev/mobile", options: {platform: "ios"}}, app: {bundleId: "com.example.app"}} +``` + +A package name is enough for an engine that needs no options. To pass +options, or to name an export, write `engine` as an object: + +| Field | Type | Does | +| --- | --- | --- | +| `package` | string | The package to load. It can carry a subpath, such as `@e2e-dev/testmu/web`. | +| `export` | string | The export to call. Default: the default export. | +| `options` | object | The options the factory takes, as in an `e2e.config.ts`. | + +- **A package name alone** calls the package's default export with no + options. `@e2e-dev/web` and `@e2e-dev/mobile` export their factory this + way. [Your own engine](/writing-an-engine#open-it-by-name) can do the same. +- **An option can be another factory call.** The server reads an option + value as a call when it has `package` and no key other than `export` and + `options`. It makes that call first and passes the result as the option. + Use it to give an engine a device or browser provider: + + ```txt + open_session { + engine: {package: "@e2e-dev/mobile", options: { + platform: "ios", + device: {package: "@e2e-dev/limrun", export: "limrun", options: {region: "eu"}} + }}, + app: {bundleId: "com.example.app"} + } + ``` + + The call above builds + `mobile({ platform: 'ios', device: limrun({ region: 'eu' }) })`. The + provider reads its API key from the server's environment. +- **Lookup.** The server looks for each package in the project, then beside + the running `e2e`, then in `~/.e2e/engines`. When it is in none of them, + the error names the command that installs it for every project: + `npm install --prefix ~/.e2e/engines @e2e-dev/web`. +- **`app`** takes `url`, `bundleId`, and `appPath`, as a target's + [`app`](/reference/config#the-app-under-test) does. `@e2e-dev/web` needs + `url`. `@e2e-dev/mobile` needs `bundleId` or `appPath`. +- **Files.** The project root is the server's working directory, and the + session writes `.e2e/` there. +- **Tests.** `run_tests` and `open_session {test}` need a config. To keep + the session's test lines as a test, write a config with + [`e2e init`](/reference/cli#e2e-init). +- **Several sessions.** Sessions opened with `engine` run side by side. A + session on a config doesn't open beside them and fails with + `CONFIG_IN_USE`. + +| Throws | When | +| --- | --- | +| `CONFIG_NOT_FOUND` | `open_session` without `engine`, and the directory has no config. | +| `ENGINE_NOT_INSTALLED` | A package that `engine` names is in none of the three lookup places. | +| `INVALID_ARGUMENT` | `package` is not a package name, `export` is not an export name, `engine` comes with `target`, `config`, or `test`, or the directory has a config. Also `app` without `engine`. | +| `INVALID_CONFIG` | The named export is not a function, a factory refused its `options`, the engine's factory returned no engine, or the engine refused `app`. For an export that is not a function, the message lists the package's functions. | + ## Output - Operator output goes to stderr, prefixed `e2e mcp:`. Stdout carries only @@ -661,7 +727,7 @@ into a scripted test drives its own session. | `SESSION_REQUIRED` | A call without `session` while several are open. The message lists the open sessions. | | `NO_SESSION` | A `session` id that is not open. For a session that is closing or ended, the message says why. | | `SESSION_OPEN` | An `open_session` past the limit, or while the server shuts down. | -| `CONFIG_IN_USE` | An `open_session` on another config while sessions are open. | +| `CONFIG_IN_USE` | An `open_session` on another config while sessions are open, or on a config beside sessions opened with `engine`. | | `ENGINE_IN_USE` | A second session on an engine created inside a package. | | `CANCELLED` | An `open_session` the client cancelled or disconnected from, or the server shut down during, once what it started has stopped. | diff --git a/docs/reference/security.mdx b/docs/reference/security.mdx index 54cbfa38aa..46213a1ae8 100644 --- a/docs/reference/security.mdx +++ b/docs/reference/security.mdx @@ -350,5 +350,5 @@ plugins, and app can make more connections. | Command | Rule | | --- | --- | -| `e2e mcp` | Uses stdio and the same action authorization as a test run. Four open sessions by default ([`--max-sessions`](/reference/mcp), 1 through 16). Closes idle sessions. | +| `e2e mcp` | Uses stdio and the same action authorization as a test run. Four open sessions by default ([`--max-sessions`](/reference/mcp), 1 through 16). Closes idle sessions. `open_session {engine}` loads the installed packages the agent names and calls the exports it names. That code runs with the server's full authority, like a config's imports. `engine` names packages only, never a path or a URL, and the server installs nothing. | | `e2e explore` | Offers every configured credential to every step. The explorer sees account names and usernames. Passwords stay handles and need an engine that supports sensitive fills; without one, the explorer skips them. With no credentials, it may sign in with made-up test credentials, typed as plain text. | diff --git a/docs/writing-an-engine.mdx b/docs/writing-an-engine.mdx index a588e9e70f..93a359d327 100644 --- a/docs/writing-an-engine.mdx +++ b/docs/writing-an-engine.mdx @@ -284,6 +284,26 @@ For platform operations the action set lacks, ship [project tools](/tools) built with `defineTool`, with `platforms` set so each shows only on compatible targets, as `@e2e-dev/mobile/tools` does. +## Open it by name + +Export the engine's factory as the package's default export. Then a coding +agent can open an [MCP session](/reference/mcp#without-a-config) on the +engine in a directory with no config: + +```ts title="src/index.ts" +export default function kiosk(options?: KioskOptions) { + return defineEngine({ /* ... */ }); +} +``` + +```txt +open_session {engine: {package: "kiosk-engine", options: {lane: 2}}} +``` + +The factory receives `options`, or `undefined` when the agent passes only +the package name. Return a new engine on every call, so each session gets +its own. + ## Verify it Run a test file against your engine. Check the lifecycle order, diff --git a/packages/e2e/src/config/load.ts b/packages/e2e/src/config/load.ts index 7f8f07a74d..55bf291caa 100644 --- a/packages/e2e/src/config/load.ts +++ b/packages/e2e/src/config/load.ts @@ -78,16 +78,17 @@ function explicitConfigNotFound(cwd: string, explicitPath: string, absolute: str * The failure for a run that found no config file: where the search looked, * a lookalike in the working directory when one exists, and otherwise the * command that creates a config. Thrown when a config file is discovered, - * never for a supplied config value. + * never for a supplied config value. `alternative` is what the caller can do + * instead of creating one: the CLI's flag by default. */ -export function missingConfigError(cwd: string): ConfigurationError { +export function missingConfigError(cwd: string, alternative = 'pass --config '): ConfigurationError { const root = path.resolve(cwd); const lookalike = CONFIG_LOOKALIKES.find((name) => existsSync(path.join(root, name))); const names = `${CONFIG_NAMES.slice(0, -1).join(', ')}, or ${CONFIG_NAMES.at(-1)}`; const searched = `no ${names} found in ${root} or its parent directories`; const remedy = lookalike === undefined - ? 'run e2e init to create one, or pass --config ' + ? `run e2e init to create one, or ${alternative}` : `found ${lookalike}, but only ${names} is loaded: rename it and keep it an ES module`; return new ConfigurationError('CONFIG_NOT_FOUND', `${searched}; ${remedy}`); } diff --git a/packages/e2e/src/mcp/config.ts b/packages/e2e/src/mcp/config.ts index c733704596..a4089d4d91 100644 --- a/packages/e2e/src/mcp/config.ts +++ b/packages/e2e/src/mcp/config.ts @@ -3,6 +3,9 @@ import path from 'node:path'; import { discoverConfig, loadConfigModule, missingConfigError } from '../config/load.ts'; import { resolveConfig, type ResolvedConfig } from '../config/resolve.ts'; +import { ConfigurationError } from '../internal/errors.ts'; +import type { TargetApp } from '../types.ts'; +import { loadEngine, type EngineLookup, type FactoryRef } from './default-engine.ts'; export interface ProjectOptions { readonly cwd: string; @@ -10,16 +13,51 @@ export interface ProjectOptions { readonly configPath?: string | undefined; } -/** A config loaded from a file, which a session always has: the file is what sessions share. */ +/** A config loaded from a file: what sessions on a project share. A session opened with `engine` has none. */ export type LoadedConfig = ResolvedConfig & { readonly configPath: string }; /** The absolute path of a project's config, found without evaluating it. */ export function locateProjectConfig(options: ProjectOptions): string { const { configPath } = discoverConfig(options.cwd, options.configPath); - if (configPath === undefined) throw missingConfigError(options.cwd); + if (configPath === undefined) { + throw missingConfigError( + options.cwd, + 'pass engine to open_session to open a session without one: {engine: "@e2e-dev/web", app: {url: ""}}, or {engine: {package: "@e2e-dev/mobile", options: {platform: "ios"}}, app: {bundleId: ""}}', + ); + } return configPath; } +/** What `open_session` names to open a session where the project has no config file: the engine's factory call, and the app. */ +export interface EngineSession extends FactoryRef { + /** The target's app: the URL it opens, or the installed app or build a device launches. */ + readonly app?: Pick | undefined; +} + +/** + * The config of a session opened with `engine`: one target on that engine + * package, with the app the call names and every other setting at its + * default. The project root is `cwd`. Refused where a config file exists, + * since that file is what a session on the project loads. + */ +export async function loadEngineConfig( + options: ProjectOptions & { readonly env: NodeJS.ProcessEnv } & Pick, + session: EngineSession, +): Promise { + const { configPath } = discoverConfig(options.cwd, options.configPath); + if (configPath !== undefined) { + throw new ConfigurationError( + 'INVALID_ARGUMENT', + `open_session: engine opens a session where there is no config file, and ${configPath} exists; leave engine out to open one of its targets`, + ); + } + const engine = await loadEngine(session, { cwd: options.cwd, enginesDir: options.enginesDir }); + return resolveConfig( + { targets: [{ engine, ...(session.app === undefined ? {} : { app: session.app }) }] }, + { projectRoot: path.resolve(options.cwd), env: options.env }, + ); +} + /** * Loads the config at `configPath` (absolute, from `locateProjectConfig`) * fresh, down to the files it imports by path: an agent edits the config, a diff --git a/packages/e2e/src/mcp/default-engine.ts b/packages/e2e/src/mcp/default-engine.ts new file mode 100644 index 0000000000..52fcaf2c08 --- /dev/null +++ b/packages/e2e/src/mcp/default-engine.ts @@ -0,0 +1,204 @@ +/** + * The engine of a session opened without a config file. `open_session` + * names it as a config would build it, a factory call written as data: + * `{package: "@e2e-dev/mobile", options: {platform: "ios", device: {package: + * "@e2e-dev/limrun"}}}` is `mobile({ platform: 'ios', device: limrun() })`. + * Core still knows no engine: every name is the caller's, and only a package + * already on disk is loaded. + */ + +import { existsSync, readFileSync } from 'node:fs'; +import { createRequire, isBuiltin } from 'node:module'; +import os from 'node:os'; +import path from 'node:path'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import { ENGINE_SPI_VERSION } from '../engine/contract.ts'; +import { isEngineHandle, type EngineHandle } from '../engine/index.ts'; +import { ConfigurationError, errorMessage, isE2EError } from '../internal/errors.ts'; + +/** npm's package name grammar with an optional subpath, so a path, a URL, or a `..` segment is never resolved. */ +const PACKAGE = /^(?:@[a-z0-9][a-z0-9._~-]*\/)?[a-z0-9][a-z0-9._~-]*(?:\/[A-Za-z0-9][A-Za-z0-9._~-]*)*$/; +const EXPORT_NAME = /^[A-Za-z_$][\w$]*$/; +const REF_KEYS: ReadonlySet = new Set(['package', 'export', 'options']); + +/** + * One factory call: `package`'s default export, or the export `export` + * names, called with `options`. An option value of this same shape is called + * first and replaced by what it returns, which is how an engine gets a + * provider (`device`, `browser`) that JSON cannot carry. + */ +export interface FactoryRef { + /** The package, with a subpath when the factory lives on one: `@e2e-dev/web`, `@e2e-dev/testmu/web`. */ + readonly package: string; + /** The export to call; the default export when omitted. */ + readonly export?: string | undefined; + /** What the factory is called with; a factory that needs none takes `undefined`. */ + readonly options?: Readonly> | undefined; +} + +export interface EngineLookup { + /** The project directory; its `node_modules` and its parents' are searched first. */ + readonly cwd: string; + /** The directory engines are installed into for every project (`userEnginesDir()`), searched last. */ + readonly enginesDir: string; +} + +/** `~/.e2e/engines`: where an engine installed once serves every project that has no config. */ +export function userEnginesDir(): string { + return path.join(os.homedir(), '.e2e', 'engines'); +} + +/** + * Builds the engine `ref` names. Each package is looked up in the project, + * then beside this copy of e2e (an `npx -p` tree, a global install), then in + * `lookup.enginesDir`. An engine that brings its own copy of e2e is fine: + * the handle's brand and contract version are all that cross. + */ +export async function loadEngine(ref: FactoryRef, lookup: EngineLookup): Promise { + const engine = await callFactory(ref, lookup, 'engine'); + if (!isEngineHandle(engine)) { + throw new ConfigurationError('INVALID_CONFIG', `${describe(ref)} did not return an engine; it must return what defineEngine or connectEngine returns`); + } + if (engine.spiVersion !== ENGINE_SPI_VERSION) { + throw new ConfigurationError( + 'INVALID_CONFIG', + `${describe(ref)} speaks engine contract ${String(engine.spiVersion)}; this e2e supports ${String(ENGINE_SPI_VERSION)}. Install the version of ${ref.package} that matches this e2e`, + ); + } + return engine; +} + +/** + * Calls the factory `ref` names with its options, the factory calls among + * them made first. `role` is where the call stands: the `engine` itself, or + * an `option` value read as a call by its shape. + */ +async function callFactory(ref: FactoryRef, lookup: EngineLookup, role: 'engine' | 'option'): Promise { + // A builtin's name resolves to itself, which a file URL would then read as a path in the project. + if (!PACKAGE.test(ref.package) || isBuiltin(ref.package)) { + throw new ConfigurationError('INVALID_ARGUMENT', `open_session: engine names an installed package such as "@e2e-dev/web", got ${JSON.stringify(ref.package)}`); + } + const name = ref.export ?? 'default'; + if (!EXPORT_NAME.test(name)) { + throw new ConfigurationError('INVALID_ARGUMENT', `open_session: export names one export of ${ref.package}, got ${JSON.stringify(name)}`); + } + const entry = resolveEntry(ref.package, lookup, role); + let exports: Record; + try { + exports = (await import(pathToFileURL(entry).href)) as Record; + } catch (cause) { + throw new ConfigurationError('INVALID_CONFIG', `package ${ref.package} (${entry}) failed to load: ${errorMessage(cause)}`, { cause }); + } + const factory = exports[name]; + if (typeof factory !== 'function') { + const callable = Object.keys(exports).filter((key) => key !== 'default' && typeof exports[key] === 'function'); + const others = callable.length === 0 ? 'it exports no function' : `its functions: ${callable.join(', ')}`; + throw new ConfigurationError( + 'INVALID_CONFIG', + `${describe(ref)} (${entry}) is not a function, so there is nothing to call; ${others}. Name one with export, or declare the target in an e2e.config.ts`, + ); + } + // The options object is the factory's argument, never a call itself, whatever its keys. + const options = ref.options === undefined ? undefined : await resolveEntries(ref.options, lookup); + try { + return await (factory as (options: unknown) => unknown)(options); + } catch (cause) { + // The factory's own refusal (`mobile()` without a platform) already says what to pass. + if (isE2EError(cause)) throw cause; + throw new ConfigurationError('INVALID_CONFIG', `${describe(ref)} refused its options: ${errorMessage(cause)}`, { cause }); + } +} + +/** `value` with every factory call in it made: objects and arrays are walked, anything else is passed as it is. */ +async function resolveOptions(value: unknown, lookup: EngineLookup): Promise { + if (Array.isArray(value)) { + const items: unknown[] = []; + for (const item of value) items.push(await resolveOptions(item, lookup)); + return items; + } + if (typeof value !== 'object' || value === null) return value; + return isFactoryRef(value) ? callFactory(value, lookup, 'option') : resolveEntries(value, lookup); +} + +/** A copy of `value` with each of its values resolved. */ +async function resolveEntries(value: object, lookup: EngineLookup): Promise> { + const entries: [string, unknown][] = []; + for (const [key, item] of Object.entries(value)) entries.push([key, await resolveOptions(item, lookup)]); + // Not assignment: a `__proto__` key from JSON stays an own property instead of becoming the object's prototype. + return Object.fromEntries(entries); +} + +/** An object that is a factory call and nothing else: a `package`, and no key a call does not have. */ +function isFactoryRef(value: object): value is FactoryRef { + const { package: name, export: exported, options } = value as Record; + return ( + typeof name === 'string' && + Object.keys(value).every((key) => REF_KEYS.has(key)) && + (exported === undefined || typeof exported === 'string') && + (options === undefined || (typeof options === 'object' && options !== null && !Array.isArray(options))) + ); +} + +/** The export a message names: `the default export of `, or ` export ""`. */ +function describe(ref: FactoryRef): string { + return ref.export === undefined ? `the default export of ${ref.package}` : `${ref.package} export ${JSON.stringify(ref.export)}`; +} + +/** + * The package's entry file, from the first place that has the package + * installed. A package that is there but has no loadable entry is an error + * of its own, never a reason to look further or to install it again. + */ +function resolveEntry(name: string, lookup: EngineLookup, role: 'engine' | 'option'): string { + const [first = '', second = ''] = name.split('/'); + const installable = name.startsWith('@') ? `${first}/${second}` : first; + for (const root of [lookup.cwd, path.dirname(fileURLToPath(import.meta.url)), lookup.enginesDir]) { + const directory = installedAt(installable, root); + if (directory === undefined) continue; + try { + // The file need not exist: it only says which node_modules directories to walk. + return createRequire(path.join(root, 'package.json')).resolve(name); + } catch (cause) { + // `require` conditions miss an entry a package exports to `import` alone. + const imported = (cause as NodeJS.ErrnoException).code === 'ERR_PACKAGE_PATH_NOT_EXPORTED' ? importEntry(directory, name.slice(installable.length)) : undefined; + if (imported !== undefined) return imported; + throw new ConfigurationError('INVALID_CONFIG', `package ${name} is installed at ${directory} but its entry cannot be resolved: ${errorMessage(cause)}`, { cause }); + } + } + // An option is only read as a call by its shape, so the message says how it was read before it names a command. + const read = role === 'engine' ? '' : `an option value with only package, export, and options keys is read as a factory call, and `; + throw new ConfigurationError( + 'ENGINE_NOT_INSTALLED', + `${read}package ${name} is not installed in ${lookup.cwd}, beside e2e, or in ${lookup.enginesDir}. Install it once for every project with: npm install --prefix ${JSON.stringify(lookup.enginesDir)} ${installable}; then call open_session again`, + ); +} + +/** The directory of `name` in the nearest `node_modules` from `root` upward, as Node.js looks a package up. */ +function installedAt(name: string, root: string): string | undefined { + for (let directory = root; ; directory = path.dirname(directory)) { + const candidate = path.join(directory, 'node_modules', name); + if (existsSync(path.join(candidate, 'package.json'))) return candidate; + if (path.dirname(directory) === directory) return undefined; + } +} + +/** The file `subpath` (`''` for the main entry) of the package at `directory` exports to `import`, when its `exports` names one. */ +function importEntry(directory: string, subpath: string): string | undefined { + const manifest: unknown = JSON.parse(readFileSync(path.join(directory, 'package.json'), 'utf8')); + const exports = typeof manifest === 'object' && manifest !== null && 'exports' in manifest ? manifest.exports : undefined; + const key = `.${subpath}`; + const keyed = typeof exports === 'object' && exports !== null && !Array.isArray(exports) && Object.keys(exports).some((entry) => entry.startsWith('.')); + const target = importTarget(keyed ? (exports as Record)[key] : key === '.' ? exports : undefined); + return target === undefined ? undefined : path.join(directory, target); +} + +/** The first target an `import` of this `exports` value selects: a path, or the first matching condition's, in the order written. */ +function importTarget(value: unknown): string | undefined { + if (typeof value === 'string') return value; + if (typeof value !== 'object' || value === null || Array.isArray(value)) return undefined; + for (const [condition, target] of Object.entries(value)) { + const selected = condition === 'import' || condition === 'node' || condition === 'default' ? importTarget(target) : undefined; + if (selected !== undefined) return selected; + } + return undefined; +} diff --git a/packages/e2e/src/mcp/server.ts b/packages/e2e/src/mcp/server.ts index 4aa5b78319..a6edf47a7d 100644 --- a/packages/e2e/src/mcp/server.ts +++ b/packages/e2e/src/mcp/server.ts @@ -14,7 +14,8 @@ import { StdioServerTransport } from '@modelcontextprotocol/server/stdio'; import { readGuide, skillTopics } from '../cli/skill.ts'; import { ConfigurationError, errorMessage } from '../internal/errors.ts'; import { processSecrets } from '../run/secrecy.ts'; -import { loadProjectConfig, locateProjectConfig } from './config.ts'; +import { loadEngineConfig, loadProjectConfig, locateProjectConfig } from './config.ts'; +import { userEnginesDir } from './default-engine.ts'; import type { McpOutput } from './output.ts'; import { TestRuns } from './run-tests.ts'; import { SessionHost } from './session.ts'; @@ -52,7 +53,7 @@ export interface ServeOptions { } const INSTRUCTIONS = `e2e is a local-first end-to-end test runner; this server drives an e2e project's app (its e2e.config.ts) live. -Call open_session (optionally with a target, a config path, and headed: true when the user wants to watch) to get a session, its tool catalog, and the first observation. Then call {tool, args} runs any catalog tool: observe, the grammar its engine honors (each action answers with the test line that repeats it), type_secret, locate, run_code (runs the test statements you are about to write), test_lines (those lines and the statements run_code passed, in order, as a draft test body), screenshot, start_recording and stop_recording when the engine records video (a video of the app for a pull request), and the project's own tools; tools lists them, tools {tool} shows one tool's arguments. close_session when done. +Call open_session (optionally with a target, a config path, and headed: true when the user wants to watch) to get a session, its tool catalog, and the first observation. A project with no e2e config file still opens one: pass engine, the engine package to drive ({engine: "@e2e-dev/web", app: {url: "http://localhost:3000"}}; {engine: {package: "@e2e-dev/mobile", options: {platform: "ios"}}, app: {bundleId: ""}}). Then call {tool, args} runs any catalog tool: observe, the grammar its engine honors (each action answers with the test line that repeats it), type_secret, locate, run_code (runs the test statements you are about to write), test_lines (those lines and the statements run_code passed, in order, as a draft test body), screenshot, start_recording and stop_recording when the engine records video (a video of the app for a pull request), and the project's own tools; tools lists them, tools {tool} shows one tool's arguments. close_session when done. Several sessions can be open at once, each with its own browser or device, so parallel agents (subagents) each open their own: pass the session id from open_session to every tools, call, and close_session. Write the test (tests/*.e2e.ts) while you walk the flow, adding each step to the file as you take it, and run it with run_tests {files: ["tests/x.e2e.ts"]} (the same run as npx e2e run ); run_tests without files runs the whole suite. To fix a failing test, open_session {test: "tests/x.e2e.ts:12"} runs it in a session and stops on the screen where it failed. Resources e2e://guide and e2e://guide/{topic} hold the writing guide.`; @@ -74,6 +75,7 @@ export async function serveMcp(options: ServeOptions): Promise { const host = new SessionHost({ locateConfig: (configPath) => locateProjectConfig({ cwd: options.cwd, configPath: configPath ?? options.configPath }), loadConfig: (configPath, graph) => loadProjectConfig(configPath, options.env, graph), + loadEngineConfig: (session) => loadEngineConfig({ cwd: options.cwd, configPath: options.configPath, env: options.env, enginesDir: userEnginesDir() }, session), withholdOutput: output === undefined ? undefined : (load) => output.withholdDuring(load), env: options.env, headed: options.headed, diff --git a/packages/e2e/src/mcp/session.ts b/packages/e2e/src/mcp/session.ts index e08f76ee09..a88a808101 100644 --- a/packages/e2e/src/mcp/session.ts +++ b/packages/e2e/src/mcp/session.ts @@ -2,7 +2,8 @@ * The live sessions behind `e2e mcp`: each one standalone attempt on one * target with one interactive agent step open on it, driven by a coding agent * through a fixed, four-tool surface. `open_session` loads the project's - * config and opens the attempt and the step; `tools` renders the session's + * config, or builds one around the engine package it names where the project + * has none, and opens the attempt and the step; `tools` renders the session's * catalog; `call` runs one catalog tool by name inside the step; * `close_session` ends the step, saves a recording still running, and tears * the attempt down. Several sessions may be open at once, up to the server's @@ -32,7 +33,8 @@ import type { AgentParams } from '../types.ts'; import { createSessionCatalog, isGrammarVerb, type SessionCatalog } from './catalog.ts'; import { appLines } from './app-lines.ts'; import { SESSION_BOUNDS } from './bounds.ts'; -import type { LoadedConfig } from './config.ts'; +import type { EngineSession, LoadedConfig } from './config.ts'; +import type { FactoryRef } from './default-engine.ts'; import { describeRecording, SessionRecorder } from './recording.ts'; import { SessionRegistry } from './sessions.ts'; import { describeTestOutcome } from './test-outcome.ts'; @@ -50,7 +52,8 @@ const SESSION_INSTRUCTION = 'Interactive session: a coding agent drives the app interface LiveSession { readonly id: string; readonly target: ResolvedTarget; - readonly configPath: string; + /** Where the session's config came from, as the opening summary says it. */ + readonly origin: string; readonly headed: boolean; readonly attempt: StandaloneAttempt; readonly step: InteractiveStep; @@ -74,8 +77,22 @@ export interface OpenSessionOptions { readonly headed?: boolean | undefined; /** One test to run in the session before it is handed over, named as `e2e run` takes a positional (`tests/a.e2e.ts:12`). */ readonly test?: string | undefined; + /** + * The engine to open on where the project has no config file: a package + * whose default export is its factory (`@e2e-dev/web`), or that factory + * call as data (`FactoryRef`). Excludes `target`, `config`, and `test`. + */ + readonly engine?: string | FactoryRef | undefined; + /** The app of an `engine` session. */ + readonly app?: EngineSession['app']; } +/** Where a session's config comes from: the project's file, or the engine package the call names. */ +type ConfigSource = { readonly file: string } | { readonly engine: EngineSession }; + +/** What sessions opened with `engine` claim as their config: they hold no secrets, so they share it whatever the engine. */ +const ENGINE_DEFAULTS = 'engine defaults, with no config file'; + export interface SessionHostOptions { /** The absolute path of the config a session loads, without evaluating it; `configPath` overrides the server's default. */ readonly locateConfig: (configPath: string | undefined) => string; @@ -85,6 +102,8 @@ export interface SessionHostOptions { * loaded in `graph`, where the session's test files load too. */ readonly loadConfig: (configPath: string, graph: string) => Promise; + /** Builds the config of a session opened with `engine`; throws where the project has a config file. */ + readonly loadEngineConfig: (session: EngineSession) => Promise; /** Holds what the process prints while `load` evaluates a config and registers its secrets; by default nothing is held. */ readonly withholdOutput?: ((load: () => Promise) => Promise) | undefined; readonly env: NodeJS.ProcessEnv; @@ -250,13 +269,13 @@ export class SessionHost { // The config is claimed before it evaluates: its top-level code resolves // secrets against the registry an open session installed, which only // knows that session's config. - const configPath = this.options.locateConfig(options.config); - this.sessions.claimConfig(id, configPath); + const source = this.configSource(options); + this.sessions.claimConfig(id, 'file' in source ? `config ${source.file}` : ENGINE_DEFAULTS); const withhold = this.options.withholdOutput ?? ((load) => load()); // The session's own module graph: a test it runs sees every project file as it is now. const graph = `mcp-${id}`; const loaded = await withhold(async () => { - const fresh = await this.options.loadConfig(configPath, graph); + const fresh = 'file' in source ? await this.options.loadConfig(source.file, graph) : await this.options.loadEngineConfig(source.engine); // Known to the process before anything can fail with one, so an open // failure and what the config printed while it loaded are redacted // like any other text; a session registers them only once its engine @@ -266,7 +285,9 @@ export class SessionHost { }); // A session is its own run: a URL declared with port 0 gets a port here, // for the opened target alone, so another target's URL cannot fail the open. - const { config, target } = await allocateTargetPorts(loaded, selectTarget(loaded, options.target ?? this.options.defaultTarget, 'pass target to open_session, or start e2e mcp --target ')); + // An engine session has one target, whatever `--target` names in a config. + const named = 'file' in source ? (options.target ?? this.options.defaultTarget) : undefined; + const { config, target } = await allocateTargetPorts(loaded, selectTarget(loaded, named, 'pass target to open_session, or start e2e mcp --target ')); usage.resolved(target.platform, target.engine); this.sessions.claimEngine(id, target.name, target.engine); // Selected before anything boots: a name that selects no test, or several, costs no engine. @@ -342,7 +363,10 @@ export class SessionHost { const live: LiveSession = { id, target, - configPath: loaded.configPath, + origin: + 'file' in source + ? `config ${source.file}, read with the e2e mcp server's environment, not your shell's.` + : `no config file: one target on ${source.engine.package}, every other setting at its default. run_tests and open_session {test} need an e2e.config.ts (npx e2e init writes one).`, headed, attempt, step, @@ -375,6 +399,24 @@ export class SessionHost { } } + /** + * The project's config file, or the engine the call names for a project + * without one. Naming an engine excludes what only a config file gives: a + * path to one, a target of it, a test it discovers. + */ + private configSource(options: OpenSessionOptions): ConfigSource { + const { engine, app } = options; + if (engine === undefined) { + if (app !== undefined) throw new ConfigurationError('INVALID_ARGUMENT', 'open_session: app describes the session engine opens; pass engine with it, or leave it out'); + return { file: this.options.locateConfig(options.config) }; + } + const needsConfig = Object.entries({ config: options.config, target: options.target, test: options.test }).find(([, value]) => value !== undefined)?.[0]; + if (needsConfig !== undefined) { + throw new ConfigurationError('INVALID_ARGUMENT', `open_session: ${needsConfig} names part of a config file, and engine opens a session without one; pass one or the other`); + } + return { engine: { ...(typeof engine === 'string' ? { package: engine } : engine), ...(app === undefined ? {} : { app }) } }; + } + /** Closes a live session that ended without a close_session: its step concluded, or it sat idle. */ private endOnItsOwn(live: LiveSession, why: string, endedBy: SessionEndedBy): void { if (!this.sessions.isLive(live.id)) return; @@ -456,7 +498,7 @@ export class SessionHost { const engine = live.target.engine; const lines = [ ...(outcome === undefined ? [] : [outcome, '']), - `Session ${live.id} open on target "${live.target.name}" (platform ${live.target.platform}, engine ${engine === undefined ? 'none' : `${engine.name} ${engine.version}`}), ${live.headed ? 'headed' : 'headless'}; config ${live.configPath}, read with the e2e mcp server's environment, not your shell's.`, + `Session ${live.id} open on target "${live.target.name}" (platform ${live.target.platform}, engine ${engine === undefined ? 'none' : `${engine.name} ${engine.version}`}), ${live.headed ? 'headed' : 'headless'}; ${live.origin}`, ]; lines.push(...appLines(live.target, outcome !== undefined)); if (config.credentials.size > 0) { @@ -518,7 +560,7 @@ export class SessionHost { return defineMcpTool({ name: 'open_session', description: - 'Open a live session on one target of an e2e project: loads the config, starts the app command the target declares (if any), boots the engine (a browser, a simulator), opens the app (its URL, or the pinned app on a device), and returns the session id, the catalog of tools it can run, and the first observation. With test, it runs that one test first and stops on the screen where the test passed or failed, with the failure and its line ahead of the catalog. Several sessions can be open at once, one per agent, each with its own engine: pass the returned session id to every later call. Then act with call and look with call {tool: "observe"}.', + 'Open a live session on one target of an e2e project: loads the config, starts the app command the target declares (if any), boots the engine (a browser, a simulator), opens the app (its URL, or the pinned app on a device), and returns the session id, the catalog of tools it can run, and the first observation. Where the project has no e2e config file, pass engine to open a session on that engine: {engine: "@e2e-dev/web", app: {url: "http://localhost:3000"}}. With test, it runs that one test first and stops on the screen where the test passed or failed, with the failure and its line ahead of the catalog. Several sessions can be open at once, one per agent, each with its own engine: pass the returned session id to every later call. Then act with call and look with call {tool: "observe"}.', inputSchema: z.object({ target: z.string().min(1).optional().describe('Target name from the config; required when the config declares several'), config: z.string().min(1).optional().describe("Path to an e2e config file, absolute or relative to the server's working directory; default: the nearest e2e.config.ts, .mts, .js, or .mjs"), @@ -531,6 +573,33 @@ export class SessionHost { .min(1) .optional() .describe('One test to run in the session first, named the way npx e2e run takes it: tests/checkout.e2e.ts:12, where 12 is the line its test( call is on (not a failing line inside it), or a file or name that selects one test. The session then stays on the screen where the test stopped; afterEach hooks and fixture teardown run at close_session. To rerun after editing the test, close_session and open_session with test again'), + engine: z + .union([ + z.string().min(1), + z + .object({ + package: z.string().min(1).describe('The package, with a subpath when the factory lives on one: @e2e-dev/mobile, @e2e-dev/testmu/web'), + export: z.string().min(1).optional().describe('The export to call; default: the default export'), + options: z + .record(z.string(), z.unknown()) + .optional() + .describe('What the factory is called with, as in an e2e.config.ts. A value of this same {package, export?, options?} shape is called first and replaced by its result'), + }) + .strict(), + ]) + .optional() + .describe( + 'Only where the project has no config file: the engine to open a session on, as a config would build it. A package name calls its default export with no options: "@e2e-dev/web". An object passes options, and an option can be another factory call, which is how an engine gets a device or browser provider: {package: "@e2e-dev/mobile", options: {platform: "ios"}}, or {package: "@e2e-dev/mobile", options: {platform: "ios", device: {package: "@e2e-dev/limrun", export: "limrun", options: {region: "eu"}}}} for mobile({platform: "ios", device: limrun({region: "eu"})}). A package not installed fails with the command that installs it. Excludes target, config, and test', + ), + app: z + .object({ + url: z.string().min(1).optional().describe('The URL the session opens, and the base of relative navigation'), + bundleId: z.string().min(1).optional().describe('An installed app a device launches: its bundle id, package, or display name'), + appPath: z.string().min(1).optional().describe("A build a device installs and launches: an .app, .ipa, or .apk, relative to the server's working directory"), + }) + .strict() + .optional() + .describe('With engine: the app under test, as a target declares it in an e2e.config.ts. @e2e-dev/web needs url; @e2e-dev/mobile needs bundleId or appPath'), }).strict(), readOnly: false, call: async (args, extra) => textResult(await this.open(args, extra.signal)), diff --git a/packages/e2e/src/mcp/sessions.ts b/packages/e2e/src/mcp/sessions.ts index e8863ef633..396ea0f795 100644 --- a/packages/e2e/src/mcp/sessions.ts +++ b/packages/e2e/src/mcp/sessions.ts @@ -30,12 +30,13 @@ type Phase = interface Entry { phase: Phase; /** - * The config the session loads. Credentials and secrets resolve through one - * process-wide registry, so every session shares one config; claimed - * before the file evaluates, since its top-level code resolves secrets - * against the registry an open session installed. + * The config the session loads, as a message names it: `config `, or + * the engine defaults of a session opened without one. Credentials and + * secrets resolve through one process-wide registry, so every session + * shares one config; claimed before the file evaluates, since its top-level + * code resolves secrets against the registry an open session installed. */ - configPath: string | undefined; + config: string | undefined; /** The target's engine instance, which runs one attempt at a time. */ engine: EngineHandle | undefined; /** Settles when the open does, whatever its outcome. */ @@ -70,7 +71,7 @@ export class SessionRegistry { if (this.draining) return Promise.reject(new ConfigurationError('SESSION_OPEN', 'the server is shutting down; no session can open')); if (this.entries.size >= this.limit) return Promise.reject(this.limitReached()); const id = uuidv7(); - const entry: Entry = { phase: { phase: 'opening' }, configPath: undefined, engine: undefined, opened: Promise.resolve() }; + const entry: Entry = { phase: { phase: 'opening' }, config: undefined, engine: undefined, opened: Promise.resolve() }; this.entries.set(id, entry); const opened = open(id).catch((cause: unknown) => { this.entries.delete(id); @@ -81,16 +82,16 @@ export class SessionRegistry { } /** Claims the config for an opening session before it loads, or refuses a config other than the one open sessions share. */ - claimConfig(id: string, configPath: string): void { - const holder = this.others(id).find(([, entry]) => entry.configPath !== undefined && entry.configPath !== configPath); + claimConfig(id: string, config: string): void { + const holder = this.others(id).find(([, entry]) => entry.config !== undefined && entry.config !== config); if (holder !== undefined) { const [holderId, entry] = holder; throw new ConfigurationError( 'CONFIG_IN_USE', - `session ${holderId} is open on config ${entry.configPath}; sessions open at once share one config, because credentials and secrets resolve process-wide; open this one on that config, or close every session on it first`, + `session ${holderId} is open on ${entry.config}; sessions open at once share one config, because credentials and secrets resolve process-wide; open this one the same way, or close every session on it first`, ); } - this.require(id).configPath = configPath; + this.require(id).config = config; } /** Claims the target's engine instance for an opening session, or refuses one another session is driving. */ diff --git a/packages/e2e/tests/integration/mcp-server.test.ts b/packages/e2e/tests/integration/mcp-server.test.ts index 0c5572774e..4cf9346868 100644 --- a/packages/e2e/tests/integration/mcp-server.test.ts +++ b/packages/e2e/tests/integration/mcp-server.test.ts @@ -6,7 +6,8 @@ * invariants, a second session open beside the first, a second session after * the first closed, an explicit config path, a config with two targets, a * custom engine, and project tools (each session's catalog is what its own - * target can do), and stdout hygiene: a config that prints to stdout must not + * target can do), a session on a named engine in a directory with no config, + * and stdout hygiene: a config that prints to stdout must not * corrupt the protocol. */ @@ -399,6 +400,11 @@ describe('e2e mcp', { timeout: 120_000 }, () => { expect(unknown.isError).toBe(true); expect(unknown.text).toContain('UNKNOWN_TARGET'); + const named = await invoke('open_session', { engine: '@e2e-dev/web', app: { url: app.url } }); + expect(named.isError).toBe(true); + expect(named.text).toContain('INVALID_ARGUMENT'); + expect(named.text).toContain(`${path.join(project.dir, 'e2e.config.ts')} exists; leave engine out`); + const absent = await invoke('open_session', { config: 'missing.config.ts' }); expect(absent.isError).toBe(true); expect(absent.text).toContain('CONFIG_NOT_FOUND'); @@ -541,6 +547,35 @@ describe('e2e mcp', { timeout: 120_000 }, () => { } }); + it('opens a session on a named engine in a directory with no config, and says how to get one otherwise', async () => { + const bare = createProject({ 'README.md': 'no config here\n' }); + const other = new Client({ name: 'e2e-mcp-bare-test', version: '0.0.0' }); + await other.connect(new StdioClientTransport({ command: process.execPath, args: [CLI, 'mcp'], cwd: bare.dir, env: { ...(process.env as Record), CI: '' }, stderr: 'ignore' })); + const open = async (args: Record): Promise<{ text: string; isError: boolean }> => { + const result = (await other.callTool({ name: 'open_session', arguments: args }, { timeout: 110_000 })) as { content: { text?: string }[]; isError?: boolean }; + return { text: result.content.map((part) => part.text ?? '').join('\n'), isError: result.isError === true }; + }; + try { + const empty = await open({}); + expect(empty.isError).toBe(true); + expect(empty.text).toContain('CONFIG_NOT_FOUND'); + expect(empty.text).toContain('pass engine to open_session to open a session without one: {engine: "@e2e-dev/web", app: {url: ""}}'); + + const absent = await open({ engine: '@e2e-dev/absent' }); + expect(absent.isError).toBe(true); + expect(absent.text).toContain('ENGINE_NOT_INSTALLED'); + expect(absent.text).toMatch(/npm install --prefix ".*engines" @e2e-dev\/absent/); + + const opened = await open({ engine: '@e2e-dev/web', app: { url: app.url } }); + expect(opened.isError, opened.text).toBe(false); + expect(opened.text).toMatch(/^Session \S+ open on target "web" \(platform web, engine web [^)]+\), headless; no config file: one target on @e2e-dev\/web, every other setting at its default\./); + expect(opened.text).toContain('button "Increment"'); + } finally { + await other.close(); + bare.cleanup(); + } + }); + it('kept stdout for the protocol: the config\'s console.log landed on stderr', () => { expect(stderr).toContain('config loaded'); expect(stderr).toContain('e2e mcp: [info] e2e mcp'); diff --git a/packages/e2e/tests/integration/mcp-session-host.test.ts b/packages/e2e/tests/integration/mcp-session-host.test.ts index b8d87220ab..0e3a420288 100644 --- a/packages/e2e/tests/integration/mcp-session-host.test.ts +++ b/packages/e2e/tests/integration/mcp-session-host.test.ts @@ -3,6 +3,7 @@ * engine, an idle session closes itself, the TTL ends a session through the * step's own deadline, sessions open side by side up to the limit, hold their * engine and config until their attempt is gone, and share one app process, + * a session opens on a named engine where the project has no config file, * a shutdown cancels sessions still opening and waits for them, a cancelled * open stops what it started, an `open_session` that fails * after the attempt opened tears the attempt down and leaves the host ready @@ -19,6 +20,7 @@ import { createFakeEngine, FAKE_APP, type FakeEngineBehavior, type FakeEngineHan import { freePort } from '../helpers/free-port.ts'; import { gate } from '../helpers/gate.ts'; import { startupLog, writeStartupScripts } from '../helpers/startup-scripts.ts'; +import type { EngineSession } from '../../src/mcp/config.ts'; import type { McpSessionSummary } from '../../src/mcp/usage.ts'; import type { RecordingMode, TargetApp } from '../../src/types.ts'; @@ -68,6 +70,8 @@ describe('SessionHost', { timeout: 60_000 }, () => { readonly tools?: Record>; /** Secrets the config declares beside the admin credential. */ readonly secrets?: Record; + /** Told what each `engine` session asked for. */ + readonly onEngineSession?: (session: EngineSession) => void; } /** @@ -93,6 +97,10 @@ describe('SessionHost', { timeout: 60_000 }, () => { ); return { ...config, configPath }; }, + loadEngineConfig: async (session) => { + options.onEngineSession?.(session); + return resolveConfig({ targets: [{ platform: 'kiosk', engine: engine().engine, ...(session.app === undefined ? {} : { app: session.app }) }] } as never, { projectRoot: dir, env: {} }); + }, env: {}, headed: options.headed ?? false, log: (level, message) => logs.push(`${level}: ${message}`), @@ -292,13 +300,39 @@ describe('SessionHost', { timeout: 60_000 }, () => { const first = sessionId(await mixed.open({})); await expect(mixed.open({ config: 'other.config.ts' })).rejects.toMatchObject({ code: 'CONFIG_IN_USE', - message: `session ${first} is open on config ${path.join(dir, 'e2e.config.ts')}; sessions open at once share one config, because credentials and secrets resolve process-wide; open this one on that config, or close every session on it first`, + message: `session ${first} is open on config ${path.join(dir, 'e2e.config.ts')}; sessions open at once share one config, because credentials and secrets resolve process-wide; open this one the same way, or close every session on it first`, }); await mixed.close('done'); expect(await mixed.open({ config: 'other.config.ts' })).toContain(`config ${path.join(dir, 'other.config.ts')}`); await mixed.close('done'); }); + it('opens sessions on a named engine without a config file, apart from sessions on one', async () => { + const asked: EngineSession[] = []; + const bare = host(engines().next, { maxSessions: 3, onEngineSession: (session) => asked.push(session) }); + const text = await bare.open({ engine: { package: '@acme/kiosk', options: { lane: 2 } }, app: FAKE_APP }); + expect(text).toContain('), headless; no config file: one target on @acme/kiosk, every other setting at its default.'); + expect(text).toContain(`App: ${FAKE_APP.url}/`); + expect(asked).toEqual([{ package: '@acme/kiosk', options: { lane: 2 }, app: FAKE_APP }]); + // Engine sessions hold no secrets, so a second one opens beside the first; a config's session does not. + const second = sessionId(await bare.open({ engine: '@acme/other' })); + await expect(bare.open({})).rejects.toMatchObject({ code: 'CONFIG_IN_USE', message: expect.stringContaining('is open on engine defaults, with no config file;') }); + expect(asked[1]).toEqual({ package: '@acme/other' }); + await bare.close('done', second); + await bare.close('done'); + }); + + it.each([ + [{ engine: '@acme/kiosk', target: 'kiosk' }, 'target names part of a config file'], + [{ engine: '@acme/kiosk', config: 'e2e.config.ts' }, 'config names part of a config file'], + [{ engine: '@acme/kiosk', test: 'tests/a.e2e.ts' }, 'test names part of a config file'], + [{ app: FAKE_APP }, 'app describes the session engine opens'], + ])('refuses %j before anything loads', async (args, message) => { + const fake = createFakeEngine(); + await expect(host(() => fake).open(args)).rejects.toMatchObject({ code: 'INVALID_ARGUMENT', message: expect.stringContaining(message) }); + expect(fake.events).toEqual([]); + }); + it('refuses a session on another config before evaluating it, so its secrets never resolve against the open session\'s', async () => { const evaluated: string[] = []; const mixed = host(engines().next, { diff --git a/packages/e2e/tests/integration/mcp-test-run.test.ts b/packages/e2e/tests/integration/mcp-test-run.test.ts index 375830e1bd..b6f3bccebd 100644 --- a/packages/e2e/tests/integration/mcp-test-run.test.ts +++ b/packages/e2e/tests/integration/mcp-test-run.test.ts @@ -390,6 +390,7 @@ describe('open_session {test}', { timeout: 120_000 }, () => { ...resolveConfig({ ...config, targets: [webTarget('web', app.url)] } as E2EConfig as never, { projectRoot: project.dir, env: {}, configPath }), configPath, }), + loadEngineConfig: () => Promise.reject(new Error('these sessions open on the config')), env: {}, headed: false, log: () => undefined, diff --git a/packages/e2e/tests/unit/mcp-default-engine.test.ts b/packages/e2e/tests/unit/mcp-default-engine.test.ts new file mode 100644 index 0000000000..3645f4570e --- /dev/null +++ b/packages/e2e/tests/unit/mcp-default-engine.test.ts @@ -0,0 +1,124 @@ +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { loadEngine } from '../../src/mcp/default-engine.ts'; + +/** A module whose default export returns what `defineEngine` brands, carrying the options it was called with. */ +const FACTORY = `export default (options) => ({ [Symbol.for('e2e.engine.v1')]: true, spiVersion: 1, name: 'fake', options });\n`; + +let root: string; +let lookup: { cwd: string; enginesDir: string }; + +/** Installs `name` under `dir/node_modules` with `source` as its entry, and `subpaths` as further entries, each exported under `condition`. */ +function install(dir: string, name: string, source: string, subpaths: Readonly> = {}, condition = 'default'): void { + const packageDir = path.join(dir, 'node_modules', name); + mkdirSync(packageDir, { recursive: true }); + const exports = Object.fromEntries([['.', { types: './index.d.ts', [condition]: './index.js' }], ...Object.keys(subpaths).map((subpath) => [`./${subpath}`, { [condition]: `./${subpath}.js` }])]); + writeFileSync(path.join(packageDir, 'package.json'), JSON.stringify({ name, type: 'module', exports })); + writeFileSync(path.join(packageDir, 'index.js'), source); + for (const [subpath, content] of Object.entries(subpaths)) writeFileSync(path.join(packageDir, `${subpath}.js`), content); +} + +beforeEach(() => { + root = mkdtempSync(path.join(os.tmpdir(), 'e2e-default-engine-')); + lookup = { cwd: path.join(root, 'project'), enginesDir: path.join(root, 'engines') }; + mkdirSync(lookup.cwd); +}); + +afterEach(() => { + rmSync(root, { recursive: true, force: true }); +}); + +describe('loadEngine', () => { + it("calls the default export of the project's package with the options", async () => { + install(lookup.cwd, '@acme/engine', FACTORY); + const engine = await loadEngine({ package: '@acme/engine', options: { platform: 'ios' } }, lookup); + expect(engine).toMatchObject({ name: 'fake', options: { platform: 'ios' } }); + }); + + it('calls a named export, and one on a subpath', async () => { + install(lookup.cwd, 'named', FACTORY.replace('export default', 'export const kiosk ='), { web: FACTORY.replace("'fake'", "'from-subpath'") }); + expect(await loadEngine({ package: 'named', export: 'kiosk' }, lookup)).toMatchObject({ name: 'fake' }); + expect(await loadEngine({ package: 'named/web' }, lookup)).toMatchObject({ name: 'from-subpath' }); + }); + + it('loads a package that exports its entries to import alone', async () => { + install(lookup.cwd, 'esm-only', FACTORY, { web: FACTORY.replace("'fake'", "'from-subpath'") }, 'import'); + expect(await loadEngine({ package: 'esm-only' }, lookup)).toMatchObject({ name: 'fake' }); + expect(await loadEngine({ package: 'esm-only/web' }, lookup)).toMatchObject({ name: 'from-subpath' }); + }); + + it('makes the factory calls among the options first, so an engine gets a provider JSON cannot carry', async () => { + install(lookup.cwd, 'engine', FACTORY); + install(lookup.enginesDir, 'cloud', `export const devices = (options) => ({ name: 'cloud', region: options?.region, acquire: () => 'leased' });\n`); + const engine = (await loadEngine( + { package: 'engine', options: { platform: 'ios', device: { package: 'cloud', export: 'devices', options: { region: 'eu' } }, pool: [{ package: 'cloud', export: 'devices' }], app: { package: 'left alone', bundleId: 'x' } } }, + lookup, + )) as unknown as { options: { device: { region: string; acquire: () => string }; pool: { name: string }[]; app: unknown } }; + expect(engine.options.device.region).toBe('eu'); + expect(engine.options.device.acquire()).toBe('leased'); + expect(engine.options.pool[0]?.name).toBe('cloud'); + // An object with any other key is an option value, not a factory call. + expect(engine.options.app).toEqual({ package: 'left alone', bundleId: 'x' }); + // Nor is the options object itself, whatever its keys. + expect(await loadEngine({ package: 'engine', options: { package: 'com.example.app' } }, lookup)).toMatchObject({ options: { package: 'com.example.app' } }); + }); + + it('keeps a __proto__ key from JSON an own property, so it cannot give the options inherited values', async () => { + install(lookup.cwd, 'engine', FACTORY); + const options = JSON.parse('{"platform":"ios","__proto__":{"device":"smuggled"}}') as Record; + const engine = (await loadEngine({ package: 'engine', options }, lookup)) as unknown as { options: Record }; + expect(Object.getPrototypeOf(engine.options)).toBe(Object.prototype); + expect(engine.options['device']).toBeUndefined(); + expect(Object.getOwnPropertyDescriptor(engine.options, '__proto__')?.value).toEqual({ device: 'smuggled' }); + }); + + it('falls back to the engines directory, and prefers the project', async () => { + install(lookup.enginesDir, 'shared-engine', FACTORY); + expect(await loadEngine({ package: 'shared-engine' }, lookup)).toMatchObject({ name: 'fake' }); + + install(lookup.enginesDir, 'both', FACTORY.replace("'fake'", "'from-engines-dir'")); + install(lookup.cwd, 'both', FACTORY.replace("'fake'", "'from-project'")); + expect(await loadEngine({ package: 'both' }, lookup)).toMatchObject({ name: 'from-project' }); + }); + + it('names the install command for a package that is nowhere, the engine or one of its options', async () => { + await expect(loadEngine({ package: '@acme/absent/web' }, lookup)).rejects.toMatchObject({ + code: 'ENGINE_NOT_INSTALLED', + message: expect.stringContaining(`npm install --prefix ${JSON.stringify(lookup.enginesDir)} @acme/absent;`), + }); + install(lookup.cwd, 'engine', FACTORY); + await expect(loadEngine({ package: 'engine', options: { device: { package: 'absent-cloud' } } }, lookup)).rejects.toMatchObject({ + code: 'ENGINE_NOT_INSTALLED', + message: expect.stringContaining('is read as a factory call, and package absent-cloud is not installed'), + }); + }); + + it('reports a package that is installed without a loadable entry, and looks no further for it', async () => { + install(lookup.cwd, 'unbuilt', FACTORY); + rmSync(path.join(lookup.cwd, 'node_modules', 'unbuilt', 'index.js')); + install(lookup.enginesDir, 'unbuilt', FACTORY); + await expect(loadEngine({ package: 'unbuilt' }, lookup)).rejects.toMatchObject({ + code: 'INVALID_CONFIG', + message: expect.stringContaining(`is installed at ${path.join(lookup.cwd, 'node_modules', 'unbuilt')} but its entry cannot be resolved`), + }); + }); + + it.each(['./engine.js', '/tmp/engine.js', 'file:///tmp/engine.js', 'engine/../../outside', '..', 'util', 'fs/promises', 'node:fs'])('refuses %s, which is not an installed package', async (name) => { + await expect(loadEngine({ package: name }, lookup)).rejects.toMatchObject({ code: 'INVALID_ARGUMENT' }); + }); + + it('refuses an export that is no factory, one that returns no engine, and an engine on another contract', async () => { + install(lookup.cwd, 'named-only', `export const engine = () => ({});\nexport const limit = 3;\n`); + install(lookup.cwd, 'not-an-engine', `export default () => ({ name: 'plain object' });\n`); + install(lookup.cwd, 'old-contract', FACTORY.replace('spiVersion: 1', 'spiVersion: 0')); + await expect(loadEngine({ package: 'named-only' }, lookup)).rejects.toMatchObject({ + code: 'INVALID_CONFIG', + message: expect.stringContaining('is not a function, so there is nothing to call; its functions: engine. Name one with export'), + }); + await expect(loadEngine({ package: 'named-only', export: 'constructor.prototype' }, lookup)).rejects.toMatchObject({ code: 'INVALID_ARGUMENT' }); + await expect(loadEngine({ package: 'not-an-engine' }, lookup)).rejects.toMatchObject({ code: 'INVALID_CONFIG', message: expect.stringContaining('did not return an engine') }); + await expect(loadEngine({ package: 'old-contract' }, lookup)).rejects.toMatchObject({ code: 'INVALID_CONFIG', message: expect.stringContaining('engine contract 0') }); + }); +}); diff --git a/packages/mobile/src/engine.ts b/packages/mobile/src/engine.ts index 3de4d9b60b..0cfbbc8fa5 100644 --- a/packages/mobile/src/engine.ts +++ b/packages/mobile/src/engine.ts @@ -69,6 +69,11 @@ export function buildEngine(surface: AgentDeviceSurface): EngineHandle { /** Creates one agent-device engine: one device session per worker; a test launches the pinned app with `app.open()`. */ export function mobile(options: MobileOptions): EngineHandle { + // The type requires it; `e2e mcp`'s `open_session {engine}` calls this with the JSON an agent wrote. + const platform: unknown = (options as Partial | undefined)?.platform; + if (platform !== 'ios' && platform !== 'android') { + throw new ConfigurationError('INVALID_CONFIG', `mobile() needs platform: 'ios' or 'android', got ${JSON.stringify(platform) ?? 'no platform'}`); + } const factory: ClientFactory = (session, connection) => { // A provider's local daemon must not inherit the user's remote-daemon connection. const local = connection?.client?.stateDir !== undefined && connection.daemon === undefined; diff --git a/packages/mobile/src/index.ts b/packages/mobile/src/index.ts index 9587f4378f..5cbb23cf4f 100644 --- a/packages/mobile/src/index.ts +++ b/packages/mobile/src/index.ts @@ -10,6 +10,8 @@ import { test as base } from 'e2e'; import type { Device } from './device.ts'; export { mobile } from './engine.ts'; +/** The engine factory again, as the default export: what `e2e mcp` calls when `open_session` names this package as its `engine`. */ +export { mobile as default } from './engine.ts'; export type { DevicePermission, LaunchPermissions, MobileOptions, MobilePlatform, PermissionState } from './options.ts'; export type { DeviceClientConfig, DeviceConnection, DeviceDaemon } from './bindings.ts'; export type { DeviceLease, DeviceProvider, DeviceReleaseContext, DeviceRequest } from './provider.ts'; diff --git a/packages/mobile/tests/unit/engine.test.ts b/packages/mobile/tests/unit/engine.test.ts index 2f951cc018..b2ef3c4483 100644 --- a/packages/mobile/tests/unit/engine.test.ts +++ b/packages/mobile/tests/unit/engine.test.ts @@ -99,10 +99,19 @@ describe('manifest', () => { expect(pinned.state).toBeUndefined(); }); + it('default-exports the factory, which is what an MCP session that names the package calls', async () => { + const entry = await import('../../src/index.ts'); + expect(entry.default).toBe(entry.mobile); + }); + it('rejects an option it does not know, naming the nearest', () => { expect(() => harness({ sesion: 'e2e-ios' } as never)).toThrow( expect.objectContaining({ code: 'INVALID_CONFIG', message: 'mobile() has unknown key "sesion"; did you mean "session"?' }), ); + // What `e2e mcp` passes when an agent names the package alone, or leaves the platform out. + for (const options of [undefined, {}, { platform: 'tvos' }]) { + expect(() => mobile(options as never)).toThrowError(expect.objectContaining({ code: 'INVALID_CONFIG', message: expect.stringContaining("mobile() needs platform: 'ios' or 'android'") })); + } expect(() => mobile({ platform: 'ios', app: 'x' } as never)).toThrowError( expect.objectContaining({ code: 'INVALID_CONFIG', message: expect.stringMatching(/^mobile\(\) has unknown key "app"/) }), ); diff --git a/packages/web/src/index.ts b/packages/web/src/index.ts index 2a44d1b827..8c45575038 100644 --- a/packages/web/src/index.ts +++ b/packages/web/src/index.ts @@ -9,6 +9,8 @@ import { test as base, type App } from 'e2e'; import type { Browser } from './browser.ts'; export { web, surfaceOf } from './engine.ts'; +/** The engine factory again, as the default export: what `e2e mcp` calls when `open_session` names this package as its `engine`. */ +export { web as default } from './engine.ts'; export type { PlaywrightLiveSurface } from './engine.ts'; export type { WebBasicAuth, WebConnectOptions, WebOptions, WebScreencastOptions } from './surface.ts'; export type { WebInitScript } from './init-scripts.ts'; diff --git a/plugins/e2e/README.md b/plugins/e2e/README.md index 3028b70f42..44a3630744 100644 --- a/plugins/e2e/README.md +++ b/plugins/e2e/README.md @@ -81,7 +81,9 @@ The plugin does not bundle the e2e MCP server, which runs from the project's own `e2e` install. `e2e init` offers to register it for Codex. In an existing project, run `e2e init --yes --skills none --mcp .codex/config.toml` with the installed CLI or add the entry below. Install dependencies, trust the project, and start a -new Codex session from the project directory: +new Codex session from the project directory. In a directory with no e2e +config, the agent can still open a session by naming the engine package: +`open_session {engine: "@e2e-dev/web", app: {url: "http://localhost:3000"}}`. ```toml [mcp_servers.e2e] diff --git a/plugins/e2e/skills/e2e/SKILL.md b/plugins/e2e/skills/e2e/SKILL.md index 6218d343f7..89f8243096 100644 --- a/plugins/e2e/skills/e2e/SKILL.md +++ b/plugins/e2e/skills/e2e/SKILL.md @@ -208,7 +208,9 @@ lines you changed: - In a project without e2e, set it up when the user requests it. Otherwise ask before adding a test framework, and name any existing framework. - Live MCP tools need the project's own `e2e mcp` server registered in the - client (topic `mcp`). Tests run with the CLI, or with the server's + client (topic `mcp`). In a directory with no e2e config, a live look still + works: `open_session {engine: "@e2e-dev/web", app: {url: "http://localhost:3000"}}` opens a session + on the engine's defaults. Tests need a config. Tests run with the CLI, or with the server's `run_tests`, the same run without a shell. Keep a session open across runs while you write or fix a web test; topic `mcp` says when a run can go beside it, and what to do on diff --git a/plugins/e2e/skills/e2e/references/debugging.md b/plugins/e2e/skills/e2e/references/debugging.md index b2d8dad830..30dad1f183 100644 --- a/plugins/e2e/skills/e2e/references/debugging.md +++ b/plugins/e2e/skills/e2e/references/debugging.md @@ -99,7 +99,9 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] | --- | --- | --- | | `CONFIG_LOAD_FAILED` | The config throws while loading (a refused engine option is `INVALID_CONFIG` instead) or imports a missing package, subpath, or removed export such as `defineConfig` | Install the dependency, or fix the import or line quoted | | `INVALID_CONFIG`, `INVALID_GLOB` | Unknown or foreign key (`app`, `webServer`, `use`, `projects`, `baseURL`), `json` with `list`; a `tests` glob with braces, classes, an absolute path, or no wildcard | The message names the key or glob to write; the app is declared in the target's `app` | -| `CONFIG_NOT_FOUND`, `CONFIG_AMBIGUOUS` | Wrong `--config` path; both `.ts` and `.mts` present | Fix the path; keep one | +| `CONFIG_NOT_FOUND` | Wrong `--config` path, or an MCP `open_session` in a directory with no config | Fix the path; in a directory with no config, pass `engine` to `open_session` (topic `mcp`) | +| `CONFIG_AMBIGUOUS` | Both `.ts` and `.mts` present | Keep one | +| `ENGINE_NOT_INSTALLED` | An MCP `open_session {engine}` named a package that is not in the project, beside e2e, or in `~/.e2e/engines` | Run the install command the message names | | `NO_TESTS` | The glob, a positional, or a filter matched nothing; the message names each empty positional and each undeclared `--tag` with the nearest declared one | Check the config `tests`, the file suffix (`.e2e.ts` by default), the tag names | | `NO_LAST_RUN` | `--last-failed` found no `.e2e/report.json` | Run once without it | | `INVALID_CHANGED_SINCE` | `--changed-since` names no commit in the checkout, shares no history with HEAD (a shallow clone, or unrelated histories), or git is missing | Run from a git checkout with git installed; check the ref; fetch it and more history (`git fetch --unshallow`, or `fetch-depth: 0` on `actions/checkout`); for unrelated histories, name a ref on HEAD's history | diff --git a/plugins/e2e/skills/e2e/references/mcp.md b/plugins/e2e/skills/e2e/references/mcp.md index 5db2ece3e9..ed84b86c67 100644 --- a/plugins/e2e/skills/e2e/references/mcp.md +++ b/plugins/e2e/skills/e2e/references/mcp.md @@ -48,9 +48,22 @@ Or declare it in the client's project config (`.mcp.json` for Claude Code, { "mcpServers": { "e2e": { "command": "npx", "args": ["e2e", "mcp"] } } } ``` -Sessions run on the project's config (`e2e.config.ts`, `.mts`, `.js`, or `.mjs`): -in a project without one, `open_session` fails with `CONFIG_NOT_FOUND`, so -set it up first (topic `setup`). No model is needed: without the optional `ai` package, sessions +Sessions run on the project's config (`e2e.config.ts`, `.mts`, `.js`, or `.mjs`). +In a directory without one, plain `open_session` fails with `CONFIG_NOT_FOUND`. +To look at an app there anyway, name the engine package: +`open_session {engine: "@e2e-dev/web", app: {url: "http://localhost:3000"}}`, or +`{engine: {package: "@e2e-dev/mobile", options: {platform: "ios"}}, app: {bundleId: ""}}`. +`engine` is the factory call a config would write: a package name calls its +default export, and `{package, export?, options?}` names another export and +passes options. An option value of that same shape is called first, which +gives an engine a device or browser provider: +`options: {platform: "ios", device: {package: "@e2e-dev/limrun", export: "limrun"}}` +is `mobile({ platform: 'ios', device: limrun() })`. That session has one +target, no credentials, secrets, or project tools, and `engine` excludes +`target`, `config`, and `test`. `ENGINE_NOT_INSTALLED` names the command that installs the package +into `~/.e2e/engines`: run it, then open again. `run_tests` and +`open_session {test}` still need a config, so set the project up before +writing tests (topic `setup`). No model is needed: without the optional `ai` package, sessions open and the deterministic tools work. A test a session runs needs one only for an agent step with no recording. @@ -69,7 +82,7 @@ tool with the same name, use the `e2e` server's. | Tool | Does | | --- | --- | -| `open_session` | Loads the config (`config` names another file; default the nearest `e2e.config.ts`), starts the declared app command if any, boots the engine, opens the app (its URL, or launches a device's pinned app; it installs no build, so a local device needs `app.bundleId`), and returns the session id, the catalog, and the first observation. Its summary names the app URL, a port assigned for `:0`, and the app command's readiness URL and `env` names; e2e starts nothing else, so a database the app uses runs outside it. The config reads the MCP server's environment, not your shell's. `target` is required when the config declares several. `headed: true` shows the browser or simulator when the user wants to watch; without it the session follows the server's `--headed`, headless by default. `test` runs one test first and leaves the session on the screen where it stopped (see Fixing a failing test). Each session has its own browser or device. | +| `open_session` | Loads the config (`config` names another file; default the nearest `e2e.config.ts`), starts the declared app command if any, boots the engine, opens the app (its URL, or launches a device's pinned app; it installs no build, so a local device needs `app.bundleId`), and returns the session id, the catalog, and the first observation. Its summary names the app URL, a port assigned for `:0`, and the app command's readiness URL and `env` names; e2e starts nothing else, so a database the app uses runs outside it. The config reads the MCP server's environment, not your shell's. `target` is required when the config declares several. `headed: true` shows the browser or simulator when the user wants to watch; without it the session follows the server's `--headed`, headless by default. `test` runs one test first and leaves the session on the screen where it stopped (see Fixing a failing test). In a directory with no config, `engine` names the engine to open on, as a package name or `{package, export?, options?}`, with `app` for the app under test (Setup); it excludes `target`, `config`, and `test`. Each session has its own browser or device. | | `tools` | The catalog: one line per tool with its argument names (`?` marks optional), the first sentence of its description, and `[read-only]` where it changes nothing. `tools {tool}` shows the full description and the JSON Schema of its arguments. | | `call` | Runs one catalog tool: `call {tool: "tap", args: {target: "n42"}}`. Arguments are checked against the tool's schema first; a wrong one fails with `INVALID_ARGUMENT` naming the field. | | `close_session` | Saves a recording still running, ends the attempt, disposes the engine, stops the app processes the session started once no other session uses them. | @@ -295,8 +308,10 @@ session runs the test and stops where it failed: ones. `NO_SESSION`: call `open_session` first, or the session named has ended (the message says why). `SESSION_REQUIRED`: several sessions are open; pass `session`. `SESSION_OPEN`: every session slot is taken (close one, or - raise `--max-sessions`). `CONFIG_IN_USE`: open sessions use another config; - open on theirs, or close them first. `ENGINE_IN_USE`: the target's engine + raise `--max-sessions`). `CONFIG_IN_USE`: open sessions use another config, + or were opened with `engine` and this one names a config; open the same + way, or close them first. `ENGINE_NOT_INSTALLED`: run the install command + the message names, then `open_session` again. `ENGINE_IN_USE`: the target's engine comes from a package every session shares; create it in the config or a file it imports by path. `UNKNOWN_TOOL`: the name is not in this session's catalog; the message lists what is. `TEST_AMBIGUOUS` and diff --git a/skills/e2e/SKILL.md b/skills/e2e/SKILL.md index 6218d343f7..89f8243096 100644 --- a/skills/e2e/SKILL.md +++ b/skills/e2e/SKILL.md @@ -208,7 +208,9 @@ lines you changed: - In a project without e2e, set it up when the user requests it. Otherwise ask before adding a test framework, and name any existing framework. - Live MCP tools need the project's own `e2e mcp` server registered in the - client (topic `mcp`). Tests run with the CLI, or with the server's + client (topic `mcp`). In a directory with no e2e config, a live look still + works: `open_session {engine: "@e2e-dev/web", app: {url: "http://localhost:3000"}}` opens a session + on the engine's defaults. Tests need a config. Tests run with the CLI, or with the server's `run_tests`, the same run without a shell. Keep a session open across runs while you write or fix a web test; topic `mcp` says when a run can go beside it, and what to do on diff --git a/skills/e2e/references/debugging.md b/skills/e2e/references/debugging.md index b2d8dad830..30dad1f183 100644 --- a/skills/e2e/references/debugging.md +++ b/skills/e2e/references/debugging.md @@ -99,7 +99,9 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] | --- | --- | --- | | `CONFIG_LOAD_FAILED` | The config throws while loading (a refused engine option is `INVALID_CONFIG` instead) or imports a missing package, subpath, or removed export such as `defineConfig` | Install the dependency, or fix the import or line quoted | | `INVALID_CONFIG`, `INVALID_GLOB` | Unknown or foreign key (`app`, `webServer`, `use`, `projects`, `baseURL`), `json` with `list`; a `tests` glob with braces, classes, an absolute path, or no wildcard | The message names the key or glob to write; the app is declared in the target's `app` | -| `CONFIG_NOT_FOUND`, `CONFIG_AMBIGUOUS` | Wrong `--config` path; both `.ts` and `.mts` present | Fix the path; keep one | +| `CONFIG_NOT_FOUND` | Wrong `--config` path, or an MCP `open_session` in a directory with no config | Fix the path; in a directory with no config, pass `engine` to `open_session` (topic `mcp`) | +| `CONFIG_AMBIGUOUS` | Both `.ts` and `.mts` present | Keep one | +| `ENGINE_NOT_INSTALLED` | An MCP `open_session {engine}` named a package that is not in the project, beside e2e, or in `~/.e2e/engines` | Run the install command the message names | | `NO_TESTS` | The glob, a positional, or a filter matched nothing; the message names each empty positional and each undeclared `--tag` with the nearest declared one | Check the config `tests`, the file suffix (`.e2e.ts` by default), the tag names | | `NO_LAST_RUN` | `--last-failed` found no `.e2e/report.json` | Run once without it | | `INVALID_CHANGED_SINCE` | `--changed-since` names no commit in the checkout, shares no history with HEAD (a shallow clone, or unrelated histories), or git is missing | Run from a git checkout with git installed; check the ref; fetch it and more history (`git fetch --unshallow`, or `fetch-depth: 0` on `actions/checkout`); for unrelated histories, name a ref on HEAD's history | diff --git a/skills/e2e/references/mcp.md b/skills/e2e/references/mcp.md index 5db2ece3e9..ed84b86c67 100644 --- a/skills/e2e/references/mcp.md +++ b/skills/e2e/references/mcp.md @@ -48,9 +48,22 @@ Or declare it in the client's project config (`.mcp.json` for Claude Code, { "mcpServers": { "e2e": { "command": "npx", "args": ["e2e", "mcp"] } } } ``` -Sessions run on the project's config (`e2e.config.ts`, `.mts`, `.js`, or `.mjs`): -in a project without one, `open_session` fails with `CONFIG_NOT_FOUND`, so -set it up first (topic `setup`). No model is needed: without the optional `ai` package, sessions +Sessions run on the project's config (`e2e.config.ts`, `.mts`, `.js`, or `.mjs`). +In a directory without one, plain `open_session` fails with `CONFIG_NOT_FOUND`. +To look at an app there anyway, name the engine package: +`open_session {engine: "@e2e-dev/web", app: {url: "http://localhost:3000"}}`, or +`{engine: {package: "@e2e-dev/mobile", options: {platform: "ios"}}, app: {bundleId: ""}}`. +`engine` is the factory call a config would write: a package name calls its +default export, and `{package, export?, options?}` names another export and +passes options. An option value of that same shape is called first, which +gives an engine a device or browser provider: +`options: {platform: "ios", device: {package: "@e2e-dev/limrun", export: "limrun"}}` +is `mobile({ platform: 'ios', device: limrun() })`. That session has one +target, no credentials, secrets, or project tools, and `engine` excludes +`target`, `config`, and `test`. `ENGINE_NOT_INSTALLED` names the command that installs the package +into `~/.e2e/engines`: run it, then open again. `run_tests` and +`open_session {test}` still need a config, so set the project up before +writing tests (topic `setup`). No model is needed: without the optional `ai` package, sessions open and the deterministic tools work. A test a session runs needs one only for an agent step with no recording. @@ -69,7 +82,7 @@ tool with the same name, use the `e2e` server's. | Tool | Does | | --- | --- | -| `open_session` | Loads the config (`config` names another file; default the nearest `e2e.config.ts`), starts the declared app command if any, boots the engine, opens the app (its URL, or launches a device's pinned app; it installs no build, so a local device needs `app.bundleId`), and returns the session id, the catalog, and the first observation. Its summary names the app URL, a port assigned for `:0`, and the app command's readiness URL and `env` names; e2e starts nothing else, so a database the app uses runs outside it. The config reads the MCP server's environment, not your shell's. `target` is required when the config declares several. `headed: true` shows the browser or simulator when the user wants to watch; without it the session follows the server's `--headed`, headless by default. `test` runs one test first and leaves the session on the screen where it stopped (see Fixing a failing test). Each session has its own browser or device. | +| `open_session` | Loads the config (`config` names another file; default the nearest `e2e.config.ts`), starts the declared app command if any, boots the engine, opens the app (its URL, or launches a device's pinned app; it installs no build, so a local device needs `app.bundleId`), and returns the session id, the catalog, and the first observation. Its summary names the app URL, a port assigned for `:0`, and the app command's readiness URL and `env` names; e2e starts nothing else, so a database the app uses runs outside it. The config reads the MCP server's environment, not your shell's. `target` is required when the config declares several. `headed: true` shows the browser or simulator when the user wants to watch; without it the session follows the server's `--headed`, headless by default. `test` runs one test first and leaves the session on the screen where it stopped (see Fixing a failing test). In a directory with no config, `engine` names the engine to open on, as a package name or `{package, export?, options?}`, with `app` for the app under test (Setup); it excludes `target`, `config`, and `test`. Each session has its own browser or device. | | `tools` | The catalog: one line per tool with its argument names (`?` marks optional), the first sentence of its description, and `[read-only]` where it changes nothing. `tools {tool}` shows the full description and the JSON Schema of its arguments. | | `call` | Runs one catalog tool: `call {tool: "tap", args: {target: "n42"}}`. Arguments are checked against the tool's schema first; a wrong one fails with `INVALID_ARGUMENT` naming the field. | | `close_session` | Saves a recording still running, ends the attempt, disposes the engine, stops the app processes the session started once no other session uses them. | @@ -295,8 +308,10 @@ session runs the test and stops where it failed: ones. `NO_SESSION`: call `open_session` first, or the session named has ended (the message says why). `SESSION_REQUIRED`: several sessions are open; pass `session`. `SESSION_OPEN`: every session slot is taken (close one, or - raise `--max-sessions`). `CONFIG_IN_USE`: open sessions use another config; - open on theirs, or close them first. `ENGINE_IN_USE`: the target's engine + raise `--max-sessions`). `CONFIG_IN_USE`: open sessions use another config, + or were opened with `engine` and this one names a config; open the same + way, or close them first. `ENGINE_NOT_INSTALLED`: run the install command + the message names, then `open_session` again. `ENGINE_IN_USE`: the target's engine comes from a package every session shares; create it in the config or a file it imports by path. `UNKNOWN_TOOL`: the name is not in this session's catalog; the message lists what is. `TEST_AMBIGUOUS` and