Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/coding-agents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ claude mcp add e2e -- e2e mcp

The server's instructions and every `open_session` result tell the agent to read the `writing-tests` guide (`e2e://guide/writing-tests`, or `e2e guide writing-tests` in a shell) before it writes or edits a test.

`open_session` names the app URL (and the port a free-port `url` was assigned) and the app command it started: its executable, readiness URL, and the names of its `env` and `inheritEnv`. It prints no `args` and no `env` values. The config reads the MCP server's environment, not the agent's shell. The `screenshot` tool takes an optional `path` to save a `.png` file inside the project. `type_secret {target, secret}` is listed when the config declares `credentials` or `secrets`: it fills a credential's password (the secret `<name>.password`) into a password field, or a `secrets` entry into any editable field, and the agent never sees the value. The opening text lists each credential's name, username, and password secret, and the test line names `Credentials.User(name).Password` or `Secrets.Get(name)`. What a test run through `open_session {test}` prints goes to the server's stderr with every secret value replaced by its `<secret:name>` marker, even when written in pieces; output printed while a config loads is held until its secrets are known, and withheld (with a warning line) if the load fails.
`open_session` names the app URL (and the port a free-port `url` was assigned) and the app command it started: its executable, readiness URL, and the names of its `env` and `inheritEnv`. It prints no `args` and no `env` values. The config reads the MCP server's environment, not the agent's shell. The `screenshot` tool takes an optional `path` to save a `.png` file inside the project. `type_secret {target, name}` is listed when the config declares `credentials` or `secrets`: it fills a credential's password (the secret `<name>.password`) into a password field, or a `secrets` entry into any editable field, and the agent never sees the value. The opening text lists each credential's name, username, and password secret, and the test line names `Credentials.User(name).Password` or `Secrets.Get(name)`. What a test run through `open_session {test}` prints goes to the server's stderr with every secret value replaced by its `<secret:name>` marker, even when written in pieces; output printed while a config loads is held until its secrets are known, and withheld (with a warning line) if the load fails.

Each node in `observe` starts its line with a node id (`#n42 button "Save"`), and an action takes that id as its `target`. `observe` shows the whole screen under a revision (`Current screen (revision b3, path /, 12 nodes):`). An action reports only what changed since the screen the agent last received: the lines that were `added`, `changed` (with what each read before), or `removed`. An id stays the same for as long as its element exists, so a node that is not listed as removed is still on screen under the id the agent has. When the screen changed mostly, the action sends it whole again.

Expand Down
3 changes: 2 additions & 1 deletion skills/e2e-dotnet/references/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -291,7 +291,8 @@ On an engine that declares `EngineCapabilities.TapModifiers` (the web engine),
extends a selection. The held keys are recorded and replayed from the cache.
A project tool adds its own name to this list (see "Project tools").
`fill_secret` fills any editable field, a rich-text editor included, with a
generic secret; a password secret needs a password field.
generic secret; a password secret needs a password field. It takes `target`
(the node id) and `name` (the secret's name), as upstream's `type_secret`.

## AssertAsync, WaitForAsync, ExtractAsync: one question

Expand Down
2 changes: 1 addition & 1 deletion skills/e2e-dotnet/references/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ config declares `credentials` or a `secrets` entry.
| `tap`, `double_tap`, `long_press`, `right_click`, `hover`, `check`, `uncheck`, `clear` `{target}` | Act on one node. `long_press` is for a control with a long-press menu, `right_click` opens a context menu, `hover` shows what the pointer resting on a node reveals. On an engine that declares `TapModifiers` (the web engine), `tap`, `double_tap`, and `right_click` also take an optional `modifiers` list (`Shift`, `Control`, `Alt`, `Meta`, `ControlOrMeta`) to hold keys for the click, as a Shift-click extends a selection |
| `drag {target, to}` | Drag one node and drop it on another, both by node id |
| `type {target, text}` | Replace the text in a field. With a keyboard (the web engine), `target` is optional: without one the text is typed into whatever has focus, and `replace` clears the field first |
| `type_secret {target, secret}` | Fill a configured secret by name: a credential's password (`<name>.password`) into a password field, a `secrets` entry into any editable field; you never see the value, and its test line names the credential or secret (`FillAsync(Credentials.User(name).Password)`, `FillAsync(Secrets.Get(name))`) |
| `type_secret {target, name}` | Fill a configured secret by name: a credential's password (`<name>.password`) into a password field, a `secrets` entry into any editable field; you never see the value, and its test line names the credential or secret (`FillAsync(Credentials.User(name).Password)`, `FillAsync(Secrets.Get(name))`) |
| `press {key, target?}` | Press a key on a node, or on whatever has focus |
| `select {target, value}` | Choose an option |
| `scroll {direction, times?, target?}`, `scroll_to {target?, text?, direction?}` | Move the viewport, or page to a node |
Expand Down
3 changes: 2 additions & 1 deletion src/E2E.Cli/Mcp/Session.cs
Original file line number Diff line number Diff line change
Expand Up @@ -434,7 +434,8 @@ [.. secrets.Select(secret => secret.Name)],
session.Engine.CanDismissKeyboard,
config.Agent.Tools,
target.Platform,
message => Log("warning", message));
message => Log("warning", message),
[.. secrets.Select(secret => new ExecutorSecret(secret.Name, secret.Purpose))]);

// The app as the session resolved it: the declared URL on the port its command was started on.
var app = TargetApps.ResolveTargetApp(target.Name, declared.App, config.ProjectRoot, Uri.TryCreate(target.App.Url, UriKind.Absolute, out var served) ? served.Port : null);
Expand Down
15 changes: 9 additions & 6 deletions src/E2E.Cli/Mcp/SessionCatalog.cs
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ private SessionCatalog(IReadOnlyList<KeyValuePair<string, CatalogTool>> entries,
/// (<see cref="IEngineSession.PointerActionKinds"/>), which the point tools (<c>tap_at</c> and the other <c>_at</c> tools) follow.
/// <paramref name="canDismissKeyboard"/> is <see cref="IEngineSession.CanDismissKeyboard"/>: <c>dismiss_keyboard</c> is a tool only where it is true.
/// </summary>
public static SessionCatalog Create(ISessionContext context, EngineCapabilities capabilities, IReadOnlyCollection<string> secrets, string projectRoot, string attemptDir, Uri? appBase = null, SessionRecorder? recorder = null, string? testRan = null, TestLineFills? fills = null, IReadOnlyCollection<string>? actionKinds = null, IReadOnlyCollection<string>? pointerKinds = null, bool canDismissKeyboard = false, IReadOnlyDictionary<string, AgentTool>? projectTools = null, string? platform = null, Action<string>? warn = null)
public static SessionCatalog Create(ISessionContext context, EngineCapabilities capabilities, IReadOnlyCollection<string> secrets, string projectRoot, string attemptDir, Uri? appBase = null, SessionRecorder? recorder = null, string? testRan = null, TestLineFills? fills = null, IReadOnlyCollection<string>? actionKinds = null, IReadOnlyCollection<string>? pointerKinds = null, bool canDismissKeyboard = false, IReadOnlyDictionary<string, AgentTool>? projectTools = null, string? platform = null, Action<string>? warn = null, IReadOnlyList<ExecutorSecret>? secretHandles = null)
{
ArgumentNullException.ThrowIfNull(context);
ArgumentNullException.ThrowIfNull(secrets);
Expand Down Expand Up @@ -229,7 +229,7 @@ public static SessionCatalog Create(ISessionContext context, EngineCapabilities

if (secrets.Count > 0 && verbs.Contains(ExecutorVerb.TypeSecret))
{
Add("type_secret", TypeSecret(context, screen));
Add("type_secret", TypeSecret(context, screen, secretHandles));
}

Add("locate", Locate(context));
Expand Down Expand Up @@ -488,15 +488,18 @@ private static CatalogTool Verb(ISessionContext context, ScreenPresenter screen,
Execute = async (args, token) => await tool.ExecuteAsync(args.ValueKind == JsonValueKind.Object ? JsonNode.Parse(args.GetRawText())!.AsObject() : throw new ConfigurationException("INVALID_ARGUMENT", "a project tool takes an object"), token).ConfigureAwait(false),
};

private static CatalogTool TypeSecret(ISessionContext context, ScreenPresenter screen)
private static CatalogTool TypeSecret(ISessionContext context, ScreenPresenter screen, IReadOnlyList<ExecutorSecret>? secretHandles)
{
return new CatalogTool
{
Description = "Fill a configured secret into an editable field by its name. You never see the value: every output shows the secret as <secret:name>.",
InputSchema = Schema(Target, ("secret", Property("string", "The secret's name from the config"), true)),
Description = "Fill a configured secret into an editable field by its name. You never see the value: every output shows the secret as <secret:name>." +
(secretHandles is { Count: > 0 }
? " Available: " + string.Join(", ", secretHandles.Select(secret => "\"" + secret.Name + "\"" + (secret.Purpose is null ? "" : " (" + secret.Purpose + ")"))) + "."
: ""),
InputSchema = Schema(Target, ("name", Property("string", "The secret's name from the config"), true)),
Execute = async (args, token) =>
{
var mapped = new JsonObject { ["target"] = args.GetProperty("target").GetString(), ["secret"] = args.GetProperty("secret").GetString() };
var mapped = new JsonObject { ["target"] = args.GetProperty("target").GetString(), ["name"] = args.GetProperty("name").GetString() };
// Marked before the fill runs: a fill that fails after typing still leaves the secret on screen.
context.SecretFilled();
var outcome = await RunAsync(context, screen, "type_secret", "fill_secret", args, token, mapped).ConfigureAwait(false);
Expand Down
2 changes: 1 addition & 1 deletion src/E2E.Cli/Mcp/TestLines.cs
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ private async Task<LinePlan> BuildAsync(string name, JsonElement args, Cancellat

case "type_secret":
{
return SecretExpression(Field(args, "secret") ?? "") is { } secret
return SecretExpression(Field(args, "name") ?? "") is { } secret
? await OnTargetAsync(target, locator => locator + ".FillAsync(" + secret + ")", token).ConfigureAwait(false)
: LinePlan.None("the secret is neither a credential password nor a config secret");
}
Expand Down
13 changes: 7 additions & 6 deletions src/E2E/Agent/ActTools.cs
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ private async Task<ToolOutcome> RunActionAsync(ModelToolCall call, IExecutorActi
"right_click" => actions.SecondaryTapAsync(target, ClickOptionsOf(arguments), token),
"hover" => actions.HoverAsync(target, token),
"fill" => actions.TypeAsync(target, Args.String(arguments, "value") ?? "", token),
"fill_secret" => actions.TypeSecretAsync(target, Args.String(arguments, "secret") ?? "", token),
"fill_secret" => actions.TypeSecretAsync(target, Args.String(arguments, "name") ?? "", token),
"select" => actions.SelectAsync(target, Args.String(arguments, "value") ?? "", token),
"check" => actions.CheckAsync(target, true, token),
"uncheck" => actions.CheckAsync(target, false, token),
Expand Down Expand Up @@ -485,11 +485,15 @@ private async Task<ToolOutcome> FailedAsync(ActionLabel label, E2EException erro

// The done and attempt leads of a node tool, upstream's wording; `reference` stands where upstream's `#id` does. fill and fill_secret
// keep the port's tool names in the attempt, where upstream's are type and type_secret. clear has no upstream tool.
private static ActionLabel SecretLabel(string secret, string reference) => new(
"Filled secret \"" + secret + "\" into " + reference + "; its value is masked in every observation.",
"fill_secret \"" + secret + "\" into " + reference,
false);

private static ActionLabel NodeLabel(string tool, JsonElement arguments, string reference)
{
var key = Args.String(arguments, "key") ?? "Enter";
var value = Args.String(arguments, "value") ?? "";
var secret = Args.String(arguments, "secret") ?? "";
return tool switch
{
"tap" => new("Tapped " + reference + ".", "tap " + reference),
Expand All @@ -498,10 +502,7 @@ private static ActionLabel NodeLabel(string tool, JsonElement arguments, string
"right_click" => new("Right-clicked " + reference + ".", "right_click " + reference),
"hover" => new("Hovered over " + reference + ".", "hover " + reference),
"fill" => new("Typed into " + reference + ".", "fill " + reference),
"fill_secret" => new(
"Filled secret \"" + secret + "\" into " + reference + "; its value is masked in every observation.",
"fill_secret \"" + secret + "\" into " + reference,
false),
"fill_secret" => SecretLabel(Args.String(arguments, "name") ?? "", reference),
"press" => new("Pressed " + key + " on " + reference + ".", "press " + key + " on " + reference, !MovesFocusOnly(key)),
"select" => new("Selected \"" + value + "\" in " + reference + ".", "select \"" + value + "\" in " + reference),
"check" => new("Checked " + reference + ".", "check " + reference),
Expand Down
6 changes: 3 additions & 3 deletions src/E2E/Agent/AgentTools.cs
Original file line number Diff line number Diff line change
Expand Up @@ -258,15 +258,15 @@ private static readonly (string Tool, ExecutorVerb Bare, ExecutorVerb Node, stri

/// <summary>
/// Upstream's <c>type_secret</c> tool (<c>agent/primitives.ts</c>), which the port names <c>fill_secret</c>. Offered only when the step
/// declared secrets and the engine can fill: its description lists each as <c>"name" (purpose)</c>. The secret's name is <c>secret</c>
/// where upstream's is <c>name</c>, which the target's own <c>name</c> field already takes.
/// declared secrets and the engine can fill: its description lists each as <c>"name" (purpose)</c>. It takes <c>target</c> and the
/// secret's <c>name</c>, as upstream.
/// </summary>
private static ModelTool FillSecret(IReadOnlyList<ExecutorSecret> secrets) => Tool(
"fill_secret",
"Fill one declared secret into an input by its name; the plaintext never passes through you. A password fills only a password field; a generic-secret fills any editable input. Available: " +
string.Join(", ", secrets.Select(secret => "\"" + secret.Name + "\"" + (secret.Purpose is null ? "" : " (" + secret.Purpose + ")"))) +
".",
WithValue(Target, "secret"));
WithValue(Target, "name"));

private static readonly ModelTool LongPress = Tool(
"long_press",
Expand Down
2 changes: 1 addition & 1 deletion tests/E2E.Cli.Tests/McpServer/E2EMcpTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ public async Task Opens_a_session_with_its_catalog_then_locates_acts_fills_a_sec
Assert.Contains("Tapped the point", pointed.Text, StringComparison.Ordinal);
Assert.DoesNotContain("No screenshot has been taken", pointed.Text, StringComparison.Ordinal);

var filled = await _served.CallAsync("type_secret", new() { ["target"] = ServedCli.RefOf(after.Text, "textbox \"Password\""), ["secret"] = "adminPassword" });
var filled = await _served.CallAsync("type_secret", new() { ["target"] = ServedCli.RefOf(after.Text, "textbox \"Password\""), ["name"] = "adminPassword" });
Assert.False(filled.IsError, filled.Text);
Assert.Contains("Filled secret \"adminPassword\" into #", filled.Text, StringComparison.Ordinal);
Assert.DoesNotContain(Served.AdminPassword, filled.Text, StringComparison.Ordinal);
Expand Down
2 changes: 1 addition & 1 deletion tests/E2E.Cli.Tests/McpTestLines/E2EMcpTestLinesTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ public async Task Answers_each_action_with_its_test_line_and_the_drafted_lines_p
(string Tool, Dictionary<string, object?> Args, string Line)[] steps =
[
("type", new() { ["target"] = ServedCli.RefOf(screen, "textbox \"Username\""), ["text"] = Username }, "await Screen.GetByRole(\"textbox\", \"Username\").FillAsync(Credentials.User(\"admin\").Username);"),
("type_secret", new() { ["target"] = ServedCli.RefOf(screen, "textbox \"Password\""), ["secret"] = "admin.password" }, "await Screen.GetByRole(\"textbox\", \"Password\").FillAsync(Credentials.User(\"admin\").Password);"),
("type_secret", new() { ["target"] = ServedCli.RefOf(screen, "textbox \"Password\""), ["name"] = "admin.password" }, "await Screen.GetByRole(\"textbox\", \"Password\").FillAsync(Credentials.User(\"admin\").Password);"),
("select", new() { ["target"] = ServedCli.RefOf(screen, "combobox \"Plan\""), ["value"] = "Pro" }, "await Screen.GetByRole(\"combobox\", \"Plan\").SelectOptionAsync(\"Pro\");"),
("check", new() { ["target"] = ServedCli.RefOf(screen, "checkbox \"Accept terms\"") }, "await Screen.GetByRole(\"checkbox\", \"Accept terms\").CheckAsync();"),
(
Expand Down
6 changes: 3 additions & 3 deletions tests/E2E.Cli.Tests/McpTestLines/SessionTestLinesTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -200,10 +200,10 @@ public async Task Partial_Writes_credentials_and_secrets_by_name_never_by_value(
NoteOf(await session.RunAsync("type", """{ "target": "n40", "text": "ada@example.test" }""")));
Assert.Equal(
"Test line: await Screen.GetByRole(\"textbox\", \"Password\").FillAsync(Credentials.User(\"admin\").Password);",
NoteOf(await session.RunAsync("type_secret", """{ "target": "n41", "secret": "admin.password" }""")));
NoteOf(await session.RunAsync("type_secret", """{ "target": "n41", "name": "admin.password" }""")));
Assert.Equal(
"Test line: await Screen.GetByRole(\"textbox\", \"API key\").FillAsync(Secrets.Get(\"apiKey\"));",
NoteOf(await session.RunAsync("type_secret", """{ "target": "n42", "secret": "apiKey" }""")));
NoteOf(await session.RunAsync("type_secret", """{ "target": "n42", "name": "apiKey" }""")));
Assert.DoesNotContain("sk-live-0000", session.Lines.Draft(), StringComparison.Ordinal);
}

Expand Down Expand Up @@ -283,7 +283,7 @@ public async Task Partial_Says_why_an_action_has_no_line_for_no_node_a_scroll_a_
NoteOf(await session.RunAsync("press", """{ "target": "n60", "key": "ArrowDown", "times": 3 }""")));
Assert.Matches(@"^No test line: a test action brings its node into view itself", NoteOf(await session.RunAsync("scroll", """{ "direction": "down" }""")));
Assert.Matches(@"^No test line: scrolling to a text has none", NoteOf(await session.RunAsync("scroll_to", """{ "text": "Row 40" }""")));
Assert.Matches(@"^No test line: the secret is neither", NoteOf(await session.RunAsync("type_secret", """{ "target": "n60", "secret": "unknown" }""")));
Assert.Matches(@"^No test line: the secret is neither", NoteOf(await session.RunAsync("type_secret", """{ "target": "n60", "name": "unknown" }""")));
Assert.Matches(@"^No test line: a locator tap takes no modifier keys", NoteOf(await session.RunAsync("tap", """{ "target": "n2", "modifiers": ["Shift"] }""")));
Assert.Matches(@"^No test line: a locator right click takes no modifier keys", NoteOf(await session.RunAsync("right_click", """{ "target": "n2", "modifiers": ["Shift"] }""")));
Assert.Equal("No test line: the drop target: node n404 is not on the newest screen.", NoteOf(await session.RunAsync("drag", """{ "target": "n2", "to": "n404" }""")));
Expand Down
Loading
Loading