Repository navigation
Features/adding features - #15
Merged
Merged
Conversation
…mination Pending in-progress work on the AddingFeatures branch (authored prior to the Phase 0 architecture work): - Library now references the single canonical proto in the Server project via relative path + Link, removing the duplicate copy (Library/Protos/uiautomation.proto). - CloseApp / CloseAppByProcessId now honour InteractionAccessGuard, so process termination obeys the same WhiteList/BlackList policy as other interactions. - CloseApp strips a trailing ".exe" for consistency with GetAppStructure / ClearByName. - Doc + framework sync (net8.0-windows) across READMEs; proto CLICK comment clarified. - Add CLAUDE.md project guidance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hread (Phase 0) Introduces UiaExecutor: one long-lived dedicated MTA background thread + bounded queue. Every UIA / global-input RPC is marshalled onto it through UiAutomationService, making all element access, simulated mouse/keyboard, and ElementCache use atomic with respect to each other. Why: gRPC dispatched calls on concurrent thread-pool threads, so multiple clients could fight over the single mouse cursor / keyboard focus and race the cached COM references (ElementCache.TryGetLive mutated shared state without a lock). MTA, not STA: UIA *client* code is MTA-friendly; STA is for providers and invites reentrancy deadlocks. - UiaExecutor.RunAsync<T>(Func<T>, CancellationToken): marshals work; re-entrancy guard runs nested calls inline (no self-deadlock); backlog over Features:MaxQueuedRequests (default 32) -> ResourceExhausted; cancellation checked at dequeue; IDisposable drains. - UiAutomationService is the single marshalling boundary. OpenApp/CloseApp/ CloseAppByProcessId/ClearCache stay off the worker (no UIA/input; CloseApp's 5s WaitForExit would starve it). - Handlers (Element/Action/Screenshot/Reflection/AppStructure) converted Task<T> -> T. PerformActionWithStructure is now synchronous and holds the worker across its 200ms settle so the action + refreshed read are atomic. - Adds ARCHITECTURE_REVIEW.md (analysis + phased plan; Phase 0 marked done). Note: ActionHandler, AppStructureHandler, Program.cs and UiAutomationService.cs also carry some pre-existing interaction-guard/config edits that were intermixed in the working tree and could not be split from the Phase 0 changes.
perf: batch GetAppStructure with CacheRequest + depth/node caps + offscreen filtering Collapses the per-node UIA cost of GetAppStructure / PerformActionWithStructure (the LLM hot path) from ~7-9 cross-process COM round-trips per element to a single batched retrieval, and bounds + trims the emitted tree. - AppStructureHandler.BuildTree activates one CacheRequest (TreeScope.Subtree, ControlViewCondition, AutomationElementMode.Full) prefetching every property the builder reads; BuildNode reads via GetCachedPropertyValue with a transparent live fallback and walks CachedChildren (falling back to the live ControlView walker). - New AppStructureOptions (config Features:AppStructure): MaxDepth (40), MaxNodes (2000), IncludeOffscreen (false), CompactJson (true). Depth/node caps stop the walk and flag the parent ChildrenTruncated; offscreen/zero-rect nodes and subtrees are skipped by default except the root window. Result message reports node count + truncation. - AppNode emits compact JSON and trims noise (nulls, empty strings, empty child lists). - ElementCache gains a no-COM CacheElement overload so the builder stores values it already read instead of re-reading .Current.* per node. - CancellationToken plumbed into the walk; surfaces as StatusCode.Cancelled on both RPCs. Verified: builds clean; live Calculator smoke test built a 65-node tree in ~257ms with compact, offscreen-filtered JSON. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @
feat: Session 0 detection + optional SendKeys element targeting Makes the "runs as both console app and Windows Service" claim honest, and gives SendKeys a real target instead of relying on ambient focus. Session 0 (#2): Program.WarnIfNonInteractiveSession runs after Build(); if SessionId == 0 or !Environment.UserInteractive it emits a hard LogWarning (routes to the Windows Event Log under UseWindowsService(), console otherwise) plus a red console line when interactive, explaining UIA/input cannot reach the interactive desktop and pointing to Scheduled-Task-at-logon / interactive-account deployment. No behavioral change to the UIA path — the limitation is surfaced honestly rather than failing silently. (Chosen over a full CreateProcessAsUser session-bridge, which is deferred.) SendKeys targeting (#3): SendKeysRequest gains optional runtime_id (proto field 3, backward-compatible). ActionHandler.SendKeys, when runtime_id is set, looks the element up via ElementCache, gates it through InteractionAccessGuard, and SetFocus()es it on the serialized UIA worker before sending, so keys land on the intended control. Missing element / focus failure -> clean {Success=false}. Empty runtime_id keeps legacy behavior. Client SDK: UiAutomationDriver.SendKeysAsync(keys, wait, runtimeId?) + VirtualKeyboard.SendToElementAsync. Verified: builds clean; live Calculator test focused an element by RuntimeId and typed "2+2=" -> "Display is 4"; bad RuntimeId returned "Element not found in cache."; untargeted send still works; Session 0 detector does not false-positive in a console run. @
test: make short-name PATH resolution test deterministic ResolvesShortName_Notepad depended on the launching shell PATH: it assumed "notepad" resolves to C:\Windows\System32\notepad.exe, but where.exe returns the first PATH match, which varies by host (Windows ships notepad.exe in both System32 and C:\Windows, and a Git-Bash PATH resolves "notepad" to C:\Program Files\Git\usr\bin\notepad). That made the test pass or fail depending on who launched the runner. Replaced with ResolvesShortName_ViaPath: drops a uniquely-named dummy .exe into a temp dir, prepends that dir to PATH (restored in finally), and asserts the short name resolves to it. Exercises the real where.exe resolution + whitelist matching deterministically, independent of any ambient system binary. Full suite now 91/91. @
#10 Unify error model: business outcomes return {Success, Message}; RpcException reserved for transport/auth/cancellation. - Add success/message to ElementResponse (proto fields 6/7, backward-compatible). - FindElement/GetProperty (ElementHandler), PerformAction (ActionHandler), and ScreenshotHandler return {Success=false} instead of throwing NotFound/ PermissionDenied/Internal. AutomationMapper.MapToResponse sets Success. - AuditInterceptor no longer echoes raw ex.Message to clients (logs detail server-side, returns an opaque Internal status) - fixes info leak. - Library UiElement surfaces the server Message on FindElement failure. #11 Bind to loopback 127.0.0.1 by default; Security:AllowRemote=true opts into 0.0.0.0 (with a warning). Applies to both HTTP and HTTPS. #13 Remove dead config (Features:AppRestrictions, RateLimiting) from appsettings.json; update CLAUDE.md + Server README; add Security:AllowRemote. #14 Broaden IsClickable beyond invoke||toggle to also include ExpandCollapse, SelectionItem, settable Value, and the Hyperlink control type (new props added to the batched CacheRequest). #12 Document the OpenApp UWP/Store launcher-PID limitation (handler + SDK XML docs, Server README note); behavior unchanged - prefer GetAppStructure by name. Verified: Server+Library build 0 errors; tests 91/91; live smoke test (loopback bind confirmed; bogus FindElement/GetProperty return {Success=false} with no RpcException; GetAppStructure(notepad) -> 24-node tree, 11 clickable incl. value-settable Edit control). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ol SDK (#7-#9) Replace the hand-rolled stdio JSON-RPC MCP server with the official ModelContextProtocol 1.3.0 C# SDK. - #7 lifecycle: delete McpServer.cs; Program.cs now uses Host.CreateApplicationBuilder + AddMcpServer().WithStdioServerTransport() .WithToolsFromAssembly(). Logging goes to stderr (stdout is the JSON-RPC channel). Protocol version negotiation, ping, cancellation, and tools/list_changed now come from the SDK. Tools are attributed methods in Tools/UiAutomationTools.cs; the gRPC client and CancellationToken are injected via DI and excluded from each tool's JSON schema. - #8 image content: take_screenshot returns ImageContentBlock.FromBytes(...) (base64 PNG the model can see) instead of a temp-file path string. - #9 gRPC stack: drop legacy native Grpc.Core 2.46.6 and Newtonsoft.Json; use only Grpc.Net.Client (auth interceptor uses transitive Grpc.Core.Api). - Tools (11): open_app, close_app, get_app_structure, find_element, get_children, get_property, perform_action, perform_action_with_structure, send_keys (with runtimeId targeting), take_screenshot, clear_cache. The new find_element/get_children/get_property/send_keys give an LLM the same element-level surface the QA Library client uses. Business outcomes map to CallToolResult.IsError mirroring the server Success flag. Server and proto are untouched, so UiAutomationGRPC.Library (QA tests) is unaffected. Builds 0 errors; verified via piped initialize + tools/list (all 11 tools, correct schemas). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Introduced a comprehensive test suite for ElementCache to validate RuntimeId resolution and caching behavior in both enabled and disabled modes. - Updated ElementCache to ensure it correctly handles element re-resolution and maintains locator metadata regardless of caching state. - Improved documentation within ElementCache to clarify the caching strategy and its implications on RuntimeId addressing. - Modified Program.cs to reflect changes in caching behavior and updated logging for cache status. - Added a new blacklist entry for Task Manager in appsettings.json to restrict interactions with specific applications.
…Cache routing, ElementCache, and TokenAuthInterceptor
- Updated README.md to clarify the interaction between LLMs and the MCP server, emphasizing the See → Think → Act loop and the use of mental maps for efficient navigation. - Revised Skill.md to streamline instructions for using the uiautomation MCP server, including loading mental maps and common operations. - Added a template for documenting app-specific automation details in apps/_template.md. - Created detailed documentation for Calculator and Notepad applications, outlining key elements, common operations, and navigation strategies. - Improved ActionHandler to support screen coordinate clicks for left, right, and double-click actions. - Enhanced AppStructureHandler to ensure only real window elements are returned, improving the reliability of UI structure retrieval. - Updated proto definitions to clarify action types and their expected parameters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.