Repository navigation
Agent loop reliability: UIA3 migration, See→Think→Act hardening, and richer agent diagnostics - #16
Merged
Merged
Conversation
- document UiaExecutor single MTA worker + bounded queue - document Features:AppStructure tuning and CacheRequest batching - add Session 0 guidance, Skill project row, broadened test coverage - fix stale claims (token-auth startup warning, per-project .sln layout) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ment Server-side polling wait replaces the client-side find/retry loop that cost an LLM round trip per probe. The wait loop runs OFF the single UIA worker: each probe is enqueued as its own short work item with Task.Delay between probes on the request thread, so long waits never starve other RPCs. ResourceExhausted during a probe counts as a miss, not a failure. Timing is normalized by WaitPolicy (default 10s/250ms, caps 120s / 50ms-5s) with unit tests, plus service-level timeout/immediate-hit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Packaged apps launched via an alias (calc) return a launcher PID that exits within a second, making every PID-addressed follow-up fail. OpenApp now snapshots visible top-level windows (EnumWindows, no UIA) before launching; when the launched process dies inside the 500 ms grace window it polls up to 3 s for a newly-appeared window and returns its owner PID instead. Classic Win32 apps keep the fast path. Best-effort: ambiguous candidates resolve to the most recently started process, and an unresolvable exit is flagged in the response message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nWithStructure New proto StructureOptions (max_depth, max_nodes, include_offscreen, scope_runtime_id, diff_mode) on AppStructureRequest and PerformActionRequest. The agent knows when it only needs a shallow or scoped view: per-request overrides merge over Features:AppStructure defaults (AppStructureOptions.MergeWith, unit-tested), and scope_runtime_id builds the tree under a known element instead of the whole window - both calls and the response shrink accordingly, which matters for huge dynamic UIs. Exposed via MCP tool parameters and SDK overloads. diff_mode is wired in the proto and implemented in a follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the fixed Thread.Sleep(200) with quiescence detection: subscribe to structure-changed and Name/Value property-changed events under the window, read the refreshed tree once no event arrives for SettleQuietMs (default 150), hard-capped at SettleMaxMs (default 1000) so continuously-updating apps (live grids/tickers) never stall the call. Falls back to the legacy fixed delay when a provider rejects subtree subscriptions. Both knobs configurable via Features:AppStructure. QuiescenceTracker timestamp math is unit-tested. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
StructureDiff compares the fresh build against the previous snapshot of the same root (matched by RuntimeId) and emits flat Added/Changed/Removed entries with a Parent id for orientation; reparenting and visibility/clickability flips count as changes. StructureSnapshotStore keeps the latest tree per window root (PID-tagged, evicted via ClearCache alongside the element cache); every build refreshes the base, so See followed by diff-mode actions works naturally. First diff call without a base returns the full tree with a note. For a one-label change in a 2000-node app this cuts the response by orders of magnitude - the difference that makes the loop usable on large dynamic UIs (e.g. trading/portfolio systems). Exposed as diffMode on the MCP get_app_structure / perform_action_with_structure tools. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… MCP channels UiAutomationDriver now applies a per-call deadline (default 30s, configurable via the new defaultTimeout ctor parameter) so a hung server fails the call with DeadlineExceeded instead of blocking a test suite forever, and every public async method accepts an optional CancellationToken. WaitForElementAsync extends its deadline to the server-side wait budget plus headroom. Both the SDK channel and the MCP server channel get a gRPC ServiceConfig retry policy scoped to pre-execution rejections only - ResourceExhausted (UIA worker queue full) and Unavailable (connection refused/shutdown) - with exponential backoff. Mid-action status codes are deliberately not retried: replaying a click or keystroke is never safe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ctions - GetServerStatus RPC reports worker queue load/capacity, cache size+mode, Windows session id and interactivity, and server version - answered off the UIA worker so it works even when the queue is saturated. Surfaced as the get_server_status MCP tool (flagged as an error result when the server runs in Session 0, where it cannot drive the desktop). - Standard grpc.health.v1 health service so clients probe readiness instead of failing their first real call. - MCP tools now carry annotations (readOnlyHint on all See/read tools, destructiveHint on close_app, idempotentHint on clear_cache) and the MCP initialize response ships See->Think->Act usage instructions, so any MCP client gets the guidance without loading the Skill file. Verified via piped initialize+tools/list: 13 tools, annotations and instructions present. Unit tests for GetServerStatus. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the WinForms SendKeys wrapper with raw SendInput while keeping the
same expression syntax. SendKeysParser (pure, 30+ unit tests) handles named
keys, +^% modifiers, (groups), {LEFT 5} repeats and all brace escapes.
Delivery rules: named keys / chords go out as VK + scan code with the
extended-key flag where required (WinForms, UWP and Chromium apps all accept
this); plain text is injected as KEYEVENTF_UNICODE, keyboard-layout
independent; characters under Ctrl/Alt resolve through VkKeyScan so ^a hits
the real accelerator. Partial injection now raises a descriptive error
instead of vanishing silently.
New WindowFocus.EnsureForeground: SetFocus alone cannot raise a background
window (foreground-lock rules), so targeted send_keys and simulated clicks
first restore + foreground the owning window, using the AttachThreadInput
escape hatch when the lock denies the switch - fixes the classic
"send_keys right after open_app did nothing" failure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…afe guard, constant-time tokens - ElementCache.TryRefind / FindByRuntimeId no longer run desktop-wide TreeScope.Descendants queries (the most expensive UIA call there is): they scan the desktop's direct children for the owning PID's top-level windows, then search only those subtrees. On a desktop full of heavy apps this turns every stale-handle recovery from "walk everything" into "walk one app". - New ElementCache.SweepDeadProcesses + a 60s background sweeper started at server startup: entries and structure snapshots of exited processes are evicted instead of lingering until individually touched. Sweeper is opt-in so tests with fake PIDs are unaffected. - InteractionAccessGuard decision cache is now keyed by (PID, process start time): Windows recycles PIDs, and a bare-PID cache could hand a new process the verdict computed for a dead one. Identity-less lookups are validated but never cached. - TokenAuthInterceptor compares tokens with CryptographicOperations .FixedTimeEquals (no early exit) instead of ordinal Contains. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…+ no keystroke leaks
New SmokeTests fixture (Category=Smoke, gated by UIA_SMOKE=1 so plain
`dotnet test` skips it) drives a real Notepad through the full pipeline:
OpenApp -> WaitForElement -> targeted SendInput typing -> read the text
back -> GetAppStructure -> CloseAppByProcessId. A dedicated GitHub Actions
job runs it on windows-latest, whose runners expose an interactive desktop.
Verified green locally against live Notepad.
Hardening found by the smoke run:
- GetProperty("Value"/"Text") now reads text through ValuePattern OR
TextPattern (Notepad's Win32 Edit supports only the latter; WinForms
textboxes only the former) before falling back to a raw property read,
and AutomationMapper.LookupProperty knows "Value". This is the read
half of SET_VALUE that agents were missing.
- Targeted SendKeys waits 50 ms after SetFocus (focus moves async) and
then VERIFIES the target window holds the foreground before injecting -
refusing with a clear error instead of leaking keystrokes into whatever
app the user is focused on.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…le, UWP PID resolution, SendInput keyboard, cache sweep, SDK deadlines/retry, smoke test
…3 (FlaUI 5.0)
The deprecated managed UIA2 stack is replaced by FlaUI.UIA3 across the whole
server - better support for modern apps (UWP/WinUI/Chromium), an actively
maintained library, and the native UIA3 COM API underneath. The proto contract,
client SDK and MCP server are untouched: this is an internal engine swap.
- UiaRuntime (new): process-wide UIA3Automation singleton + desktop root +
stale-element detection (FlaUI ElementNotAvailableException / COM 0x80040201)
- AutomationMapper: FlaUI conditions/scopes/PropertyIds; control-type condition
values now accept enum names ("Window"), programmatic names
("ControlType.Window") and native UIA ids (50032); responses keep emitting
the UIA2-style "ControlType.X" strings so existing clients/skills see no change
- ElementCache: liveness probe + scoped re-find ported; RuntimeId handles keep
the same comma-joined format, so cached ids stay wire-compatible
- AppStructureHandler: CacheRequest now stays active for the whole tree build
(FlaUI routes property reads through the cache while activated); per-property
fallback defaults instead of cached/live read pairs
- UiSettler: structure/property-changed subscriptions via FlaUI event handlers
with per-handler unregistration
- WindowFocus.EnsureForeground hardened: UIA2's managed SetFocus raised the
window via the client-side Win32 proxy as a side effect, UIA3's does not -
activation now attaches to BOTH input queues (foreground owner + target),
uses BringWindowToTop, and falls back to the ALT-key foreground-lock release.
Without this, targeted SendKeys always refused ("could not be brought to the
foreground") - caught by the live smoke test, fixed and re-verified green
- ReflectionHandler: property/pattern discovery via FlaUI libraries
- csproj: + FlaUI.UIA3 5.0.0, - UseWPF; assembly-level SupportedOSPlatform
attribute added (GenerateAssemblyInfo=false suppressed the SDK-generated one,
which made the platform analyzer flag every FlaUI call with CA1416)
- Tests ported (mapper/cache fixtures) + new coverage for control-type value
mapping, incl. an Enum.TryParse numeric-string pitfall found by the new tests
Verified: 286 unit tests green; live Notepad smoke test (UIA_SMOKE=1) green:
open -> wait -> find edit -> type via SendInput -> read back via TextPattern ->
structure tree -> close. Library/MCP/Client all build against the same proto.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…conventions, foreground activation) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live Calculator runs exposed three real failure modes in UwpPidResolver, all reproduced on a Win10 desktop: 1. The fixed 500 ms launcher grace misclassified calc.exe: the alias stub stays alive ~1 s before handing off to the Store app and exiting, so OpenApp returned the stub's PID, which died moments later. The grace is replaced by evidence: alive + owns a new visible window -> classic app (returns as soon as the window shows, faster than the old fixed wait); alive + window-less past 1.5 s -> background/CLI app; exited -> resolve. 2. The window diff compared owner PIDs, but UWP windows belong to the long-lived ApplicationFrameHost whose PID is in the pre-launch snapshot too - the new Calculator frame was invisible to the diff. Now diffs by window HANDLE and maps new handles to owner PIDs. 3. UWP startup briefly shows the app's CoreWindow as a visible top-level window before reparenting it into the frame; sampling once could return that transient owner. Resolution now debounces (300 ms resample, prefer what persists) and verifies the picked PID is alive. Resolution budget raised 3 s -> 10 s (cold UWP starts take 5-6 s; the happy path returns immediately so fast apps never pay it). New smoke test Calculator_TwoPlusThree_ShowsFive: OpenApp calc (exercises the resolver) -> GetAppStructure poll -> Invoke num2/plus/num3/equal by AutomationId (no keyboard, no foreground dependency) -> read CalculatorResults -> close the real CalculatorApp PID (never ApplicationFrameHost). Ignores (not fails) when the UWP Calculator is absent, e.g. on CI server images. Verified live: Notepad + Calculator smokes green; full suite 289/289. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…direct-to-file approach
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch makes the gRPC UI Automation service substantially more reliable for LLM-driven See → Think → Act loops. The headline change is migrating the engine from the deprecated managed UIA2 (System.Windows.Automation) to UIA3 via FlaUI 5.0, alongside a set of features that remove the slow, token-hungry client-side polling patterns agents previously had to use and make element addressing, input injection, and process resolution far more robust.
+3468 / −444 across 44 files, including a new live UI smoke test suite and a CI job to run it.