Skip to content

Add semantic act-wait-observe execution - #39

Draft
wolfiesch wants to merge 1 commit into
mainfrom
review/semantic-engine
Draft

Add semantic act-wait-observe execution#39
wolfiesch wants to merge 1 commit into
mainfrom
review/semantic-engine

Conversation

@wolfiesch

Copy link
Copy Markdown
Owner

Why

Agents currently have to combine brittle node references, fixed sleeps, and repeated snapshots. That is slower, less ergonomic, and particularly unreliable when the user leaves a task unattended.

What changes

  • Adds stable semantic accessibility refs for unique actionable role + accessible name targets.
  • Re-resolves a semantic ref against the live accessibility tree so SPA DOM replacement does not require a new snapshot.
  • Fails closed with actionable candidates when a target becomes ambiguous or disappears.
  • Binds resolution to the exact live document identity and page revision before and after the AX query, preventing a stale ref from crossing a navigation/origin boundary.
  • Adds event/hybrid waits for load, URL, DOM text/selector, network idle, and download completion.
  • Anchors network-idle quiet time to each wait and wakes once an active request burst drains.
  • Filters irrelevant tab metadata events and coalesces MutationObserver scans to bound page-churn overhead.
  • Adds TypeScript/Python actWaitObserve helpers that stop correctly on Commit/Your Turn outcomes.
  • Enables optional debugger domains lazily, recovers from domain-disable/failure, and removes global observation as a barrier to later independent writes.

Review decisions

  • Semantic identity: refs intentionally encode revision, role, and name—not CSS structure. Duplicate matches fail closed instead of guessing.
  • Wait semantics: network idle uses a 500 ms quiet window and DOM identity is revalidated; these are practical heuristics rather than a transaction with the page.
  • Helper atomicity: act → wait → observe is a documented multi-RPC workflow. Another actor/page script may still change state between phases.
  • Ordering: a global observation waits for earlier writes but no longer blocks unrelated writes submitted afterward.

Validation

  • Workspace TypeScript typecheck/build and extension build
  • Semantic uniqueness, SPA replacement, ambiguity, and cross-document race regressions
  • Load/status/URL wake and 100-event storm regressions
  • DOM observer coalescing/options regressions
  • Network-idle anchoring/burst tests
  • Lazy debugger-domain and Developer-disable recovery tests
  • TypeScript and Python helper outcome tests

Why draft

This is a substantial execution-model and scheduling change. It should be evaluated with representative real sites and agent traces before it becomes the default recommended workflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant