Skip to content

feat(browser): operate controls inside embedded documents #40

Closed

Description

@trmosala

What to build

The agent navigates and edits embedded calendar-like interfaces through the granted tab using pointer, keyboard, selection and scrolling operations.

Acceptance criteria

  • Click, hover, drag, fill, key, select and scroll target controls in nested and cross-origin documents through the same tab grant.
  • Controls remain targetable after supported scrolling and transforms, using current native identity and coordinate mapping.
  • An embedded input receives input in its actual document; unsupported targets never silently use the top frame.
  • Replacement, revocation, focus changes and stale references prevent follow-on input; already-dispatched effects retain accurate outcome status.
  • Native fixtures exercise calendar navigation, opening an event, editing synthetic inputs and scrolling without modifying real account data.
  • Verify actual authenticated Teams week navigation, scrolling and opening an event for observation without another approval sequence. Do not edit/create/send calendar events or enter/change/submit timesheets.

Blocked by

Context

Source: revised CM_Browser review and whole-tab Agent Access requirement, 2 October 2026. This is follow-up work to the previously completed bounded browser implementation; existing issues remain closed.

Preserve main-process ownership, explicit revocation, stale-target protection and isolation from unrelated tabs. Coordinate with ongoing browser changes and the separate Microsoft sign-in fix. Native validation must distinguish fixture results from actual authenticated-site evidence. Do not restart the installed app or server, modify real calendar events, or enter/change/submit timesheets.

Activity

  1. added
    enhancementNew feature or request
    ready-for-agentScoped work with acceptance criteria and recorded blockers
    on Oct 2, 2026
  2. trmosala commented on Oct 2, 2026

    @trmosala
    OwnerAuthor

    Implementation checkpoint saved; remaining acceptance work stays open.

    Embedded input implementation and synthetic/native qualification are committed. Actual authenticated Teams week navigation, scrolling and event-open observation remain unverified.

    Local commit: 7f746e8fcb7d2ad819b107aac2c0d7d9462d1361 on branch browser-gaps. This commit has not been pushed, merged or released.

    Fresh checkpoint validation: 410 tests / 6,620 assertions pass; all five affected package typechecks and the browser plugin build pass; native navigation and leave-confirmation reruns pass; scoped lint has 0 errors and 356 style warnings; formatting and whitespace checks pass. Earlier native fixture results are documented in the committed cm-browser README. These checks do not qualify authenticated Teams, macOS or a packaged app.

    Native UI screenshot and validation archive. No installed app/server was restarted and no real account data was modified.

  3. trmosala commented on Oct 3, 2026

    @trmosala
    OwnerAuthor

    All six acceptance criteria are complete. The remaining authenticated checks passed on 3 October 2026 through the rebuilt CookieMonster instance, bundled browser plugin, sidecar and main-process routing using cookiemonster/CM_Opus5.5-High.

    • Week navigation changed 28 September–4 October to 21–27 September and restored the starting week. Separate reads and native-window observations verified both changes.
    • An element-anchored embedded wheel operation with deltaY: 300 moved the visible calendar hours from approximately 10–13 to 14–17. Screenshots and native-window observations confirmed movement independently of dispatch status.
    • One existing timed event opened for observation and was closed with Escape after reacquiring its current frame binding. Separate reads and native-window observations verified the preview opening and closure.
    • Day view on 3 October was restored, with the preview and menus closed. No additional origin grant or operation approval appeared after the user's whole-tab grant. No calendar data, timesheets or chat messages were changed or sent.

    Authenticated qualification and recovery details include the exact launch hashes, transitional uncertain outcomes, overlay dismissal and fresh-frame recovery. These are Windows development results; macOS and packaged builds are not qualified. Private calendar content and authenticated screenshots are not published.

    Synthetic/native qualification covers nested cross-origin input, transforms, clipping, native identity, replacement, stale-reference, revocation and focus safeguards, including 170 tests / 1,647 assertions and the native fixtures. Earlier native review images remain available.

    Evidence is pushed as 1b03c41978 on browser-gaps; local and remote commits match. Formatting and whitespace checks passed, and the push hook completed all 31 typecheck tasks successfully, with 30 cached. No merge or release is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestready-for-agentScoped work with acceptance criteria and recorded blockers

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions