Skip to content

Port observe's pixels option (capturePixels) to the engine session - #915

Merged
kblok merged 1 commit into
mainfrom
issue-908-observe-pixels
Nov 1, 2026
Merged

kblok merged 1 commit into
mainfrom
issue-908-observe-pixels

Conversation

@kblok

@kblok kblok commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

An observation could not carry the screen's pixels, so the three budget tests of #892 ran the viewport probe and ScreenshotAsync instead of upstream's path and kept a Partial_ prefix. The engine session now has upstream's observe(operation, { pixels, comparable }), with the screenshot taken under the observation's one budget. The three tests port as written.

Upstream change ported: fix(mobile): bound reads by the operation budget, 57f9ed9 (e2e-inc/e2e#1464): the capturePixels step of observe in packages/mobile/src/surface.ts (the screenshot shares the budget, OPERATION_TIMEOUT is rethrown, any other failure costs the observation its image), and artifacts.screenshot(label, operation).

Public API change: IEngineSession.ObserveAsync(ObserveOptions? options, TimeSpan timeout, ct), ObserveOptions { Pixels, Comparable } (upstream's EngineObserveOptions), Observation.Pixels (an EngineScreenshot: upstream's pixels and maskedRegionCount), and ScreenshotArtifactAsync(label, TimeSpan timeout, ct) on IEngineSession and DeviceSession. The agent still takes its screenshot through Capture; it does not read Observation.Pixels yet.

Tests (Engine/ObservationTests): no partial or skipped tests. 'bounds screenshot by operation budget', 'rethrows OPERATION_TIMEOUT from capturePixels during observe', and 'shares one budget across observe steps so screenshot cannot take a fresh budget' lose Partial_. Newly ported: 'captures pixels on request, masks every secure node, and degrades to tree-only when it cannot'. 'holds the status bar in a fixed state for a screenshot taken for comparison, and only then' and 'withholds the pixels of a screen with a secure field when the image cannot be read or decoded' now run through ObserveAsync with Pixels, as upstream does.

.NET-only symbols:

  • Default bodies of ObserveAsync(ObserveOptions?, TimeSpan, ct) and ScreenshotArtifactAsync(label, TimeSpan, ct) on IEngineSession drop the options/budget, so engines that take none keep compiling.
  • DeviceSession.CapturePixelsAsync is capturePixels (Async suffix); it does not swallow a caller cancel, where upstream catches every non-timeout error.

Docs: SKILL.md, COMPATIBILITY.md, and the regenerated API pages for the touched types.

Closes #908

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ls2e1yaZFgZ3FEJkYScfkK

…(upstream observe(operation, { pixels, comparable }))

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ls2e1yaZFgZ3FEJkYScfkK
@coderabbitai

coderabbitai Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 37 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
Commits

Reviewing files that changed from the base of the PR and between 58c28fc and 99fcb59.

Files selected for processing: 13
  • COMPATIBILITY.md
  • docs-site/public/api/E2E.Engine.IEngineSession.html
  • docs-site/public/api/E2E.Engine.ObserveOptions.html
  • docs-site/src/content/docs/api/E2E.Engine/IEngineSession.md
  • docs-site/src/content/docs/api/E2E.Engine/Observation.md
  • docs-site/src/content/docs/api/E2E.Engine/ObserveOptions.md
  • docs-site/src/content/docs/api/E2E.Engine/index.md
  • skills/e2e-dotnet/SKILL.md
  • src/E2E.Mobile/DeviceSession.Screen.cs
  • src/E2E.Mobile/DeviceSession.cs
  • src/E2E/Engine/IEngine.cs
  • src/E2E/Engine/Observation.cs
  • tests/E2E.Mobile.Tests/Engine/ObservationTests.cs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kblok

kblok commented Nov 1, 2026

Copy link
Copy Markdown
Member Author

Review of this PR: no code changes. Most findings describe upstream's own design, which the port follows:

  • Redaction against the observe tree (interactive-only, truncated, sparse), the snapshot taken before the screenshot, and generation bound before capturePixels run: observe in packages/mobile/src/surface.ts does the same. Not fixed here. The first two are possible gaps in upstream's redaction; reported to the orchestrator.
  • capturePixels catching every non-timeout error: upstream does too. The port only adds that a caller cancel is not swallowed.
  • Defaults that drop the options or the budget: .NET-only, named in the description, so engines that take neither keep compiling.
  • Observation.Pixels uses init: E2E.Mobile builds the Observation, so an internal set cannot work.
  • Duplicated scale logic with MaskedScreenshotAsync: upstream has the same two paths (maskedScreenshot and capturePixels).
  • The viewport-probe screenshot is still covered by the other tests in this class (the known-viewport test, and 'Takes no second snapshot...').

@kblok
kblok merged commit dd8aacd into main Nov 1, 2026
4 checks passed
@kblok
kblok deleted the issue-908-observe-pixels branch November 1, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port observe's pixels option (upstream capturePixels) to the engine session

1 participant