fix(visual): mask context-token counter in electron sidebar snapshot

[edelauna]

Related GitHub Issue

Closes: #1679

Description

The extension-host-visual job has failed on every PR since run #2252. PR #1505 changed the generated system prompt, which reduced the token count shown in the live context-token counter from 4.2k / 1.0m to 3.1k / 1.0m. That difference caused the electron-chat-dark-sidebar.png snapshot comparison to fail on all three Playwright retries.

The root cause is structural: the visual regression workflow does not monitor src/core/prompts/**, so the breakage shipped undetected.

Fix — use Playwright's mask option to overlay the two token counter elements before comparing the sidebar screenshot:

const webviewFrame = running.page.frameLocator(
    'iframe[src*="extensionId=ZooCodeOrganization.zoo-code"]',
)
const tokenCountMask = webviewFrame
    .frameLocator("iframe")
    .locator('[data-testid="context-tokens-count"],[data-testid="context-window-size"]')
await expect(sidebar).toHaveScreenshot(`electron-${scenario.name}-sidebar.png`, {
    mask: [tokenCountMask],
})

The mask replaces the dynamic token values with Playwright's default pink overlay at screenshot time. It does not touch the DOM, so there are no timing or CSP issues. The progress bar and surrounding layout remain in the snapshot. The welcome-dark and welcome-high-contrast scenarios are unaffected — Playwright ignores a mask locator that matches no elements.

The baseline is updated to match the new rendering (pink overlay over token values).

Test Procedure

1. The extension-host-visual CI job on this PR passes — all 3 repeats produce an identical screenshot (96bd934c).
2. Subsequent PRs that change src/core/prompts/** no longer cause this job to fail.

Pre-Submission Checklist

• Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
• Scope: My changes are focused on the linked issue (one major feature/fix per PR).
• Self-Review: I have performed a thorough self-review of my code.
• Testing: The updated baseline was generated from a deterministic 3-repeat CI run; all hashes matched.
• Documentation Impact: No documentation updates required.
• Contribution Guidelines: I have read and agree to the Contributor Guidelines.

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: Zoo-Code-Org/Zoo-Code/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 8b2a41d3-81e0-4445-aca5-4d473b6f3df8

📥 Commits

Reviewing files that changed from the base of the PR and between 4bd3a4c and 92679b6.

⛔ Files ignored due to path filters (1)

• apps/vscode-e2e/src/visual/__screenshots__/electron-chat-dark-sidebar.png is excluded by !**/*.png

📒 Files selected for processing (1)

• apps/vscode-e2e/src/visual/electron.visual.ts

Included review availability: Your plan provides up to 4 included reviews per hour; 2 remain after this review.

📜 Recent review details

🧰 Additional context used

📓 Path-based instructions (4)

Check strict typing and exhaustive behavior across normal, boundary, error, cancellation, retry, and compatibility paths.

⚙️ CodeRabbit configuration file

Files:

• apps/vscode-e2e/src/visual/electron.visual.ts

Reserve end-to-end coverage for behavior that requires the real VS Code host, workspace APIs, extension activation, webview messaging, file watchers, or a full workflow.

⚙️ CodeRabbit configuration file

Files:

• apps/vscode-e2e/src/visual/electron.visual.ts

Act as an adversarial second-opinion reviewer.

⚙️ CodeRabbit configuration file

Files:

• apps/vscode-e2e/src/visual/electron.visual.ts

Keep the Electron matrix limited to high-value shell integration scenes that require the real VS Code host.

📄 CodeRabbit inference engine (apps/vscode-e2e/AGENTS.md)

Files:

• apps/vscode-e2e/src/visual/electron.visual.ts

🔇 Additional comments (1)

apps/vscode-e2e/src/visual/electron.visual.ts (1)

242-250: LGTM!

---

📝 Summary

Summary by CodeRabbit

• Tests
  • Improved Electron visual smoke tests by excluding dynamic token-count indicators from screenshot comparisons.
  • Screenshot validation is now more reliable when system-prompt changes occur without affecting the rendered layout or styling.
  • Visual checks now focus on meaningful interface changes rather than expected fluctuations in token usage indicators.

Walkthrough

The Electron visual smoke test masks dynamic context-token counter elements when it captures the sidebar screenshot. The mask targets both token count and context window size elements inside the Zoo Code webview frames.

Changes

Visual Snapshot Stabilization

Layer / File(s)	Summary
Mask dynamic token counters apps/vscode-e2e/src/visual/electron.visual.ts	The sidebar screenshot assertion resolves [data-testid="context-tokens-count"] and [data-testid="context-window-size"] through nested webview frames and passes the locator to toHaveScreenshot as mask.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

🚥 Pre-merge checks | ✅ 8

✅ Passed checks (8 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Issue #1679 requires prompt-only token-count changes to stop causing visual failures while the surrounding progress UI remains covered. apps/vscode-e2e/src/visual/electron.visual.ts now passes a Pla…
Out of Scope Changes check	✅ Passed	The changed visual-test masking code and the sidebar baseline update directly implement issue #1679. The unchanged welcome scenarios remain within scope because they have no matching elements. No unre…
Regression Evidence	✅ Passed	PASS. The pull request changes only the existing Electron Playwright visual test and its electron-chat-dark-sidebar.png baseline; it does not change product UI behavior. The changed test adds focuse…
Security Boundaries	✅ Passed	PASS: The pull request changes only the Electron visual test and its sidebar baseline. The new code uses fixed Playwright frame and data-testid locators to mask the two token-counter elements during…
Persistence Integrity	✅ Passed	PASS. The pull request changes only a screenshot baseline and the toHaveScreenshot options in apps/vscode-e2e/src/visual/electron.visual.ts. The new code creates a Playwright mask locator and awai…
Lifecycle Resource Cleanup	✅ Passed	The pull request changes only the sidebar screenshot assertion. It creates Playwright FrameLocator/Locator objects and passes one locator to toHaveScreenshot as a mask. This path does not add li…
Title check	✅ Passed	The title clearly and concisely describes the main change: masking the dynamic context-token counter in the Electron sidebar visual snapshot.
Description check	✅ Passed	The description identifies the linked issue, explains the failure and root cause, describes the Playwright mask implementation, documents test results, and completes the relevant checklist items. It o…

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Review status

Thanks for contributing. This comment tracks the review sequence and the next action.

Current step: The required review sequence passed. Remaining merge requirements apply.

Review-state labels are managed by this workflow; do not edit them manually.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!