Check automatic approval end to end, against a stubbed AWS SSO - #89
Open
popen2 wants to merge 1 commit into
Open
Check automatic approval end to end, against a stubbed AWS SSO#89popen2 wants to merge 1 commit into
popen2 wants to merge 1 commit into
Conversation
Automatic approval is a script clicking buttons on pages Frost does not
own, in a window the user cannot see, and every way it can be wrong is
quiet: a button that is never found (the refresh hangs until the device
code expires), a button that should not have been clicked (the request is
denied), a hand-over that never happens (the user waits in front of
nothing). None of that is visible to a type-check, and none of it is
really testable against the matching rules alone — what makes it work is
the whole path.
So `npm run check:auto-approve` drives the real `refresh()`, the same
entry point the tray, the hotkey and the timer use, and asserts on what
the user would have seen. Two interceptions make that possible without
the app knowing it is under test:
- `AWS_ENDPOINT_URL_SSO_OIDC` / `AWS_ENDPOINT_URL_SSO`, an AWS SDK
feature, point the SDK at a stub HTTP service. The device
authorization, the polling and its AuthorizationPendingException are
the real client speaking a real protocol.
- `session.protocol.handle("https", ...)` serves the verification pages
at their real names, so the renderer gets `https://d-….awsapps.com`, a
secure context, and a genuine cross-origin redirect to the identity
provider. Served from localhost it would prove nothing — the host rule
is the point.
`HOME` and the electron-store move to a temp directory, so a run touches
nothing of the developer's. Three scenarios, one per outcome: a live
identity provider session finishes with no window ever shown; a page
asking for a password shows the window and finishes the approval once the
check signs in on the page; and a page with nothing recognisable —
including a refusal wearing `cli_login_button`'s id — is not clicked at
all, and the window comes up.
It fails when the feature breaks, which is the part worth stating:
disabling the host rule fails both approval scenarios, and disabling the
refusal rule fails the third by clicking the trap. 16 seconds, and CI
runs it under xvfb next to the overlay check.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018XK1mi3rSPLzxu7XUhyC6a
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #85, which merged with only its matching rules covered. This adds the regression test that was missing:
npm run check:auto-approve, in CI next tocheck:overlay.Why
Automatic approval is a script clicking buttons on pages Frost does not own, in a window the user cannot see, and every way it can be wrong is quiet — a button never found (the refresh hangs until the device code expires), a button that should not have been clicked (the request is denied), a hand-over that never happens (the user waits in front of nothing). None of that is visible to a type-check, and testing the matching rules alone misses the thing that actually makes it work: the whole path.
So this drives the real
refresh()— the same entry point the tray, the hotkey and the timer use — and asserts on what the user would have seen.How it stubs AWS SSO
Two interceptions, neither of which asks the app to know it is under test:
AWS_ENDPOINT_URL_SSO_OIDC/AWS_ENDPOINT_URL_SSO— an AWS SDK feature, so the device authorization, the polling and itsAuthorizationPendingExceptionare the real client speaking a real protocol to a stub HTTP service. The token only becomes redeemable when the stub's/approvedpage is actually fetched, so nothing passes without a real click.session.protocol.handle("https", …)— serves the verification pages at their real names. The renderer getshttps://d-1234567890.awsapps.com, a secure context, and a genuine cross-origin 302 to the identity provider. Served from localhost this would prove nothing; the host rule is the point.HOMEand the electron-store move to a temp directory, so a run touches nothing of the developer's.The three scenarios
cli_login_button's id — and the window comes upIt fails when the feature breaks
Worth stating, since a green test that cannot go red is worse than none. Mutating the built driver:
isApprovalHost→false: both approval scenarios fail.isDenial→false: the third fails by clicking the "Cancel request" trap.16 seconds for a passing run. CI runs it under xvfb, the same way as
check:overlay.Still not covered
A real desktop: no security key, no keychain, no dock, and no real AWS pages — so the id and label matching against what AWS actually serves today still wants a smoke test on a real machine. Default-browser hand-over is also untested here, since asserting it would mean really opening a browser on the runner.
Generated by Claude Code