Repository navigation
feat(audit): reusable /v1/cua capability re-audit harness - #954
Merged
Merged
Conversation
…atrix + trajectory scanner) The cua-issues audit (R1-R3) ran against real LinkedIn/Reddit — where anti-bot from Modal IPs dominates the signal and where the audited builds predated the #948/#951/#952 fixes. This gives a clean capability read on any current build: - run_cua_audit.py: fires a category matrix against the mantis-linkedin sim env (deterministic, no anti-bot) — read / read-profile / search-type / contenteditable-write / multi-step-nav. Non-destructive (the write task types into the composer but never sends; nothing logs in or submits). - scan_cua_audit.py: reads the per-run trajectories off the Modal volume and prints the REAL per-category verdict (focus / typed / type-landed-verified / clicked / done) — wire status alone lies here (the sim-env search-results page is a stub, so a perfect type still 'halts'). First run on 69af1a6: reads work; search type-landing FIXED (text landed, verified) — the audit's "search 0/7" was stale (pre-#948); multi-step click+nav and contenteditable-write still fail (both gated by the multi-step click chain). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mercurialsolo
added a commit
that referenced
this pull request
Jul 1, 2026
…-tab drift (#956) Audit finding: running many tasks on one warm profile piles up 40-50 tabs (the agent's target=_blank clicks / mid-task link-opens never get cleaned up). Both current_url and the CDP navigate/read target pick "the FIRST page tab" from /json/list, so once stale tabs accumulate they land on a PRIOR run's page — the serial-same-profile drift the auditor saw (S09 showed S04's URL, S03 drifted to github, Q02 to the Reddit ToS). This also affects the warm-container profile-reuse path in prod, not just the serial audit. reset() now calls _close_stale_tabs() at the START of a fresh navigation (a new task with a start_url) — keeps the active tab (Chrome lists it first), closes the rest via CDP /json/close/<id>. Gated to the navigation branch, so tabs the agent opens WITHIN a run are untouched; best-effort so a CDP hiccup never breaks reset. Tests: test_cua_close_stale_tabs.py — closes all-but-active, no-op on single tab, best-effort on CDP failure. 19 adjacent tests green; ruff clean. Note: the auditor is right that fresh-profile-per-run is the cleaner test setup (the re-audit harness #954 already does this and shows typing+multi-step work); this fix hardens the warm-reuse path so the drift can't recur there either. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The cua-issues audit (R1–R3) ran against real LinkedIn/Reddit, where anti-bot
from Modal IPs dominates the signal and where the audited builds predated
the #948/#951/#952 fixes (dead brain + type-stall). This harness gets a clean
capability read on any current build by exercising the same categories
against the deterministic sim env.
Scripts
run_cua_audit.py— fires a category matrix at the mantis-linkedin simenv: read / read-profile / search-type / contenteditable-write /
multi-step-nav. Non-destructive (the write task types into the composer
but never Sends; nothing logs in or submits a form).
scan_cua_audit.py— reads the per-run trajectories off the Modal volumeand prints the REAL per-category verdict (focus / typed / type-landed
(verified) / clicked / done). Needed because wire status lies here — the
sim-env search-results page is a stub, so a perfect type still
halts.First run on
69af1a6→ The audit's "search 0/7" was stale (pre-#948); the real remaining fault
is multi-step click-to-navigate, which also gates the write/composer flow.
🤖 Generated with Claude Code