diff --git a/CHANGELOG.md b/CHANGELOG.md index f747adeae..118113e7f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,8 +4,25 @@ All notable changes to OriginWeave are documented in this file. The format follo ## [Unreleased] +- Keep WebDriver remote HTTP bodies, W3C error/message text, last-response startup detail, and mismatched remote `browserVersion` capability values out of CI exception strings while preserving fail-closed command/readiness/version decisions and the response-size bound. +- Publish success-shaped MV3/Agent Task compatibility JSON only after both owned loopback fixture servers complete their shutdown post-conditions; browser/trial gate failures still emit bounded diagnostic evidence before raising. +- Require loopback fixture-server cleanup to observe helper-thread termination after the bounded join, so a timed join cannot be treated as cleanup success while an owned server thread remains live. +- Require the controlled Agent Task lane to observe an empty input `value` after native WebDriver Element Clear and before Send Keys, so clear command acknowledgement cannot substitute for browser-observed action evidence. +- Require the controlled Agent Task lane to read the browser-observed input value after native WebDriver typing and before click, so send-keys command acknowledgement cannot substitute for evidence that the synthetic input actually reached the controlled field. +- Require controlled Agent Task success to re-observe the browser URL after submitted-state and exact-echo verification, so URL stability is proven through the accepted outcome boundary rather than only immediately after click. +- Require the controlled Agent Task lane to observe the canonical idle result baseline before native clear/type/click and to carry that baseline witness into repeatability surface completeness, preventing a pre-fired fixture from being accepted as an action-caused success. +- Separate Agent Task extension-isolation launch intent from verified browser surfaces: `--disable-extensions` is recorded as requested configuration only and no longer counts toward observed success completeness. +- Keep page-derived browser post-condition text, observed URLs, and extension/page datasets out of CI exception payloads while preserving their local comparison role and fail-closed browser evidence semantics. - Refreshed the product-gap queue to 126 open pull requests (54 ready, 72 draft) after #190, #188, #185, #192, #182, #184, #115, #181, #116, #117, #118, #183, #114, #127, #112, #109, #186, #110, #108, #111, #174, and #113 were merged into their immediate stacked prerequisites. PRs #147, #146, #145, #144, #143, #142, #141, #139, #136, #132, #129, and #128 moved to ready after exact-head checks and thread review; these are queue-consolidation results, not protected-main shipment. +- Classified bounded WebDriver HTTP cleanup failures, including truncated responses, as typed cleanup evidence while preserving the primary browser error. +- Record truncated WebDriver trial responses as bounded failed-trial evidence instead of aborting the complete MV3 compatibility run. +- Close the first pinned-Chrome fixture server when startup of the second server fails, and attempt both shutdowns when one cleanup fails, preventing partial compatibility runs from leaking loopback server threads. +- The fixture-shutdown contract now exercises successful MV3 and Agent Task trial paths before asserting reverse-order server cleanup. +- The fixture-shutdown success double now includes both browser-computed semantic verification surfaces, so cleanup failures cannot mask incomplete Agent Task evidence. +- Preserve Chromium's process sandbox in the controlled Agent Task evidence lane by removing that lane's `--no-sandbox` launch argument after a test-first source contract reproduced the defect; sandbox-enabled pinned-browser exact-head execution remains required before the repaired lane is promoted to security evidence. +- Classify Agent Task browser-session startup failures in credential-free trial evidence without exposing ChromeDriver text; this fixture evidence does not claim a shipped OriginWeave browser adapter. +- Keep Chrome for Testing sandbox-helper installation in the separately governed workflow activation; this workflow-free slice does not claim hosted sandbox execution. ### Added - Corrected the 2026-08-26 product-gap snapshot with current #229 presentation-identity evidence, stacked-only #205 integration evidence, current base/head pairs, the 126-PR queue count, explicit root-versus-child merge ordering, and the active GitHub counted-approval gate. - Refreshed the product and technical gap baseline onto the 2026-08-26 live inventory: 126 open pull requests (54 ready, 72 draft), protected-main promotion of #168/#194/#196/#216/#151, a verified maintenance-loop record (supersession closure of #153, conflict reconciliations on #37/#149/#152/#173/#175, issue #212 option-(b) authorization on #43, Strix vuln-0001 homoglyph remediation on #124), provider-rerun outcome evidence, an organization review-pipeline congestion record, and refreshed merge-order queue guidance. Documentation evidence contracts were aligned to the same snapshot so the baseline, its dated markers, and the pinned exact-head rows cannot silently diverge. @@ -36,6 +53,8 @@ All notable changes to OriginWeave are documented in this file. The format follo - Real loopback TCP integration proof plus deterministic timeout, refusal, retry, peer-inspection, peer-mismatch, canonicalization, IPv6 metadata, and single-use replay tests. - Real loopback rustls integration covering trusted DNS SAN, Common-Name fallback rejection, wrong-name and untrusted-root rejection, fixed-time expiry and not-yet-valid failures, exact IPv4 and IPv6 SANs, TLS 1.2/TLS 1.3, required and optional ALPN, and transport-origin binding. - Cumulative interactive-first RAM, VRAM, batch, local-model, admission, pause, and compositor-pressure mitigation plans, including active-consumer reduction at exact hard limits. +- Real pinned-Chrome WebDriver evidence for the controlled Agent Task fixture: the CI lane uses an isolated profile, requests extensions disabled as launch configuration, types and submits synthetic text, observes the same-document post-condition, and proves profile cleanup; extension disablement itself is not promoted to a verified browser surface without an independent observation. Predecessor sandbox-disabled success is compatibility evidence only until the repaired sandbox-enabled exact head executes successfully. +- Active pinned-Chrome Agent Task evidence verifies browser-computed role/name for controlled input and submit targets before action; this remains test-harness semantic evidence and does not claim a product semantic observer or authority. - Universally value-redacted network evidence with explicit path, metadata, and provenance bounds; ambiguous path rejection; validated source URLs; lowercase SHA-256 identifiers; and verification state. - Versioned schema-bound extraction contracts with bounded identifiers and field counts, typed value/cardinality metadata, explicit duplicate-free reviewed source channels, fail-closed schema validation, and deterministic `Display`/`std::error::Error` contracts for public schema failures. - Rust 1.97.1 build contract, strict Clippy and rustdoc gates, and exact production function, line, region, and branch coverage enforcement. diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md index 69f603252..53c7e71a4 100644 --- a/docs/DOCUMENTATION_FITNESS.md +++ b/docs/DOCUMENTATION_FITNESS.md @@ -158,9 +158,9 @@ Active #64 makes a successful action-outcome value require existing verified pro ### 3.19 Controlled Agent Task fixture -Active #65 supplies a deterministic synthetic local web fixture with a labelled semantic input, submit control, same-document post-condition and explicitly hidden/untrusted prompt-injection text. The fixture contains no credential collection surface and requires no live third-party site. +Active #65 supplies a deterministic synthetic local web fixture with a labelled semantic input, submit control, same-document post-condition and explicitly hidden/untrusted prompt-injection text. Active #70 executes that fixture through real WebDriver on pinned Chrome with an isolated profile, an extension-disable launch request, synthetic input, same-document post-condition verification and profile cleanup. The launch request is recorded as configuration metadata and is not a verified browser surface without independent observation. If the second fixture server cannot start, the runner also closes the first one; cleanup attempts both servers even when one shutdown fails, so partial setup does not leak loopback threads. Active #71 verifies browser-computed role/name for the controlled input and submit button before action. The fixture contains no credential collection surface and requires no live third-party site. -**Resolution:** the fixture makes the future real Chromium vertical slice reproducible without turning a third-party site into a test dependency. It is not a browser adapter, semantic extractor, input dispatcher, policy engine, trusted clock, process-attribution source or proof of real Chromium execution. +**Resolution:** the #65/#70/#71 lane makes controlled browser-level and browser-computed role/name evidence reproducible without turning a third-party site into a test dependency. It is not a browser adapter, product semantic extractor, OriginWeave input-dispatch authority, policy engine, trusted clock, process-attribution source or proof of the shipped product runtime. ### 3.20 Bounded browser process-set resource evidence @@ -273,4 +273,4 @@ The graph becomes **PROTECTED-MAIN-SUFFICIENT** only when: 8. security, test, operability, privacy and release docs agree on shipped-vs-planned boundaries; and 9. this documentation reconciliation itself reaches protected main through repository governance and is re-evaluated against whatever feature heads actually integrated. -Until then, OriginWeave is **design-documented but not protected-main documentation-closed**. That finding must never be used as an excuse to stop unrelated safe implementation work. +Until then, OriginWeave is **design-documented but not protected-main documentation-closed**. That finding must never be used as an excuse to stop unrelated safe implementation work. \ No newline at end of file diff --git a/docs/TEST_STRATEGY.md b/docs/TEST_STRATEGY.md index ba3c31624..60e085fe0 100644 --- a/docs/TEST_STRATEGY.md +++ b/docs/TEST_STRATEGY.md @@ -78,6 +78,45 @@ session creation -> task close/recovery ``` +Draft PR #288 carries the current controlled Agent Task fixture lane on pinned +Chrome for Testing without owning the workflow that activates Chromium. The lane +uses browser-computed role/name evidence, real WebDriver clear/type/click, +browser-observed input state, URL-stability observation, exact synthetic echo and +profile cleanup. The controlled input deliberately starts with a non-empty +synthetic value, so the lane requires Element Clear to be followed by a browser +Get Element Property observation proving `value == ""` before Send Keys. It then +requires a second Get Element Property observation proving the synthetic typed +value after Send Keys and before click. A successful command acknowledgement for +either input action is insufficient. + +The lane also requires a browser-observed `#task-result` baseline +(`data-state=idle` and rendered `idle`) before clear/type, and observes the same +idle result baseline again after typing and submit-target semantic verification, +immediately before the native click. Successful evidence therefore carries +`pre_action_baseline_verified`, `clear_value_verified`, `input_value_verified`, +and `pre_click_baseline_verified`; all four are mandatory in repeatability surface +completeness. This prevents a pre-fired fixture, a no-op or incomplete clear, a +failed/partial send-keys operation accepted only from command ACK, or a regression +that pre-satisfies the post-condition during typing from being accepted as +click-caused success. + +URL stability is sampled immediately after the native click and again only after +the submitted-state and exact synthetic-echo post-condition has been observed. +The trial may emit `url_unchanged: true` only when both observations equal the +original controlled fixture URL. This keeps a delayed navigation from escaping +the accepted outcome boundary merely because an earlier post-click URL sample +was still unchanged. + +The two idle baselines, browser-observed clear/typed input values, post-condition +and URL observations are compared locally and unexpected page-controlled values +are not echoed into CI diagnostics. The lane remains active-PR fixture evidence, +not a shipped OriginWeave browser adapter. CSS locators are harness selectors; the +work does not establish OriginWeave semantic node authority, policy-authorized +production dispatch, WebDriver BiDi/CDP authority translation, or protected-main +runtime acceptance. Workflow/sandbox activation remains #212 authority and +ChromeDriver process/protocol diagnostics remain #148 authority. +Draft-policy-skipped CI/MV3 runs are not browser GREEN. + ### 3.5 Buyer acceptance Versioned task packs measure repeatable product outcomes rather than one lucky agent run. The benchmark artifact records browser build, OriginWeave version, model/provider/reasoning configuration, seed where supported, policy profile, hardware profile and source fixtures. diff --git a/docs/doctoring/chromium-sandbox-evidence-boundary.md b/docs/doctoring/chromium-sandbox-evidence-boundary.md new file mode 100644 index 000000000..2ac97666b --- /dev/null +++ b/docs/doctoring/chromium-sandbox-evidence-boundary.md @@ -0,0 +1,41 @@ +# Chromium sandbox evidence boundary + +Status: active verification evidence for PR #288. This note does not claim protected-main delivery or a passing browser gate. + +## Problem + +OriginWeave uses pinned real Chromium as compatibility and Agent Task evidence. A browser test that succeeds only after disabling Chromium process sandboxing does not prove the security posture expected from the governed-browser runtime. The transferred PR #288 generation originally inherited two real-browser launch paths in `scripts/ci/run_mv3_compatibility.py`: the Agent Task path kept sandboxing enabled, while the ordinary Manifest V3 compatibility path passed `--no-sandbox`. That source defect is repaired: both ordinary Manifest V3 and Agent Task real-browser paths omit `--no-sandbox`. + +Chromium's current Linux security guidance states that `--no-sandbox` disables critical security features and recommends installing/configuring a sandbox helper for developer builds instead. Chromium's Linux debugging guidance likewise says sandbox testing is needed on automated waterfall bots rather than routinely running them without the sandbox. The SUID sandbox development guidance documents the helper ownership/mode and `CHROME_DEVEL_SANDBOX` setup used when the normal user-namespace sandbox is unavailable. + +## Constraints and rejected alternatives + +The scheduled OriginWeave product writer does not own `.github/**`; workflow setup belongs to issue #212. Disabling Ubuntu/AppArmor restrictions runner-wide, retaining `--no-sandbox`, reducing browser trials, or treating ChromeDriver command acknowledgement as product success would weaken the evidence boundary and is rejected. + +Copying the complete #43 runner is also rejected. #43 contains unrelated downloads and diagnostic work. The reviewed causal precedent is narrower: commit `a45c83e4d8988fe89920ecb6a9eac469815f5b9b` removes the single `--no-sandbox` launch override and records that a sandbox-incompatible environment must fail instead of weakening Chromium isolation. + +## Selected repair path + +1. Keep `tests/test_mv3_browser_sandbox_contract.py` as the product-side invariant for both ordinary MV3 and Agent Task real-browser paths, including the documentation/source consistency guard. +2. Preserve the repaired ordinary `_run_browser_pass` launch without `--no-sandbox`; do not import unrelated #43 product delta. +3. Let #212 own the canonical workflow helper setup needed by the pinned Chrome for Testing archive, including root ownership/mode and environment wiring when the chosen Linux sandbox requires it. +4. Run the full repository contract suite on the repaired exact head, then execute fresh pinned-Chromium compatibility and Agent Task trials with sandboxing enabled. A skipped Draft job, command ACK, mock-fixture JSON, source inspection, or predecessor run is not GREEN browser evidence. + +## Exact active evidence + +- Protected base when the regression was recorded: `87c4daa1830bac5a5228b6036752ad5633232085`. +- Test-first RED generation for the launch defect: `101470a19b370bd30533ab3db330a882a2c25bc3`. +- Product repair commit: `6c072c3cd035319a57977d012007898c29602402`, which removes the ordinary `--no-sandbox` override. +- Diagnostic-restoration/source-verification generation: `0f434bc29d468127412366b6864b733b40b83c4d`; both real-browser option lists omit the forbidden override there. Its reported 173 Python/full-Rust/numeric-coverage verification is source/test evidence, not an actual Chromium execution; the `agent_task` JSON produced by the tests is a mock fixture. +- Documentation-consistency RED commit: `e74acd7b83aac2c7768ba21a2a9b5d9ea2b8a31b`; the new focused contract rejects the stale current-tense `--no-sandbox` statement that this revision repairs. +- Reviewed causal precedent: #43 commit `a45c83e4d8988fe89920ecb6a9eac469815f5b9b`. +- Canonical workflow-owner path: issue #212. +- Prior Agent Task hosted RED remains #70 MV3 run `33887386759`, job `101070423144`; it is predecessor evidence only and does not establish the current head as GREEN. + +## References + +The Chromium Authors. (2026). *AppArmor user namespace restrictions vs. Chromium developer builds*. Chromium source documentation. https://chromium.googlesource.com/chromium/src/+/main/docs/security/apparmor-userns-restrictions.md + +The Chromium Authors. (2026). *Linux SUID sandbox development*. Chromium source documentation. https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md + +The Chromium Authors. (2026). *Tips for debugging on Linux*. Chromium source documentation. https://chromium.googlesource.com/chromium/src/+/main/docs/linux_debugging.md diff --git a/docs/doctoring/mv3-compatibility.md b/docs/doctoring/mv3-compatibility.md index 571c49329..96ad5f7dc 100644 --- a/docs/doctoring/mv3-compatibility.md +++ b/docs/doctoring/mv3-compatibility.md @@ -1,13 +1,21 @@ # Manifest V3 compatibility evidence baseline - **Status:** Active implementation evidence for issue #27 -- **Reviewed:** 2026-08-11 +- **Reviewed:** 2026-09-03 - **Pinned browser:** Chrome for Testing `150.0.7871.129`, Chromium revision `r1639810` OriginWeave uses Chromium as its compatibility kernel, so browser-extension compatibility must be demonstrated with executable Chromium evidence rather than inferred from architecture alone. The protected-main lane exercises a controlled unpacked Manifest V3 extension against one exact Chrome for Testing build and proves service-worker, content-script, storage, declarative-network-request, tabs, windows, scripting, commands, side-panel, bookmarks/history read compatibility, restart persistence, repeatability, and one real WebDriver click/post-condition. Active stacked compatibility work adds downloads, bounded bookmark/history mutation, profile isolation, explicit extension update/version-migration evidence, and an exact content-script isolated-world check. OriginWeave does **not claim 100% Chrome extension compatibility**. The checked-in fixture is intentionally local-only. Its host permission is limited to loopback HTTP used by the deterministic test server. It contains no remote code, user credential, model call, external content, native-messaging host, or production PII. Chrome permissions remain distinct from the explicit OriginWeave extension-to-Agent grant implemented in `originweave-core`. Compatibility mutation tests create only controlled synthetic state inside the ephemeral test profile and must clean it up; successful API compatibility never grants the OriginWeave Agent ambient bookmarks/history/downloads authority. +The runner treats expected `http.client.HTTPException` transport failures, including truncated ChromeDriver responses, as failed trials and continues to emit bounded aggregate evidence. It does not classify such a run as successful: the repeatability gate still fails when the required trial count is not met. + +## Chromium sandbox evidence boundary + +Chromium's process sandbox is part of the browser security boundary, not a test decoration. Chromium's current sandbox library describes sandboxing as privilege/capability restriction for code that handles untrustworthy data, implemented with platform-specific OS isolation. Chromium's Linux debugging guidance treats `--no-sandbox` as a temporary debugging escape and explicitly warns against using it on waterfall bots because sandbox testing is required. + +PR #70 therefore separates compatibility evidence from security evidence. Its test-first contract rejects `--no-sandbox` in `_run_agent_task_browser_pass`. The pre-repair source failed that contract; repair commit `60b697095be510a129cfb61a3fd97790cf7a0679` removes exactly that Agent Task launch argument while leaving the separate MV3 compatibility pass unchanged. Chrome for Testing arrives as a raw archive, so the separately governed workflow activation must install its bundled `chrome_sandbox` helper with Chromium's required root ownership and `4755` mode before execution. This workflow-free slice verifies only the runner boundary; it does not assert that protected-main CI already performs that installation. A focused source-contract GREEN is not a substitute for browser execution: the repaired Agent Task lane remains Draft until the pinned sandbox-enabled Chromium job actually runs and preserves semantic role/name observation, native input/click, same-document post-condition, exact echo/URL, and profile cleanup. If a hosted environment cannot start sandboxed Chromium, that environment may support a narrowly labelled compatibility lane but cannot be promoted to governed-browser security evidence by weakening the assertion. + ## Supported-capability evidence matrix This matrix separates protected-main executable evidence from active, non-shipped evidence and from genuinely unproven surfaces. A row marked **ACTIVE_PR** is never a release claim; exact head/run provenance belongs in `docs/evidence/2026-08-10-active-pr-maturity.md` and must be refreshed when the branch changes. @@ -27,6 +35,7 @@ This matrix separates protected-main executable evidence from active, non-shippe | History add/read/delete lifecycle | **ACTIVE_PR #59** | Controlled synthetic loopback visit is added, exactly read back, deleted in `finally`, and required to be absent afterward. | Compatibility only; no Agent history capability. | | Downloads | **ACTIVE_PR #43** | Controlled loopback payload is downloaded and validated through pinned Chromium. | No general download persistence, unsafe filename, or Agent filesystem authority claim. | | Per-trial Agent Task profile isolation | **ACTIVE_PR #49** | Compatibility trials use isolated ephemeral profiles rather than ambient human state. | Full production Agent Task browser orchestration remains issue #28 work. | +| Agent Task Chromium sandbox preservation | **ACTIVE_PR #70** | Source contract rejects `--no-sandbox`; exact repair removes it only from Agent Task launch. | Sandbox-enabled pinned-browser exact-head execution is still required before security evidence is claimed; MV3 compatibility launch remains a separate lane. | | Extension update/version migration | **ACTIVE_PR #60** | Trial-local extension copy transitions `1.0.0` → `1.0.1` on the same ephemeral profile; versioned storage state is required to migrate and real pinned-Chromium evidence reports the update-migration surface. | No Chrome Web Store updater, enterprise deployment channel, arbitrary downgrade, or protected-main release claim. | | Managed enterprise extension policy | **PLANNED** | No protected-main executable compatibility proof yet. | Do not infer managed-policy support from Chromium ancestry alone. | | Native messaging | **PLANNED / SECURITY-GATED** | No compatibility claim. | Future support requires an explicit host-managed allow-list and process boundary. | @@ -52,6 +61,12 @@ The CI lane downloads the exact Chrome/ChromeDriver version from the official Ch ## Primary references — APA 7th +Chromium Authors. (n.d.). *Sandbox library*. Chromium source. Retrieved September 3, 2026, from https://chromium.googlesource.com/chromium/src/sandbox/ + +Chromium Authors. (n.d.). *Linux SUID sandbox development*. Chromium source. Retrieved September 4, 2026, from https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md + +Chromium Authors. (n.d.). *Tips for debugging on Linux*. Chromium source. Retrieved September 3, 2026, from https://chromium.googlesource.com/chromium/src/+/main/docs/linux/debugging.md + Chrome for Developers. (n.d.). *Extensions / Manifest V3*. Google. Retrieved August 9, 2026, from https://developer.chrome.com/docs/extensions/develop/migrate/what-is-mv3 Chrome for Developers. (2023, May 2). *Extension service worker basics*. Google. https://developer.chrome.com/docs/extensions/develop/concepts/service-workers/basics diff --git a/docs/traceability/action-postcondition-evidence.md b/docs/traceability/action-postcondition-evidence.md index 23a6e764d..cbbea7181 100644 --- a/docs/traceability/action-postcondition-evidence.md +++ b/docs/traceability/action-postcondition-evidence.md @@ -55,6 +55,24 @@ On that unchanged exact head, CI run `31445201739` succeeds; Rust contracts job This remains controlled test infrastructure rather than browser-execution evidence. The fixture itself does not establish WebDriver BiDi/CDP transport, Chromium semantic extraction, policy dispatch, native input, post-condition provenance, profile teardown or process attribution. +### PR #70 — pinned Chrome execution of the controlled Agent Task fixture + +**Capability maturity:** `IMPLEMENTED_ON_ACTIVE_PR` + +PR #70 reuses the existing pinned Chrome for Testing workflow and executes the #65 fixture through loopback ChromeDriver with extensions disabled and a fresh temporary profile. Each bounded trial performs real WebDriver clear/type/click operations, observes the `submitted` state and synthetic value through element endpoints, verifies that submission preserves the loaded URL, and proves that the temporary profile is removed after teardown. The fixture-shutdown contract also exercises successful MV3 and Agent Task trial paths before proving reverse-order cleanup after one server stop fails. The runner emits credential-free repeatability evidence and fails the lane when any trial or post-condition is incomplete. + +Security review `5098613778` found that the Agent Task lane had been launched with Chromium `--no-sandbox`, so predecessor browser success is compatibility evidence rather than production-security evidence. The test-only sandbox contract then failed against the unchanged launch source. Exact repair commit `60b697095be510a129cfb61a3fd97790cf7a0679` removes only the Agent Task lane's `--no-sandbox` argument; the separate MV3 compatibility pass is intentionally unchanged. The focused source contract now passes, but a sandbox-enabled pinned-Chromium exact-head run is still required before this active PR can claim repaired browser security evidence. + +This is real WebDriver evidence for a controlled local fixture, not a product browser adapter. It does not establish WebDriver BiDi/CDP authority translation, OriginWeave semantic observation or node handles, policy-authorized typed action dispatch, trusted browser-process attribution, or protected-main product runtime completion. + +### PR #71 — browser-computed semantic role/name evidence before action + +**Capability maturity:** `IMPLEMENTED_ON_ACTIVE_PR` + +PR #71 extends the pinned-Chrome fixture lane by reading WebDriver's browser-computed role and accessible name for the controlled input and submit button before sending input or clicking. The exact expected values are `textbox` / `Task text` and `button` / `Submit task`; the repeatability gate requires both semantic checks in every successful trial, and the cleanup regression double supplies both success surfaces so teardown failures cannot mask incomplete evidence. + +This is bounded browser-computed evidence for a synthetic test target, not the OriginWeave semantic observation adapter. CSS locators remain test-harness selectors, and the lane does not create OriginWeave node handles, source-channel provenance, policy authority, or permission to execute page-advertised actions. + ## 4. Non-transitive success semantics The intended first-slice chain is: @@ -77,10 +95,10 @@ Unverified -/> successful action completion Rejected -/> successful action completion caller-supplied timestamp ordering -/> proof of trusted clock provenance VerifiedActionOutcomeEvidence type existence -/> proof of real Chromium execution -controlled fixture success -/> proof of real Chromium execution +controlled fixture success -/> proof of an OriginWeave product browser runtime ``` -PR #64 now rejects a caller-supplied observation timestamp that predates caller-supplied dispatch time, but the type cannot independently prove the clock source, that a real browser actually dispatched the action, that the supplied provenance belongs to the claimed browser target/node, or that the observed state was caused by that action. PR #65 supplies deterministic hostile input and a post-condition target but no browser execution. Those claims remain the responsibility of the real adapter/runtime composition under issue #28. +PR #64 now rejects a caller-supplied observation timestamp that predates caller-supplied dispatch time, but the type cannot independently prove the clock source, that a real browser actually dispatched the action, that the supplied provenance belongs to the claimed browser target/node, or that the observed state was caused by that action. PR #70 proves real Chromium execution against the controlled fixture and PR #71 adds browser-computed role/name evidence, but their test-harness CSS locators and direct WebDriver calls are not the OriginWeave adapter/runtime composition required under issue #28. ## 5. Active prerequisite graph for issue #28 @@ -93,15 +111,17 @@ The first real Chromium vertical slice remains distributed across bounded active - PR #49 — ephemeral compatibility-profile lifecycle regression stacked on #43; - PR #51 — bounded browser-task telemetry plus one explicitly supplied Linux PID `VmRSS` sampler; Chromium process discovery/process-set attribution remains outside that slice; - PR #64 — verified and caller-timestamp-ordered post-condition action-outcome evidence; and -- PR #65 — controlled hostile local Agent Task workflow fixture, gate-clean and Ready for review. +- PR #65 — controlled hostile local Agent Task workflow fixture; and +- PR #70 — real WebDriver execution of that fixture on pinned Chrome, without claiming a product browser adapter; and +- PR #71 — browser-computed role/name evidence before controlled action, without claiming a product semantic observer. -These active PRs are non-shipped evidence. They do not themselves compose WebDriver BiDi/CDP transport, trusted Chromium process attribution, policy-authorized real input dispatch, causal post-condition observation, or deterministic end-to-end teardown/recovery into one protected-main runtime. +These active PRs are non-shipped evidence. PR #70/#71 prove bounded browser-level and semantic evidence, but the active set does not itself compose WebDriver BiDi/CDP transport, OriginWeave authority translation, trusted Chromium process attribution, policy-authorized real input dispatch, causal post-condition observation, or deterministic end-to-end teardown/recovery into one protected-main runtime. ## 6. Remaining issue #28 boundary This dossier does **not** close issue #28. Material remaining work includes: -- pinned stock Chromium exercised as one reproducible end-to-end Agent Task runtime path, not only extension compatibility fixtures; +- a production Agent Task runtime path that composes pinned stock Chromium with OriginWeave authority, rather than only the controlled #70 fixture and extension compatibility fixtures; - isolated Agent Task profile/context lifecycle and cleanup in the production vertical path; - versioned WebDriver BiDi adapter plus explicitly bounded CDP observation fallback where needed; - real semantic observation feeding typed query and policy-authorized typed action; @@ -113,4 +133,4 @@ This dossier does **not** close issue #28. Material remaining work includes: ## 7. Documentation fitness consequence -The ADR/PRD/TRD/Architecture/UML/ERD graph remains **DESIGN-SUFFICIENT / PROTECTED-MAIN-PARTIAL**. PR #64 narrows a typed evidence gap already governed by existing provenance/action-success decisions, while PR #65 supplies controlled test infrastructure for the eventual real-browser proof. Neither introduces a new trust domain, deployed component, persistence owner, database schema, or independent architecture decision, so a new ADR or physical ERD entity would overstate the implementation. Detailed real-Chromium dispatch/post-condition sequence diagrams should be reconciled when the executable adapter chain stabilizes rather than manufacturing as-built detail before that runtime exists. +The ADR/PRD/TRD/Architecture/UML/ERD graph remains **DESIGN-SUFFICIENT / PROTECTED-MAIN-PARTIAL**. PR #64 narrows a typed evidence gap, PR #65 supplies the controlled fixture, and PR #70/#71 supply real WebDriver and browser-computed semantic evidence for that fixture. None of these introduces a new trust domain, deployed component, persistence owner, database schema, or independent architecture decision, so a new ADR or physical ERD entity would overstate the implementation. Detailed real-Chromium dispatch/post-condition sequence diagrams should be reconciled when the executable adapter chain stabilizes rather than manufacturing as-built detail before that runtime exists. diff --git a/docs/traceability/agent-task-action-transition-evidence.md b/docs/traceability/agent-task-action-transition-evidence.md new file mode 100644 index 000000000..bbf497589 --- /dev/null +++ b/docs/traceability/agent-task-action-transition-evidence.md @@ -0,0 +1,85 @@ +# Agent Task action-transition evidence + +- **Status:** Active-PR evidence; not protected-main shipped truth +- **Canonical browser-domain owner:** OriginWeave +- **Active PR:** #288 +- **Protected base:** `main@87c4daa1830bac5a5228b6036752ad5633232085` +- **Initial causal observation repair:** `8b7aa28ecf7acb1e3f3b2dcadd4cb3cbf59ea01f` +- **Initial causal acceptance repair:** `e1dd50999cd3a52977179047e8d5d77a2e85aef9` +- **Immediate pre-click RED:** `07540f0cdb3178998d305382d9176cccaeabce57` +- **Immediate pre-click repair:** `942e4c1a44119384d01ee4c7ec4168e6c5ab38b5` +- **Accepted-outcome URL RED:** `4306f38c45f65a7fba7ccd842ae42b2ed4646e29` +- **Accepted-outcome URL repair:** `cac13ab79cd41ca55315622c3148126bee5643f8` +- **Typed-input observation RED:** `ac78a46dd550e64993e48571d72eee2d8bf2d0f5` +- **Typed-input observation repair:** `dfbc0f13e0e7c4f381fc333f95c41127583bfc7a` +- **Clear post-condition RED:** `7081d021def0d740a86771b5b1b77ff36885e433` +- **Clear post-condition repair:** `1fa16927a33cb0a151aa479587bb54654e48b538` +- **Successful evidence-double alignment:** `f6c43aaad5b1c0acf7e3ca04a3f9ec872d08c83b` +- **Workflow/sandbox owner:** #212 +- **WebDriver protocol-diagnostic owner:** #148 + +## Problem + +The controlled Agent Task lane already used browser-computed role/name evidence, native WebDriver clear/type/click commands, page-observed transition state, exact synthetic text echo, URL stability, and profile cleanup. The first causal repair added an `idle` / `idle` observation before the input sequence, preventing a fixture that was already successful at navigation time from satisfying the buyer gate. + +That was still insufficient to attribute the final transition specifically to the submit click. A fixture regression could remain idle at the first observation, mutate `#task-result` while the WebDriver value command types the synthetic input, and then present `submitted` plus the expected echo before the click. The later post-condition would still look successful even though the click did not cause the transition. + +A separate timing gap remained after that repair: the runner sampled `post_submit_url` immediately after click, before it verified the submitted state and exact echo. A click handler could synchronously satisfy those post-conditions, schedule a navigation, let the early URL sample remain unchanged, and still be accepted without proving that the controlled URL remained stable through the accepted outcome boundary. + +The typing path also needed an independent browser observation. The runner issued the WebDriver element value command and later checked the submitted result echo, but a controlled-fixture regression could hard-code the expected output and allow a failed or partial send-keys operation to be accepted from command acknowledgement alone. The typed-value repair therefore reads the input element's `value` property after typing and before click. + +One earlier interaction in the same sequence remained acknowledgement-only. The controlled fixture deliberately starts `#task-text` with the non-empty synthetic value `synthetic order 42`. The runner issued Element Clear and immediately issued Element Send Keys without observing the value after clear. A no-op or incomplete clear could therefore be accepted from command completion even though the fixture provides a deterministic post-condition that can prove the clear action itself. + +## Decision + +The controlled fixture has one canonical unsuccessful result state: `#task-result` is `data-state="idle"` and rendered text `idle`. The runner observes that state twice through WebDriver: once before clear/type and again after typing plus submit-target semantic verification, immediately before the native click. Both observations reuse `_validate_agent_task_pre_action_state`, so unexpected page-controlled values fail closed without being serialized into diagnostics. + +The controlled text input also has explicit action post-conditions. Immediately after Element Clear, the runner uses Get Element Property on `value` and requires the browser-observed value to be the empty string before any send-keys command is issued. `_validate_agent_task_cleared_value` emits only `Agent Task clear verification failed` on mismatch; the observed input value is not serialized. Successful evidence carries `clear_value_verified: true`, and surface completeness requires that witness in every successful trial. + +After Element Send Keys, the runner again uses Get Element Property for the same controlled input and requires the browser-observed `value` to equal `AGENT_TASK_INPUT_VALUE`. `_validate_agent_task_typed_value` emits only `Agent Task typed input verification failed` on mismatch. Successful evidence carries `input_value_verified: true`, and repeatability surface completeness requires that witness independently of the clear witness. + +Only after the empty-after-clear witness, typed-value witness, and second browser-observed idle baseline succeed does the runner issue the WebDriver click. The runner retains the immediate post-click URL check, then verifies `data-state="submitted"` and the exact synthetic echo, and finally reads the browser URL again before returning successful evidence. `url_unchanged` is true only when both post-click URL observations equal the original controlled fixture URL. The final mismatch uses the closed diagnostic `Agent Task URL changed before accepted outcome` and does not serialize the observed page URL. + +Successful trial evidence therefore carries `pre_action_baseline_verified: true`, `clear_value_verified: true`, `input_value_verified: true`, and `pre_click_baseline_verified: true`; `_agent_task_surfaces_complete` requires all four witnesses in every successful trial. The accepted observed sequence is `idle before input → native clear → browser-observed empty value → native type → browser-observed typed value → idle immediately before click → click → immediate URL check → submitted/exact echo → accepted-outcome URL check`, rather than inferring success from command acknowledgement, from a post-condition that may already have been true, or from a URL sample that precedes the accepted outcome. + +The baseline values, input properties, and observed URLs are used only for local comparison. Unexpected page-controlled state, text, input values, or URLs are never serialized into CI diagnostics. + +## Test-first evidence + +The regression sequence is intentionally non-destructive: + +- `b9707975a605347b573b992cfe178150feda6a95` introduced the original causal-transition contract. +- `651d7fe89a1e6ebd811607683eb50f3b17a5822e` pinned the fixture's actual `idle` / `idle` baseline. +- `4d61c2f82a82048727980f6638b7e95e93699fd6` first required the baseline witness to propagate into per-trial evidence without changing the existing gate. +- `8b7aa28ecf7acb1e3f3b2dcadd4cb3cbf59ea01f` added the first runner observation repair. +- `42a9a129ff80edc698c3c097a049246d538c63bc` strengthened the regression so a trial lacking that baseline witness cannot satisfy surface completeness. +- `ed6af6ebf825a1571f16aaf1bc1d1bfdea4327a4` aligned the then-current successful-trial doubles. +- `e1dd50999cd3a52977179047e8d5d77a2e85aef9` made `pre_action_baseline_verified is True` mandatory in `_agent_task_surfaces_complete`. +- `07540f0cdb3178998d305382d9176cccaeabce57` added the second-baseline RED after the WebDriver value command and before click. +- `942e4c1a44119384d01ee4c7ec4168e6c5ab38b5` repaired that gap by re-observing the existing result element immediately before click and requiring `pre_click_baseline_verified`. +- `3d3166ec1e3a7c5aaee1f2dae92f09a7acc294f4` and `efca7d69c3bbc459bc218b9142a6dd4e58828076` aligned successful evidence doubles with the stronger two-baseline contract instead of weakening the predicate. +- `4306f38c45f65a7fba7ccd842ae42b2ed4646e29` added the accepted-outcome URL-order RED: after the exact echo is accepted, a browser URL read must still occur before successful evidence is returned. +- `cac13ab79cd41ca55315622c3148126bee5643f8` performed the minimum URL repair by retaining the immediate post-click URL check and adding a second URL equality observation after submitted-state/exact-echo verification. +- `ac78a46dd550e64993e48571d72eee2d8bf2d0f5` added the typed-input RED: command acknowledgement is insufficient until the browser-observed input `value` property is read and validated before click, and surface completeness requires the resulting witness. +- `dfbc0f13e0e7c4f381fc333f95c41127583bfc7a` implemented the typing repair with Get Element Property, a closed/non-echoing validator, `input_value_verified`, trial propagation, and mandatory completeness. +- `7081d021def0d740a86771b5b1b77ff36885e433` added the clear-action RED: the controlled fixture's preloaded input makes Element Clear independently observable, so a browser-observed empty `value` must occur after `/clear` and before `/value`, and surface completeness must require `clear_value_verified`. +- `1fa16927a33cb0a151aa479587bb54654e48b538` implemented the minimum clear repair with Get Element Property, a closed/non-echoing validator, trial propagation, and mandatory completeness without changing the fixture or browser action sequence. +- `f6c43aaad5b1c0acf7e3ca04a3f9ec872d08c83b` aligned existing successful-trial doubles with both already-required `input_value_verified` and the new `clear_value_verified` witness rather than weakening the acceptance predicate. + +These commits do not change browser version, trial denominator, semantic-target checks, click/post-condition semantics, cleanup, workflow, sandbox configuration, extension-isolation semantics, or #148 protocol-diagnostic authority. + +Because #288 is Draft, CI and Manifest V3 Compatibility may skip before executing this exact lineage. A source-semantic/test-first RED or code inspection is not a substitute for a fresh pinned-Chromium run. Browser acceptance still requires the #212 workflow/sandbox owner path to execute the unchanged three-trial lane on the exact successor head. + +## Standards traceability + +The latest published WebDriver draft remains **W3C Working Draft, 2 July 2026** as of 9 September 2026. It defines WebDriver as an out-of-process browser-control protocol and separately defines Get Element Property, Element Clear, Element Send Keys, element-state retrieval, and current-URL retrieval commands. The controlled Agent Task lane uses those commands as transport-level observation and interaction mechanisms; OriginWeave's stronger causal acceptance rule is a product evidence invariant layered above the protocol. WebDriver command completion alone does not establish OriginWeave task success. + +For the clear boundary, the runner performs Element Clear and then separately performs Get Element Property for `value`; it does not infer an empty value from successful completion of the clear command. It repeats Get Element Property after Element Send Keys to prove the controlled typed value before click. Get Current URL is observed before action, immediately after click, and again after the accepted state/echo observation. These reads bind interaction evidence to browser-observed state rather than to command-completion instants. + +### APA 7th + +World Wide Web Consortium. (2026, July 2). *WebDriver* (W3C Working Draft). https://www.w3.org/TR/2026/WD-webdriver2-20260702/ + +## Remaining acceptance + +This repair does not claim a production Agent Task adapter, WebDriver BiDi authority translation, semantic-node authority, policy authorization, or sandbox-enabled Chromium GREEN. Those remain separate bounded responsibilities. The next browser acceptance must preserve the protected workflow generation, least-privilege sandbox setup, Chrome/ChromeDriver `150.0.7871.129`, three independent trials, both idle baselines, the browser-observed empty-after-clear witness, browser-observed typed-value witness, both post-click URL observations, page-derived diagnostic redaction, session/profile cleanup, and exact-head evidence. \ No newline at end of file diff --git a/docs/traceability/fixture-server-shutdown-postcondition.md b/docs/traceability/fixture-server-shutdown-postcondition.md new file mode 100644 index 000000000..00a4ef057 --- /dev/null +++ b/docs/traceability/fixture-server-shutdown-postcondition.md @@ -0,0 +1,53 @@ +# Fixture-server shutdown post-condition + +## Problem + +The controlled MV3 and Agent Task browser lanes own two loopback `ThreadingHTTPServer` instances. `_stop_fixture_server` previously called `shutdown()`, `server_close()`, and `thread.join(timeout=5)` and then returned without observing whether the helper thread had actually terminated. A timed `join` is only a bounded wait; it does not itself prove the target thread stopped. That left cleanup evidence vulnerable to the same acknowledgement-versus-post-condition error that the browser action lane already rejects. + +A second evidence-ordering defect remained after the liveness repair: `main()` serialized and printed a success-shaped compatibility document before either fixture server entered its shutdown `finally` block. A later cleanup failure therefore produced a non-zero process result but could leave an already-emitted JSON document that looked like completed browser acceptance. For evidence consumers, successful publication must be downstream of owned teardown rather than merely adjacent to it. + +## Constraints + +- Keep both fixture servers loopback-only and preserve the existing reverse-order cleanup attempt. +- Do not add retries, unbounded waits, process-wide thread manipulation, or workflow changes. +- Do not weaken the three-trial browser evidence denominator or sandbox requirements. +- Do not move WebDriver/ChromeDriver protocol diagnostic authority out of PR #148 or workflow/sandbox authority out of issue #212. +- Cleanup diagnostics must remain stable and must not include filesystem paths, page content, credentials, or remote-driver text. +- Preserve bounded JSON evidence for a browser/trial gate failure even though successful publication moves behind cleanup. + +## Decision + +After the existing five-second `join`, `_stop_fixture_server` checks `thread.is_alive()`. If the helper is still alive, cleanup fails closed with the fixed diagnostic `fixture server thread did not stop`. + +A focused regression uses a stalled thread double to prove that `shutdown()`, `server_close()`, and the bounded join are attempted but are not accepted as successful cleanup unless liveness becomes false. A companion success case proves that an observed stopped thread is accepted. + +Successful compatibility JSON is now emitted only after both reverse-order fixture shutdown calls return successfully. Gate-failure paths still emit their bounded evidence immediately before raising, so the diagnostic artifact for an unsuccessful browser/trial run is not lost. If a run otherwise satisfies the browser gates but either owned fixture server fails cleanup, no success-shaped JSON is published. The success-path `duration_ms` is refreshed after teardown so its measured interval includes the owned fixture-server shutdown boundary. + +This follows Python's documented `Thread.join(timeout)` contract: `join()` returns `None` whether the target terminated or the timeout expired, so callers must inspect `is_alive()` after a timed join to determine whether the timeout occurred. + +## Alternatives rejected + +An unbounded `join()` was rejected because a failed fixture server could hang the CI lane indefinitely. Repeating `shutdown()` or sleeping before a second join was rejected because it would obscure the causal cleanup defect and add timing-dependent behavior. Silently recording the thread as cleaned after a timed join was rejected because command completion is not a post-condition. + +Publishing success JSON before cleanup and relying only on the process exit code was rejected because the artifact itself is a first-class evidence surface and may be retained or inspected separately from the runner status. Suppressing all pre-cleanup JSON was also rejected because failed browser/trial gates still need bounded diagnostic evidence. The chosen ordering distinguishes unsuccessful diagnostic publication from successful acceptance publication without changing browser behavior. + +## Risk and effect + +The liveness check can turn a previously silent helper-thread leak into an explicit test failure. The publication-order repair can remove a success-shaped JSON artifact from runs that would previously have printed it and then failed during teardown. Both effects are intentional: a browser evidence run is not complete while an owned fixture service remains live. The changes do not alter browser navigation, semantic observation, native interaction, sandbox configuration, profile cleanup, workflow topology, or trial counts. + +## Exact evidence + +- Fixture liveness test-first commit: `586bf780a6ebeb565feb0e5325afae3053937496`. +- Fixture liveness minimal production repair: `aed62d356721ee5abf3cc84414c87bea15eaa09e`. +- Success-publication test-first commit: `2c7e5df3c13229614419733491cdf85b0d1838eb`. +- Success-publication minimal production repair: `d3de2a26c51f2d0dbf2362c8da36330cf5b2b443`. +- Review finding anchoring the publication defect: `5148970300` on exact predecessor `63d57081cb591a93661e483a9fb7b7712adb3b2d`. +- The new publication regression proves three boundaries: successful evidence follows both fixture stops; a cleanup failure leaves no success-shaped evidence; and a browser/trial gate failure retains bounded evidence before raising. + +## Primary runtime reference + +Python Software Foundation. (2026). *threading — Thread-based parallelism* (Python 3.14.7 documentation). https://docs.python.org/3/library/threading.html#threading.Thread.join + +## Follow-up acceptance + +Keep PR #288 Draft until exact-head repository gates execute and issue #212 supplies the authorized current-generation sandbox-helper workflow. The pinned Chrome/ChromeDriver `150.0.7871.129` replay must still prove the existing browser surfaces and cleanup. Successful JSON publication is now downstream of fixture-server teardown, but Draft-policy skips remain non-evidence and are not executable GREEN. \ No newline at end of file diff --git a/docs/traceability/webdriver-response-diagnostic-boundary.md b/docs/traceability/webdriver-response-diagnostic-boundary.md new file mode 100644 index 000000000..6d9a8c877 --- /dev/null +++ b/docs/traceability/webdriver-response-diagnostic-boundary.md @@ -0,0 +1,52 @@ +# WebDriver response diagnostic boundary + +## Problem + +The controlled MV3/Agent Task evidence runner consumes WebDriver Classic HTTP responses from the local ChromeDriver remote end. Before this repair, `_json_request` copied the complete body of an HTTP error and the remote `value.error` / `value.message` fields into `RuntimeError`. `_wait_for_driver` then copied the last exception string into its timeout error. Those values are useful for protocol conformance at the remote boundary, but they are not trustworthy CI log content: the W3C WebDriver error object explicitly contains implementation-defined `message` and `stacktrace` text and may carry additional `data`, including user-prompt text. + +The same boundary applies to successful New Session capability data. W3C WebDriver defines `browserVersion` as a standard capability returned by the remote end to identify the user-agent version. OriginWeave must compare that value against the pinned Chrome version, but a mismatched remote value is comparison state rather than CI diagnostic payload. + +The browser evidence lane therefore had a provenance mismatch. Remote response data was allowed to decide failure and also become operator-visible diagnostic text without an explicit redaction boundary. + +## Constraints + +- Keep non-success HTTP status and defensive error-shaped JSON responses fail-closed. +- Keep the existing response-size bound and JSON/object validation. +- Keep exact equality with `PINNED_CHROME_VERSION` mandatory in both MV3 and Agent Task browser passes. +- Do not serialize the remote `browserVersion` value into version-mismatch diagnostics. +- Do not reinterpret remote response text or capabilities as browser policy, product authority, or success evidence. +- Do not add ChromeDriver process/startup reason classification here; PR #148 remains the canonical owner of that diagnostic semantics. +- Do not change Chrome arguments, sandboxing, browser version, workflow activation, retry behavior, or the three-trial denominators. +- Do not weaken browser-observed action, post-condition, URL, session/profile, fixture-cleanup, or success-publication evidence. + +## Alternatives + +1. Preserve raw WebDriver bodies/messages or mismatched capability values in CI and attempt pattern-based secret filtering. Rejected because remote fields are open-ended data; a denylist cannot establish a closed disclosure boundary. +2. Allowlist W3C error codes and publish them. Rejected for this slice because error-code interpretation overlaps the richer #148 protocol/startup diagnostic authority and is unnecessary to preserve fail-closed behavior. +3. Keep only the bounded local HTTP status for transport-level failures, use fixed diagnostics for remote response/capability failures, and retain the remote values only for local comparison. Selected because it preserves each failure decision while preventing remote payload serialization and keeps this runner generic. + +## Decision and test-first evidence + +Test-first commit `8faef5967d8df770e9fc84ba358846ecd4dd1062` extends `tests/test_mv3_page_diagnostic_redaction_contract.py` with three hostile-response contracts: + +- a non-success WebDriver HTTP body containing `buyer-secret-marker-must-not-reach-ci` must fail as `WebDriver HTTP request failed with status 500` without retaining the body; +- a defensive error-shaped JSON object on a success-status response must fail as `WebDriver command failed` without retaining the remote `error` or `message` fields; +- driver-readiness timeout must be `ChromeDriver did not become ready` even if the last request exception contains the hostile marker. + +Exact predecessor production still serialized those values, so this commit is a source-semantic RED. Draft admission prevents representing it as a hosted executed RED. Corrective test-only commit `356764db9ef6fe53c0bba2f02c3f01c58a80d845` makes the second case explicit rather than inaccurately describing an HTTP 200 response as a conforming W3C error response. + +Production commit `4f09563520c9ba8565fa2c269db3a3de45fa7ca0` makes the minimum generic response repair: HTTP failures retain only the numeric status, an error-shaped decoded JSON object uses one fixed command-failure diagnostic, and readiness timeout no longer interpolates the last exception. Response bounds, JSON validation and failure behavior remain intact. + +Test-first commit `d682b641cfa30e0040c1b445c6aa45d211cdf0a8` then injects the hostile marker as the remote `browserVersion` in both browser-pass variants. The predecessor still performed the required pinned-version comparison but echoed the mismatched capability into `RuntimeError`, establishing the next source-semantic RED. Production commit `32953588084d72eb669ed53b02ccc7d5264dc00e` preserves both exact equality checks and replaces only their diagnostics with `unexpected Chrome version` and `unexpected Agent Task Chrome version`. + +## Risks and follow-up + +The tighter diagnostic vocabulary intentionally removes remote troubleshooting text from ordinary CI. When ChromeDriver startup/process causality is needed, the bounded diagnostic mechanism owned by PR #148 must supply typed evidence rather than reopening raw remote-text logging here. + +This change is not browser acceptance. Exact-head repository gates and sandbox-enabled pinned-Chromium three-trial execution still depend on the current workflow/sandbox owner path in issue #212. A skipped Draft workflow is not GREEN evidence. + +## Primary standard + +World Wide Web Consortium. (2026, July 2). *WebDriver* (Working Draft). https://www.w3.org/TR/webdriver2/ + +Section 6.6 defines conforming WebDriver errors as HTTP 4xx/5xx responses with a JSON `value` object containing an error code plus implementation-defined `message` and `stacktrace`, with optional additional `data`. Section 7 defines `browserVersion` as a standard string capability identifying the user-agent version and states that the remote end uses capabilities to describe the session feature set. OriginWeave therefore treats descriptive error fields and remote capability values as untrusted observations: they can cause a failed result but are not serialized into the CI diagnostic surface. diff --git a/scripts/ci/run_mv3_compatibility.py b/scripts/ci/run_mv3_compatibility.py index 28a3fb1e2..d9dcd5883 100755 --- a/scripts/ci/run_mv3_compatibility.py +++ b/scripts/ci/run_mv3_compatibility.py @@ -1,17 +1,20 @@ #!/usr/bin/env python3 -"""Run bounded repeatable Manifest V3 compatibility evidence against pinned Chromium. +"""Run bounded repeatable real-browser evidence against pinned Chromium. This is a release/CI evidence runner, not a product browser adapter. It uses the W3C WebDriver HTTP protocol only to prove that a real Chrome for Testing build can load the controlled MV3 fixture and repeatedly exercise service-worker, content-script, storage, declarative-net-request, tabs, windows, scripting, commands, side-panel, bookmarks, history, real browser-click, and -restart-persistence behavior. +restart-persistence behavior. It also executes the controlled Agent Task fixture +while requesting extensions disabled in a fresh profile, verifies browser-computed +role/name for the controlled action targets, performs real WebDriver input and +click operations, verifies the observable post-condition, and proves profile +cleanup without treating page content as instruction or authority. """ from __future__ import annotations -import contextlib import http.client import http.server import json @@ -20,6 +23,7 @@ import socket import string import subprocess +import sys import tempfile import threading import time @@ -27,9 +31,12 @@ ROOT = pathlib.Path(__file__).resolve().parents[2] FIXTURE = ROOT / "tests" / "fixtures" / "mv3_basic" +AGENT_TASK_FIXTURE = ROOT / "tests/fixtures/agent_task_basic" PINNED_CHROME_VERSION = "150.0.7871.129" PINNED_CHROME_REVISION = "r1639810" REPEATABILITY_TRIALS = 3 +AGENT_TASK_REPEATABILITY_TRIALS = 3 +AGENT_TASK_INPUT_VALUE = "originweave controlled input" REQUEST_TIMEOUT_SECONDS = 5.0 STARTUP_TIMEOUT_SECONDS = 20.0 FIXTURE_TIMEOUT_SECONDS = 20.0 @@ -45,6 +52,34 @@ def log_message(self, _format: str, *args: object) -> None: """Suppress request logs because the fixture contains no diagnostic value.""" +class BrowserSessionCleanupError(RuntimeError): + """Report bounded WebDriver-session cleanup failure without echoing remote text.""" + + def __init__(self, cleanup_error: BaseException) -> None: + self.cleanup_error_type = type(cleanup_error).__name__ + super().__init__( + "WebDriver session cleanup failed; see the chained causal browser failure" + ) + + +class BrowserProfileCleanupError(RuntimeError): + """Report bounded profile cleanup failure without exposing filesystem details.""" + + def __init__(self, cleanup_error: BaseException) -> None: + self.cleanup_error_type = type(cleanup_error).__name__ + super().__init__( + "browser profile cleanup failed; see the chained causal browser failure" + ) + + +class AgentTaskSessionStartError(RuntimeError): + """Classify a failed Agent Task browser session without exposing driver text.""" + + def __init__(self, session_error: BaseException) -> None: + self.session_error_type = type(session_error).__name__ + super().__init__("Agent Task browser session failed to start") + + def _free_loopback_port() -> int: """Reserve and release one loopback TCP port for a short-lived local service.""" @@ -108,8 +143,9 @@ def _json_request( if len(raw) > MAX_WEBDRIVER_RESPONSE_BYTES: raise RuntimeError("WebDriver response exceeded the bounded JSON limit") if response.status >= 400: - detail = raw.decode("utf-8", errors="replace") - raise RuntimeError(f"WebDriver HTTP {response.status}: {detail}") + raise RuntimeError( + f"WebDriver HTTP request failed with status {response.status}" + ) finally: connection.close() @@ -118,7 +154,7 @@ def _json_request( raise RuntimeError("WebDriver returned a non-object JSON payload") value = decoded.get("value") if isinstance(value, dict) and value.get("error"): - raise RuntimeError(f"WebDriver error: {value.get('error')}: {value.get('message')}") + raise RuntimeError("WebDriver command failed") return decoded @@ -126,16 +162,15 @@ def _wait_for_driver(driver_port: int) -> None: """Wait for the exact local ChromeDriver process to become ready.""" deadline = time.monotonic() + STARTUP_TIMEOUT_SECONDS - last_error: Exception | None = None while time.monotonic() < deadline: try: status = _json_request(driver_port, "GET", "/status", timeout=1.0) if status.get("value", {}).get("ready") is True: return - except (OSError, ValueError, RuntimeError, json.JSONDecodeError) as exc: - last_error = exc + except (OSError, ValueError, RuntimeError, json.JSONDecodeError): + pass time.sleep(0.1) - raise RuntimeError(f"ChromeDriver did not become ready: {last_error}") + raise RuntimeError("ChromeDriver did not become ready") def _execute(driver_port: int, session_id: str, script: str) -> Any: @@ -150,6 +185,83 @@ def _execute(driver_port: int, session_id: str, script: str) -> Any: return response.get("value") +def _find_element(driver_port: int, session_id: str, selector: str) -> str: + """Find one fixture element and return its validated ChromeDriver identifier.""" + + found = _json_request( + driver_port, + "POST", + _webdriver_path(session_id, "/element"), + {"using": "css selector", "value": selector}, + ) + element = found.get("value", {}) + element_id = element.get(W3C_ELEMENT_KEY) if isinstance(element, dict) else None + if not isinstance(element_id, str): + raise RuntimeError("WebDriver did not return a W3C element identifier") + return _path_token(element_id, "element identifier") + + +def _element_command_path(session_id: str, element_id: str, suffix: str) -> str: + """Build a bounded WebDriver element command path from validated identifiers.""" + + safe_element = _path_token(element_id, "element identifier") + return _webdriver_path(session_id, f"/element/{safe_element}{suffix}") + + +def _get_element_semantics( + driver_port: int, + session_id: str, + element_id: str, +) -> tuple[str, str]: + """Read one controlled element's browser-computed role and accessible name.""" + + role = _json_request( + driver_port, + "GET", + _element_command_path(session_id, element_id, "/computedrole"), + ).get("value") + label = _json_request( + driver_port, + "GET", + _element_command_path(session_id, element_id, "/computedlabel"), + ).get("value") + if not isinstance(role, str) or not isinstance(label, str): + raise RuntimeError("WebDriver returned malformed element semantics") + return role, label + + +def _cleanup_browser_session(driver_port: int, session_id: str) -> None: + """Delete one WebDriver session through the fixed loopback authority.""" + + _json_request( + driver_port, + "DELETE", + _webdriver_path(session_id, ""), + ) + + +def _cleanup_browser_session_preserving_primary( + driver_port: int, + session_id: str, + primary_error: BaseException | None, +) -> None: + """Fail closed on expected cleanup errors while retaining an earlier causal failure.""" + + try: + _cleanup_browser_session(driver_port, session_id) + except ( + OSError, + ValueError, + RuntimeError, + http.client.HTTPException, + json.JSONDecodeError, + ) as cleanup_error: + bounded_error = BrowserSessionCleanupError(cleanup_error) + if primary_error is None: + raise bounded_error from cleanup_error + raise bounded_error from primary_error + + def _wait_for_extension_evidence( driver_port: int, session_id: str, @@ -212,48 +324,27 @@ def _wait_for_extension_evidence( ): return latest time.sleep(0.1) - raise RuntimeError( - f"MV3 fixture did not converge: expected={expected!r}, observed={latest!r}" - ) + raise RuntimeError("MV3 fixture did not converge") def _exercise_real_click(driver_port: int, session_id: str) -> str: """Use the WebDriver element-click command and verify the DOM post-condition.""" - found = _json_request( - driver_port, - "POST", - _webdriver_path(session_id, "/element"), - {"using": "css selector", "value": "#fixture-button"}, - ) - element = found.get("value", {}) - element_id = element.get(W3C_ELEMENT_KEY) if isinstance(element, dict) else None - if not isinstance(element_id, str): - raise RuntimeError("WebDriver did not return a W3C element identifier") - safe_element = _path_token(element_id, "element identifier") + safe_element = _find_element(driver_port, session_id, "#fixture-button") _json_request( driver_port, "POST", - _webdriver_path(session_id, f"/element/{safe_element}/click"), + _element_command_path(session_id, safe_element, "/click"), {}, ) - output = _json_request( - driver_port, - "POST", - _webdriver_path(session_id, "/element"), - {"using": "css selector", "value": "#fixture-output"}, - ).get("value", {}) - output_id = output.get(W3C_ELEMENT_KEY) if isinstance(output, dict) else None - if not isinstance(output_id, str): - raise RuntimeError("WebDriver did not return the fixture output element") - safe_output = _path_token(output_id, "element identifier") + safe_output = _find_element(driver_port, session_id, "#fixture-output") text = _json_request( driver_port, "GET", - _webdriver_path(session_id, f"/element/{safe_output}/text"), + _element_command_path(session_id, safe_output, "/text"), ).get("value") if text != "clicked": - raise RuntimeError(f"real click post-condition failed: {text!r}") + raise RuntimeError("real click post-condition failed") return str(text) @@ -293,7 +384,6 @@ def _run_browser_pass( "--disable-component-update", "--disable-sync", "--disable-dev-shm-usage", - "--no-sandbox", f"--user-data-dir={profile_dir}", f"--disable-extensions-except={FIXTURE}", f"--load-extension={FIXTURE}", @@ -314,10 +404,7 @@ def _run_browser_pass( capabilities.get("browserVersion") if isinstance(capabilities, dict) else None ) if browser_version != PINNED_CHROME_VERSION: - raise RuntimeError( - f"unexpected Chrome version: expected {PINNED_CHROME_VERSION}, " - f"got {browser_version!r}" - ) + raise RuntimeError("unexpected Chrome version") _json_request( driver_port, @@ -353,20 +440,21 @@ def _run_browser_pass( }, } finally: - if session_id is not None: - with contextlib.suppress(Exception): - _json_request( + primary_error = sys.exc_info()[1] + try: + if session_id is not None: + _cleanup_browser_session_preserving_primary( driver_port, - "DELETE", - _webdriver_path(session_id, ""), - {}, + session_id, + primary_error, ) - driver.terminate() - try: - driver.wait(timeout=5) - except subprocess.TimeoutExpired: - driver.kill() - driver.wait(timeout=5) + finally: + driver.terminate() + try: + driver.wait(timeout=5) + except subprocess.TimeoutExpired: + driver.kill() + driver.wait(timeout=5) def _run_restart_trial( @@ -433,8 +521,371 @@ def _run_restart_trial( } +def _validate_agent_task_submitted_state(state: object) -> None: + """Require the controlled submitted marker without echoing page-controlled data.""" + + if state != "submitted": + raise RuntimeError("Agent Task state post-condition failed") + + +def _validate_agent_task_pre_action_state(state: object, text: object) -> None: + """Require the controlled idle baseline without echoing page-controlled data.""" + + if state != "idle" or text != "idle": + raise RuntimeError("Agent Task pre-action baseline was already satisfied") + + +def _validate_agent_task_cleared_value(value: object) -> None: + """Require the browser-observed clear post-condition without echoing input data.""" + + if value != "": + raise RuntimeError("Agent Task clear verification failed") + + +def _validate_agent_task_typed_value(value: object) -> None: + """Require the browser-observed typed value without echoing page-controlled data.""" + + if value != AGENT_TASK_INPUT_VALUE: + raise RuntimeError("Agent Task typed input verification failed") + + +def _cleanup_agent_task_browser_session(driver_port: int, session_id: str) -> None: + """Delete one Agent Task WebDriver session without suppressing cleanup failures.""" + + _cleanup_browser_session(driver_port, session_id) + + +def _run_agent_task_browser_pass( + chrome_bin: pathlib.Path, + chromedriver_bin: pathlib.Path, + fixture_url: str, + profile_dir: str, +) -> dict[str, Any]: + """Execute one synthetic Agent Task through real WebDriver input in pinned Chrome.""" + + started = time.monotonic() + driver_port = _free_loopback_port() + session_id: str | None = None + driver = subprocess.Popen( + [str(chromedriver_bin), f"--port={driver_port}", "--allowed-ips=127.0.0.1"], + stdout=subprocess.DEVNULL, + stderr=subprocess.STDOUT, + text=True, + ) + try: + _wait_for_driver(driver_port) + try: + session = _json_request( + driver_port, + "POST", + "/session", + { + "capabilities": { + "alwaysMatch": { + "browserName": "chrome", + "goog:chromeOptions": { + "binary": str(chrome_bin), + "args": [ + "--headless=new", + "--no-first-run", + "--disable-default-apps", + "--disable-component-update", + "--disable-sync", + "--disable-dev-shm-usage", + "--disable-extensions", + f"--user-data-dir={profile_dir}", + ], + }, + } + } + }, + ).get("value", {}) + except ( + OSError, + ValueError, + RuntimeError, + http.client.HTTPException, + json.JSONDecodeError, + ) as session_error: + raise AgentTaskSessionStartError(session_error) from session_error + if not isinstance(session, dict): + raise RuntimeError("ChromeDriver Agent Task session response is malformed") + raw_session_id = session.get("sessionId") + capabilities = session.get("capabilities", {}) + if not isinstance(raw_session_id, str): + raise RuntimeError("ChromeDriver did not return an Agent Task session id") + session_id = _path_token(raw_session_id, "session identifier") + browser_version = ( + capabilities.get("browserVersion") if isinstance(capabilities, dict) else None + ) + if browser_version != PINNED_CHROME_VERSION: + raise RuntimeError("unexpected Agent Task Chrome version") + + _json_request( + driver_port, + "POST", + _webdriver_path(session_id, "/url"), + {"url": fixture_url}, + ) + initial_url = _json_request( + driver_port, + "GET", + _webdriver_path(session_id, "/url"), + ).get("value") + if initial_url != fixture_url: + raise RuntimeError("Agent Task initial URL mismatch") + input_element = _find_element(driver_port, session_id, "#task-text") + input_role, input_name = _get_element_semantics( + driver_port, + session_id, + input_element, + ) + if input_role != "textbox" or input_name != "Task text": + raise RuntimeError("Agent Task input semantic evidence mismatch") + pre_action_result_element = _find_element(driver_port, session_id, "#task-result") + pre_action_state = _json_request( + driver_port, + "GET", + _element_command_path( + session_id, + pre_action_result_element, + "/attribute/data-state", + ), + ).get("value") + pre_action_text = _json_request( + driver_port, + "GET", + _element_command_path(session_id, pre_action_result_element, "/text"), + ).get("value") + _validate_agent_task_pre_action_state(pre_action_state, pre_action_text) + _json_request( + driver_port, + "POST", + _element_command_path(session_id, input_element, "/clear"), + {}, + ) + cleared_value = _json_request( + driver_port, + "GET", + _element_command_path(session_id, input_element, "/property/value"), + ).get("value") + _validate_agent_task_cleared_value(cleared_value) + _json_request( + driver_port, + "POST", + _element_command_path(session_id, input_element, "/value"), + {"text": AGENT_TASK_INPUT_VALUE, "value": list(AGENT_TASK_INPUT_VALUE)}, + ) + typed_value = _json_request( + driver_port, + "GET", + _element_command_path(session_id, input_element, "/property/value"), + ).get("value") + _validate_agent_task_typed_value(typed_value) + submit_element = _find_element( + driver_port, + session_id, + "#agent-task-form button[type=submit]", + ) + submit_role, submit_name = _get_element_semantics( + driver_port, + session_id, + submit_element, + ) + if submit_role != "button" or submit_name != "Submit task": + raise RuntimeError("Agent Task submit semantic evidence mismatch") + pre_click_state = _json_request( + driver_port, + "GET", + _element_command_path( + session_id, + pre_action_result_element, + "/attribute/data-state", + ), + ).get("value") + pre_click_text = _json_request( + driver_port, + "GET", + _element_command_path(session_id, pre_action_result_element, "/text"), + ).get("value") + _validate_agent_task_pre_action_state(pre_click_state, pre_click_text) + _json_request( + driver_port, + "POST", + _element_command_path(session_id, submit_element, "/click"), + {}, + ) + post_submit_url = _json_request( + driver_port, + "GET", + _webdriver_path(session_id, "/url"), + ).get("value") + url_unchanged = post_submit_url == initial_url + if not url_unchanged: + raise RuntimeError("Agent Task URL changed during submission") + result_element = _find_element(driver_port, session_id, "#task-result") + state = _json_request( + driver_port, + "GET", + _element_command_path(session_id, result_element, "/attribute/data-state"), + ).get("value") + text = _json_request( + driver_port, + "GET", + _element_command_path(session_id, result_element, "/text"), + ).get("value") + _validate_agent_task_submitted_state(state) + if text != AGENT_TASK_INPUT_VALUE: + raise RuntimeError("Agent Task result did not match the synthetic typed value") + accepted_outcome_url = _json_request( + driver_port, + "GET", + _webdriver_path(session_id, "/url"), + ).get("value") + url_unchanged = url_unchanged and accepted_outcome_url == initial_url + if not url_unchanged: + raise RuntimeError("Agent Task URL changed before accepted outcome") + return { + "browser_version": browser_version, + "pre_action_baseline_verified": True, + "clear_value_verified": True, + "input_value_verified": True, + "pre_click_baseline_verified": True, + "post_condition": True, + "input_echo_verified": True, + "url_unchanged": url_unchanged, + "input_semantics_verified": True, + "submit_semantics_verified": True, + "extensions_disabled_requested": True, + "duration_ms": round((time.monotonic() - started) * 1000), + } + finally: + primary_error = sys.exc_info()[1] + try: + if session_id is not None: + _cleanup_browser_session_preserving_primary( + driver_port, + session_id, + primary_error, + ) + finally: + driver.terminate() + try: + driver.wait(timeout=5) + except subprocess.TimeoutExpired: + driver.kill() + driver.wait(timeout=5) + + +def _run_agent_task_trial( + chrome_bin: pathlib.Path, + chromedriver_bin: pathlib.Path, + fixture_url: str, + trial_number: int, +) -> dict[str, Any]: + """Run one isolated Agent Task browser trial and prove its profile is removed.""" + + trial_started = time.monotonic() + temporary_profile = tempfile.TemporaryDirectory( + prefix=f"originweave-agent-task-trial-{trial_number}-" + ) + profile_path = pathlib.Path(temporary_profile.name) + profile_observed_before_cleanup = profile_path.is_dir() + if not profile_observed_before_cleanup: + raise RuntimeError(f"Agent Task profile was not created in trial {trial_number}") + + try: + result = _run_agent_task_browser_pass( + chrome_bin, + chromedriver_bin, + fixture_url, + temporary_profile.name, + ) + finally: + primary_error = sys.exc_info()[1] + try: + temporary_profile.cleanup() + except OSError as cleanup_error: + bounded_error = BrowserProfileCleanupError(cleanup_error) + if primary_error is None: + raise bounded_error from cleanup_error + raise bounded_error from primary_error + + profile_cleaned = profile_observed_before_cleanup and not profile_path.exists() + if not profile_cleaned: + raise RuntimeError(f"Agent Task profile cleanup failed in trial {trial_number}") + + return { + "trial_number": trial_number, + "passed": True, + "browser_version": result["browser_version"], + "pre_action_baseline_verified": result["pre_action_baseline_verified"], + "clear_value_verified": result["clear_value_verified"], + "input_value_verified": result["input_value_verified"], + "pre_click_baseline_verified": result["pre_click_baseline_verified"], + "post_condition": result["post_condition"], + "input_echo_verified": result["input_echo_verified"], + "url_unchanged": result["url_unchanged"], + "input_semantics_verified": result["input_semantics_verified"], + "submit_semantics_verified": result["submit_semantics_verified"], + "extensions_disabled_requested": result["extensions_disabled_requested"], + "profile_cleaned": profile_cleaned, + "duration_ms": round((time.monotonic() - trial_started) * 1000), + } + + +def _agent_task_surfaces_complete(agent_task_trials: list[dict[str, Any]]) -> bool: + """Require every recorded Agent Task trial to contain every observed success surface.""" + + if not agent_task_trials: + return False + return all( + trial.get("passed") is True + and trial.get("pre_action_baseline_verified") is True + and trial.get("clear_value_verified") is True + and trial.get("input_value_verified") is True + and trial.get("pre_click_baseline_verified") is True + and trial.get("post_condition") is True + and trial.get("input_echo_verified") is True + and trial.get("url_unchanged") is True + and trial.get("input_semantics_verified") is True + and trial.get("submit_semantics_verified") is True + and trial.get("profile_cleaned") is True + for trial in agent_task_trials + ) + + +def _start_fixture_server(directory: pathlib.Path) -> tuple[http.server.ThreadingHTTPServer, threading.Thread]: + """Start one loopback-only static fixture server for a bounded browser lane.""" + + server = http.server.ThreadingHTTPServer( + ("127.0.0.1", 0), + lambda *args, **kwargs: QuietFixtureHandler( + *args, + directory=str(directory), + **kwargs, + ), + ) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + return server, thread + + +def _stop_fixture_server( + server: http.server.ThreadingHTTPServer, + thread: threading.Thread, +) -> None: + """Stop one bounded fixture server and join its helper thread.""" + + server.shutdown() + server.server_close() + thread.join(timeout=5) + if thread.is_alive(): + raise RuntimeError("fixture server thread did not stop") + + def main() -> int: - """Run three independent restart trials and emit bounded repeatability evidence.""" + """Run bounded MV3 and Agent Task trials and emit credential-free evidence.""" chrome_bin = pathlib.Path(os.environ.get("CHROME_BIN", "")) chromedriver_bin = pathlib.Path(os.environ.get("CHROMEDRIVER_BIN", "")) @@ -444,16 +895,20 @@ def main() -> int: raise SystemExit("CHROMEDRIVER_BIN must point to the matching pinned ChromeDriver") if not (FIXTURE / "manifest.json").is_file(): raise SystemExit("MV3 fixture manifest is missing") + if not (AGENT_TASK_FIXTURE / "index.html").is_file(): + raise SystemExit("Agent Task fixture is missing") - fixture_server = http.server.ThreadingHTTPServer( - ("127.0.0.1", 0), - lambda *args, **kwargs: QuietFixtureHandler( - *args, directory=str(FIXTURE), **kwargs - ), - ) - fixture_thread = threading.Thread(target=fixture_server.serve_forever, daemon=True) - fixture_thread.start() + fixture_server, fixture_thread = _start_fixture_server(FIXTURE) + try: + agent_task_server, agent_task_thread = _start_fixture_server(AGENT_TASK_FIXTURE) + except BaseException as startup_error: + try: + _stop_fixture_server(fixture_server, fixture_thread) + except BaseException as cleanup_error: + raise cleanup_error from startup_error + raise started = time.monotonic() + evidence: dict[str, Any] | None = None try: fixture_url = f"http://127.0.0.1:{fixture_server.server_port}/page.html" @@ -468,7 +923,13 @@ def main() -> int: trial_number, ) ) - except (OSError, ValueError, RuntimeError, json.JSONDecodeError): + except ( + OSError, + ValueError, + RuntimeError, + http.client.HTTPException, + json.JSONDecodeError, + ): trial_results.append( { "trial_number": trial_number, @@ -496,6 +957,43 @@ def main() -> int: for name in first_surfaces } + agent_task_url = ( + f"http://127.0.0.1:{agent_task_server.server_port}/index.html" + ) + agent_task_trials: list[dict[str, Any]] = [] + for trial_number in range(1, AGENT_TASK_REPEATABILITY_TRIALS + 1): + try: + agent_task_trials.append( + _run_agent_task_trial( + chrome_bin, + chromedriver_bin, + agent_task_url, + trial_number, + ) + ) + except ( + OSError, + ValueError, + RuntimeError, + http.client.HTTPException, + json.JSONDecodeError, + ) as error: + agent_task_trials.append( + { + "trial_number": trial_number, + "passed": False, + "failure_type": type(error).__name__, + } + ) + + agent_task_successful_trials = sum( + 1 for trial in agent_task_trials if trial.get("passed") is True + ) + agent_task_trial_pass_rate = ( + agent_task_successful_trials / AGENT_TASK_REPEATABILITY_TRIALS + ) + agent_task_surfaces_complete = _agent_task_surfaces_complete(agent_task_trials) + evidence = { "chrome_version": PINNED_CHROME_VERSION, "chrome_revision": PINNED_CHROME_REVISION, @@ -509,22 +1007,45 @@ def main() -> int: if successful_results else [] ), + "agent_task": { + "repeatability_trials": AGENT_TASK_REPEATABILITY_TRIALS, + "successful_trials": agent_task_successful_trials, + "trial_pass_rate": agent_task_trial_pass_rate, + "trial_results": agent_task_trials, + }, "duration_ms": round((time.monotonic() - started) * 1000), } - print(json.dumps(evidence, sort_keys=True)) if successful_trials != REPEATABILITY_TRIALS: + print(json.dumps(evidence, sort_keys=True)) raise RuntimeError( "Manifest V3 repeatability gate failed: " f"{successful_trials}/{REPEATABILITY_TRIALS} trials passed" ) if not common_surfaces or not all(common_surfaces.values()): + print(json.dumps(evidence, sort_keys=True)) raise RuntimeError("Manifest V3 repeatability surfaces were incomplete") - return 0 + if agent_task_successful_trials != AGENT_TASK_REPEATABILITY_TRIALS: + print(json.dumps(evidence, sort_keys=True)) + raise RuntimeError( + "Agent Task repeatability gate failed: " + f"{agent_task_successful_trials}/{AGENT_TASK_REPEATABILITY_TRIALS} " + "trials passed" + ) + if not agent_task_surfaces_complete: + print(json.dumps(evidence, sort_keys=True)) + raise RuntimeError("Agent Task repeatability surfaces were incomplete") finally: - fixture_server.shutdown() - fixture_server.server_close() - fixture_thread.join(timeout=5) + try: + _stop_fixture_server(agent_task_server, agent_task_thread) + finally: + _stop_fixture_server(fixture_server, fixture_thread) + + if evidence is None: + raise RuntimeError("browser compatibility evidence was not materialized") + evidence["duration_ms"] = round((time.monotonic() - started) * 1000) + print(json.dumps(evidence, sort_keys=True)) + return 0 if __name__ == "__main__": - raise SystemExit(main()) + raise SystemExit(main()) \ No newline at end of file diff --git a/tests/test_agent_task_action_transition_evidence_contract.py b/tests/test_agent_task_action_transition_evidence_contract.py new file mode 100644 index 000000000..4dde74097 --- /dev/null +++ b/tests/test_agent_task_action_transition_evidence_contract.py @@ -0,0 +1,167 @@ +"""Contract for causal Agent Task action-transition evidence in pinned Chrome.""" + +from __future__ import annotations + +import inspect +import pathlib +import runpy +import unittest + +ROOT = pathlib.Path(__file__).resolve().parents[1] +RUNNER = ROOT / "scripts" / "ci" / "run_mv3_compatibility.py" + + +class AgentTaskActionTransitionEvidenceContractTests(unittest.TestCase): + """Require false baselines before accepting an action-caused success state.""" + + def test_pre_action_baseline_validator_is_closed_and_non_echoing(self) -> None: + """A pre-fired fixture must fail without echoing page-controlled state or text.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_transition_contract") + self.assertIn("_validate_agent_task_pre_action_state", namespace) + validate = namespace["_validate_agent_task_pre_action_state"] + + validate("idle", "idle") + hostile = "buyer-secret-marker-must-not-reach-ci" + for state, text in (("submitted", "idle"), ("idle", hostile)): + with self.subTest(state=state, text=text), self.assertRaisesRegex( + RuntimeError, + r"^Agent Task pre-action baseline was already satisfied$", + ) as raised: + validate(state, text) + self.assertNotIn(hostile, str(raised.exception)) + + def test_cleared_input_validator_is_closed_and_non_echoing(self) -> None: + """Clear evidence must fail closed without echoing browser-observed input.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_clear_value_contract") + self.assertIn("_validate_agent_task_cleared_value", namespace) + validate = namespace["_validate_agent_task_cleared_value"] + + validate("") + hostile = "buyer-secret-marker-must-not-reach-ci" + with self.assertRaisesRegex( + RuntimeError, + r"^Agent Task clear verification failed$", + ) as raised: + validate(hostile) + self.assertNotIn(hostile, str(raised.exception)) + + def test_typed_input_validator_is_closed_and_non_echoing(self) -> None: + """Typed-value evidence must fail closed without echoing page-controlled input.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_typed_value_contract") + self.assertIn("_validate_agent_task_typed_value", namespace) + validate = namespace["_validate_agent_task_typed_value"] + + validate(namespace["AGENT_TASK_INPUT_VALUE"]) + hostile = "buyer-secret-marker-must-not-reach-ci" + with self.assertRaisesRegex( + RuntimeError, + r"^Agent Task typed input verification failed$", + ) as raised: + validate(hostile) + self.assertNotIn(hostile, str(raised.exception)) + + def test_pre_action_observation_happens_before_native_click(self) -> None: + """Evidence must prove baseline→native action→post-condition ordering.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_transition_order") + source = inspect.getsource(namespace["_run_agent_task_browser_pass"]) + baseline = source.index("_validate_agent_task_pre_action_state") + native_click = source.index('"/click"') + post_condition = source.index("_validate_agent_task_submitted_state") + self.assertLess(baseline, native_click) + self.assertLess(native_click, post_condition) + self.assertIn('"pre_action_baseline_verified": True', source) + + def test_clear_value_is_observed_before_typing(self) -> None: + """Native clear acknowledgement must not substitute for observed empty state.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_clear_value_order") + source = inspect.getsource(namespace["_run_agent_task_browser_pass"]) + clear = source.index('"/clear"') + typing = source.index('"/value"', clear) + value_observation = source.find('"/property/value"', clear, typing) + clear_validation = source.find("_validate_agent_task_cleared_value", clear, typing) + self.assertNotEqual(value_observation, -1) + self.assertNotEqual(clear_validation, -1) + self.assertLess(clear, value_observation) + self.assertLess(value_observation, clear_validation) + self.assertLess(clear_validation, typing) + self.assertIn('"clear_value_verified": True', source) + + def test_typed_input_value_is_observed_before_pre_click_baseline(self) -> None: + """Native send-keys acknowledgement must not substitute for observed input state.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_typed_value_order") + source = inspect.getsource(namespace["_run_agent_task_browser_pass"]) + typing = source.index('"/value"') + value_observation = source.find('"/property/value"', typing) + typed_validation = source.find("_validate_agent_task_typed_value", typing) + pre_click_baseline = source.find("_validate_agent_task_pre_action_state", typing) + native_click = source.index('"/click"') + self.assertNotEqual(value_observation, -1) + self.assertNotEqual(typed_validation, -1) + self.assertLess(typing, value_observation) + self.assertLess(value_observation, typed_validation) + self.assertLess(typed_validation, pre_click_baseline) + self.assertLess(pre_click_baseline, native_click) + self.assertIn('"input_value_verified": True', source) + + def test_pre_click_baseline_happens_after_typing_and_before_click(self) -> None: + """Typing must not be able to pre-satisfy the submit post-condition.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_pre_click_order") + source = inspect.getsource(namespace["_run_agent_task_browser_pass"]) + typing = source.index('"/value"') + pre_click_baseline = source.find("_validate_agent_task_pre_action_state", typing) + native_click = source.index('"/click"') + self.assertNotEqual(pre_click_baseline, -1) + self.assertLess(typing, pre_click_baseline) + self.assertLess(pre_click_baseline, native_click) + self.assertIn('"pre_click_baseline_verified": True', source) + + def test_url_stability_is_observed_through_the_accepted_post_condition(self) -> None: + """A delayed navigation must not escape the accepted action-outcome boundary.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_final_url_order") + source = inspect.getsource(namespace["_run_agent_task_browser_pass"]) + native_click = source.index('"/click"') + post_condition = source.index("_validate_agent_task_submitted_state", native_click) + exact_echo = source.index("if text != AGENT_TASK_INPUT_VALUE", post_condition) + final_url_read = source.find('"/url"', exact_echo) + success_evidence = source.index("return {", exact_echo) + self.assertNotEqual(final_url_read, -1) + self.assertLess(exact_echo, final_url_read) + self.assertLess(final_url_read, success_evidence) + self.assertIn("Agent Task URL changed before accepted outcome", source) + + def test_surface_completeness_requires_transition_baseline_evidence(self) -> None: + """Post-condition evidence must include clear, typed-value, and baseline witnesses.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_transition_surface") + complete = namespace["_agent_task_surfaces_complete"] + trial = { + "trial_number": 1, + "passed": True, + "post_condition": True, + "input_echo_verified": True, + "url_unchanged": True, + "input_semantics_verified": True, + "submit_semantics_verified": True, + "profile_cleaned": True, + } + self.assertFalse(complete([trial])) + trial["pre_action_baseline_verified"] = True + self.assertFalse(complete([trial])) + trial["clear_value_verified"] = True + self.assertFalse(complete([trial])) + trial["input_value_verified"] = True + self.assertFalse(complete([trial])) + trial["pre_click_baseline_verified"] = True + self.assertTrue(complete([trial])) + + +if __name__ == "__main__": + unittest.main() \ No newline at end of file diff --git a/tests/test_agent_task_chromium_sandbox_contract.py b/tests/test_agent_task_chromium_sandbox_contract.py new file mode 100644 index 000000000..3ea9258f2 --- /dev/null +++ b/tests/test_agent_task_chromium_sandbox_contract.py @@ -0,0 +1,26 @@ +"""Contract for preserving Chromium's process sandbox in Agent Task evidence.""" + +from __future__ import annotations + +import inspect +import pathlib +import runpy +import unittest + +ROOT = pathlib.Path(__file__).resolve().parents[1] +RUNNER = ROOT / "scripts" / "ci" / "run_mv3_compatibility.py" + + +class AgentTaskChromiumSandboxContractTests(unittest.TestCase): + """Keep the governed-browser Agent Task evidence on a sandboxed Chrome process.""" + + def test_agent_task_browser_pass_does_not_disable_chromium_sandbox(self) -> None: + """Security evidence must not launch the Agent Task browser with ``--no-sandbox``.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_sandbox_contract") + browser_pass_source = inspect.getsource(namespace["_run_agent_task_browser_pass"]) + + self.assertNotIn('"--no-sandbox"', browser_pass_source) + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_agent_task_extension_isolation_evidence_contract.py b/tests/test_agent_task_extension_isolation_evidence_contract.py new file mode 100644 index 000000000..31eb60b6f --- /dev/null +++ b/tests/test_agent_task_extension_isolation_evidence_contract.py @@ -0,0 +1,56 @@ +"""Contract for honest extension-isolation evidence in controlled Agent Task trials.""" + +from __future__ import annotations + +import inspect +import pathlib +import runpy +import unittest + +ROOT = pathlib.Path(__file__).resolve().parents[1] +RUNNER = ROOT / "scripts" / "ci" / "run_mv3_compatibility.py" + + +class AgentTaskExtensionIsolationEvidenceContractTests(unittest.TestCase): + """Separate requested Chrome launch isolation from browser-observed success evidence.""" + + def test_launch_request_is_not_reported_as_observed_extension_isolation(self) -> None: + """A command-line request must not become a verified Agent Task success surface.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_extension_evidence_contract") + browser_pass_source = inspect.getsource(namespace["_run_agent_task_browser_pass"]) + surface_source = inspect.getsource(namespace["_agent_task_surfaces_complete"]) + + self.assertIn('"--disable-extensions"', browser_pass_source) + self.assertIn('"extensions_disabled_requested": True', browser_pass_source) + self.assertNotIn('"extensions_disabled": True', browser_pass_source) + self.assertNotIn('trial.get("extensions_disabled") is True', surface_source) + + def test_requested_extension_isolation_is_metadata_not_success_evidence(self) -> None: + """Verified surface completeness must not depend on unobserved launch intent.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_extension_surface_contract") + surfaces_complete = namespace["_agent_task_surfaces_complete"] + observed_success = { + "trial_number": 1, + "passed": True, + "pre_action_baseline_verified": True, + "pre_click_baseline_verified": True, + "post_condition": True, + "input_echo_verified": True, + "url_unchanged": True, + "input_semantics_verified": True, + "submit_semantics_verified": True, + "profile_cleaned": True, + } + + self.assertTrue(surfaces_complete([observed_success])) + self.assertTrue( + surfaces_complete( + [{**observed_success, "extensions_disabled_requested": False}] + ) + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_agent_task_pinned_chrome_contract.py b/tests/test_agent_task_pinned_chrome_contract.py new file mode 100644 index 000000000..24c48776e --- /dev/null +++ b/tests/test_agent_task_pinned_chrome_contract.py @@ -0,0 +1,426 @@ +"""Contract for executing the controlled Agent Task fixture on pinned Chrome.""" + +from __future__ import annotations + +import io +import http.client +import inspect +import json +import os +import pathlib +import runpy +import unittest +from contextlib import redirect_stdout +from unittest.mock import patch + +ROOT = pathlib.Path(__file__).resolve().parents[1] +RUNNER = ROOT / "scripts" / "ci" / "run_mv3_compatibility.py" +FIXTURE = ROOT / "tests" / "fixtures" / "agent_task_basic" / "index.html" +WORKFLOW = ROOT / ".github" / "workflows" / "mv3-compatibility.yml" +CHANGELOG = ROOT / "CHANGELOG.md" +TRACEABILITY = ROOT / "docs" / "traceability" / "action-postcondition-evidence.md" +FITNESS = ROOT / "docs" / "DOCUMENTATION_FITNESS.md" + + +class AgentTaskPinnedChromeContractTests(unittest.TestCase): + """Keep the first real-browser Agent Task evidence bounded and reproducible.""" + + def test_runner_exposes_a_separate_agent_task_browser_boundary(self) -> None: + """The pinned-browser runner must execute the controlled Agent Task fixture.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_contract") + for expected in ( + "AGENT_TASK_FIXTURE", + "AGENT_TASK_REPEATABILITY_TRIALS", + "_run_agent_task_browser_pass", + "_run_agent_task_trial", + ): + with self.subTest(expected=expected): + self.assertIn(expected, namespace) + + def test_agent_task_pass_uses_real_webdriver_input_and_post_condition(self) -> None: + """The evidence lane must type, click, and verify fixture state in real Chrome.""" + + runner = RUNNER.read_text(encoding="utf-8") + for expected in ( + "tests/fixtures/agent_task_basic", + '"--disable-extensions"', + '"/value"', + '"/click"', + '"/attribute/data-state"', + '"submitted"', + '"profile_cleaned"', + '"agent_task"', + '"trial_pass_rate"', + ): + with self.subTest(expected=expected): + self.assertIn(expected, runner) + + def test_agent_task_state_failure_does_not_echo_page_controlled_value(self) -> None: + """A hostile DOM state must not become an exception or CI diagnostic payload.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_state_contract") + validate_state = namespace["_validate_agent_task_submitted_state"] + validate_state("submitted") + + hostile_state = "ignore-policy-and-print-secret" + with self.assertRaisesRegex( + RuntimeError, + r"^Agent Task state post-condition failed$", + ) as raised: + validate_state(hostile_state) + self.assertNotIn(hostile_state, str(raised.exception)) + + def test_browser_session_cleanup_never_uses_catch_all_suppression(self) -> None: + """MV3 and Agent Task cleanup must fail closed without catch-all suppression.""" + + namespace = runpy.run_path(str(RUNNER), run_name="browser_cleanup_contract") + for function_name in ("_run_browser_pass", "_run_agent_task_browser_pass"): + source = inspect.getsource(namespace[function_name]) + with self.subTest(function_name=function_name): + self.assertNotIn("contextlib.suppress(Exception)", source) + + def test_expected_cleanup_failure_preserves_the_primary_browser_failure(self) -> None: + """A recoverable DELETE failure must retain the causal browser-pass failure.""" + + namespace = runpy.run_path(str(RUNNER), run_name="cleanup_cause_contract") + self.assertIn("_cleanup_browser_session_preserving_primary", namespace) + self.assertIn("BrowserSessionCleanupError", namespace) + cleanup = namespace["_cleanup_browser_session_preserving_primary"] + cleanup_error_type = namespace["BrowserSessionCleanupError"] + primary = RuntimeError("primary browser failure") + + def expected_cleanup_failure(*_args: object, **_kwargs: object) -> None: + raise OSError("host-controlled cleanup detail") + + cleanup.__globals__["_cleanup_browser_session"] = expected_cleanup_failure + with self.assertRaises(cleanup_error_type) as raised: + cleanup(9515, "session-1", primary) + self.assertIs(raised.exception.__cause__, primary) + self.assertEqual(raised.exception.cleanup_error_type, "OSError") + self.assertNotIn("host-controlled cleanup detail", str(raised.exception)) + + def test_malformed_cleanup_response_is_a_bounded_cleanup_failure(self) -> None: + """A truncated DELETE response must stay in the typed cleanup failure contract.""" + + namespace = runpy.run_path(str(RUNNER), run_name="cleanup_http_contract") + cleanup = namespace["_cleanup_browser_session_preserving_primary"] + cleanup_error_type = namespace["BrowserSessionCleanupError"] + primary = RuntimeError("primary browser failure") + + def malformed_cleanup_response(*_args: object, **_kwargs: object) -> None: + raise http.client.IncompleteRead(b"partial", 32) + + cleanup.__globals__["_cleanup_browser_session"] = malformed_cleanup_response + with self.assertRaises(cleanup_error_type) as raised: + cleanup(9515, "session-1", primary) + self.assertIs(raised.exception.__cause__, primary) + self.assertEqual(raised.exception.cleanup_error_type, "IncompleteRead") + self.assertNotIn("partial", str(raised.exception)) + + def test_agent_task_response_failure_is_recorded_as_a_failed_trial(self) -> None: + """A truncated WebDriver response must become bounded failed-trial evidence.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_trial_failure_contract") + main_globals = namespace["main"].__globals__ + + class FakeServer: + server_port = 9515 + + servers_started = 0 + + def start_fixture_server(_directory: pathlib.Path) -> tuple[FakeServer, object]: + nonlocal servers_started + servers_started += 1 + return FakeServer(), object() + + def successful_restart_trial(*_args: object, **_kwargs: object) -> dict[str, object]: + return {"trial_number": 1, "passed": True, "surfaces": {"worker": True}} + + def truncated_agent_task_response( + *_args: object, **_kwargs: object + ) -> dict[str, object]: + raise http.client.IncompleteRead(b"partial", 32) + + main_globals.update( + { + "_start_fixture_server": start_fixture_server, + "_stop_fixture_server": lambda *_args: None, + "_run_restart_trial": successful_restart_trial, + "_run_agent_task_trial": truncated_agent_task_response, + "REPEATABILITY_TRIALS": 1, + "AGENT_TASK_REPEATABILITY_TRIALS": 1, + } + ) + output = io.StringIO() + session_start_error = namespace["AgentTaskSessionStartError"] + + def failed_session_start( + *_args: object, **_kwargs: object + ) -> dict[str, object]: + raise session_start_error(RuntimeError("host-controlled browser detail")) + + main_globals["_run_agent_task_trial"] = failed_session_start + with patch.dict( + os.environ, + {"CHROME_BIN": "/bin/sh", "CHROMEDRIVER_BIN": "/bin/sh"}, + ), redirect_stdout(output), self.assertRaisesRegex( + RuntimeError, + r"^Agent Task repeatability gate failed: 0/1 trials passed$", + ): + namespace["main"]() + + self.assertEqual(servers_started, 2) + evidence = json.loads(output.getvalue()) + failed_trial = evidence["agent_task"]["trial_results"][0] + self.assertEqual(failed_trial["failure_type"], "AgentTaskSessionStartError") + self.assertNotIn("host-controlled browser detail", output.getvalue()) + + def test_unexpected_cleanup_programming_failure_is_not_normalized(self) -> None: + """Programming failures in cleanup must propagate rather than enter fallback handling.""" + + namespace = runpy.run_path(str(RUNNER), run_name="cleanup_programming_contract") + self.assertIn("_cleanup_browser_session_preserving_primary", namespace) + cleanup = namespace["_cleanup_browser_session_preserving_primary"] + primary = RuntimeError("primary browser failure") + + def unexpected_cleanup_failure(*_args: object, **_kwargs: object) -> None: + raise AssertionError("unexpected cleanup programming failure") + + cleanup.__globals__["_cleanup_browser_session"] = unexpected_cleanup_failure + with self.assertRaisesRegex( + AssertionError, + r"^unexpected cleanup programming failure$", + ): + cleanup(9515, "session-1", primary) + + def test_agent_task_profile_cleanup_evidence_records_a_real_transition(self) -> None: + """Profile cleanup evidence must prove the profile existed before it became absent.""" + + namespace = runpy.run_path(str(RUNNER), run_name="profile_cleanup_contract") + trial_source = inspect.getsource(namespace["_run_agent_task_trial"]) + self.assertIn("profile_observed_before_cleanup", trial_source) + self.assertIn("temporary_profile.cleanup()", trial_source) + self.assertNotIn("with tempfile.TemporaryDirectory", trial_source) + + def test_agent_task_surface_completeness_is_non_vacuous(self) -> None: + """Surface completeness must be false for empty or failed-trial evidence.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_surface_contract") + self.assertIn("_agent_task_surfaces_complete", namespace) + surfaces_complete = namespace["_agent_task_surfaces_complete"] + self.assertFalse(surfaces_complete([])) + self.assertFalse(surfaces_complete([{"trial_number": 1, "passed": False}])) + self.assertTrue( + surfaces_complete( + [ + { + "trial_number": 1, + "passed": True, + "pre_action_baseline_verified": True, + "clear_value_verified": True, + "input_value_verified": True, + "pre_click_baseline_verified": True, + "post_condition": True, + "input_echo_verified": True, + "url_unchanged": True, + "input_semantics_verified": True, + "submit_semantics_verified": True, + "extensions_disabled_requested": True, + "profile_cleaned": True, + } + ] + ) + ) + + def test_agent_task_session_cleanup_never_suppresses_programming_failures(self) -> None: + """Unexpected cleanup defects must fail closed instead of becoming successful evidence.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_cleanup_contract") + self.assertIn("_cleanup_agent_task_browser_session", namespace) + cleanup_session = namespace["_cleanup_agent_task_browser_session"] + browser_pass_source = inspect.getsource(namespace["_run_agent_task_browser_pass"]) + self.assertNotIn("contextlib.suppress(Exception)", browser_pass_source) + + def unexpected_cleanup_failure(*_args: object, **_kwargs: object) -> dict[str, object]: + raise AssertionError("unexpected cleanup programming failure") + + cleanup_session.__globals__["_json_request"] = unexpected_cleanup_failure + with self.assertRaisesRegex( + AssertionError, + r"^unexpected cleanup programming failure$", + ): + cleanup_session(9515, "session-1") + + def test_second_fixture_server_start_cleans_up_the_first(self) -> None: + """A partial fixture startup must not leak the server already created.""" + + namespace = runpy.run_path(str(RUNNER), run_name="fixture_startup_cleanup_contract") + first_server = object() + first_thread = object() + starts = 0 + stopped: list[tuple[object, object]] = [] + + def start_fixture_server(_directory: pathlib.Path) -> tuple[object, object]: + nonlocal starts + starts += 1 + if starts == 1: + return first_server, first_thread + raise RuntimeError("second fixture startup failed") + + def stop_fixture_server(server: object, thread: object) -> None: + stopped.append((server, thread)) + + namespace["main"].__globals__["_start_fixture_server"] = start_fixture_server + namespace["main"].__globals__["_stop_fixture_server"] = stop_fixture_server + with patch.dict( + os.environ, + {"CHROME_BIN": "/bin/sh", "CHROMEDRIVER_BIN": "/bin/sh"}, + ), self.assertRaisesRegex(RuntimeError, r"^second fixture startup failed$"): + namespace["main"]() + + self.assertEqual(stopped, [(first_server, first_thread)]) + + def test_fixture_shutdown_attempts_both_servers_when_first_stop_fails(self) -> None: + """A cleanup failure for one fixture must not skip the other fixture.""" + + namespace = runpy.run_path(str(RUNNER), run_name="fixture_shutdown_contract") + + class FakeServer: + server_port = 9515 + + first_server = FakeServer() + first_thread = object() + second_server = FakeServer() + second_thread = object() + starts = 0 + stopped: list[tuple[object, object]] = [] + + def start_fixture_server(_directory: pathlib.Path) -> tuple[object, object]: + nonlocal starts + starts += 1 + return ( + (first_server, first_thread) + if starts == 1 + else (second_server, second_thread) + ) + + def stop_fixture_server(server: object, thread: object) -> None: + stopped.append((server, thread)) + if server is second_server: + raise OSError("agent-task fixture shutdown failed") + + namespace["main"].__globals__["_start_fixture_server"] = start_fixture_server + namespace["main"].__globals__["_stop_fixture_server"] = stop_fixture_server + namespace["main"].__globals__["_run_restart_trial"] = ( + lambda *_args, **_kwargs: { + "trial_number": 1, + "passed": True, + "surfaces": {"worker": True}, + } + ) + + def successful_agent_task_trial(*_args: object, **_kwargs: object) -> dict[str, object]: + return { + "trial_number": 1, + "passed": True, + "pre_action_baseline_verified": True, + "clear_value_verified": True, + "input_value_verified": True, + "pre_click_baseline_verified": True, + "post_condition": True, + "input_echo_verified": True, + "url_unchanged": True, + "input_semantics_verified": True, + "submit_semantics_verified": True, + "extensions_disabled_requested": True, + "profile_cleaned": True, + } + + self.assertTrue( + namespace["_agent_task_surfaces_complete"]([successful_agent_task_trial()]) + ) + namespace["main"].__globals__["_run_agent_task_trial"] = successful_agent_task_trial + namespace["main"].__globals__["REPEATABILITY_TRIALS"] = 1 + namespace["main"].__globals__["AGENT_TASK_REPEATABILITY_TRIALS"] = 1 + with patch.dict( + os.environ, + {"CHROME_BIN": "/bin/sh", "CHROMEDRIVER_BIN": "/bin/sh"}, + ), self.assertRaisesRegex(OSError, r"^agent-task fixture shutdown failed$"): + namespace["main"]() + + self.assertEqual( + stopped, + [ + (second_server, second_thread), + (first_server, first_thread), + ], + ) + + def test_agent_task_submission_preserves_the_loaded_url(self) -> None: + """Submission must prove that the controlled action did not navigate away.""" + + runner = RUNNER.read_text(encoding="utf-8") + for expected in ( + "initial_url", + "post_submit_url", + "url_unchanged", + ): + with self.subTest(expected=expected): + self.assertIn(expected, runner) + self.assertIn("Agent Task URL changed during submission", runner) + + def test_agent_task_observes_computed_role_and_name_before_action(self) -> None: + """Real-browser evidence must bind the controlled targets to semantic role/name.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_semantics_contract") + runner = RUNNER.read_text(encoding="utf-8") + self.assertIn("_get_element_semantics", namespace) + for expected in ( + '"/computedrole"', + '"/computedlabel"', + '"textbox"', + '"Task text"', + '"button"', + '"Submit task"', + '"input_semantics_verified"', + '"submit_semantics_verified"', + ): + with self.subTest(expected=expected): + self.assertIn(expected, runner) + + def test_agent_task_fixture_runs_under_the_existing_pinned_chrome_job(self) -> None: + """No floating browser or second workflow may be introduced for this slice.""" + + workflow = WORKFLOW.read_text(encoding="utf-8") + runner = RUNNER.read_text(encoding="utf-8") + self.assertTrue(FIXTURE.is_file()) + self.assertIn('CHROME_VERSION: "150.0.7871.129"', workflow) + self.assertIn("run_mv3_compatibility.py", workflow) + self.assertIn("150.0.7871.129", runner) + self.assertNotIn("google-chrome-stable", runner.lower()) + self.assertNotIn("COPILOT_GITHUB_TOKEN", runner) + self.assertNotIn("NVIDIA_NIM_API_KEY", runner) + + def test_documentation_separates_active_browser_evidence_from_product_runtime(self) -> None: + """Documentation must record the real fixture evidence without shipping the adapter claim.""" + + changelog = CHANGELOG.read_text(encoding="utf-8") + traceability = TRACEABILITY.read_text(encoding="utf-8") + fitness = FITNESS.read_text(encoding="utf-8") + self.assertIn("Real pinned-Chrome WebDriver evidence", changelog) + self.assertIn("does not claim a shipped OriginWeave browser adapter", changelog) + self.assertIn("PR #70", traceability) + self.assertIn("real WebDriver", traceability) + self.assertIn("not a product browser adapter", traceability) + self.assertIn("None of these introduces a new trust domain", traceability) + self.assertIn("pinned Chrome", fitness) + self.assertIn("not a browser adapter", fitness) + self.assertIn("browser-computed role/name", changelog) + self.assertIn("PR #71", traceability) + self.assertIn("computed role/name", traceability) + self.assertIn("computed role/name", fitness) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_fixture_server_evidence_publication_contract.py b/tests/test_fixture_server_evidence_publication_contract.py new file mode 100644 index 000000000..f0056e0b6 --- /dev/null +++ b/tests/test_fixture_server_evidence_publication_contract.py @@ -0,0 +1,133 @@ +"""Contract for publishing browser evidence only after owned fixture teardown.""" + +from __future__ import annotations + +import os +import pathlib +import runpy +import tempfile +import unittest +import unittest.mock + +ROOT = pathlib.Path(__file__).resolve().parents[1] +RUNNER = ROOT / "scripts" / "ci" / "run_mv3_compatibility.py" + + +class FixtureServerEvidencePublicationContractTests(unittest.TestCase): + """Keep success-shaped evidence behind both fixture-server cleanup post-conditions.""" + + def _run_main_with_controlled_servers( + self, + *, + fail_agent_cleanup: bool = False, + fail_trials: bool = False, + ) -> tuple[list[str], Exception | None]: + namespace = runpy.run_path( + str(RUNNER), + run_name="fixture_evidence_publication_contract", + ) + events: list[str] = [] + + class FakeServer: + def __init__(self, label: str, port: int) -> None: + self.label = label + self.server_port = port + + class FakeThread: + pass + + def start_fixture_server(directory: pathlib.Path) -> tuple[FakeServer, FakeThread]: + if directory == namespace["FIXTURE"]: + return FakeServer("mv3", 19001), FakeThread() + if directory == namespace["AGENT_TASK_FIXTURE"]: + return FakeServer("agent", 19002), FakeThread() + raise AssertionError("unexpected fixture directory") + + def stop_fixture_server(server: FakeServer, _thread: FakeThread) -> None: + events.append(f"stopped:{server.label}") + if fail_agent_cleanup and server.label == "agent": + raise RuntimeError("fixture server thread did not stop") + + def run_restart_trial( + _chrome_bin: pathlib.Path, + _chromedriver_bin: pathlib.Path, + _fixture_url: str, + trial_number: int, + ) -> dict[str, object]: + return { + "trial_number": trial_number, + "passed": not fail_trials, + "browser_version": namespace["PINNED_CHROME_VERSION"], + "surfaces": {"controlled": not fail_trials}, + "browser_passes": [], + } + + def run_agent_task_trial( + _chrome_bin: pathlib.Path, + _chromedriver_bin: pathlib.Path, + _fixture_url: str, + trial_number: int, + ) -> dict[str, object]: + return { + "trial_number": trial_number, + "passed": not fail_trials, + } + + namespace["_start_fixture_server"] = start_fixture_server + namespace["_stop_fixture_server"] = stop_fixture_server + namespace["_run_restart_trial"] = run_restart_trial + namespace["_run_agent_task_trial"] = run_agent_task_trial + namespace["_agent_task_surfaces_complete"] = lambda _trials: not fail_trials + namespace["print"] = lambda *_args, **_kwargs: events.append("evidence") + + error: Exception | None = None + with tempfile.TemporaryDirectory() as temp_dir: + chrome_bin = pathlib.Path(temp_dir) / "chrome" + chromedriver_bin = pathlib.Path(temp_dir) / "chromedriver" + chrome_bin.touch() + chromedriver_bin.touch() + with unittest.mock.patch.dict( + os.environ, + { + "CHROME_BIN": str(chrome_bin), + "CHROMEDRIVER_BIN": str(chromedriver_bin), + }, + ): + try: + self.assertEqual(namespace["main"](), 0) + except Exception as caught: + error = caught + return events, error + + def test_success_evidence_is_emitted_after_both_fixture_servers_stop(self) -> None: + """A success artifact must follow, not precede, the owned teardown witnesses.""" + + events, error = self._run_main_with_controlled_servers() + + self.assertIsNone(error) + self.assertEqual(events, ["stopped:agent", "stopped:mv3", "evidence"]) + + def test_cleanup_failure_cannot_leave_success_shaped_evidence(self) -> None: + """A teardown failure must fail closed before successful evidence publication.""" + + events, error = self._run_main_with_controlled_servers(fail_agent_cleanup=True) + + self.assertIsInstance(error, RuntimeError) + self.assertEqual(str(error), "fixture server thread did not stop") + self.assertEqual(events, ["stopped:agent", "stopped:mv3"]) + + def test_failed_trial_gate_keeps_bounded_evidence_before_raising(self) -> None: + """Moving success publication must not suppress bounded failure diagnostics.""" + + events, error = self._run_main_with_controlled_servers(fail_trials=True) + + self.assertIsInstance(error, RuntimeError) + self.assertEqual( + str(error), + "Manifest V3 repeatability gate failed: 0/3 trials passed", + ) + self.assertEqual(events, ["evidence", "stopped:agent", "stopped:mv3"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_fixture_server_shutdown_postcondition_contract.py b/tests/test_fixture_server_shutdown_postcondition_contract.py new file mode 100644 index 000000000..e1e024a61 --- /dev/null +++ b/tests/test_fixture_server_shutdown_postcondition_contract.py @@ -0,0 +1,81 @@ +"""Contract for proving loopback fixture-server shutdown actually completes.""" + +from __future__ import annotations + +import pathlib +import runpy +import unittest + +ROOT = pathlib.Path(__file__).resolve().parents[1] +RUNNER = ROOT / "scripts" / "ci" / "run_mv3_compatibility.py" + + +class FixtureServerShutdownPostconditionContractTests(unittest.TestCase): + """Reject cleanup evidence that relies on bounded join acknowledgement alone.""" + + def test_fixture_server_shutdown_fails_closed_if_thread_is_still_alive(self) -> None: + """A timed join must be followed by an explicit thread-termination observation.""" + + namespace = runpy.run_path(str(RUNNER), run_name="fixture_shutdown_postcondition_contract") + stop_fixture_server = namespace["_stop_fixture_server"] + + class FakeServer: + def __init__(self) -> None: + self.shutdown_called = False + self.close_called = False + + def shutdown(self) -> None: + self.shutdown_called = True + + def server_close(self) -> None: + self.close_called = True + + class StalledThread: + def __init__(self) -> None: + self.join_timeout: float | None = None + + def join(self, timeout: float | None = None) -> None: + self.join_timeout = timeout + + def is_alive(self) -> bool: + return True + + server = FakeServer() + thread = StalledThread() + with self.assertRaisesRegex( + RuntimeError, + r"^fixture server thread did not stop$", + ): + stop_fixture_server(server, thread) + + self.assertTrue(server.shutdown_called) + self.assertTrue(server.close_called) + self.assertEqual(thread.join_timeout, 5) + + def test_fixture_server_shutdown_accepts_observed_thread_termination(self) -> None: + """Successful cleanup requires the helper thread to be observed stopped.""" + + namespace = runpy.run_path(str(RUNNER), run_name="fixture_shutdown_success_contract") + stop_fixture_server = namespace["_stop_fixture_server"] + + class FakeServer: + def shutdown(self) -> None: + return None + + def server_close(self) -> None: + return None + + class StoppedThread: + def join(self, timeout: float | None = None) -> None: + self.join_timeout = timeout + + def is_alive(self) -> bool: + return False + + thread = StoppedThread() + stop_fixture_server(FakeServer(), thread) + self.assertEqual(thread.join_timeout, 5) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_mv3_browser_sandbox_contract.py b/tests/test_mv3_browser_sandbox_contract.py new file mode 100644 index 000000000..1d01915d5 --- /dev/null +++ b/tests/test_mv3_browser_sandbox_contract.py @@ -0,0 +1,42 @@ +"""Regression contract for preserving Chromium sandboxing in browser evidence.""" + +from __future__ import annotations + +import inspect +import pathlib +import runpy +import unittest + +ROOT = pathlib.Path(__file__).resolve().parents[1] +RUNNER = ROOT / "scripts" / "ci" / "run_mv3_compatibility.py" +DOCTORING = ROOT / "docs" / "doctoring" / "chromium-sandbox-evidence-boundary.md" + + +class BrowserSandboxContractTests(unittest.TestCase): + """Require every real-Chromium compatibility path to keep the process sandbox enabled.""" + + def test_real_browser_passes_do_not_disable_chromium_sandbox(self) -> None: + """Neither MV3 nor Agent Task evidence may launch Chrome with ``--no-sandbox``.""" + + namespace = runpy.run_path(str(RUNNER), run_name="browser_sandbox_contract") + for function_name in ("_run_browser_pass", "_run_agent_task_browser_pass"): + with self.subTest(function_name=function_name): + source = inspect.getsource(namespace[function_name]) + self.assertNotIn('"--no-sandbox"', source) + + def test_doctoring_describes_the_repaired_sandbox_boundary(self) -> None: + """Doctoring must not describe the repaired ordinary launch as still unsandboxed.""" + + text = DOCTORING.read_text(encoding="utf-8") + self.assertNotIn( + "ordinary Manifest V3 compatibility path still passes `--no-sandbox`", + text, + ) + self.assertIn( + "both ordinary Manifest V3 and Agent Task real-browser paths omit `--no-sandbox`", + text, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_mv3_page_diagnostic_redaction_contract.py b/tests/test_mv3_page_diagnostic_redaction_contract.py new file mode 100644 index 000000000..5be6e8663 --- /dev/null +++ b/tests/test_mv3_page_diagnostic_redaction_contract.py @@ -0,0 +1,286 @@ +"""Regression tests for credential-safe page-derived browser evidence failures.""" + +from __future__ import annotations + +import pathlib +import runpy +import unittest +from unittest.mock import patch + +ROOT = pathlib.Path(__file__).resolve().parents[1] +RUNNER = ROOT / "scripts" / "ci" / "run_mv3_compatibility.py" +HOSTILE_PAGE_VALUE = "buyer-secret-marker-must-not-reach-ci" + + +class Mv3PageDiagnosticRedactionContractTests(unittest.TestCase): + """Page-controlled observations may decide failure but must not become CI text.""" + + def test_real_click_failure_does_not_echo_page_text(self) -> None: + """A mismatched DOM post-condition must keep page text out of diagnostics.""" + + namespace = runpy.run_path(str(RUNNER), run_name="mv3_click_diagnostic_contract") + exercise = namespace["_exercise_real_click"] + element_ids = iter(("button-element", "output-element")) + request_count = 0 + + def find_element(*_args: object, **_kwargs: object) -> str: + return next(element_ids) + + def json_request(*_args: object, **_kwargs: object) -> dict[str, object]: + nonlocal request_count + request_count += 1 + if request_count == 1: + return {"value": None} + return {"value": HOSTILE_PAGE_VALUE} + + exercise.__globals__["_find_element"] = find_element + exercise.__globals__["_json_request"] = json_request + + with self.assertRaises(RuntimeError) as captured: + exercise(9515, "session-1") + + self.assertEqual(str(captured.exception), "real click post-condition failed") + self.assertNotIn(HOSTILE_PAGE_VALUE, str(captured.exception)) + + def test_mv3_convergence_failure_does_not_echo_page_dataset(self) -> None: + """Untrusted extension/page dataset values must not be serialized into CI errors.""" + + namespace = runpy.run_path(str(RUNNER), run_name="mv3_surface_diagnostic_contract") + wait_for_evidence = namespace["_wait_for_extension_evidence"] + wait_for_evidence.__globals__["FIXTURE_TIMEOUT_SECONDS"] = 0.5 + wait_for_evidence.__globals__["_execute"] = lambda *_args, **_kwargs: { + "content": HOSTILE_PAGE_VALUE, + "workerStartCount": "1", + } + + time_module = wait_for_evidence.__globals__["time"] + with patch.object(time_module, "monotonic", side_effect=(0.0, 0.0, 1.0)), patch.object( + time_module, + "sleep", + return_value=None, + ), self.assertRaises(RuntimeError) as captured: + wait_for_evidence(9515, "session-1", "initialized") + + self.assertEqual(str(captured.exception), "MV3 fixture did not converge") + self.assertNotIn(HOSTILE_PAGE_VALUE, str(captured.exception)) + + def test_agent_task_initial_url_mismatch_does_not_echo_observed_url(self) -> None: + """A browser-observed URL mismatch must not serialize page-controlled URL data.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_url_diagnostic_contract") + browser_pass = namespace["_run_agent_task_browser_pass"] + request_count = 0 + hostile_url = f"https://example.invalid/?value={HOSTILE_PAGE_VALUE}" + + class FakeDriver: + def terminate(self) -> None: + return None + + def wait(self, *, timeout: float) -> int: + del timeout + return 0 + + def json_request(*_args: object, **_kwargs: object) -> dict[str, object]: + nonlocal request_count + request_count += 1 + if request_count == 1: + return { + "value": { + "sessionId": "session-1", + "capabilities": {"browserVersion": namespace["PINNED_CHROME_VERSION"]}, + } + } + if request_count == 2: + return {"value": None} + return {"value": hostile_url} + + browser_pass.__globals__["_wait_for_driver"] = lambda *_args, **_kwargs: None + browser_pass.__globals__["_json_request"] = json_request + browser_pass.__globals__["_cleanup_browser_session_preserving_primary"] = ( + lambda *_args, **_kwargs: None + ) + subprocess_module = browser_pass.__globals__["subprocess"] + + with patch.object(subprocess_module, "Popen", return_value=FakeDriver()), self.assertRaises( + RuntimeError + ) as captured: + browser_pass( + pathlib.Path("/controlled/chrome"), + pathlib.Path("/controlled/chromedriver"), + "http://127.0.0.1:8080/index.html", + "/controlled/profile", + ) + + self.assertEqual(str(captured.exception), "Agent Task initial URL mismatch") + self.assertNotIn(HOSTILE_PAGE_VALUE, str(captured.exception)) + + def test_mv3_version_mismatch_does_not_echo_remote_capability(self) -> None: + """A remote browserVersion mismatch must remain local comparison state.""" + + namespace = runpy.run_path(str(RUNNER), run_name="mv3_version_diagnostic_contract") + browser_pass = namespace["_run_browser_pass"] + + class FakeDriver: + def terminate(self) -> None: + return None + + def wait(self, *, timeout: float) -> int: + del timeout + return 0 + + browser_pass.__globals__["_wait_for_driver"] = lambda *_args, **_kwargs: None + browser_pass.__globals__["_json_request"] = lambda *_args, **_kwargs: { + "value": { + "sessionId": "session-1", + "capabilities": {"browserVersion": HOSTILE_PAGE_VALUE}, + } + } + browser_pass.__globals__["_cleanup_browser_session_preserving_primary"] = ( + lambda *_args, **_kwargs: None + ) + subprocess_module = browser_pass.__globals__["subprocess"] + + with patch.object(subprocess_module, "Popen", return_value=FakeDriver()), self.assertRaises( + RuntimeError + ) as captured: + browser_pass( + pathlib.Path("/controlled/chrome"), + pathlib.Path("/controlled/chromedriver"), + "http://127.0.0.1:8080/page.html", + "/controlled/profile", + "initialized", + ) + + self.assertEqual(str(captured.exception), "unexpected Chrome version") + self.assertNotIn(HOSTILE_PAGE_VALUE, str(captured.exception)) + + def test_agent_task_version_mismatch_does_not_echo_remote_capability(self) -> None: + """Agent Task version gating must not serialize remote capability data.""" + + namespace = runpy.run_path(str(RUNNER), run_name="agent_task_version_diagnostic_contract") + browser_pass = namespace["_run_agent_task_browser_pass"] + + class FakeDriver: + def terminate(self) -> None: + return None + + def wait(self, *, timeout: float) -> int: + del timeout + return 0 + + browser_pass.__globals__["_wait_for_driver"] = lambda *_args, **_kwargs: None + browser_pass.__globals__["_json_request"] = lambda *_args, **_kwargs: { + "value": { + "sessionId": "session-1", + "capabilities": {"browserVersion": HOSTILE_PAGE_VALUE}, + } + } + browser_pass.__globals__["_cleanup_browser_session_preserving_primary"] = ( + lambda *_args, **_kwargs: None + ) + subprocess_module = browser_pass.__globals__["subprocess"] + + with patch.object(subprocess_module, "Popen", return_value=FakeDriver()), self.assertRaises( + RuntimeError + ) as captured: + browser_pass( + pathlib.Path("/controlled/chrome"), + pathlib.Path("/controlled/chromedriver"), + "http://127.0.0.1:8080/index.html", + "/controlled/profile", + ) + + self.assertEqual(str(captured.exception), "unexpected Agent Task Chrome version") + self.assertNotIn(HOSTILE_PAGE_VALUE, str(captured.exception)) + + def test_webdriver_http_failure_does_not_echo_remote_body(self) -> None: + """A non-success HTTP response may select failure but must not become CI payload.""" + + namespace = runpy.run_path(str(RUNNER), run_name="webdriver_http_diagnostic_contract") + json_request = namespace["_json_request"] + + class FakeResponse: + status = 500 + + def read(self, _limit: int) -> bytes: + return f'{{"value":{{"message":"{HOSTILE_PAGE_VALUE}"}}}}'.encode() + + class FakeConnection: + def request(self, *_args: object, **_kwargs: object) -> None: + return None + + def getresponse(self) -> FakeResponse: + return FakeResponse() + + def close(self) -> None: + return None + + http_client = json_request.__globals__["http"].client + with patch.object(http_client, "HTTPConnection", return_value=FakeConnection()), self.assertRaises( + RuntimeError + ) as captured: + json_request(9515, "GET", "/status") + + self.assertEqual(str(captured.exception), "WebDriver HTTP request failed with status 500") + self.assertNotIn(HOSTILE_PAGE_VALUE, str(captured.exception)) + + def test_error_shaped_success_response_does_not_echo_remote_error_text(self) -> None: + """A defensive error-shaped success response must not retain remote error text.""" + + namespace = runpy.run_path(str(RUNNER), run_name="webdriver_protocol_diagnostic_contract") + json_request = namespace["_json_request"] + + class FakeResponse: + status = 200 + + def read(self, _limit: int) -> bytes: + return ( + '{"value":{"error":"javascript error","message":"' + + HOSTILE_PAGE_VALUE + + '"}}' + ).encode() + + class FakeConnection: + def request(self, *_args: object, **_kwargs: object) -> None: + return None + + def getresponse(self) -> FakeResponse: + return FakeResponse() + + def close(self) -> None: + return None + + http_client = json_request.__globals__["http"].client + with patch.object(http_client, "HTTPConnection", return_value=FakeConnection()), self.assertRaises( + RuntimeError + ) as captured: + json_request(9515, "POST", "/session", {}) + + self.assertEqual(str(captured.exception), "WebDriver command failed") + self.assertNotIn("javascript error", str(captured.exception)) + self.assertNotIn(HOSTILE_PAGE_VALUE, str(captured.exception)) + + def test_driver_readiness_timeout_does_not_echo_last_exception(self) -> None: + """Startup timeout must not serialize the last remote diagnostic into CI text.""" + + namespace = runpy.run_path(str(RUNNER), run_name="webdriver_startup_diagnostic_contract") + wait_for_driver = namespace["_wait_for_driver"] + wait_for_driver.__globals__["STARTUP_TIMEOUT_SECONDS"] = 0.5 + wait_for_driver.__globals__["_json_request"] = ( + lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError(HOSTILE_PAGE_VALUE)) + ) + time_module = wait_for_driver.__globals__["time"] + + with patch.object(time_module, "monotonic", side_effect=(0.0, 0.0, 1.0)), patch.object( + time_module, + "sleep", + return_value=None, + ), self.assertRaises(RuntimeError) as captured: + wait_for_driver(9515) + + self.assertEqual(str(captured.exception), "ChromeDriver did not become ready") + self.assertNotIn(HOSTILE_PAGE_VALUE, str(captured.exception)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_webdriver_session_cleanup_contract.py b/tests/test_webdriver_session_cleanup_contract.py new file mode 100644 index 000000000..6e647cefe --- /dev/null +++ b/tests/test_webdriver_session_cleanup_contract.py @@ -0,0 +1,47 @@ +"""Protocol contract for bounded WebDriver session cleanup.""" + +from __future__ import annotations + +import inspect +import pathlib +import runpy +import unittest + +ROOT = pathlib.Path(__file__).resolve().parents[1] +RUNNER = ROOT / "scripts" / "ci" / "run_mv3_compatibility.py" + + +class WebDriverSessionCleanupContractTests(unittest.TestCase): + """Keep Delete Session aligned with the body-free WebDriver command.""" + + def test_delete_session_omits_request_body(self) -> None: + """DELETE /session/{id} must not manufacture an empty JSON payload.""" + + namespace = runpy.run_path(str(RUNNER), run_name="webdriver_cleanup_body_contract") + cleanup = namespace["_cleanup_browser_session"] + calls: list[tuple[tuple[object, ...], dict[str, object]]] = [] + + def record_request(*args: object, **kwargs: object) -> dict[str, object]: + calls.append((args, kwargs)) + return {"value": None} + + cleanup.__globals__["_json_request"] = record_request + cleanup(9515, "session-1") + + self.assertEqual( + calls, + [((9515, "DELETE", "/session/session-1"), {})], + ) + + def test_agent_task_cleanup_uses_shared_production_path_only(self) -> None: + """The executed Agent Task finalizer must use the shared cleanup path directly.""" + + namespace = runpy.run_path(str(RUNNER), run_name="webdriver_cleanup_path_contract") + browser_pass_source = inspect.getsource(namespace["_run_agent_task_browser_pass"]) + self.assertIn("_cleanup_browser_session_preserving_primary(", browser_pass_source) + self.assertNotIn("_cleanup_agent_task_browser_session(", browser_pass_source) + self.assertIn("_cleanup_browser_session_preserving_primary", namespace) + + +if __name__ == "__main__": + unittest.main()