feat(contracts): add image_qa assertion to the outcome contract DSL

[shaun0927]

Summary

Stacked on #1441 (Part 1).

• New image_qa leaf assertion in the contract DSL: { question, expected_pattern }.
• Evaluator goes through a new optional EvalContext.imageQaSample hook so the runtime can route to the image_qa MCP tool (and hence to host sampling). Absent hook or unsupported_by_host reply → inconclusive (passed:false).
• 10 tests: validator (4), evaluator (5), dispatch via evaluate() (1).

SSOT (#1359) alignment

• No server-side LLM. Evaluator is inconclusive without host wiring.
• ReDoS guard via validateRegexPattern at author time and compileSafeRegex at evaluation time.

Review-pass changes

• Documented the image_qa kind in docs/contracts.md and added it to the oc_assert contract schema description (hosts can now discover it).
• Routed the evaluator's runtime regex through compileSafeRegex for consistency/defense-in-depth.
• Added a validator test for a missing expected_pattern.

Part 2 of #1432. Depends on #1441.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[shaun0927]

Goal-review verdict — #1445

Intent vs SSOT (#1359)

• Pillar C (contract-verifiable browser work) + Pillar D (portable verified memory). Adds an image_qa clause to the outcome contract DSL so a contract can cite a vision-mediated answer as evidence. Crucially:
  • P2 (harness, not agent) preserved: the evaluator only fires when the runtime has wired EvalContext.imageQaSample. With no hook OR unsupported_by_host, the result is inconclusive (passed: false). OpenChrome never invokes a model itself.
  • P4 (facts before decisions): the evidence record carries question, answer, and expected_pattern verbatim, plus a reason enum (host_runtime_did_not_wire_imageQaSample, unsupported_by_host, invalid_expected_pattern, no_screenshot_available) so the host can act on the failure mode without re-deriving it.
  • P1/P7 (host-neutral MCP, core/pilot split): pure data-layer addition; the image_qa MCP tool from feat(tools): add image_qa primitive via MCP sampling #1441 is the only path to a host LLM, and the contract evaluator can be tested with a fake hook.

Code-level review

• src/contracts/types.ts: ImageQaAssertion joins LeafAssertion. Assertion['kind'] therefore now includes 'image_qa' — this is the change that triggered the inherited TS2366 in oc-assert.ts:expectedActualFor. Fix landed as fix(oc_assert): handle image_qa kind in expectedActualFor switch (commit c96aaa69, now fb7546da after rebase) — emits { question, expected_pattern } / passthrough details, matching the pattern used by screenshot_class.
• src/contracts/validator.ts: validateImageQa reuses validateRegexPattern for ReDoS safety. Good.
• src/contracts/evaluators/image-qa.ts: handles all four inconclusive paths and regex.test on the answer. The 9/9 tests in tests/contracts/image-qa.test.ts pass locally.
• src/contracts/evaluate.ts: dispatch case wired; the try/catch in evaluate() still wraps the evaluator so any thrown errors become passed: false with details.error.

CI fixes applied

• Rebased on the updated parent feat/1432-image-qa-v2 head 8f72cba8 (parent feat(tools): add image_qa primitive via MCP sampling #1441 fix: REQUIRED prefix on required field descriptions + image_qa in TOOL_CAPABILITY_MAP and the v1.11 snapshot). Combined with the local expectedActualFor switch fix, build and the schema-lint / capability-filter test surface both clear locally.

Status

Merge-ready pending parent #1441 landing. No further changes expected.

[shaun0927]

Analysis & merge summary (automated review)

Intent / direction. Part 2 of #1432: adds an image_qa leaf assertion { kind, question, expected_pattern } to the outcome-contract DSL (validator + evaluator + evaluate dispatch + oc_assert expectedActualFor). It asks the host LLM about the latest screenshot via an optional EvalContext.imageQaSample hook and regex-matches the answer.

#1359 (SSOT) alignment — consistent. No server-side LLM. The evaluator only calls the host-provided hook; when the hook is absent / returns unsupported_by_host / no screenshot exists, it degrades deterministically to passed: false. This matches P2 (harness, not agent) and P4 (facts before decisions); the regex is ReDoS-guarded at parse time via validateRegexPattern.

Note on the diff. Against current develop the two-dot diff appeared to delete browser-lanes.ts / oc-lane.ts code — that was a develop-ahead artifact (the branch's merge-base predates #1439's scratch-profile-lane work). The two real commits only touch contract/image_qa files; a 3-way merge does not revert #1439. I rebased onto develop to confirm: the net change is exactly the 7 image_qa files, and it builds clean against current develop.

Review findings (both addressed in the follow-up #1452, by design):

• P1 — ReDoS: the evaluator compiled new RegExp(expected_pattern) directly. In this PR alone the regex path is unreachable in production (the evaluator returns at the !imageQaSample gate before any hook is wired), so the fix lands in feat(contracts): inject imageQaSample hook into oc_assert runtime #1452 where the hook is actually activated — there it swaps in compileSafeRegex.
• P2 — verdict translation: missing-hook outcomes here read as fail rather than inconclusive; translating them to inconclusive (via details.error) is exactly feat(contracts): inject imageQaSample hook into oc_assert runtime #1452's purpose.

Verification. tsc build clean; 114 contract tests pass.

Merged to develop.

[shaun0927]

Review analysis — merge-readiness

Intent & direction. Part 2 of #1432: adds an image_qa leaf to the Outcome Contract DSL so a contract can name a vision-Q&A clause as evidence, routed to the host via the image_qa primitive (#1441) / MCP sampling. Sound and matches the issue.

#1359 alignment — PASS. No path where OpenChrome itself invokes a model: the evaluator returns inconclusive (passed:false) when the runtime doesn't wire imageQaSample or when the host replies unsupported_by_host. The single-call oc_assert surface deliberately does not wire the hook, so image_qa is only decidable inside a sampling-capable host runtime — exactly the host-neutral, deterministic-fallback shape #1359 requires.

Findings & fixes applied this pass:

• (was P1) Host-facing discoverability. The kind was added to the validator/evaluator but docs/contracts.md omitted it and the oc_assert contract inputSchema description left it out of the kind list — hosts had no portable way to discover it. Added a DSL reference section + schema-description entry (commit af621294).
• (was P2) Regex safety consistency. The evaluator recompiled expected_pattern with a bare new RegExp — the only pattern-matching evaluator in src/contracts not routed through the safe-regex guard. Switched to compileSafeRegex for defense-in-depth on the hot path (commit d2d0217f).
• (was P2) Added a validator test for a missing expected_pattern.

Exhaustiveness checked: KNOWN_KINDS, evaluate() dispatch, LeafAssertion union, and oc-assert expectedActualFor are all updated; task-signature treats assertions opaquely (no switch gap).

Tests: image-qa.test.ts 10/10 pass locally; typecheck clean.

Verdict: MERGE-READY once CI is green. Targets develop; dependency #1441 already merged.