Skip to content

feat(cli): add -f/--format to browser verify - #351

Open
Agnik47 wants to merge 2 commits into
agentrhq:mainfrom
Agnik47:feat/175-browser-verify-format
Open

feat(cli): add -f/--format to browser verify#351
Agnik47 wants to merge 2 commits into
agentrhq:mainfrom
Agnik47:feat/175-browser-verify-format

Conversation

@Agnik47

@Agnik47 Agnik47 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Refs #175.

Problem

browser verify is the command the adapter-authoring loop is built around — webcmd-adapter-author describes the whole workflow as "a 30-minute loop from zero context to a passing webcmd browser verify". It produces a genuinely structured result: a row count, row-shape violations, fixture match failures, and a site-memory check.

All of it was emoji prose, with no -f/--format:

🔍 Verifying hn/top...

  Loading: /home/u/.webcmd/clis/hn/top.js
  Executing: webcmd hn top --limit 3

  → 3 rows

  ✗ Adapter output does not match fixture:
    - [rowCount] got 3 rows, expected at least 5

So an agent driving that loop had to scrape / out of stdout to learn whether its adapter passed, and had no way to read which rule failed without parsing free text. That is the same failure mode #175 was opened for, and browser verify is listed in its scope.

After

Default table output is unchanged — byte for byte, on every path. Any other format returns the result as data:

$ webcmd browser verify hn/top -f json
{
  "ok": false,
  "site": "hn",
  "command": "top",
  "rowCount": 3,
  "fixture": {
    "path": "/home/u/.webcmd/sites/hn/verify/top.json",
    "exists": true,
    "action": "none"
  },
  "matchFailures": [
    { "rule": "rowCount", "detail": "got 3 rows, expected at least 5" }
  ]
}

A passing run carries ok: true, rowCount, the fixture block, and a memory block mirroring the existing site-memory check. Failures swap in shapeFailures (row-shape violations) or matchFailures (fixture mismatches). When the adapter cannot be run or read at all, the report is an error with a code: ADAPTER_NOT_FOUND, ADAPTER_EXEC_FAILED, or ADAPTER_OUTPUT_NOT_JSON.

Notes on the shape

shapeFailures and matchFailures are the existing ValidationFailure rows returned by validateRowShape/validateRows in src/browser/verify-fixture.ts, and memory is the existing SiteMemoryReport from checkSiteMemory. Nothing new is invented for the wire — the structured path emits the objects the command already computed and was formatting into text.

Exit codes are untouched on every path, including the --strict-memory escalation, so adding -f json to an existing verification step cannot change whether CI passes.

Prose progress lines are gated behind a small notice() helper rather than deleted, which is what keeps table mode byte-identical. -f/--format resolves through the same resolveOutputFormat/render pair as every other formatted built-in, so alias and unknown-format handling comes along for free (-f xml is a usage error, exit 2, before the adapter subprocess is spawned).

Not included

The browser verify entry in browserCommandCatalog and hosted-contract.json are untouched: this is local CLI presentation, and the hosted surface renders its own output.

While scoping this I found the "non-streaming browser reads" bullet in #175 (tab list, state, frames, console, analyze, find, get title|url|text|value|attributes, eval, extract, network) no longer corresponds to live local commands — those verbs survive only in the retired-grammar rejection list in src/cli-argv-preprocess.ts and as hosted protocol action names in src/hosted/types.ts. The local surface consolidated them into browser run and browser snapshot, both of which already emit unconditional JSON through rawBrowserAction. I've left a comment on #175 so a maintainer can re-scope or close that bullet.

Verification

  • npx tsc --noEmit clean.
  • npx vitest run src/cli.test.ts --project unit: 107 passed. The 6 pre-existing browser verify tests assert on prose substrings and still pass untouched, which is the regression guard for table mode.
  • 5 new tests cover the structured path: fixture-less pass, row-shape violation, fixture mismatch, missing adapter, and -f xml rejected before the subprocess spawns. Each asserts the report shape, the exit code, and that emoji/prose does not reach stdout.
  • npm run check:typed-error-lint and npm run check:silent-column-drop: 0 new violations.
  • npm run check:hosted-contract reports pre-existing drift on this checkout; it reports the same on a clean main, so it is unrelated to this change.

`browser verify` was the last data-returning built-in with no structured
output. It reported a full verification result — row count, shape
violations, fixture match failures, site-memory state — as emoji prose
only, so an agent driving the adapter authoring loop had to scrape ✓/✗
text to find out whether the adapter passed.

The default `table` rendering is unchanged, byte for byte. Any other
format returns the result as data instead:

    webcmd browser verify hn/top -f json

The report carries `ok`, `site`, `command`, `rowCount`, a `fixture`
block (`path`, `exists`, `action`), and a `memory` block. Failures
replace the prose with `shapeFailures`, `matchFailures`, or an `error`
with a code (`ADAPTER_NOT_FOUND`, `ADAPTER_EXEC_FAILED`,
`ADAPTER_OUTPUT_NOT_JSON`). `shapeFailures`/`matchFailures` reuse the
existing `ValidationFailure` rows from verify-fixture.ts and `memory`
reuses `SiteMemoryReport`, so nothing new is invented for the wire.

Exit codes are untouched on every path, so `-f json` is safe to add to
an existing verification step.

Refs agentrhq#175.
@github-actions

Copy link
Copy Markdown
Contributor

🟠 Maintainer review suggested — low confidence

The automated review could not reach a fully supported conclusion.

Limitations

  • The automated review returned an invalid structured result.

This review is advisory and does not block merging.

The skill pipeline added in agentrhq#347 builds skills/ from skill-src/, and
`make verify` rebuilds and compares hashes. My previous commit edited
the published skills/webcmd-usage/SKILL.md directly, so a fresh build
regenerated it from the untouched source and the hashes diverged.

Same text, applied to skill-src/webcmd-usage/SKILL.src.md. The published
file already carries it, and the only source-to-published transformation
in this file is the trailing author-only <!-- @ --> block litprompt
strips, so the two are back in sync.
@Agnik47

Agnik47 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Unit tests (windows-latest, shard 2/2) is red on a timeout, not an assertion:

× src/plugin-pr-scope.test.ts > plugin-only PR scope guard
  > rejects a new plugin plus an existing-plugin edit   6095ms
  → Test timed out in 5000ms.

Tests  1 failed | 1268 passed | 66 skipped (1335)

It is the only failure in the shard, and it is unrelated to this diff — this PR touches src/cli.ts, src/cli.test.ts, docs/cli-reference.mdx, and one skill source. It adds no plugin, and changes nothing under plugins/ or in the scope-guard code the test exercises.

The test builds throwaway git repos and shells out to real git, which is slow on the Windows runner. Its siblings in the same file on the same run took 3479ms and 1084ms against a 5000ms default, so this case — which adds a plugin and edits an existing one, i.e. the most fixture setup of the group — sits closest to the limit and crossed it. Plugin tests (windows-latest) and both other Windows jobs passed on this commit.

Locally on Windows the file passes 14/14 in 8.96s wall clock.

A re-run of just that shard should clear it. I can push an amend to pick up fresh runners if that is easier than a manual re-run, but I did not want to re-trigger the full matrix unprompted for a timing flake.

Happy to open a separate issue proposing an explicit testTimeout for plugin-pr-scope.test.ts if this shows up on other PRs — it looks like it would recur on any sufficiently loaded Windows runner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant