feat(cli): add -f/--format to browser verify - #351
Conversation
`browser verify` was the last data-returning built-in with no structured
output. It reported a full verification result — row count, shape
violations, fixture match failures, site-memory state — as emoji prose
only, so an agent driving the adapter authoring loop had to scrape ✓/✗
text to find out whether the adapter passed.
The default `table` rendering is unchanged, byte for byte. Any other
format returns the result as data instead:
webcmd browser verify hn/top -f json
The report carries `ok`, `site`, `command`, `rowCount`, a `fixture`
block (`path`, `exists`, `action`), and a `memory` block. Failures
replace the prose with `shapeFailures`, `matchFailures`, or an `error`
with a code (`ADAPTER_NOT_FOUND`, `ADAPTER_EXEC_FAILED`,
`ADAPTER_OUTPUT_NOT_JSON`). `shapeFailures`/`matchFailures` reuse the
existing `ValidationFailure` rows from verify-fixture.ts and `memory`
reuses `SiteMemoryReport`, so nothing new is invented for the wire.
Exit codes are untouched on every path, so `-f json` is safe to add to
an existing verification step.
Refs agentrhq#175.
🟠 Maintainer review suggested — low confidenceThe automated review could not reach a fully supported conclusion. Limitations
This review is advisory and does not block merging. |
The skill pipeline added in agentrhq#347 builds skills/ from skill-src/, and `make verify` rebuilds and compares hashes. My previous commit edited the published skills/webcmd-usage/SKILL.md directly, so a fresh build regenerated it from the untouched source and the hashes diverged. Same text, applied to skill-src/webcmd-usage/SKILL.src.md. The published file already carries it, and the only source-to-published transformation in this file is the trailing author-only <!-- @ --> block litprompt strips, so the two are back in sync.
|
It is the only failure in the shard, and it is unrelated to this diff — this PR touches The test builds throwaway git repos and shells out to real Locally on Windows the file passes 14/14 in 8.96s wall clock. A re-run of just that shard should clear it. I can push an amend to pick up fresh runners if that is easier than a manual re-run, but I did not want to re-trigger the full matrix unprompted for a timing flake. Happy to open a separate issue proposing an explicit |
Refs #175.
Problem
browser verifyis the command the adapter-authoring loop is built around —webcmd-adapter-authordescribes the whole workflow as "a 30-minute loop from zero context to a passingwebcmd browser verify". It produces a genuinely structured result: a row count, row-shape violations, fixture match failures, and a site-memory check.All of it was emoji prose, with no
-f/--format:So an agent driving that loop had to scrape
✓/✗out of stdout to learn whether its adapter passed, and had no way to read which rule failed without parsing free text. That is the same failure mode #175 was opened for, andbrowser verifyis listed in its scope.After
Default
tableoutput is unchanged — byte for byte, on every path. Any other format returns the result as data:A passing run carries
ok: true,rowCount, thefixtureblock, and amemoryblock mirroring the existing site-memory check. Failures swap inshapeFailures(row-shape violations) ormatchFailures(fixture mismatches). When the adapter cannot be run or read at all, the report is anerrorwith a code:ADAPTER_NOT_FOUND,ADAPTER_EXEC_FAILED, orADAPTER_OUTPUT_NOT_JSON.Notes on the shape
shapeFailuresandmatchFailuresare the existingValidationFailurerows returned byvalidateRowShape/validateRowsinsrc/browser/verify-fixture.ts, andmemoryis the existingSiteMemoryReportfromcheckSiteMemory. Nothing new is invented for the wire — the structured path emits the objects the command already computed and was formatting into text.Exit codes are untouched on every path, including the
--strict-memoryescalation, so adding-f jsonto an existing verification step cannot change whether CI passes.Prose progress lines are gated behind a small
notice()helper rather than deleted, which is what keeps table mode byte-identical.-f/--formatresolves through the sameresolveOutputFormat/renderpair as every other formatted built-in, so alias and unknown-format handling comes along for free (-f xmlis a usage error, exit 2, before the adapter subprocess is spawned).Not included
The
browser verifyentry inbrowserCommandCatalogandhosted-contract.jsonare untouched: this is local CLI presentation, and the hosted surface renders its own output.While scoping this I found the "non-streaming browser reads" bullet in #175 (
tab list,state,frames,console,analyze,find,get title|url|text|value|attributes,eval,extract,network) no longer corresponds to live local commands — those verbs survive only in the retired-grammar rejection list insrc/cli-argv-preprocess.tsand as hosted protocol action names insrc/hosted/types.ts. The local surface consolidated them intobrowser runandbrowser snapshot, both of which already emit unconditional JSON throughrawBrowserAction. I've left a comment on #175 so a maintainer can re-scope or close that bullet.Verification
npx tsc --noEmitclean.npx vitest run src/cli.test.ts --project unit: 107 passed. The 6 pre-existingbrowser verifytests assert on prose substrings and still pass untouched, which is the regression guard for table mode.-f xmlrejected before the subprocess spawns. Each asserts the report shape, the exit code, and that emoji/prose does not reach stdout.npm run check:typed-error-lintandnpm run check:silent-column-drop: 0 new violations.npm run check:hosted-contractreports pre-existing drift on this checkout; it reports the same on a cleanmain, so it is unrelated to this change.