Agent QA Harness

A lightweight, file-based QA harness that lets AI agents test your project in real browser sessions.

You bring the product. The agent reads a structured runbook, asks a few questions about your project, generates test cases, runs them, logs every result, and delivers a development handoff with bugs and UX observations — all without writing a line of test code.

Quick Start

1. Clone this repo
2. Open it in Claude Code, Codex, or Claude Desktop
3. Say "start onboarding"

The agent reads AGENTS.md, detects that project.config.md is unfilled, and walks you through setup. It asks about your product, checks your tooling, optionally explores your codebase, and generates an initial test_cases.jsonl. Then it runs the tests.

Two Modes

Setup Mode — runs automatically on first use (when project.config.md has [FILL IN] placeholders)

• Asks 6 questions about your product
• Checks tooling (browser, product accessible, auth state)
• Optionally explores your project repo to generate more accurate test cases
• Writes project.config.md and generates test_cases.jsonl
• Confirms with you before proceeding

Test Mode — runs when setup is complete

• Reads project.config.md for product name, platforms, and UI cues
• Executes test_cases.jsonl in order
• Appends each result immediately to test_results_YYYY-MM-DD.jsonl
• Flags usability friction, not just pass/fail
• Updates DEVELOPMENT_HANDOFF.md with bugs and improvements
• Generates test_report_YYYY-MM-DD.md

File Glossary

File	Purpose
AGENTS.md	Master runbook — all agent behavior flows from here
CLAUDE.md	Claude Code entry point (delegates to AGENTS.md)
project.config.md	Your project details, filled during onboarding
test_cases.jsonl	Test spec — one JSON object per line, generated during onboarding
test_results_YYYY-MM-DD.jsonl	Raw result log for a dated run
test_report_YYYY-MM-DD.md	Human-readable report for a dated run
DEVELOPMENT_HANDOFF.md	Bugs, UX issues, and improvements found during testing
PATTERNS.md	Known problem patterns and established solutions, discovered through real harness use

Common Prompts

Start onboarding.
Run tests on [platform].
Run a full test.
Run TC-010 through TC-020 on [platform].
Summarize failures from the latest report.

Worked Example

PromptMate is a Chrome extension that injects a prompt-management sidebar into Claude, ChatGPT, DeepSeek, and Kimi. Its full Agent QA Harness lives at promptmate-agent-qa-harness — 86 test cases across 18 categories, real test results, development handoff notes, and reports from live runs.

It is a complete, end-to-end example of this pattern in production use.

---

Roadmap — CLI Tool

The file-based harness works today with any AI agent. A future CLI layer will make setup and discovery faster, without changing the underlying pattern.

Planned:

• agent-qa init — interactive scaffolding: asks the same 6 onboarding questions, writes project.config.md and a starter test_cases.jsonl from the terminal, no agent required
• agent-qa check — validates project.config.md is complete and test_cases.jsonl is valid JSONL before handing off to an agent
• agent-qa add-test — appends a new test case interactively, auto-incrementing the ID
• agent-qa report — renders the latest test_report_YYYY-MM-DD.md as a terminal summary (pass/fail counts, failure list)
• agent-qa diff — compares two dated result files and surfaces regressions

The CLI is additive — the file format and AGENTS.md runbook stay unchanged. Anything the CLI produces, an agent can also produce, and vice versa.