Skip to content

feat(bitwarden-testing-tools): add requirement readiness and test plan skills (QA-2463) #282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
awiester-bw merged 8 commits into from
Oct 28, 2026
Merged

feat(bitwarden-testing-tools): add requirement readiness and test plan skills (QA-2463) #282

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
43f187e
feat(bitwarden-testing-tools): add requirement readiness and test pla…
awiester-bw Oct 21, 2026
3309cf4
fix(bitwarden-testing-tools): address review findings, add test plan …
awiester-bw Oct 21, 2026
78c23ec
fix(bitwarden-testing-tools): address second-round review findings (Q…
awiester-bw Oct 21, 2026
5672d6f
fix(bitwarden-testing-tools): return unreadable design-only criteria …
awiester-bw Oct 21, 2026
41a112b
fix(bitwarden-testing-tools): list unreadable design criteria in ever…
awiester-bw Oct 21, 2026
f92c662
fix(bitwarden-testing-tools): name the fallback exclusion reason (QA-…
awiester-bw Oct 21, 2026
52633de
fix(bitwarden-testing-tools): route checklist fails to gaps, define t…
awiester-bw Oct 21, 2026
e55f5fe
fix(bitwarden-testing-tools): fix Figma discovery, epic exclusions, a…
awiester-bw Oct 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/marketplace.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@
{
"name": "bitwarden-testing-tools",
"source": "./plugins/bitwarden-testing-tools",
"version": "1.2.0",
"version": "1.3.0",
"description": "Testing tools for Bitwarden — analyzing and improving test quality across its repositories."
},
{
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ A curated collection of plugins for AI-assisted development at Bitwarden. Enable
| [bitwarden-security-engineer](plugins/bitwarden-security-engineer/) | 2.1.0 | Application security engineering: vulnerability triage, threat modeling, and secure code analysis |
| [bitwarden-software-engineer](plugins/bitwarden-software-engineer/) | 1.1.0 | Software engineer agent for a Bitwarden product team. Implements stories, tasks, and bugs with code quality, performance, security, and team comms in mind. |
| [bitwarden-stripe-tools](plugins/bitwarden-stripe-tools/) | 1.0.0 | Read-only Stripe test-mode queries, invoice previews, and test-clock advancement through a guarded CLI wrapper |
| [bitwarden-testing-tools](plugins/bitwarden-testing-tools/) | 1.2.0 | Testing tools for analyzing and improving test quality across Bitwarden's repositories. |
| [bitwarden-testing-tools](plugins/bitwarden-testing-tools/) | 1.3.0 | Testing tools for analyzing and improving test quality across Bitwarden's repositories. |
| [claude-config-validator](plugins/claude-config-validator/) | 2.2.1 | Validates Claude Code configuration files for security, structure, and quality |
| [claude-retrospective](plugins/claude-retrospective/) | 1.1.1 | Analyze Claude Code sessions to identify successful patterns and improvement opportunities |

Expand Down
2 changes: 1 addition & 1 deletion plugins/bitwarden-testing-tools/.claude-plugin/plugin.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "bitwarden-testing-tools",
"version": "1.2.0",
"version": "1.3.0",
"description": "Testing tools for Bitwarden — analyzing and improving test quality across its repositories.",
"author": {
"name": "Bitwarden",
Expand Down
8 changes: 8 additions & 0 deletions plugins/bitwarden-testing-tools/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@ All notable changes to the Bitwarden Testing Tools Plugin will be documented in
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.3.0] - 2026-10-21

### Added

- `assessing-requirement-readiness` skill: judges whether a Jira story is ready to build and test, and returns READY or NOT READY with the gaps to fix. Given an epic, it judges each child story.
- `designing-test-plans` skill: builds a test plan from a READY story, or from an epic's READY child stories, mapping each test to its acceptance criterion with its layer, test kind, and method (automated or manual).
- Trigger evals and baselines for both skills.

## [1.2.0] - 2026-09-14

### Added
Expand Down
14 changes: 9 additions & 5 deletions plugins/bitwarden-testing-tools/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,22 @@ A set of skills that support Bitwarden's testing and quality work with evidence

## Skills

| Skill | What It Does |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `assessing-test-coverage` | Determines what a change is **already tested by**. From a PR, Jira key, Tech Breakdown, or Testmo CSV, resolves the change surface, finds the existing tests PRs-first, buckets each by layer (unit / integration / E2E), cites it as a stable GitHub permalink, and records untested behaviors as gaps — writing a self-contained markdown report under `${CLAUDE_PLUGIN_DATA}/coverage-reports/`. |
| `writing-manual-test-cases` | Authors the **new manual test cases** a change needs. From a Jira ticket, PR, or feature description, gap-checks the requirements, plans the scenario coverage for approval, then drafts Gherkin cases — each classified Smoke / Regression / Functional with a matching Automation Type. Delivers a plain-text file for review and a Testmo-importable CSV under `${CLAUDE_PLUGIN_DATA}/writing-manual-test-cases/`. |
| `recommending-test-layers` | Recommends **which tests to add and at which layer**. From a Jira key, Testmo CSV, an `assessing-test-coverage` report, a PR, or a feature description, places each behavior at its lowest sufficient deterministic layer (static / unit / component / contract), then adds non-deterministic layers (integration, E2E, smoke, synthetic monitoring, exploratory) where each one's trigger is met, grading criticality against Bitwarden's Severity guide. Writes a markdown recommendation report under `${CLAUDE_PLUGIN_DATA}/recommending-test-layers/`. |
| Skill | What It Does |
| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `assessing-test-coverage` | Determines what a change is **already tested by**. From a PR, Jira key, Tech Breakdown, or Testmo CSV, resolves the change surface, finds the existing tests PRs-first, buckets each by layer (unit / integration / E2E), cites it as a stable GitHub permalink, and records untested behaviors as gaps — writing a self-contained markdown report under `${CLAUDE_PLUGIN_DATA}/coverage-reports/`. |
| `writing-manual-test-cases` | Authors the **new manual test cases** a change needs. From a Jira ticket, PR, or feature description, gap-checks the requirements, plans the scenario coverage for approval, then drafts Gherkin cases — each classified Smoke / Regression / Functional with a matching Automation Type. Delivers a plain-text file for review and a Testmo-importable CSV under `${CLAUDE_PLUGIN_DATA}/writing-manual-test-cases/`. |
| `recommending-test-layers` | Recommends **which tests to add and at which layer**. From a Jira key, Testmo CSV, an `assessing-test-coverage` report, a PR, or a feature description, places each behavior at its lowest sufficient deterministic layer (static / unit / component / contract), then adds non-deterministic layers (integration, E2E, smoke, synthetic monitoring, exploratory) where each one's trigger is met, grading criticality against Bitwarden's Severity guide. Writes a markdown recommendation report under `${CLAUDE_PLUGIN_DATA}/recommending-test-layers/`. |
| `assessing-requirement-readiness` | Judges whether a Jira story is **READY to build and test**; given an epic, enumerates its child stories and judges each. Checks each acceptance criterion for clarity, testability, and completeness, reads linked issues, sibling stories, and any Figma design, and returns READY or NOT READY with the specific gaps to fix. Reviews only and does not write the specification. |
| `designing-test-plans` | Designs the **tests a READY story or epic needs**, derived from its acceptance criteria. Gates on `assessing-requirement-readiness` first, then maps each acceptance criterion to the tests that cover it, with traceability from criterion to test. Designs only and writes no test code. |
Comment thread
awiester-bw marked this conversation as resolved.
Resolved

## Cross-Plugin Integration

| Plugin | How It's Used |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bitwarden-atlassian-tools` | **Recommended** — the primary way to drive analysis from Jira tickets and linked Confluence requirements, via its `researching-jira-issues` skill and Atlassian MCP tools. `recommending-test-layers` also uses its `get_confluence_page` tool to grade criticality against the Bitwarden Defect Severity Classification Guide, for any input type. Optional by design: if absent, drive the analysis from the PR / CSV / tech-breakdown / description instead and grade criticality by judgment. A Jira ticket input, however, requires the plugin — without it, stop and ask the user to install and configure it. |

`assessing-requirement-readiness` and `designing-test-plans` also require `bitwarden-atlassian-tools`. The readiness skill reads the story with `get_issue` and gets linked issues and Confluence context through `researching-jira-issues`, and its input is always a Jira story or epic key. Without the plugin, stop and ask the user to install and configure it.

## Installation

```bash
Expand Down
12 changes: 8 additions & 4 deletions plugins/bitwarden-testing-tools/evals/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ timeout — `--timeout` bounds anything that stalls.
a non-trigger and warns on stderr, so a slow run is not silently read as a real failure.
- `--model` (default `claude-opus-4-8`) — pass the same model the baseline was recorded on so
verdicts are comparable. Each `baseline.json` records its model in a top-level `model` field;
match it. Both current baselines were recorded on `claude-opus-4-8`, the model most users run.
match it. All four current baselines were recorded on `claude-opus-4-8`, the model most users run.

## Running

Expand All @@ -68,8 +68,9 @@ python3 ../../../evals/run_real_eval.py \

A 20-query set at `--runs-per-query 7` is 140 `claude -p` invocations; with 3 workers it takes
several minutes. `--num-workers` is the memory knob (see _Arguments_). The two coverage skills
share trigger space, so after any description change to one, re-run the other's eval too to
confirm neither cannibalizes the other's triggers.
share trigger space, and so do `assessing-requirement-readiness` and `designing-test-plans`. After
a description change to either skill in a pair, re-run both evals in that pair to confirm neither
cannibalizes the other's triggers.

## Regression check

Expand All @@ -94,7 +95,10 @@ description change.

## Known below-threshold queries

None. On `claude-opus-4-8` both baselines pass every query: `recommending-test-layers` at 10/10
- `assessing-requirement-readiness`: `is CL-1221 actually testable or is it hand-wavy`, 2/7 should-trigger on `claude-opus-4-8`. Too short and vague to route.
- `assessing-requirement-readiness`: `the autofill story we're refining next has a figma link and two lines of description. will that hold up or should I send it back to the PO now?`, 2/7 should-trigger on `claude-opus-4-8`. Names no Jira ticket, and the skill starts from a Jira story.

On `claude-opus-4-8` the `recommending-test-layers` and `assessing-test-coverage` baselines pass every query: `recommending-test-layers` at 10/10
should-trigger and 10/10 should-not-trigger, `assessing-test-coverage` likewise 10/10 and 10/10.
The mechanism still allows a baseline to _record_ a should-trigger query below threshold and diff
clean; when the model was `claude-sonnet-5`, `assessing-test-coverage`'s branch-audit phrasing
Expand Down
Loading
Loading