Skip to content

[QA-2464] feat(bitwarden-testing-tools): review existing Testmo cases in assessing-test-coverage #307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
awiester-bw wants to merge 9 commits into
base: main
Choose a base branch
from
+4,374 −76
Open

[QA-2464] feat(bitwarden-testing-tools): review existing Testmo cases in assessing-test-coverage #307

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f538d6d
feat(bitwarden-testing-tools): review existing Testmo cases in assess…
awiester-bw Oct 30, 2026
b9a77c3
fix(bitwarden-testing-tools): update MCP SDK and zod to patched versi…
awiester-bw Oct 30, 2026
0df9f6f
fix(bitwarden-testing-tools): address Validate review findings in ass…
awiester-bw Oct 30, 2026
f3821a0
fix(bitwarden-testing-tools): add step J fields to entry layouts and …
awiester-bw Nov 2, 2026
f2ec3d5
fix(bitwarden-testing-tools): cover every section in the no-changes l…
awiester-bw Nov 2, 2026
8c8b754
fix(bitwarden-testing-tools): rename the Testmo key variable, add a g…
awiester-bw Nov 4, 2026
7314eb8
fix(bitwarden-testing-tools): read SetProperty("Covers", ...) in the …
awiester-bw Nov 4, 2026
f501a54
fix(bitwarden-testing-tools): stop reporting every GitHub 403 as a ra…
awiester-bw Nov 4, 2026
bd311e5
Merge remote-tracking branch 'origin/main' into skill-c-testmo-poc
awiester-bw Nov 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude-plugin/marketplace.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,8 @@
{
"name": "bitwarden-testing-tools",
"source": "./plugins/bitwarden-testing-tools",
"version": "1.4.0",
"description": "Testing tools for Bitwarden — analyzing and improving test quality across its repositories."
"version": "1.5.0",
"description": "Testing tools for Bitwarden: test coverage and Testmo case review, test layer recommendations, manual test cases, requirement readiness, test plans, and QA manual regression runs, plus the bundled bitwarden-test-automation MCP server."
},
{
"name": "bitwarden-doc-parity",
Expand Down
1 change: 1 addition & 0 deletions .cspell.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"AKIA",
"alnum",
"anthropics",
"apos",
"appsec",
"appsettings",
"architecting",
Expand Down
133 changes: 133 additions & 0 deletions .github/workflows/test-automation-mcp-server-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
name: Test Automation MCP Server Testing

on:
pull_request:
types: [opened, synchronize, reopened]
paths:
- "plugins/bitwarden-testing-tools/mcp/bitwarden-test-automation-mcp-server/**"
- "plugins/bitwarden-testing-tools/.mcp.json"
- ".github/workflows/test-automation-mcp-server-test.yml"
- ".nvmrc"
push:
branches:
- main
paths:
- "plugins/bitwarden-testing-tools/mcp/bitwarden-test-automation-mcp-server/**"
- "plugins/bitwarden-testing-tools/.mcp.json"
- ".github/workflows/test-automation-mcp-server-test.yml"
- ".nvmrc"
workflow_dispatch:

permissions: {}

defaults:
run:
working-directory: plugins/bitwarden-testing-tools/mcp/bitwarden-test-automation-mcp-server

jobs:
testing:
name: Run tests
runs-on: ubuntu-24.04
permissions:
contents: read

steps:
- name: Check out repo
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Get Node version
id: retrieve-node-version
working-directory: ${{ github.workspace }}
run: |
NODE_NVMRC=$(cat .nvmrc)
NODE_VERSION=${NODE_NVMRC/v/''}
echo "node_version=$NODE_VERSION" >> "$GITHUB_OUTPUT"

- name: Set up Node
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: ${{ steps.retrieve-node-version.outputs.node_version }}

- name: Set up pnpm via Corepack
run: |
COREPACK_VERSION=$(jq -er '.devDependencies.corepack' package.json)
mkdir -p "$RUNNER_TEMP/corepack-bin"
npm exec --yes --package=corepack@"$COREPACK_VERSION" -- corepack enable --install-directory "$RUNNER_TEMP/corepack-bin"
echo "$RUNNER_TEMP/corepack-bin" >> "$GITHUB_PATH"

- name: Get pnpm store directory
id: pnpm-store
run: echo "path=$(pnpm store path --silent)" >> "$GITHUB_OUTPUT"

- name: Cache pnpm store
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ steps.pnpm-store.outputs.path }}
key: ${{ runner.os }}-pnpm-store-${{ hashFiles('plugins/bitwarden-testing-tools/mcp/bitwarden-test-automation-mcp-server/pnpm-lock.yaml') }}
restore-keys: ${{ runner.os }}-pnpm-store-

- name: Install dependencies
run: pnpm install --frozen-lockfile

- name: Run tests
run: pnpm run test

launch:
name: Launch via .mcp.json
runs-on: ubuntu-24.04
permissions:
contents: read
strategy:
fail-fast: false
matrix:
# Node 24 bundles Corepack and Node 26 does not. The launch path has to
# work on both without anything being installed or enabled first.
node-version: ["24", "26"]

steps:
- name: Check out repo
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Set up Node
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: ${{ matrix.node-version }}

- name: Build a PATH holding only node and npm
working-directory: ${{ github.workspace }}
run: |
mkdir -p "$RUNNER_TEMP/node-only-bin"
ln -sf "$(command -v node)" "$RUNNER_TEMP/node-only-bin/node"
ln -sf "$(command -v npm)" "$RUNNER_TEMP/node-only-bin/npm"

- name: Start the server through .mcp.json and require an MCP handshake
working-directory: plugins/bitwarden-testing-tools
run: |
set -eu

# Read the command from the shipped artifact so this test cannot drift from it.
CMD=$(jq -er '.mcpServers["bitwarden-test-automation"].args[1]' .mcp.json)
printf '%s\n' "Launch command under test:" "$CMD"

# Fail loudly if the runner image puts Corepack back on PATH, rather
# than assuming a given Node line lacks it.
{
printf '%s\n' 'if command -v corepack >/dev/null; then echo "corepack leaked into PATH" >&2; exit 1; fi'
printf '%s\n' "$CMD"
} > "$RUNNER_TEMP/launch.sh"

REQ='{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"ci","version":"1.0"}}}'
OUT=$(printf '%s\n' "$REQ" | env -i \
HOME="$HOME" \
PATH="$RUNNER_TEMP/node-only-bin:/usr/bin:/bin" \
RUNNER_TEMP="$RUNNER_TEMP" \
CLAUDE_PLUGIN_ROOT="$PWD" \
bash "$RUNNER_TEMP/launch.sh" | head -1)

printf '%s\n' "$OUT"
printf '%s' "$OUT" | jq -e '.result.serverInfo.name == "bitwarden-test-automation-mcp"' >/dev/null \
|| { echo "unexpected initialize result" >&2; exit 1; }
50 changes: 25 additions & 25 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,31 +4,31 @@ A curated collection of plugins for AI-assisted development at Bitwarden. Enable

## Available Plugins

| Plugin | Version | Description |
| ----------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [bitwarden-abacus-tools](plugins/bitwarden-abacus-tools/) | 1.0.0 | Abacus profile: which Bitwarden skills Abacus uses at each step: Jira issues to plan, commit and PR conventions to land. |
| [bitwarden-ai-telemetry](plugins/bitwarden-ai-telemetry/) | 1.5.1 | Claude Code hooks emitting metadata-only AI-usage telemetry (session, identity, git-linkage, MCP) tagged with the signed-in account email via OTLP |
| [bitwarden-tech-lead](plugins/bitwarden-tech-lead/) | 3.1.0 | Tech lead for technical planning, architecture coherence, and surfacing patterns to Technical Strategy Ideas |
| [bitwarden-shepherd](plugins/bitwarden-shepherd/) | 1.0.1 | Champion of a technical strategy — shepherds a TSI through evaluation into the funnel, then through to adoption |
| [bitwarden-adr-tools](plugins/bitwarden-adr-tools/) | 1.0.0 | Architecture Decision Record tools: checking designs and plans against the ADR catalog, locating and summarizing ADRs |
| [bitwarden-atlassian-tools](plugins/bitwarden-atlassian-tools/) | 2.8.2 | Atlassian access via MCP server with deep Jira issue research skill and opt-in Jira write tools |
| [bitwarden-code-review](plugins/bitwarden-code-review/) | 2.7.0 | Autonomous code review agent following Bitwarden engineering standards with GitHub integration |
| [bitwarden-delivery-tools](plugins/bitwarden-delivery-tools/) | 3.5.2 | Delivery lifecycle skills: initiative funnel navigation, work transitions, architectural judgment, feature flags, commits, PRs, preflight, labeling, Jira ticket filing |
| [bitwarden-designer](plugins/bitwarden-designer/) | 0.1.0 | Product designer persona: Code of Conduct and 30/60/90 critique, critique facilitation; dispatches into bitwarden-design-tools |
| [bitwarden-design-tools](plugins/bitwarden-design-tools/) | 0.1.0 | Design toolkit: content style guide, Figma Dev Mode MCP, Bitwarden brand application, handoff prep, Design System governance, Product and Design Jira |
| [bitwarden-devops-engineer](plugins/bitwarden-devops-engineer/) | 0.3.1 | DevOps engineering assistant: workflow compliance linting, action security auditing, and org-wide CI/CD remediation |
| [bitwarden-doc-parity](plugins/bitwarden-doc-parity/) | 1.0.1 | Keeps docs current with the code they describe: session instructions, a Stop-hook tripwire, a semantic verification skill, and a CI review face |
| [bitwarden-init](plugins/bitwarden-init/) | 1.2.2 | Initialize and enhance CLAUDE.md files with Bitwarden's standardized template format |
| [bitwarden-mailcatcher-tools](plugins/bitwarden-mailcatcher-tools/) | 1.0.0 | Reads verification links, magic links, and other action links from the local Bitwarden Mailcatcher inbox |
| [bitwarden-product-analyst](plugins/bitwarden-product-analyst/) | 0.1.7 | Product analyst agent for creating comprehensive Bitwarden requirements documents from multiple sources, and writing user-facing release notes |
| [bitwarden-retrospective-tools](plugins/bitwarden-retrospective-tools/) | 1.1.0 | Daily work recap built from your Claude Code session logs, GitHub activity, and Jira and Confluence activity |
| [bitwarden-security-engineer](plugins/bitwarden-security-engineer/) | 2.1.0 | Application security engineering: vulnerability triage, threat modeling, and secure code analysis |
| [bitwarden-software-engineer](plugins/bitwarden-software-engineer/) | 1.1.0 | Software engineer agent for a Bitwarden product team. Implements stories, tasks, and bugs with code quality, performance, security, and team comms in mind. |
| [bitwarden-stripe-tools](plugins/bitwarden-stripe-tools/) | 1.0.0 | Read-only Stripe test-mode queries, invoice previews, and test-clock advancement through a guarded CLI wrapper |
| [bitwarden-testing-tools](plugins/bitwarden-testing-tools/) | 1.4.0 | Testing tools for analyzing and improving test quality across Bitwarden's repositories, and setting up the QA manual regression runs for a release. |
| [bitwarden-testmo-tools](plugins/bitwarden-testmo-tools/) | 1.0.0 | Testmo API access: create test runs from reviewable filter specs, one or many per pass, with dry-run, milestone linkage, and duplicate checks |
| [claude-config-validator](plugins/claude-config-validator/) | 2.3.0 | Validates Claude Code configuration files for security, structure, and quality |
| [claude-retrospective](plugins/claude-retrospective/) | 1.1.1 | Analyze Claude Code sessions to identify successful patterns and improvement opportunities |
| Plugin | Version | Description |
| ----------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [bitwarden-abacus-tools](plugins/bitwarden-abacus-tools/) | 1.0.0 | Abacus profile: which Bitwarden skills Abacus uses at each step: Jira issues to plan, commit and PR conventions to land. |
| [bitwarden-ai-telemetry](plugins/bitwarden-ai-telemetry/) | 1.5.1 | Claude Code hooks emitting metadata-only AI-usage telemetry (session, identity, git-linkage, MCP) tagged with the signed-in account email via OTLP |
| [bitwarden-tech-lead](plugins/bitwarden-tech-lead/) | 3.1.0 | Tech lead for technical planning, architecture coherence, and surfacing patterns to Technical Strategy Ideas |
| [bitwarden-shepherd](plugins/bitwarden-shepherd/) | 1.0.1 | Champion of a technical strategy — shepherds a TSI through evaluation into the funnel, then through to adoption |
| [bitwarden-adr-tools](plugins/bitwarden-adr-tools/) | 1.0.0 | Architecture Decision Record tools: checking designs and plans against the ADR catalog, locating and summarizing ADRs |
| [bitwarden-atlassian-tools](plugins/bitwarden-atlassian-tools/) | 2.8.2 | Atlassian access via MCP server with deep Jira issue research skill and opt-in Jira write tools |
| [bitwarden-code-review](plugins/bitwarden-code-review/) | 2.7.0 | Autonomous code review agent following Bitwarden engineering standards with GitHub integration |
| [bitwarden-delivery-tools](plugins/bitwarden-delivery-tools/) | 3.5.2 | Delivery lifecycle skills: initiative funnel navigation, work transitions, architectural judgment, feature flags, commits, PRs, preflight, labeling, Jira ticket filing |
| [bitwarden-designer](plugins/bitwarden-designer/) | 0.1.0 | Product designer persona: Code of Conduct and 30/60/90 critique, critique facilitation; dispatches into bitwarden-design-tools |
| [bitwarden-design-tools](plugins/bitwarden-design-tools/) | 0.1.0 | Design toolkit: content style guide, Figma Dev Mode MCP, Bitwarden brand application, handoff prep, Design System governance, Product and Design Jira |
| [bitwarden-devops-engineer](plugins/bitwarden-devops-engineer/) | 0.3.1 | DevOps engineering assistant: workflow compliance linting, action security auditing, and org-wide CI/CD remediation |
| [bitwarden-doc-parity](plugins/bitwarden-doc-parity/) | 1.0.1 | Keeps docs current with the code they describe: session instructions, a Stop-hook tripwire, a semantic verification skill, and a CI review face |
| [bitwarden-init](plugins/bitwarden-init/) | 1.2.2 | Initialize and enhance CLAUDE.md files with Bitwarden's standardized template format |
| [bitwarden-mailcatcher-tools](plugins/bitwarden-mailcatcher-tools/) | 1.0.0 | Reads verification links, magic links, and other action links from the local Bitwarden Mailcatcher inbox |
| [bitwarden-product-analyst](plugins/bitwarden-product-analyst/) | 0.1.7 | Product analyst agent for creating comprehensive Bitwarden requirements documents from multiple sources, and writing user-facing release notes |
| [bitwarden-retrospective-tools](plugins/bitwarden-retrospective-tools/) | 1.1.0 | Daily work recap built from your Claude Code session logs, GitHub activity, and Jira and Confluence activity |
| [bitwarden-security-engineer](plugins/bitwarden-security-engineer/) | 2.1.0 | Application security engineering: vulnerability triage, threat modeling, and secure code analysis |
| [bitwarden-software-engineer](plugins/bitwarden-software-engineer/) | 1.1.0 | Software engineer agent for a Bitwarden product team. Implements stories, tasks, and bugs with code quality, performance, security, and team comms in mind. |
| [bitwarden-stripe-tools](plugins/bitwarden-stripe-tools/) | 1.0.0 | Read-only Stripe test-mode queries, invoice previews, and test-clock advancement through a guarded CLI wrapper |
| [bitwarden-testing-tools](plugins/bitwarden-testing-tools/) | 1.5.0 | Testing tools for Bitwarden: test coverage and Testmo case review, test layer recommendations, manual test cases, requirement readiness, test plans, and QA manual regression runs, plus the bundled bitwarden-test-automation MCP server. |
| [bitwarden-testmo-tools](plugins/bitwarden-testmo-tools/) | 1.0.0 | Testmo API access: create test runs from reviewable filter specs, one or many per pass, with dry-run, milestone linkage, and duplicate checks |
| [claude-config-validator](plugins/claude-config-validator/) | 2.3.0 | Validates Claude Code configuration files for security, structure, and quality |
| [claude-retrospective](plugins/claude-retrospective/) | 1.1.1 | Analyze Claude Code sessions to identify successful patterns and improvement opportunities |

## Usage

Expand Down
Loading
Loading