diff --git a/.ai/spec/what/agent-modes.md b/.ai/spec/what/agent-modes.md index 2df31913a..e50df58db 100644 --- a/.ai/spec/what/agent-modes.md +++ b/.ai/spec/what/agent-modes.md @@ -165,6 +165,22 @@ behavior; other specs reference it rather than redefining mode rules. mode. TROUBLESHOOTING mode uses its own dedicated agent instructions regardless of model family. +## E2E Test Coverage + +### Troubleshooting Scenario Tests + +Scenario-based e2e tests verify that `mode=troubleshooting` produces structurally correct responses when diagnosing real cluster problems. Tests are gated behind the `cluster_scenario` pytest marker and require `oc` access to a live OpenShift cluster. + +Each test: +1. Injects a broken cluster state via setup script +2. Queries OLS with `mode=troubleshooting` +3. Asserts HTTP 200, non-empty response, and presence of problem-domain keywords +4. Cleans up cluster state + +Scenarios are defined under `eval/troubleshooting/scenarios/`. The test module is `tests/e2e/test_troubleshooting_scenarios.py`. + +[PLANNED] LLM-graded response quality evaluation via `lightspeed-eval` will layer on top of these structural tests. + ## Planned Changes | Jira Key | Summary |