Skip to content

Add e2e test coverage spec for troubleshooting mode scenarios #2983

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JoaoFula wants to merge 1 commit into
base: main
Choose a base branch
from
+16 −0
Open

Add e2e test coverage spec for troubleshooting mode scenarios #2983

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .ai/spec/what/agent-modes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,22 @@ behavior; other specs reference it rather than redefining mode rules.
mode. TROUBLESHOOTING mode uses its own dedicated agent instructions
regardless of model family.

## E2E Test Coverage

### Troubleshooting Scenario Tests

Scenario-based e2e tests verify that `mode=troubleshooting` produces structurally correct responses when diagnosing real cluster problems. Tests are gated behind the `cluster_scenario` pytest marker and require `oc` access to a live OpenShift cluster.

Each test:
1. Injects a broken cluster state via setup script
2. Queries OLS with `mode=troubleshooting`
3. Asserts HTTP 200, non-empty response, and presence of problem-domain keywords
4. Cleans up cluster state

Scenarios are defined under `eval/troubleshooting/scenarios/`. The test module is `tests/e2e/test_troubleshooting_scenarios.py`.
Comment on lines +172 to +180

@coderabbitai coderabbitai Bot Jul 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
rg -n "cluster_scenario|lseval|test_troubleshooting_scenarios|test_lseval_troubleshooting" .ai/spec/what tests/e2e

Repository: openshift/lightspeed-service

Length of output: 5052

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '\n--- .ai/spec/what/agent-modes.md (lines 168-183) ---\n'
sed -n '168,183p' .ai/spec/what/agent-modes.md

printf '\n--- tests/e2e/evaluation/test_lseval_troubleshooting.py (lines 150-210) ---\n'
sed -n '150,210p' tests/e2e/evaluation/test_lseval_troubleshooting.py

printf '\n--- tests/e2e/pytest.ini ---\n'
sed -n '1,80p' tests/e2e/pytest.ini

Repository: openshift/lightspeed-service

Length of output: 3693

Clarify whether troubleshooting coverage is current or planned.
This section points to a cluster_scenario-gated tests/e2e/test_troubleshooting_scenarios.py, but the troubleshooting eval is still a commented-out @pytest.mark.lseval placeholder in tests/e2e/evaluation/test_lseval_troubleshooting.py. Update the spec to match the current layout, or mark this as planned future work so readers do not follow a non-existent entrypoint.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.ai/spec/what/agent-modes.md around lines 172 - 180, The troubleshooting
test coverage description is out of sync with the current code layout: it
references a `cluster_scenario`-gated
`tests/e2e/test_troubleshooting_scenarios.py`, while the actual troubleshooting
eval entrypoint is still the commented-out `@pytest.mark.lseval` placeholder in
`tests/e2e/evaluation/test_lseval_troubleshooting.py`. Update the spec section
to match the current implementation and use the existing
`tests/e2e/evaluation/test_lseval_troubleshooting.py`/`@pytest.mark.lseval`
symbols, or explicitly label the scenario-based coverage as planned future work
so the documentation does not point to a non-existent test entrypoint.


[PLANNED] LLM-graded response quality evaluation via `lightspeed-eval` will layer on top of these structural tests.

## Planned Changes

| Jira Key | Summary |
Expand Down