forked from containers/kubernetes-mcp-server
-
Notifications
You must be signed in to change notification settings - Fork 56
add LVMS toolset for storage diagnostics #370
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mmakwana30
wants to merge
1
commit into
openshift:main
Choose a base branch
from
mmakwana30:add-lvms-toolset
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # LVMS Task Library | ||
|
|
||
| Evaluation tasks for the LVMS (Logical Volume Manager Storage) toolset. These tasks exercise storage troubleshooting workflows on OpenShift clusters using LVMS. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - LVMS operator installed on the cluster | ||
| - LVMCluster CR configured (typically in `openshift-lvm-storage` namespace) | ||
| - Available block devices on nodes for volume group creation | ||
|
|
||
| ## Toolset Design | ||
|
|
||
| LVMS is a **prompts-only toolset** — it provides no dedicated tools. This design follows the toolset-design principle that CRUD operations on Kubernetes resources should use the core toolset's generic tools (`resources_list`, `resources_get`, etc.). | ||
|
|
||
| ### What the LVMS toolset provides: | ||
|
|
||
| - **`lvms-troubleshoot` prompt** — Pre-fetches and presents: | ||
| - LVMCluster status and device class configuration | ||
| - LVMVolumeGroupNodeStatus per node | ||
| - vg-manager pod health and logs | ||
| - **Node-level LVM data** (vgs, lvs, pvs output via debug pods) | ||
| - Events in the LVMS namespace | ||
| - Domain knowledge (vg_attr/lv_attr field interpretation) | ||
|
|
||
| ### What eval tasks use: | ||
|
|
||
| | Tool | Source | Purpose | | ||
| |------|--------|---------| | ||
| | `resources_list` / `resources_get` | core | Query LVMS CRDs (LVMCluster, LVMVolumeGroup, etc.) | | ||
| | `nodes_debug_exec` | cluster-diagnostics | Run LVM commands on nodes (vgs, lvs, pvs, lsblk) | | ||
| | `lvms-troubleshoot` prompt | lvms | Guided diagnostics with pre-fetched data | | ||
|
|
||
| ## Task Categories | ||
|
|
||
| | Task | Difficulty | Description | | ||
| |------|------------|-------------| | ||
| | check-lvmcluster-status | Easy | Verify LVMCluster health and device class configuration | | ||
| | check-capacity | Medium | Analyze thin pool utilization and storage capacity | | ||
| | check-thin-pool-utilization | Medium | Check thin pool data% and metadata% thresholds | | ||
| | list-node-block-devices | Easy | List available block devices on nodes | | ||
| | show-volume-groups | Easy | Display volume group status | | ||
| | show-lvm-physical-volumes | Easy | Display physical volume status | | ||
| | diagnose-no-space-left | Hard | Diagnose thin pool exhaustion issues | | ||
| | diagnose-disk-not-used | Hard | Investigate why a disk isn't being used by LVMS | | ||
| | troubleshoot-stuck-pvc | Hard | Diagnose why a PVC is stuck in Pending state | | ||
| | troubleshoot-prompt | Medium | Use the lvms-troubleshoot prompt for guided diagnosis | | ||
| | forced-cleanup | Hard | Perform forced cleanup of stuck LVMS resources | | ||
|
|
||
| ## Running Tasks | ||
|
|
||
| ```bash | ||
| # Run all LVMS tasks | ||
| mcpchecker check evals/claude-code/eval.yaml --label-selector suite=lvms | ||
|
|
||
| # Run specific difficulty level | ||
| mcpchecker check evals/claude-code/eval.yaml --label-selector suite=lvms,difficulty=hard | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| kind: Task | ||
| metadata: | ||
| labels: | ||
| suite: lvms | ||
| project: lvms | ||
| requires: lvms | ||
| annotations: | ||
| project-name: "LVMS" | ||
| project-url: "https://github.com/openshift/lvm-operator" | ||
| name: "check-lvms-capacity" | ||
| difficulty: medium | ||
| description: "Check LVMS storage capacity and thin pool utilization across nodes" | ||
| steps: | ||
| setup: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| NS="openshift-lvm-storage" | ||
|
|
||
| # Verify LVMS is installed | ||
| if ! kubectl get crd lvmclusters.lvm.topolvm.io > /dev/null 2>&1; then | ||
| echo "LVMS CRD not found - LVMS must be installed" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Check if an LVMCluster exists and is ready | ||
| CLUSTER=$(kubectl get lvmclusters -n "$NS" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) | ||
| if [ -z "$CLUSTER" ]; then | ||
| echo "No LVMCluster found - LVMS must be configured" | ||
| exit 1 | ||
| fi | ||
|
|
||
| STATE=$(kubectl get lvmcluster "$CLUSTER" -n "$NS" -o jsonpath='{.status.state}' 2>/dev/null) | ||
| if [ "$STATE" != "Ready" ]; then | ||
| echo "LVMCluster '$CLUSTER' is not ready (state: $STATE)" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "LVMS is configured with LVMCluster: $CLUSTER" | ||
| verify: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| NS="openshift-lvm-storage" | ||
|
|
||
| echo "=== Verification: Checking agent capacity analysis ===" | ||
|
|
||
| # List node statuses | ||
| kubectl get lvmvolumegroupnodestatuses -n "$NS" -o custom-columns='NODE:.metadata.name,STATUS:.status' 2>/dev/null || true | ||
|
|
||
| echo "" | ||
| echo "=== Capacity Eval Complete ===" | ||
| echo "The agent should have:" | ||
| echo " 1. Queried LVMS volume group status across nodes" | ||
| echo " 2. Reported per-node storage capacity (total, allocatable)" | ||
| echo " 3. Identified any nodes with low available space" | ||
| echo " 4. Provided capacity planning recommendations if applicable" | ||
| echo "" | ||
|
|
||
| exit 0 | ||
| cleanup: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| # No cleanup needed - read-only operation | ||
| exit 0 | ||
| prompt: | ||
| inline: |- | ||
| We're planning to deploy several new applications that will need persistent storage. | ||
| How much LVMS storage capacity do we have available across the cluster, and are there | ||
| any nodes running low on space? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| kind: Task | ||
| metadata: | ||
| labels: | ||
| suite: lvms | ||
| project: lvms | ||
| requires: lvms | ||
| annotations: | ||
| project-name: "LVMS" | ||
| project-url: "https://github.com/openshift/lvm-operator" | ||
| name: "check-lvms-cluster-status" | ||
| difficulty: easy | ||
| description: "Check the status of the LVMCluster and verify it is healthy" | ||
| steps: | ||
| setup: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| NS="openshift-lvm-storage" | ||
|
|
||
| # Verify LVMS is installed | ||
| if ! kubectl get crd lvmclusters.lvm.topolvm.io > /dev/null 2>&1; then | ||
| echo "LVMS CRD not found - LVMS must be installed" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Check if an LVMCluster exists | ||
| if [ -z "$(kubectl get lvmclusters -n "$NS" -o name 2>/dev/null)" ]; then | ||
| echo "No LVMCluster found in $NS - creating test cluster" | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| cat <<EOF | kubectl apply -f - | ||
| apiVersion: lvm.topolvm.io/v1alpha1 | ||
| kind: LVMCluster | ||
| metadata: | ||
| name: eval-lvmcluster | ||
| namespace: $NS | ||
| spec: | ||
| storage: | ||
| deviceClasses: | ||
| - name: vg1 | ||
| default: true | ||
| fstype: xfs | ||
| thinPoolConfig: | ||
| name: thin-pool-1 | ||
| sizePercent: 90 | ||
| overprovisionRatio: 10 | ||
| EOF | ||
| echo "Created eval-lvmcluster - waiting for reconciliation" | ||
| sleep 30 | ||
| fi | ||
| verify: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| NS="openshift-lvm-storage" | ||
|
|
||
| echo "=== Verification: Checking agent findings ===" | ||
|
|
||
| # The agent should have used list_lvmclusters or describe_lvmcluster | ||
| # and reported the cluster status | ||
|
|
||
| CLUSTER=$(kubectl get lvmclusters -n "$NS" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) | ||
| if [ -n "$CLUSTER" ]; then | ||
| STATE=$(kubectl get lvmcluster "$CLUSTER" -n "$NS" -o jsonpath='{.status.state}' 2>/dev/null || echo "Unknown") | ||
| READY=$(kubectl get lvmcluster "$CLUSTER" -n "$NS" -o jsonpath='{.status.ready}' 2>/dev/null || echo "Unknown") | ||
| echo "LVMCluster '$CLUSTER' state: $STATE, ready: $READY" | ||
| fi | ||
|
|
||
| echo "" | ||
| echo "=== Eval Complete ===" | ||
| echo "The agent should have:" | ||
| echo " 1. Queried the LVMCluster resource" | ||
| echo " 2. Reported the cluster name, state, and ready status" | ||
| echo " 3. Listed any device classes configured" | ||
|
|
||
| exit 0 | ||
| cleanup: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| # Only clean up the eval cluster if we created it | ||
| kubectl delete lvmcluster eval-lvmcluster -n openshift-lvm-storage --ignore-not-found | ||
| prompt: | ||
| inline: Is LVMS storage configured and working properly on this cluster? What storage options are available? | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| kind: Task | ||
| metadata: | ||
| labels: | ||
| suite: lvms | ||
| project: lvms | ||
| requires: lvms | ||
| annotations: | ||
| project-name: "LVMS" | ||
| project-url: "https://github.com/openshift/lvm-operator" | ||
| name: "check-thin-pool-utilization" | ||
| difficulty: medium | ||
| description: "Check actual thin pool data and metadata utilization on nodes" | ||
| steps: | ||
| setup: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| NS="openshift-lvm-storage" | ||
|
|
||
| # Verify LVMS is installed and ready | ||
| if ! kubectl get crd lvmclusters.lvm.topolvm.io > /dev/null 2>&1; then | ||
| echo "LVMS CRD not found - LVMS must be installed" | ||
| exit 1 | ||
| fi | ||
|
|
||
| CLUSTER=$(kubectl get lvmclusters -n "$NS" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) | ||
| if [ -z "$CLUSTER" ]; then | ||
| echo "No LVMCluster found - LVMS must be configured" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "LVMS is configured. This task requires checking LIVE thin pool utilization." | ||
| verify: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| echo "=== Verification ===" | ||
| echo "The agent should have:" | ||
| echo " 1. Run 'lvs' command on the node to get thin pool data" | ||
| echo " 2. Reported data_percent and metadata_percent for thin pools" | ||
| echo " 3. This CANNOT be done with resources_list/resources_get" | ||
| echo "" | ||
| echo "If the agent only queried LVMVolumeGroupNodeStatus CRD," | ||
| echo "it did NOT get live thin pool utilization - that requires lvs on node." | ||
| exit 0 | ||
| cleanup: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| exit 0 | ||
| prompt: | ||
| inline: |- | ||
| What is the current thin pool utilization on each node? I need the actual | ||
| data_percent and metadata_percent values from the LVM thin pools, not just | ||
| the Kubernetes CRD status. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| kind: Task | ||
| metadata: | ||
| labels: | ||
| suite: lvms | ||
| project: lvms | ||
| requires: lvms | ||
| annotations: | ||
| project-name: "LVMS" | ||
| project-url: "https://github.com/openshift/lvm-operator" | ||
| name: "diagnose-disk-not-used" | ||
| difficulty: hard | ||
| description: "Diagnose why a specific disk is not being used by LVMS" | ||
| steps: | ||
| setup: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| NS="openshift-lvm-storage" | ||
|
|
||
| # Verify LVMS is installed | ||
| if ! kubectl get crd lvmclusters.lvm.topolvm.io > /dev/null 2>&1; then | ||
| echo "LVMS CRD not found - LVMS must be installed" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "LVMS installed. This task tests disk filtering knowledge." | ||
| echo "" | ||
| echo "The agent must know LVMS device filtering rules:" | ||
| echo "- Read-only devices are excluded" | ||
| echo "- Devices with children (partitions) are excluded" | ||
| echo "- Devices with reserved partition labels are excluded" | ||
| echo "- LVM2_member fstype devices are excluded unless explicitly listed" | ||
| echo "- Loop devices in use by Kubernetes are excluded" | ||
| verify: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| echo "=== Verification ===" | ||
| echo "The agent should have:" | ||
| echo " 1. Identified the node where the disk exists" | ||
| echo " 2. Run lsblk to inspect disk properties" | ||
| echo " 3. Checked LVMCluster deviceSelector configuration" | ||
| echo " 4. Identified which LVMS filtering rule excludes the disk" | ||
| echo " 5. Suggested how to make the disk available (if possible)" | ||
| exit 0 | ||
| cleanup: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| exit 0 | ||
| prompt: | ||
| inline: |- | ||
| A user reports that /dev/sdb on node worker-1 is not being used by LVMS | ||
| even though they expected it to be available for storage. | ||
|
|
||
| Investigate why this disk is not being used by LVMS and explain | ||
| the reason to the user. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.