Skip to content

test(e2e): add k8s storage SP API, NATS, and cluster checks (FLPATH-4… - #44

Open
LinskId wants to merge 1 commit into
dcm-project:mainfrom
LinskId:FLPATH-4434_e2e_storage_testing
Open

test(e2e): add k8s storage SP API, NATS, and cluster checks (FLPATH-4…#44
LinskId wants to merge 1 commit into
dcm-project:mainfrom
LinskId:FLPATH-4434_e2e_storage_testing

Conversation

@LinskId

@LinskId LinskId commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

FLPATH-4434

Summary

  • Add k8s storage SP direct API E2E tests (CRUD, validation, PVC/hints, label filtering)
  • Add NATS status-event coverage (CloudEvents, RUNNING lifecycle, delete, concurrency)
  • Add optional env-agent registration placeholder tests (storage && registration, skipped without DCM_ENVIRONMENT_AGENT_URL)
  • Extend deploy-dcm.sh with storage-class prerequisite validation and E2E SP default wiring
  • Add make test-storage-sp / make test-storage-sp-registration

Scope

SP-direct tests on :8089 — simulates catalog hints via tests/e2e/fixtures/basic-pvc-catalog-item.yaml.
Full catalog → instance → SP flow is out of scope (covered in control-plane SpecBuilder tests).

Test plan

./scripts/deploy-dcm.sh --k8s-storage-service-provider --kubeconfig ~/.kube/config
make test-storage-sp

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Add storage SP API, NATS, and Kubernetes E2E coverage

🧪 Tests ⚙️ Configuration changes 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Adds storage SP API, Kubernetes PVC, and NATS lifecycle end-to-end coverage.
• Adds optional environment-agent registration and idempotency checks with graceful skips.
• Adds focused Make targets and storage-class handling for portable cluster runs.
Diagram

graph TD
  Runner["Test targets"] -->|filters| Suite["Storage E2E"] -->|HTTP API| SP["Storage SP"] -->|manages| PVC["Kubernetes PVC"]
  Suite -->|cluster checks| CLI["kubectl or oc"] -->|inspects| PVC
  SP -->|status events| NATS["NATS"]
  SP -->|registers| Agent["Environment Agent"]
Loading
High-Level Assessment

The PR appropriately extends the repository's existing Ginkgo service-provider patterns with direct integration checks and graceful dependency-based skips. Mocked Kubernetes, NATS, or registration tests were considered but would not validate the cross-system behavior targeted by this E2E suite.

Files changed (9) +1110 / -6

Tests (5) +1062 / -0
api_helpers_test.goInitialize storage SP and environment-agent probes +2/-0

Initialize storage SP and environment-agent probes

• Extends suite setup to detect storage SP and environment-agent availability before running their labeled tests.

tests/e2e/api_helpers_test.go

sp_storage_api_test.goCover storage SP API and Kubernetes PVC behavior +372/-0

Cover storage SP API and Kubernetes PVC behavior

• Adds health, CRUD, pagination, validation, not-found, and deletion contract tests. Cluster-labeled cases verify PVC identity, labels, provider hints, defaults, and filtering of unmanaged claims.

tests/e2e/sp_storage_api_test.go

sp_storage_helpers_test.goAdd reusable storage, cluster, and registration test helpers +382/-0

Add reusable storage, cluster, and registration test helpers

• Provides storage SP probing and HTTP helpers, volume fixtures, kubectl operations, PVC consumer management, and asynchronous RUNNING checks. Also supports paginated environment-agent provider discovery and storage registration filtering.

tests/e2e/sp_storage_helpers_test.go

sp_storage_registration_test.goValidate environment-agent storage registration +48/-0

Validate environment-agent storage registration

• Adds optional checks that exactly one storage provider registers with the expected endpoint and remains idempotent across periodic re-registration.

tests/e2e/sp_storage_registration_test.go

sp_storage_status_test.goValidate storage lifecycle CloudEvents over NATS +258/-0

Validate storage lifecycle CloudEvents over NATS

• Adds CloudEvent schema, status progression, deletion, unmanaged-PVC filtering, and concurrent-volume isolation coverage on the storage NATS subject.

tests/e2e/sp_storage_status_test.go

Documentation (2) +12 / -5
CLAUDE.mdDocument storage SP test groups and environment controls +6/-2

Document storage SP test groups and environment controls

• Adds storage API and registration suites to the E2E test matrix. Documents registration endpoints, label filters, and graceful skip behavior for unavailable dependencies.

CLAUDE.md

README.mdDocument storage E2E commands and label filters +6/-3

Document storage E2E commands and label filters

• Updates service-provider test commands and provides separate examples for regular storage checks and registration checks.

README.md

Other (2) +36 / -1
MakefileAdd focused storage SP test targets +7/-1

Add focused storage SP test targets

• Adds targets for the default storage API/NATS suite and the optional environment-agent registration suite using Ginkgo label filters.

Makefile

deploy-dcm.shResolve storage classes for portable E2E provisioning +29/-0

Resolve storage classes for portable E2E provisioning

• Detects or honors the storage class used by storage E2E tests and reports its binding mode. This supports clusters whose default class requires a consumer before PVC binding.

scripts/deploy-dcm.sh

@qodo-code-review

qodo-code-review Bot commented Aug 26, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (4) 📘 Rule violations (1) 📜 Skill insights (0)

Grey Divider


Action required

1. StorageClass export is lost 🐞 Bug ≡ Correctness
Description
ensure_storage_class_for_e2e exports the detected class only inside deploy-dcm.sh, which
tests/run-e2e.sh executes as a child process; the later Ginkgo process therefore falls back to
standard. On clusters whose selected class is not named standard, the new volume specs request
the wrong class and storage API/NATS tests fail instead of using the class deployment detected.
Code

scripts/deploy-dcm.sh[366]

+    export K8S_STORAGE_SP_DEFAULT_STORAGE_CLASS="${default_sc}"
Evidence
The deployment hook exports the detected value, but the harness launches that script as a separate
executable and only afterward launches Ginkgo. The new test helper reads the variable from Ginkgo's
environment and otherwise hard-codes standard, which is then sent in volume creation requests.

scripts/deploy-dcm.sh[358-366]
tests/run-e2e.sh[245-249]
tests/run-e2e.sh[293-305]
tests/e2e/sp_storage_helpers_test.go[102-106]
tests/e2e/sp_storage_helpers_test.go[153-155]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The auto-detected StorageClass is exported only by the child deployment script, so it is unavailable when the parent E2E runner starts Ginkgo.

## Issue Context
Make the selected class persist across the deploy-script boundary, then export it before running the Go tests.

## Fix Focus Areas
- scripts/deploy-dcm.sh[358-366]
- tests/run-e2e.sh[245-305]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Missing classes become standard 🐞 Bug ☼ Reliability
Description
When the cluster has no StorageClasses, the fallback command substitutes the literal standard and
the function continues without verifying that it exists. Deployment consequently succeeds with an
invalid selected class, and the new tests later create PVCs against that nonexistent class rather
than failing prerequisite validation.
Code

scripts/deploy-dcm.sh[R360-362]

+    if [[ -z "${default_sc}" ]]; then
+        default_sc=$("${cli}" --kubeconfig="${kubeconfig}" get sc -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "standard")
+    fi
Evidence
The first-class lookup falls back to standard on command failure, the subsequent binding-mode
lookup suppresses errors, and the unverified value is exported. The storage test helper materially
uses that value as the StorageClass hint.

scripts/deploy-dcm.sh[358-366]
tests/e2e/sp_storage_helpers_test.go[102-106]
tests/e2e/sp_storage_helpers_test.go[153-155]
tests/e2e/sp_storage_helpers_test.go[177-199]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
StorageClass discovery silently changes an empty or failed lookup into `standard`, even when that class does not exist.

## Issue Context
Validate the selected class with the cluster CLI and fail deployment with a clear prerequisite error when no usable class exists. Apply the same validation to an explicitly configured class.

## Fix Focus Areas
- scripts/deploy-dcm.sh[353-366]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

3. Failed delete test leaks volume 🐞 Bug ☼ Reliability
Description
The delete-status context closes only its NATS collector in AfterAll, so any failure before the
explicit DELETE completes leaves the created PVC and consumer pod behind. Timeouts waiting for
RUNNING or DELETED can therefore contaminate subsequent tests and later runs.
Code

tests/e2e/sp_storage_status_test.go[R147-149]

+		AfterAll(func() {
+			collector.Close()
+		})
Evidence
This context creates a volume and has several assertions/timeouts before and after its explicit
deletion, but its teardown omits the cleanup helper used by the neighboring contexts.
deleteTestVolume already removes both the consumer pod and volume and tolerates an already-gone
resource.

tests/e2e/sp_storage_status_test.go[143-175]
tests/e2e/sp_storage_status_test.go[64-69]
tests/e2e/sp_storage_status_test.go[116-120]
tests/e2e/sp_storage_helpers_test.go[142-151]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The delete-event test has no failure-safe cleanup for the volume it creates.

## Issue Context
Use the existing idempotent cleanup helper from `AfterAll` when `volumeID` is nonempty; a 404 after successful deletion is already safe for cleanup.

## Fix Focus Areas
- tests/e2e/sp_storage_status_test.go[147-149]
- tests/e2e/sp_storage_helpers_test.go[142-151]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. Storage class override undocumented ✗ Dismissed 📘 Rule violation ⚙ Maintainability
Description
K8S_STORAGE_SP_DEFAULT_STORAGE_CLASS is now a user-configurable deployment override, but neither
README.md nor CLAUDE.md documents it or the new automatic StorageClass selection behavior. Users
cannot reliably discover how to control storage E2E provisioning on clusters whose selected class is
unsuitable.
Code

scripts/deploy-dcm.sh[R353-356]

+    if [[ -n "${K8S_STORAGE_SP_DEFAULT_STORAGE_CLASS:-}" ]]; then
+        info "Using configured storage class: ${K8S_STORAGE_SP_DEFAULT_STORAGE_CLASS}"
+        return 0
+    fi
Evidence
PR Compliance ID 2901048 requires new environment variables and changed runtime behavior to be
described in README.md or CLAUDE.md. The script adds the override at
scripts/deploy-dcm.sh[353-356] and automatic selection at scripts/deploy-dcm.sh[358-370], while
repository searches confirm the variable appears only in the script and storage tests, not in either
required documentation file.

Rule 2901048: Update project documentation and cursor configs when behavior or conventions change
scripts/deploy-dcm.sh[353-370]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Document the new `K8S_STORAGE_SP_DEFAULT_STORAGE_CLASS` override and automatic StorageClass-selection behavior in project documentation.

## Issue Context
The deployment script now honors this environment variable and otherwise selects a cluster StorageClass, but the updated `README.md` and `CLAUDE.md` do not mention either behavior.

## Fix Focus Areas
- scripts/deploy-dcm.sh[343-370]
- README.md[123-133]
- CLAUDE.md[204-218]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


5. Cursor storage guidance remains stale 📘 Rule violation ⚙ Maintainability
Description
The PR introduces the sp_storage_* test family, storage/registration labels, storage-specific
guards, and new environment variables, but .cursor/rules/e2e-tests.mdc still enumerates only the
older provider conventions. Cursor-assisted contributions will therefore receive incomplete naming,
labeling, setup, and configuration guidance for the new storage suite.
Code

tests/e2e/sp_storage_api_test.go[R14-16]

+var _ = Describe("Storage SP API", Label("sp", "storage"), func() {
+	BeforeEach(func() {
+		requireStorageSP()
Evidence
PR Compliance ID 2901048 requires relevant .cursor/ files to be updated when contributor
conventions or assistant-referenced commands change. The new storage suite establishes `Label("sp",
"storage") and requireStorageSP() at tests/e2e/sp_storage_api_test.go[14-16]`, while
.cursor/rules/e2e-tests.mdc[13-45] is specifically scoped to E2E tests yet omits the storage
prefix, labels, guard functions, and storage/registration environment variables.

Rule 2901048: Update project documentation and cursor configs when behavior or conventions change
tests/e2e/sp_storage_api_test.go[14-16]
.cursor/rules/e2e-tests.mdc[13-45]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Update the Cursor E2E rule to cover the new storage SP test conventions and configuration.

## Issue Context
`.cursor/rules/e2e-tests.mdc` governs `tests/e2e/**/*_test.go` but its provider table, labels, optional-infrastructure guards, environment-variable table, and compose guidance omit the newly introduced storage and registration suite behavior.

## Fix Focus Areas
- .cursor/rules/e2e-tests.mdc[13-45]
- .cursor/rules/e2e-tests.mdc[103-106]
- tests/e2e/sp_storage_api_test.go[14-16]
- tests/e2e/sp_storage_helpers_test.go[37-106]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View medium (1)
6. Cluster tests bypass label 🐞 Bug ≡ Correctness
Description
The healthy-lifecycle and delete-event status tests call ensurePVCConsumer, which requires kubectl
and creates a pod, but neither context carries the cluster label. As a result,
--label-filter='!cluster' still selects these cluster-dependent tests and then skips or performs
cluster work, defeating the documented label contract and silently omitting status coverage when
kubectl is unavailable.
Code

tests/e2e/sp_storage_status_test.go[R76-77]

+			By("scheduling a PVC consumer pod for WaitForFirstConsumer storage classes")
+			ensurePVCConsumer(volumeID)
Evidence
Both tests invoke the helper that immediately calls requireKubectl and applies a consumer Pod.
Other new kubectl-dependent storage contexts are explicitly labeled cluster, and the project
documentation defines that label for tests requiring kubectl/oc access.

tests/e2e/sp_storage_status_test.go[56-80]
tests/e2e/sp_storage_status_test.go[139-157]
tests/e2e/sp_storage_helpers_test.go[248-273]
tests/e2e/sp_storage_api_test.go[137-143]
CLAUDE.md[182-184]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Two status-event tests require kubectl through `ensurePVCConsumer` but are not marked with the `cluster` label.

## Issue Context
Apply `Label("cluster")` at the relevant contexts/specs, or redesign PVC binding so these tests genuinely do not require local cluster tooling.

## Fix Focus Areas
- tests/e2e/sp_storage_status_test.go[56-80]
- tests/e2e/sp_storage_status_test.go[139-157]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 2 rules
Review mode: ⚖️ Balanced: Downgraded extended -> standard: change is below the extended eligibility bar (hunks 12/18, lines 1116/200; both must reach the floor). Router rationale: This adds substantial new behavior across multiple E2E API, Kubernetes, NATS, registration, deployment, and test-helper paths, creating many independent opportunities for subtle defects.

Grey Divider

Tip of the day
💡 Did you know, you can ask Qodo to dismiss a finding you disagree with, with your reason on record

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread scripts/deploy-dcm.sh
Comment thread tests/e2e/sp_storage_api_test.go
Comment thread scripts/deploy-dcm.sh Outdated
Comment thread scripts/deploy-dcm.sh Outdated
Comment thread tests/e2e/sp_storage_status_test.go
Comment thread tests/e2e/sp_storage_status_test.go
@jenniferubah

Copy link
Copy Markdown

Please, can you fix the PR description

@LinskId
LinskId force-pushed the FLPATH-4434_e2e_storage_testing branch from 3ade7bd to 26ed681 Compare August 27, 2026 13:34
…434)

Assisted-by: Cursor
Signed-off-by: igavra <igavra@redhat.com>
Expect(err).NotTo(HaveOccurred())
Expect(resp.StatusCode).To(Equal(http.StatusOK))

var body map[string]interface{}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't use unstructured data. There must be a struct in the business layer that we can reuse to unmarshal into. Using unstructured data has 2 problems:

  • Harder to maintain (no automatic refactoring available)
  • Does not guarantee field drift in the original structure, which means extensions to the struct would not be covered in this evaluation. Comparing against a given structure (not field by field) provides such guarantee.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k8s-storage-service-provider already generates a client for this — pkg/client/client.gen.go (oapi-codegen, same pattern as container/ACM cluster, wired into generate-api/check-generate-api). Its GetHealthWithResponse() decodes straight into a typed Health struct (Status, Path, Type, Uptime, Version), so no hand-written struct or map[string]interface{} is needed at all:

import (
    storageclient "github.com/dcm-project/k8s-storage-service-provider/pkg/client"
)

client, err := storageclient.NewClientWithResponses(storageSPBaseURL)
Expect(err).NotTo(HaveOccurred())

resp, err := client.GetHealthWithResponse(ctx)
Expect(err).NotTo(HaveOccurred())
Expect(resp.StatusCode()).To(Equal(http.StatusOK))
Expect(resp.JSON200.Status).To(Equal("healthy"))

Would need adding k8s-storage-service-provider as a go.mod dep in tests/e2e/. Same fix I flagged on #47 for the container/ACM cluster SPs — worth applying consistently rather than hand-writing structs per SP.


var page2 map[string]interface{}
decodeJSON(resp2, &page2)
Expect(page2).To(HaveKey("volumes"))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about the key's value? There's not test value to ensure the key exists only.

@jordigilh

Copy link
Copy Markdown

@LinskId please review my comment about using the generate go client binding and redo the tests. Also use structured data (go structs) and not unstructured (map[string]interface{}). Import the structures from the business layer rather than duplicate it so that you can focus on testing what adds value and not on the boilerplate (http connectivity and structure consistency (e.g. does this unstructured map has this key?)).
I recommend you review your tests to ensure they provide behavioral assurance against business acceptance criteria and verify business-level behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants