Skip to content

feat(e2e): add environment-agent deploy and embedded storage registration tests (FLPATH-4803) - #45

Open
LinskId wants to merge 1 commit into
dcm-project:mainfrom
LinskId:FLPATH-4803_embed_storage_agent
Open

feat(e2e): add environment-agent deploy and embedded storage registration tests (FLPATH-4803)#45
LinskId wants to merge 1 commit into
dcm-project:mainfrom
LinskId:FLPATH-4803_embed_storage_agent

Conversation

@LinskId

@LinskId LinskId commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

And dcm-project/environment-agent#31

Wire environment-agent with embedded k8s storage SP into the utilities E2E stack
so registration can be validated without a standalone storage SP container.
Deploy:

  • Add providers/environment-agent.conf and compose override (port 8090,
    AGENT_EMBEDDED_SPS=storage, DCM_REGISTRATION_URL without /api/v1alpha1 path)
  • Extend deploy-dcm.sh: --environment-agent flag, validation hook, .dcm-e2e.env
    exports (DCM_ENVIRONMENT_AGENT_URL, K8S_STORAGE_SP_REGISTERED_ENDPOINT)
  • Use pull_policy: missing so locally built agent images are not overwritten
    by Quay while embedded storage is not yet on main
    Tests:
  • Assert embedded storage registers at embedded://storage via agent /providers
  • Fix providers list parsing (results, not providers)
  • Resolve expected endpoint before standalone SP health probe so registration
    tests work with --environment-agent only (no :8089)
    Docs: README, CLAUDE.md, e2e-tests.mdc

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Add environment-agent and Kubernetes storage E2E coverage

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Deploy environment-agent with embedded Kubernetes storage and persisted E2E configuration.
• Validate embedded registration, storage APIs, PVC behavior, and NATS lifecycle events.
• Document storage test modes, environment variables, and targeted Makefile workflows.
Diagram

sequenceDiagram
    participant Deploy as Deploy Harness
    participant K8s as Kubernetes
    participant Agent as Environment Agent
    participant CP as Control Plane
    participant Env as E2E Env
    participant Tests as Ginkgo Tests
    participant SP as Storage SP
    participant NATS as NATS
    Deploy->>K8s: Validate storage class
    Deploy->>Agent: Start embedded storage
    Agent->>CP: Register embedded endpoint
    Deploy->>Env: Persist URLs and defaults
    Env->>Tests: Load test configuration
    Tests->>Agent: List registered providers
    Tests->>SP: Exercise volume API
    SP->>K8s: Manage PVC lifecycle
    SP->>NATS: Publish status events
    Tests->>NATS: Verify event streams
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Test standalone storage only
  • ➕ Keeps deployment topology simpler
  • ➕ Avoids dependence on environment-agent changes
  • ➖ Cannot validate embedded provider registration
  • ➖ Misses the production-style agent integration path
2. Pass exports only to child processes
  • ➕ Avoids a generated repository-local environment file
  • ➕ Eliminates stale configuration between manual runs
  • ➖ Separate Make invocations lose deploy-time StorageClass discovery
  • ➖ Requires users to reconstruct several coupled environment variables

Recommendation: Keep the provider-registry and Compose-overlay approach because it follows existing deployment extension points and tests the embedded registration contract directly. Persisting .dcm-e2e.env is justified for separate test invocations; atomic replacement, teardown cleanup, and gitignore coverage mitigate stale-file risk.

Files changed (15) +1441 / -12

Enhancement (3) +176 / -3
environment-agent.confRegister environment-agent with the deploy provider registry +25/-0

Register environment-agent with the deploy provider registry

• Defines the environment-agent flag, Compose overlay, Kubernetes CLI requirement, storage namespace reuse, kubeconfig export, and validation hook.

providers/environment-agent.conf

deploy-dcm.shDeploy environment-agent and persist storage test settings +101/-0

Deploy environment-agent and persist storage test settings

• Adds environment-agent versioning and validation, including StorageClass discovery for embedded or standalone storage. Atomically writes test URLs, namespace, expected registration endpoint, and storage defaults to '.dcm-e2e.env', then removes it during teardown.

scripts/deploy-dcm.sh

run-e2e.shWire storage and environment-agent flags into the E2E runner +50/-3

Wire storage and environment-agent flags into the E2E runner

• Forwards storage namespace and environment-agent deployment options, tracks storage test selection, and conditionally loads generated exports. It also configures storage and NATS endpoints for relevant runs.

tests/run-e2e.sh

Tests (5) +1163 / -0
api_helpers_test.goInitialize storage and environment-agent probes +2/-0

Initialize storage and environment-agent probes

• Extends suite initialization to detect standalone storage and environment-agent availability before labeled tests execute.

tests/e2e/api_helpers_test.go

sp_storage_api_test.goCover Kubernetes storage provider API and PVC behavior +369/-0

Cover Kubernetes storage provider API and PVC behavior

• Adds health, CRUD, pagination, validation, provider-hint, defaults, label filtering, and deletion coverage. Cluster-labeled cases verify the resulting Kubernetes PVC resources.

tests/e2e/sp_storage_api_test.go

sp_storage_helpers_test.goAdd shared storage, Kubernetes, and agent test helpers +481/-0

Add shared storage, Kubernetes, and agent test helpers

• Introduces endpoint probing, request/spec builders, StorageClass validation, PVC lifecycle utilities, and WaitForFirstConsumer support. Adds paginated environment-agent provider parsing from the 'results' field and storage-provider filtering.

tests/e2e/sp_storage_helpers_test.go

sp_storage_registration_test.goVerify embedded storage provider registration +48/-0

Verify embedded storage provider registration

• Asserts the environment agent registers exactly one 'embedded://storage' provider and keeps registration idempotent over time.

tests/e2e/sp_storage_registration_test.go

sp_storage_status_test.goValidate storage lifecycle events over NATS +263/-0

Validate storage lifecycle events over NATS

• Adds CloudEvent contract, status progression, deletion, label isolation, and concurrent volume event tests. Cluster cases coordinate PVC consumer pods for delayed-binding StorageClasses.

tests/e2e/sp_storage_status_test.go

Documentation (3) +44 / -7
e2e-tests.mdcDocument storage E2E conventions and configuration +14/-2

Document storage E2E conventions and configuration

• Adds storage test naming, labels, helper requirements, environment variables, and Compose override guidance. Clarifies the distinction between catalog storage hints and storage-provider defaults.

.cursor/rules/e2e-tests.mdc

CLAUDE.mdDescribe storage and embedded registration test workflows +10/-2

Describe storage and embedded registration test workflows

• Documents storage test labels, environment-agent registration coverage, StorageClass settings, namespaces, and graceful skip behavior.

CLAUDE.md

README.mdDocument storage deployment and test commands +20/-3

Document storage deployment and test commands

• Adds commands and environment settings for direct storage tests and embedded environment-agent registration. Explains local agent image requirements and '.dcm-e2e.env' behavior.

README.md

Other (4) +58 / -2
.gitignoreIgnore generated E2E environment exports +1/-0

Ignore generated E2E environment exports

• Excludes '.dcm-e2e.env', which is generated after deployment to share discovered storage configuration with later test processes.

.gitignore

MakefileAdd targeted storage E2E test targets +11/-2

Add targeted storage E2E test targets

• Adds standalone storage and embedded registration targets with Ginkgo label filters. Storage-related targets source generated deployment exports before running tests.

Makefile

compose-environment-agent-local.yamlPreserve local environment-agent image compatibility +7/-0

Preserve local environment-agent image compatibility

• Provides a backward-compatible Compose overlay that keeps locally built environment-agent images when explicitly supplied.

tests/compose-environment-agent-local.yaml

compose-environment-agent-test.yamlDefine embedded storage environment-agent service +39/-0

Define embedded storage environment-agent service

• Adds the E2E environment-agent service on port 8090 with embedded storage, kubeconfig access, storage defaults, NATS connectivity, persistence, and control-plane registration. Uses 'pull_policy: missing' to preserve local development images.

tests/compose-environment-agent-test.yaml

@qodo-code-review

qodo-code-review Bot commented Aug 27, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Shared namespace flag loses value ✓ Resolved 🐞 Bug ≡ Correctness
Description
environment-agent reuses the standalone storage provider's k8s-storage-namespace registry flag,
but argument matching updates only the first matching provider; when both are enabled, the later
standalone provider exports its unchanged default and the agent compose service runs in default
instead of the requested namespace. This also makes validation target one namespace while the
resulting container is configured for another.
Code

providers/environment-agent.conf[R16-18]

+NAMESPACE_FLAG="k8s-storage-namespace"
+NAMESPACE_ENV="K8S_STORAGE_SP_NAMESPACE"
+NAMESPACE_DEFAULT="default"
Evidence
Both provider definitions now use the same namespace flag and export variable. Provider files are
loaded lexically with environment-agent before k8s-storage; match_provider_flag returns on the
first match, while the later validation/export loop writes each provider's value to the shared
environment variable, so the standalone provider's default overwrites the agent's requested
namespace before Compose starts.

providers/environment-agent.conf[15-22]
providers/k8s-storage.conf[16-23]
scripts/deploy-dcm.sh[66-102]
scripts/deploy-dcm.sh[757-773]
scripts/deploy-dcm.sh[839-848]
scripts/deploy-dcm.sh[920-950]
tests/compose-environment-agent-test.yaml[20-24]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The environment-agent and standalone storage provider expose the same namespace flag, but the registry parser updates only one matching entry and later compose exports overwrite the requested value.

## Issue Context
A custom `--k8s-storage-namespace` must configure both enabled storage implementations consistently, or the environment-agent must use a distinct flag without sharing the same exported variable.

## Fix Focus Areas
- providers/environment-agent.conf[15-18]
- scripts/deploy-dcm.sh[757-773]
- scripts/deploy-dcm.sh[839-848]
- scripts/deploy-dcm.sh[920-950]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Re-registration cycle never occurs ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
The test claiming to verify idempotency across re-registration only polls the read-only provider
list and never restarts the agent or submits another registration. It therefore passes even if an
actual re-registration cycle would create duplicate storage providers.
Code

tests/e2e/sp_storage_registration_test.go[R42-45]

+	It("does not duplicate the storage registration across a re-registration cycle", func() {
+		Consistently(func() int {
+			return len(storageProvidersFromAgent())
+		}, 90*time.Second, 10*time.Second).Should(Equal(1),
Evidence
The test body contains only Consistently calls to storageProvidersFromAgent; that helper
ultimately performs GET /providers, so no state-changing operation occurs. Existing agent
idempotency coverage demonstrates a real cycle by issuing a second POST and checking that the same
registration ID is returned.

tests/e2e/sp_storage_registration_test.go[41-46]
tests/e2e/sp_storage_helpers_test.go[419-480]
tests/e2e/api_agents_test.go[85-104]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The idempotency test observes a stable provider count without causing the re-registration whose behavior it claims to validate.

## Issue Context
Perform a deterministic agent restart or invoke the supported registration action, wait for registration to complete, and then assert that exactly one embedded storage provider remains. If the E2E harness cannot safely trigger that operation, rename/remove the test rather than claiming re-registration coverage.

## Fix Focus Areas
- tests/e2e/sp_storage_registration_test.go[41-46]
- tests/e2e/sp_storage_helpers_test.go[419-480]
- tests/e2e/api_agents_test.go[85-104]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 2 rules
Review mode: ⚖️ Balanced

Grey Divider

Tip of the day
💡 Did you know, you can type 'qodo, fix this' on a finding and the fix lands right on your PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread providers/environment-agent.conf
Comment thread tests/e2e/sp_storage_registration_test.go Outdated
@LinskId
LinskId force-pushed the FLPATH-4803_embed_storage_agent branch 3 times, most recently from 57bab78 to 2841832 Compare August 28, 2026 16:05
…s (FLPATH-4434, FLPATH-4803)

Add standalone storage SP E2E (API, NATS status, cluster checks) and wire
environment-agent deploy with embedded storage registration validation.

Deploy: --environment-agent provider, compose override on port 8090,
pull_policy missing for local agent builds, .dcm-e2e.env exports.

Tests: embedded storage at embedded://storage via agent /providers;
keep DCM_NATS_URL in run-e2e.sh --help for CI validate-scripts.

Assisted-by: Cursor
Signed-off-by: igavra <igavra@redhat.com>
@LinskId
LinskId force-pushed the FLPATH-4803_embed_storage_agent branch from 495ec9d to 86f0807 Compare August 31, 2026 14:18
CLI_REQUIREMENT="oc-or-kubectl"

# Reuse storage namespace flag/env for embedded SP_K8S_NAMESPACE.
NAMESPACE_FLAG="k8s-storage-namespace"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: what do you think to prefix this variable with the K8S_STORAGE_SP_ so that it's clear these are specific for the SP and not required by the environment agent?

Expect(err).NotTo(HaveOccurred())
Expect(resp.StatusCode).To(Equal(http.StatusOK))

var body map[string]interface{}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we already have a defined structure to use instead of unstructured data?

var body map[string]interface{}
decodeJSON(resp, &body)
Expect(body["status"]).To(Equal("healthy"))
Expect(body).To(HaveKey("uptime"))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there an expected value to check for all 3 fields?

name := uniqueName("e2e-vol")
body := createTestVolume(volumeSpec(name, defaultTestCapacity))

id := body["id"].(string)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason why not use structured data?


var body map[string]interface{}
decodeJSON(resp, &body)
Expect(body).To(HaveKey("id"))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you used structured data, you would not need to check this. Perhaps check that the field has a value that matches the format from the rest call?

ExpectWithOffset(1, err).NotTo(HaveOccurred(), "failed to get PVC %s", name)

var pvc map[string]interface{}
ExpectWithOffset(1, json.Unmarshal([]byte(out), &pvc)).To(Succeed())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're not validating the pvc, just that there is one. You should also validate that the PVC is ready.


// runStorageKubectl executes kubectl/oc in the storage SP namespace.
func runStorageKubectl(args ...string) (string, error) {
fullArgs := append([]string{"-n", storageSPNamespace}, args...)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why cli and not using the kubernetes go client?


// applyStorageManifest applies a Kubernetes manifest in the storage SP namespace.
func applyStorageManifest(manifest string) error {
cmd := exec.Command(kubectlBin, "-n", storageSPNamespace, "apply", "-f", "-")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here: why cli and not the kubernetes go client bindings?


// doEnvironmentAgentRequest sends a request to the environment agent API.
func doEnvironmentAgentRequest(method, path string, body string) (*http.Response, error) {
url := environmentAgentBaseURL + path

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use the environment agent bindings? using pure http client is not testing the env agent real path. And it makes this tests brittle the moment the rest api changes (new fields, path, etc..), which should not make these tests fail

Comment thread tests/run-e2e.sh

# Export SP URLs when providers are enabled.
if [[ "${ENABLE_CONTAINER_SP}" == "true" ]] || [[ "${ENABLE_ACM_CLUSTER_SP}" == "true" ]]; then
if [[ "${ENABLE_CONTAINER_SP}" == "true" ]] || [[ "${ENABLE_STORAGE_SP}" == "true" ]] || [[ "${ENABLE_ACM_CLUSTER_SP}" == "true" ]]; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we planning on creating lanes for each SP so that they run isolated?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants