feat(e2e): add environment-agent deploy and embedded storage registration tests (FLPATH-4803)

.cursor/rules/e2e-tests.mdc

@@ -15,6 +15,7 @@ Files follow `{layer}_{subject}_test.go` with a matching `{layer}_helpers_test.g
 | `api_` | Control plane HTTP API (`doRequest` → `localhost:8080`) | `api_helpers_test.go` |
 | `cli_` | CLI binary (`runDCM` → `os/exec`) | `cli_helpers_test.go` |
 | `sp_container_` | Container SP direct API + NATS (`doContainerSPRequest` → `localhost:8082`) | `sp_helpers_test.go` |
+| `sp_storage_` | Storage SP direct API + NATS (`doStorageSPRequest` → `localhost:8089`) | `sp_storage_helpers_test.go` |
 | `sp_acm_cluster_` | ACM Cluster SP direct API (`doAcmClusterSPRequest` → `localhost:8083`) | `sp_acm_cluster_helpers_test.go` |
 | `core_` | Cross-service provisioning flow via control plane (`doRequest`) | `api_helpers_test.go`, `sp_helpers_test.go` |
 
@@ -23,9 +24,9 @@ When adding a new service provider, add `sp_{provider}_{concern}_test.go` (e.g.
 ## Go Conventions
 - All files start with `//go:build e2e` and `package e2e_test`
 - Dot-import Ginkgo and Gomega: `. "github.com/onsi/ginkgo/v2"`
-- Use Ginkgo labels for selective runs: `Label("sp", "container")`, `Label("sp", "acm-cluster")`, `Label("core", "platform")`, `Label("cli")`, `Label("smoke")`, `Label("cluster")`, `Label("disruptive")`
+- Use Ginkgo labels for selective runs: `Label("sp", "container")`, `Label("sp", "storage")`, `Label("sp", "storage", "registration")`, `Label("sp", "acm-cluster")`, `Label("core", "platform")`, `Label("cli")`, `Label("smoke")`, `Label("cluster")`, `Label("disruptive")`, `Label("nats")`
 - Ordered contexts need `AfterAll` cleanup; standalone tests use `BeforeEach`/`AfterEach`
-- Tests that need optional infrastructure call `requireCLI()`, `requireContainerSP()`, `requireAcmClusterSP()`, `requireKubectl()`, or `requirePodman()` to skip gracefully
+- Tests that need optional infrastructure call `requireCLI()`, `requireContainerSP()`, `requireStorageSP()`, `requireEnvironmentAgent()`, `requireAcmClusterSP()`, `requireKubectl()`, or `requirePodman()` to skip gracefully
 - `BeforeSuite` in `api_helpers_test.go` is the single initialization point — add `init*()` calls there for new subsystems
 - Tests that modify infrastructure (e.g. stop NATS) use `Label("disruptive")` so they can be excluded from normal runs
 - Tests that require cluster access use `Label("cluster")` so they can be filtered when no cluster is available
@@ -39,10 +40,20 @@ When adding a new service provider, add `sp_{provider}_{concern}_test.go` (e.g.
 |----------|---------|---------|
 | `DCM_GATEWAY_URL` | `http://localhost:8080/api/v1alpha1` | Control plane API |
 | `DCM_CONTAINER_SP_URL` | `http://localhost:8082/api/v1alpha1` | Container SP direct API |
+| `DCM_STORAGE_SP_URL` | `http://localhost:8089/api/v1alpha1` | Storage SP direct API |
 | `DCM_ACM_CLUSTER_SP_URL` | `http://localhost:8083/api/v1alpha1` | ACM Cluster SP direct API |
 | `DCM_NATS_URL` | `nats://localhost:4222` | NATS for status events |
 | `DCM_CLI_PATH` | (auto-resolved) | CLI binary path |
 | `DCM_CONTAINER_PROVIDER_NAME` | (first container provider) | Target a specific container provider in core platform tests |
+| `DCM_ENVIRONMENT_AGENT_URL` | (from `.dcm-e2e.env`) | Environment agent for embedded storage registration tests (`storage && registration`) |
+| `K8S_STORAGE_SP_NAMESPACE` | `default` | Namespace for storage SP PVCs |
+| `K8S_STORAGE_SP_DEFAULT_STORAGE_CLASS` | (from `.dcm-e2e.env`) | E2E harness: SP fallback when requests omit `provider_hints` (`SP_K8S_DEFAULT_STORAGE_CLASS`) |
+| `E2E_CATALOG_STORAGE_CLASS` | `standard` | Optional override of simulated catalog hint `storage_class` in SP-direct E2E |
+| `K8S_STORAGE_SP_REGISTERED_ENDPOINT` | `embedded://storage` | Expected embedded storage provider endpoint in registration tests |
+
+Storage class sources in storage E2E (do not conflate):
+- **Catalog hints** (`provider_hints.kubernetes.storage_class`): `defaultCatalogStorageClass` constant (`standard`) via `catalogStorageClassHint()`; optional `E2E_CATALOG_STORAGE_CLASS` override
+- **SP default** (no hints, TC-2.1.6): from `K8S_STORAGE_SP_DEFAULT_STORAGE_CLASS` via `spDefaultStorageClass()`; set by `deploy-dcm.sh` and `.dcm-e2e.env`
 
 ## Code Quality Patterns
 
@@ -103,4 +114,5 @@ When testing message ordering via NATS, comment whether the test relies on singl
 ## Compose Overrides
 SP compose overrides are managed by the provider registry (`providers/*.conf`). Each provider's `COMPOSE_OVERRIDE` key specifies its overlay file, which `deploy-dcm.sh` injects automatically when the provider is enabled:
 - `tests/compose-sp-test.yaml` — publishes container SP port 8082 (via `k8s-container.conf`)
+- `tests/compose-storage-sp-test.yaml` — publishes storage SP port 8089 (via `k8s-storage.conf`)
 - `tests/compose-acm-cluster-sp.yaml` — adds the ACM cluster SP service on port 8083 (via `acm-cluster.conf`)

.gitignore

@@ -29,5 +29,6 @@ Thumbs.db
 # Env / secrets
 .env
 .env.*
+.dcm-e2e.env
 *.pem
 *.key

CLAUDE.md

@@ -178,6 +178,8 @@ All test targets support JUnit XML output: `make test-e2e JUNIT_REPORT=results.x
 | **Core platform tests** | Full provisioning flow through control plane | `core`, `platform` |
 | **API tests** | HTTP CRUD operations against the control plane | (none) |
 | **SP tests** | Container SP direct API + NATS status events | `sp`, `container` |
+| **Storage SP tests** | K8s storage SP direct API + NATS status events | `sp`, `storage` |
+| **Storage SP registration** | Embedded storage in environment-agent (`embedded://storage`) | `sp`, `storage`, `registration` |
 | **ACM SP tests** | ACM Cluster SP API (health, registration, validation, CRUD) | `sp`, `acm-cluster` |
 | **Cluster tests** | Tests requiring `kubectl`/`oc` cluster access | `cluster` |
 | **Disruptive tests** | Tests that stop/start infrastructure (e.g. NATS) | `disruptive` |
@@ -204,12 +206,18 @@ CLI tests are skipped (not failed) if no binary is available.
 - `DCM_GATEWAY_URL` env var overrides the control plane API endpoint (default: `http://localhost:8080/api/v1alpha1`)
 - `DCM_CONTAINER_SP_URL` env var overrides the container SP endpoint (default: `http://localhost:8082/api/v1alpha1`)
 - `DCM_STORAGE_SP_URL` env var overrides the storage SP endpoint (default: `http://localhost:8089/api/v1alpha1`)
+- `K8S_STORAGE_SP_DEFAULT_STORAGE_CLASS` — **E2E harness only:** SP fallback when `provider_hints` omit `storage_class`; auto-detected by `deploy-dcm.sh`, written to `.dcm-e2e.env` (loaded by `make test-storage-sp`, `make test-sp`, and `run-e2e.sh` when storage tests run)
+- `K8S_STORAGE_SP_DEFAULT_ACCESS_MODE` — **E2E harness only:** SP fallback when hints omit `access_mode` (compose default `ReadWriteOnce`; not cluster-detected)
+- Catalog `provider_hints.kubernetes.storage_class` in storage E2E defaults to `standard` (simulated catalog hint for SP-direct tests; override with `E2E_CATALOG_STORAGE_CLASS`)
+- `K8S_STORAGE_SP_NAMESPACE` — PVC namespace for storage SP tests (from deploy `--k8s-storage-namespace`, default `default`)
+- `DCM_ENVIRONMENT_AGENT_URL` enables storage registration tests (`registration` label); set by deploy with `--environment-agent`
+- `K8S_STORAGE_SP_REGISTERED_ENDPOINT` overrides expected embedded storage endpoint in registration tests (default: `embedded://storage`)
 - `DCM_ACM_CLUSTER_SP_URL` env var overrides the ACM cluster SP endpoint (default: `http://localhost:8083/api/v1alpha1`)
 - `DCM_NATS_URL` env var overrides the NATS server (default: `nats://localhost:4222`)
 - `DCM_CLI_PATH` env var specifies the CLI binary path
 - `DCM_CONTAINER_PROVIDER_NAME` env var overrides which container provider to target in core platform tests (default: first `service_type=container` provider found)
-- Ginkgo labels (`smoke`, `cli`, `sp`, `container`, `acm-cluster`, `nats`, `cluster`, `disruptive`, `core`, `platform`, `rehydration`, `happy-path`, `failover`, `policy`, `negative`, `integrity`, `contract`) enable selective test runs via `--label-filter`
-- SP tests skip gracefully if the container SP or ACM cluster SP isn't reachable (no hard failure)
+- Ginkgo labels (`smoke`, `cli`, `sp`, `container`, `storage`, `registration`, `acm-cluster`, `nats`, `cluster`, `disruptive`, `core`, `platform`, `rehydration`, `happy-path`, `failover`, `policy`, `negative`, `integrity`, `contract`) enable selective test runs via `--label-filter`
+- SP tests skip gracefully if the container SP, storage SP, or ACM cluster SP isn't reachable (no hard failure); storage registration tests skip when `DCM_ENVIRONMENT_AGENT_URL` is unset
 - Cluster tests skip gracefully if `kubectl`/`oc` is unavailable or the cluster is unreachable
 - Disruptive tests skip if `podman` is unavailable; exclude from normal runs with `--label-filter '!disruptive'`
 

Makefile

@@ -1,4 +1,4 @@
-.PHONY: help e2e-up test-e2e test-smoke test-cli test-sp test-acm-sp test-kubevirt-sp test-core test-rehydration test-rehydration-safe test-rehydration-cli e2e-down test-e2e-full download-cli cli-version lint
+.PHONY: help e2e-up test-e2e test-smoke test-cli test-sp test-storage-sp test-storage-sp-registration test-acm-sp test-kubevirt-sp test-core test-rehydration test-rehydration-safe test-rehydration-cli e2e-down test-e2e-full download-cli cli-version lint
 
 # Set JUNIT_REPORT to a filename to produce JUnit XML output.
 # Example: make test-e2e JUNIT_REPORT=results.xml
@@ -9,6 +9,9 @@ ifdef JUNIT_REPORT
 GINKGO_BASE += --junit-report=$(JUNIT_REPORT)
 endif
 
+# Source storage SP deploy exports when present (storage E2E targets only).
+STORAGE_E2E_ENV_LOAD = set -a; [ -f .dcm-e2e.env ] && . ./.dcm-e2e.env; set +a;
+
 help: ## Show all available targets
 	@grep -hE '^[a-zA-Z0-9_-]+:.*## ' $(MAKEFILE_LIST) | awk -F ':.*## ' '{printf "  %-18s %s\n", $$1, $$2}'
 
@@ -25,7 +28,13 @@ test-cli: ## Run CLI tests only (stack must be running)
 	cd tests/e2e && $(GINKGO_BASE) --label-filter=cli .
 
 test-sp: ## Run all service provider tests (SPs must be deployed with ports published)
-	cd tests/e2e && $(GINKGO_BASE) --label-filter=sp .
+	@$(STORAGE_E2E_ENV_LOAD) cd tests/e2e && $(GINKGO_BASE) --label-filter=sp .
+
+test-storage-sp: ## Run k8s storage SP API + NATS tests (excludes registration placeholder)
+	@$(STORAGE_E2E_ENV_LOAD) cd tests/e2e && $(GINKGO_BASE) --label-filter='storage && !registration' .
+
+test-storage-sp-registration: ## Run k8s storage SP env-agent registration placeholder only
+	@$(STORAGE_E2E_ENV_LOAD) cd tests/e2e && $(GINKGO_BASE) --label-filter='storage && registration' .
 
 test-acm-sp: ## Run ACM cluster SP tests only
 	cd tests/e2e && $(GINKGO_BASE) --label-filter=acm-cluster .

README.md

@@ -101,8 +101,10 @@ make e2e-up        # Deploy the stack
 make test-e2e      # Run all tests (stack must be running)
 make test-smoke    # Run health checks + CLI version only
 make test-cli      # Run CLI tests only
-make test-sp       # Run container SP tests (SP must be deployed)
-make test-acm-sp   # Run ACM cluster SP tests (ACM SP must be deployed)
+make test-sp          # Run all SP tests (SPs must be deployed)
+make test-storage-sp            # Storage SP API + NATS (default; excludes registration placeholder)
+make test-storage-sp-registration # Embedded storage registration via environment-agent
+make test-acm-sp      # Run ACM cluster SP tests (ACM SP must be deployed)
 make test-core     # Run core platform tests (full provisioning flow)
 make e2e-down      # Tear down
 make download-cli  # Download latest DCM CLI without running tests
@@ -127,9 +129,23 @@ make help
 | `DCM_STORAGE_SP_URL` | `http://localhost:8089/api/v1alpha1` | Storage SP direct URL (requires published port) |
 | `DCM_ACM_CLUSTER_SP_URL` | `http://localhost:8083/api/v1alpha1` | ACM Cluster SP direct URL (requires published port) |
 | `DCM_NATS_URL` | `nats://localhost:4222` | NATS server URL for status event tests |
+| `K8S_STORAGE_SP_DEFAULT_STORAGE_CLASS` | (auto-detected at deploy) | **E2E harness only:** sets `SP_K8S_DEFAULT_STORAGE_CLASS` on the storage SP for requests that omit `provider_hints` (TC-2.1.6). Not used for catalog hints. Written to `.dcm-e2e.env` so `make test-storage-sp` sees the same SP config after deploy exits. |
+| `K8S_STORAGE_SP_DEFAULT_ACCESS_MODE` | `ReadWriteOnce` | **E2E harness only:** sets `SP_K8S_DEFAULT_ACCESS_MODE` when requests omit access_mode hints (compose default; not auto-detected). |
+| `E2E_CATALOG_STORAGE_CLASS` | `standard` | **Tests only:** optional override of simulated catalog `provider_hints.kubernetes.storage_class` in SP-direct E2E (default `standard`, common on kind). Deploy does not set this. |
+| `K8S_STORAGE_SP_NAMESPACE` | `default` | Namespace for storage PVCs (from deploy `--k8s-storage-namespace`) |
 | `DCM_CLI_PATH` | (auto-resolved) | Path to `dcm` CLI binary |
 | `JUNIT_REPORT` | (none) | JUnit XML report filename (e.g. `make test-e2e JUNIT_REPORT=results.xml`) |
 
+When storage E2E providers are deployed, `deploy-dcm.sh` writes `.dcm-e2e.env` (gitignored) with storage class, SP URLs, and (with `--environment-agent`) `DCM_ENVIRONMENT_AGENT_URL`. `make test-storage-sp`, `make test-storage-sp-registration`, and `make test-sp` source it.
+
+**Direct SP API tests** (`storage && !registration`): deploy with `--k8s-storage-service-provider` (standalone SP on port 8089).
+
+**Embedded registration tests** (`storage && registration`): deploy with `--environment-agent --kubeconfig ~/.kube/config`. The agent runs with `AGENT_EMBEDDED_SPS=storage` and registers `embedded://storage`. Until environment-agent merges embedded storage to `main`, build the image locally:
+
+```bash
+cd ../environment-agent && make image-build CONTAINER_IMAGE_TAG=main
+```
+
 ### Test Harness Flags
 
 The test harness (`tests/run-e2e.sh`) supports additional flags for fine-grained control:
@@ -146,7 +162,8 @@ The test harness (`tests/run-e2e.sh`) supports additional flags for fine-grained
 # Service provider tests
 ./tests/run-e2e.sh --k8s-container-service-provider --cluster-api https://api.example.com:6443
 ./tests/run-e2e.sh --k8s-storage-service-provider --kubeconfig ~/.kube/config
-./tests/run-e2e.sh --skip-deploy --label-filter "sp && container"
+./tests/run-e2e.sh --skip-deploy --label-filter "storage && !registration"
+./tests/run-e2e.sh --skip-deploy --label-filter "storage && registration"   # needs --environment-agent deploy
 
 # ACM cluster SP tests
 ./tests/run-e2e.sh --acm-cluster-service-provider --kubeconfig ~/.kube/config

providers/environment-agent.conf

@@ -0,0 +1,25 @@
+# Environment Agent for DCM E2E (current compose override enables embedded storage SP).
+# Requires kubeconfig access to the target cluster when embedded SPs need the Kubernetes API.
+
+PROVIDER_LABEL="environment-agent"
+PROVIDER_FLAG="environment-agent"
+PROVIDER_DESCRIPTION="Enable the DCM environment agent"
+
+# No compose profile in control-plane — full service is defined in the override.
+COMPOSE_PROFILE=""
+COMPOSE_OVERRIDE="tests/compose-environment-agent-test.yaml"
+
+# CLI — accepts oc or kubectl (prefers oc)
+CLI_REQUIREMENT="oc-or-kubectl"
+
+# Reuse storage namespace flag/env for embedded SP_K8S_NAMESPACE.
+NAMESPACE_FLAG="k8s-storage-namespace"
+NAMESPACE_ENV="K8S_STORAGE_SP_NAMESPACE"
+NAMESPACE_DEFAULT="default"
+
+# Env vars exported for compose.yaml substitution
+KUBECONFIG_EXPORT="AGENT_KUBECONFIG"
+NAMESPACE_EXPORT="K8S_STORAGE_SP_NAMESPACE"
+
+# Validation function (defined in deploy-dcm.sh)
+VALIDATE_HOOK="validate_environment_agent_provider"