Skip to content

feat(deploy): add compose and in-cluster Kind deployment - #30

Open
jenniferubah wants to merge 8 commits into
dcm-project:mainfrom
jenniferubah:agent-deployment
Open

feat(deploy): add compose and in-cluster Kind deployment#30
jenniferubah wants to merge 8 commits into
dcm-project:mainfrom
jenniferubah:agent-deployment

Conversation

@jenniferubah

@jenniferubah jenniferubah commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add standalone compose stack (deploy/compose.yaml) for NATS + environment-agent with Kind helpers pointing to utilities scripts
  • Add in-cluster Kind deployment (deploy/k8s/) with make k8s-deploy, NodePort access, and k8s-host-urls.sh for verify/publish
  • Add sample create requests (deploy/samples/, make publish-creates / make k8s-publish-creates) and deployment docs (DEPLOY.md, compose-kind.md, in-cluster.md)

Type of Change

  • New feature (deployment)

Stacked PR Status

  • Yes, this is part of a stack:
    • Parent PR: #48 (Base PR - Open for review)

Assisted-By: Cursor AI

Assisted-By: Cursor AI

Signed-off-by: Jennifer Ubah <cju.cipher@gmail.com>
Assisted-By: Cursor AI

Signed-off-by: Jennifer Ubah <cju.cipher@gmail.com>
Assisted-By: Cursor AI

Signed-off-by: Jennifer Ubah <cju.cipher@gmail.com>
Assisted-By: Cursor AI

Signed-off-by: Jennifer Ubah <cju.cipher@gmail.com>
@jenniferubah jenniferubah changed the title Deployment set up + testing examples feat(deploy): add compose and in-cluster Kind deployment Aug 31, 2026
@jenniferubah
jenniferubah marked this pull request as ready for review August 31, 2026 21:03
@jenniferubah
jenniferubah requested a review from a team as a code owner August 31, 2026 21:03
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Add Compose and in-cluster Kind deployment workflows

✨ Enhancement 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Adds Compose and in-cluster Kind stacks for NATS and the environment agent.
• Automates networking, KubeVirt installation, deployment, verification, and sample request
 publishing.
• Documents both deployment models, configuration, RBAC, troubleshooting, and teardown.
Diagram

graph TD
  OP["Developer"] --> MAKE["Make targets"]
  MAKE --> COMPOSE["Compose stack"] --> AGENT["Environment agent"] --> API["Kind API"] --> WORK["SP workloads"]
  MAKE --> K8S["Kubernetes manifests"] --> AGENT
  COMPOSE --> NATS["NATS JetStream"] --> AGENT
  K8S --> NATS
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Helm-based in-cluster deployment
  • ➕ Provides reusable values, release lifecycle management, and easier environment-specific configuration.
  • ➕ Could align deployment packaging with the control-plane Helm stack.
  • ➖ Adds chart maintenance and templating overhead for a local Kind workflow.
  • ➖ May duplicate or couple this repository to control-plane deployment concerns.
2. kubectl port-forward host access
  • ➕ Works consistently across Kind runtimes without relying on NodePort routing.
  • ➕ Avoids fixed cluster-wide NodePort assignments.
  • ➖ Requires persistent background processes for verification and publishing.
  • ➖ Complicates automation and endpoint discovery across multiple services.

Recommendation: The Compose plus Kustomize approach is appropriate for repository-local development because it remains explicit, lightweight, and easy to inspect. Retain fixed NodePorts for the current one-command workflow, but consider optional port-forward support if Docker Desktop or Podman networking proves inconsistent.

Files changed (26) +1350 / -3

Enhancement (1) +2 / -1
ContainerfilePrepare writable agent persistence directory +2/-1

Prepare writable agent persistence directory

• Creates '/app/data' before ownership and permission changes so the non-root agent can persist embedded SP registrations.

Containerfile

Documentation (4) +432 / -0
README.mdLink local Compose and in-cluster deployment guides +19/-0

Link local Compose and in-cluster deployment guides

• Adds a concise Kind and Compose quick start and links to the detailed deployment models.

README.md

DEPLOY.mdDocument supported local deployment models +116/-0

Document supported local deployment models

• Introduces quick starts, configuration guidance, sample publishing, teardown, and script references for Compose and in-cluster deployments.

deploy/DEPLOY.md

compose-kind.mdExplain Compose-to-Kind connectivity +82/-0

Explain Compose-to-Kind connectivity

• Documents kubeconfig rewriting, shared runtime networking, KubeVirt prerequisites, service exposure, teardown, and troubleshooting.

deploy/docs/compose-kind.md

in-cluster.mdGuide in-cluster agent deployment +215/-0

Guide in-cluster agent deployment

• Documents Kind deployment, in-cluster authentication, workload RBAC, persistence, verification, sample requests, and troubleshooting.

deploy/docs/in-cluster.md

Other (21) +916 / -2
.gitignoreIgnore local deployment secrets and generated kubeconfig +4/-0

Ignore local deployment secrets and generated kubeconfig

• Excludes the deployment environment file and Compose-specific Kind kubeconfig from version control.

.gitignore

MakefileAdd Compose and Kind deployment automation +57/-2

Add Compose and Kind deployment automation

• Adds targets for Compose lifecycle, Kind networking, KubeVirt, in-cluster deployment, verification, and sample publishing. It also validates container-engine discovery and explicitly builds from 'Containerfile'.

Makefile

.env.exampleProvide local deployment configuration template +53/-0

Provide local deployment configuration template

• Documents environment overrides for agent identity, DCM connectivity, embedded providers, persistence, and image selection.

deploy/.env.example

compose.yamlDefine standalone NATS and environment-agent stack +75/-0

Define standalone NATS and environment-agent stack

• Adds a Compose stack with JetStream persistence, one-shot stream initialization, agent configuration, host control-plane access, and Kind kubeconfig mounting.

deploy/compose.yaml

environment-agent.yamlDeploy and expose the in-cluster environment agent +70/-0

Deploy and expose the in-cluster environment agent

• Defines the agent Deployment with embedded container and VM providers, ServiceAccount authentication, registration storage, and a fixed NodePort service.

deploy/k8s/environment-agent.yaml

kustomization.yamlAssemble in-cluster deployment resources +7/-0

Assemble in-cluster deployment resources

• Collects namespace, NATS, RBAC, and environment-agent manifests into one Kustomize application.

deploy/k8s/kustomization.yaml

namespace.yamlCreate dedicated DCM namespace +4/-0

Create dedicated DCM namespace

• Defines the namespace used by the local in-cluster NATS and agent resources.

deploy/k8s/namespace.yaml

nats-init-job.yamlInitialize required JetStream streams +26/-0

Initialize required JetStream streams

• Adds an idempotent Job that creates the control-plane-owned request, status, and response streams after NATS starts.

deploy/k8s/nats-init-job.yaml

nats.yamlDeploy NATS JetStream with NodePort access +47/-0

Deploy NATS JetStream with NodePort access

• Defines a single-node NATS deployment with ephemeral JetStream storage and host-reachable client access.

deploy/k8s/nats.yaml

rbac.yamlGrant embedded providers workload permissions +36/-0

Grant embedded providers workload permissions

• Creates the agent ServiceAccount and binds namespaced permissions for container and KubeVirt workload management in 'default'.

deploy/k8s/rbac.yaml

container-create-spec.jsonAdd sample container create specification +19/-0

Add sample container create specification

• Provides an nginx container request with representative CPU and memory constraints for local routing checks.

deploy/samples/container-create-spec.json

vm-create-spec.jsonAdd sample VM create specification +23/-0

Add sample VM create specification

• Provides a Fedora VM request with CPU, memory, and boot disk settings for KubeVirt routing checks.

deploy/samples/vm-create-spec.json

install-kubevirt.shAutomate KubeVirt installation on Kind +18/-0

Automate KubeVirt installation on Kind

• Applies a configurable KubeVirt release to the active Kind context and waits for availability.

deploy/scripts/install-kubevirt.sh

k8s-deploy.shAutomate image build and in-cluster rollout +55/-0

Automate image build and in-cluster rollout

• Builds and optionally loads the agent image into Kind, applies manifests, initializes JetStream, and waits for service readiness.

deploy/scripts/k8s-deploy.sh

k8s-host-urls.shResolve host-reachable in-cluster endpoints +81/-0

Resolve host-reachable in-cluster endpoints

• Detects whether Kind NodePorts are reachable through localhost or the node IP and exports matching agent and NATS URLs.

deploy/scripts/k8s-host-urls.sh

kind-connect.shAttach Kind to the Compose network +23/-0

Attach Kind to the Compose network

• Connects the active Kind control-plane container to the Compose network with a certificate-compatible Kubernetes alias.

deploy/scripts/kind-connect.sh

kind-disconnect.shDetach Kind before Compose teardown +43/-0

Detach Kind before Compose teardown

• Finds the Kind container runtime and safely disconnects the node from known Compose networks before network removal.

deploy/scripts/kind-disconnect.sh

kind-env.shCentralize Kind context and runtime detection +79/-0

Centralize Kind context and runtime detection

• Provides shared helpers to derive the Kind node from kubectl context and select a runtime that sees both the node and Compose network.

deploy/scripts/kind-env.sh

kubeconfig-for-compose.shGenerate a container-reachable Kind kubeconfig +43/-0

Generate a container-reachable Kind kubeconfig

• Flattens the active Kind kubeconfig and rewrites its API endpoint to the Compose network alias, with ownership and path safeguards.

deploy/scripts/kubeconfig-for-compose.sh

publish-create-requests.shPublish sample create CloudEvents +114/-0

Publish sample create CloudEvents

• Builds container and VM create events from sample specs and publishes them through the NATS CLI or a containerized fallback.

deploy/scripts/publish-create-requests.sh

verify.shVerify agent health and embedded providers +39/-0

Verify agent health and embedded providers

• Checks the configured agent endpoint, reports container diagnostics on failure, and prints health and provider responses.

deploy/scripts/verify.sh

@qodo-code-review

qodo-code-review Bot commented Aug 31, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Init uses PID variables ✓ Resolved 🐞 Bug ≡ Correctness
Description
The Kubernetes Job runs $$1, $$2, and $$3 directly under /bin/sh, where $$ expands to the
shell PID rather than escaping Compose interpolation. It therefore looks up or creates streams with
values such as <pid>1 instead of the requested stream names, and k8s-deploy.sh fails while
waiting for this Job.
Code

deploy/k8s/nats-init-job.yaml[R18-21]

+                nats --server nats://nats:4222 stream info "$$1" >/dev/null 2>&1 && return 0
+                echo "Creating JetStream stream $$1 (subjects: $$2)"
+                nats --server nats://nats:4222 stream add "$$1" \
+                  --subjects "$$2" --storage file --retention "$$3" \
Evidence
The Job invokes /bin/sh -ec directly, while its function references use $$N; the deployment
script applies this manifest directly and waits for completion. The sibling Compose script needs
doubled dollars only because Compose performs an extra interpolation pass.

deploy/k8s/nats-init-job.yaml[14-26]
deploy/scripts/k8s-deploy.sh[39-42]
deploy/compose.yaml[38-50]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The directly applied Kubernetes Job uses Compose-style dollar escaping, causing `/bin/sh` to expand the PID instead of function positional parameters.

## Issue Context
Unlike `deploy/compose.yaml`, this manifest is not interpolated by Compose. Use `$1`, `$2`, and `$3` in the Job script.

## Fix Focus Areas
- deploy/k8s/nats-init-job.yaml[17-22]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Unconditional container-engine check breaks all make targets ✓ Resolved 🐞 Bug ≡ Correctness
Description
The new top-level ifeq ($(CONTAINER_ENGINE),) / $(error ...) block is evaluated at Makefile
parse time for every invocation, so environments without Podman or Docker cannot run
container-independent targets such as build, test, lint, fmt, vet, or generate-*.
Container availability should not be a global prerequisite because the existing scoped
check-container-engine target already validates it for image-build.
Code

Makefile[R11-13]

+ifeq ($(CONTAINER_ENGINE),)
+$(error No supported container engine found. Please install podman or docker, or set CONTAINER_ENGINE explicitly.)
+endif
Evidence
The conditional block is at the top level outside any recipe, so GNU Make evaluates $(error ...)
while parsing the Makefile, before selecting the requested target. The Makefile defines many
non-container targets that invoke only Go tooling, while image-build already depends on the
dedicated check-container-engine target further down, demonstrating that runtime validation can be
limited to commands that actually require a container engine.

Makefile[11-13]
Makefile[213-220]
Makefile[3-13]
Makefile[33-37]
Makefile[81-90]
Makefile[217-220]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description

The top-level `ifeq ($(CONTAINER_ENGINE),)` / `$(error ...)` block is evaluated unconditionally at parse time, preventing every Make target from running on machines without Docker or Podman, including Go-only development and CI targets such as `build`, `test`, `lint`, `fmt`, `vet`, and `generate-*`.

## Issue Context

Container detection should remain lazy and apply only to commands that require a container runtime. The Makefile already defines a scoped `check-container-engine` target used as a prerequisite for `image-build`; retain that approach or use an equivalent target-specific check rather than making container availability a global prerequisite.

## Fix Focus Areas

- Makefile[3-13]
- Makefile[213-220]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. NATS init has no readiness wait in compose ✓ Resolved 🐞 Bug ☼ Reliability
Description
nats-init depends on nats only through condition: service_started, which means the container
process has launched rather than NATS being ready on port 4222, while the service has no healthcheck
and the initializer performs no retry or wait around its CLI calls. On slower systems, set -ec can
terminate this one-shot container after connection failures, and because restart: "no" is set and
environment-agent requires service_completed_successfully, initialization and the entire Compose
stack can remain blocked or fail to start.
Code

deploy/compose.yaml[R32-50]

+  nats-init:
+    image: docker.io/natsio/nats-box:0.14.3
+    depends_on:
+      nats:
+        condition: service_started
+    restart: "no"
+    entrypoint: ["/bin/sh", "-ec"]
+    command:
+      - |
+        ensure() {
+          nats --server nats://nats:4222 stream info "$$1" >/dev/null 2>&1 && return 0
+          echo "Creating JetStream stream $$1 (subjects: $$2)"
+          nats --server nats://nats:4222 stream add "$$1" \
+            --subjects "$$2" --storage file --retention "$$3" \
+            --discard old --max-age 72h --defaults
+        }
+        ensure dcm-agent-requests 'dcm.agent.>' limits
+        ensure dcm-status 'dcm.*' limits
+        ensure dcm-agent-responses dcm.agents.responses work
Evidence
The nats service has no healthcheck, and nats-init waits only for service_started, which
guarantees container startup but not that NATS is accepting connections. Its ensure() function
makes a single stream info attempt followed by stream add; if both fail to connect during the
normal startup window, /bin/sh -ec exits the initializer, which does not restart, while
environment-agent explicitly waits for it to complete successfully.

deploy/compose.yaml[23-50]
deploy/compose.yaml[65-69]
deploy/compose.yaml[23-37]
deploy/compose.yaml[38-50]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`nats-init` in `deploy/compose.yaml` can fail intermittently because it starts when the `nats` container process launches, before NATS is necessarily listening on port 4222, and its `ensure()` function does not retry connection or stream-creation failures. Because the initializer uses `set -ec`, has `restart: "no"`, and `environment-agent` depends on its successful completion, this race can break `make compose-up` and leave the agent blocked.

## Issue Context
The Compose configuration gates initialization only on `condition: service_started` and defines no NATS healthcheck, so it does not establish actual service readiness. Add a NATS healthcheck and depend on `service_healthy`, or retry readiness and stream creation inside `nats-init` before giving up; unlike Compose, the Kubernetes path explicitly waits for `deployment/nats` to become available before running the equivalent initialization Job and also uses `backoffLimit` retries as a safety net.

## Fix Focus Areas
- deploy/compose.yaml[23-37]
- deploy/compose.yaml[41-50]
- deploy/compose.yaml[65-69]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

4. kind-disconnect hardcoded network list ignores custom project name ✓ Resolved 🐞 Bug ≡ Correctness
Description
kind-disconnect.sh's fallback network list (`environment-agent_default control-plane_default
deploy_default) is hardcoded and unrelated to COMPOSE_PROJECT_NAME`, unlike the Makefile's
COMPOSE_NETWORK which is derived from it. If a user overrides COMPOSE_PROJECT_NAME (e.g. `make
compose-down COMPOSE_PROJECT_NAME=myproject), the actual network myproject_default` is never
disconnected because it's absent from the script's fallback list.
Code

deploy/scripts/kind-disconnect.sh[12]

+NETWORKS="${COMPOSE_NETWORKS:-environment-agent_default control-plane_default deploy_default}"
Evidence
Makefile derives COMPOSE_NETWORK from COMPOSE_PROJECT_NAME (Makefile:16-17) and exports
COMPOSE_PROJECT_NAME (Makefile:23), but kind-disconnect.sh's NETWORKS fallback at line 12 is a
separate hardcoded list unrelated to COMPOSE_PROJECT_NAME, so a custom project name's network is not
included.

deploy/scripts/kind-disconnect.sh[12]
Makefile[15-17]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
deploy/scripts/kind-disconnect.sh's NETWORKS fallback list is hardcoded and does not account for a custom COMPOSE_PROJECT_NAME, so overriding the project name leaves the actual compose network never disconnected.

## Fix Focus Areas
- deploy/scripts/kind-disconnect.sh[12]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


5. Default Kind NodePorts unreachable ✓ Resolved 🐞 Bug ≡ Correctness
Description
The documented kind create cluster --name dcm-local creates no host port mappings, yet
verification relies exclusively on localhost or the node container's InternalIP. On Docker Desktop
hosts the container-network InternalIP is not host-routable and localhost has no mapped NodePort, so
make k8s-verify and make k8s-publish-creates cannot reach this deployment.
Code

deploy/scripts/k8s-host-urls.sh[R20-27]

+	# Docker Desktop Kind sometimes exposes NodePorts on localhost.
+	if agent_healthy "http://127.0.0.1:${AGENT_NODE_PORT}"; then
+		echo "http://127.0.0.1:${AGENT_NODE_PORT}"
+		return 0
+	fi
+
+	node_ip="$(node_internal_ip)"
+	if [[ -n "${node_ip}" ]] && agent_healthy "http://${node_ip}:${AGENT_NODE_PORT}"; then
Evidence
Both services expose fixed NodePorts, and the resolver tests only localhost and the first node's
InternalIP. The supplied cluster creation command has no Kind extraPortMappings, so the localhost
branch has no configured mapping and the workflow has no fallback such as port-forwarding.

deploy/scripts/k8s-host-urls.sh[17-32]
deploy/k8s/environment-agent.yaml[62-70]
deploy/k8s/nats.yaml[39-47]
deploy/docs/in-cluster.md[26-31]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The default Kind creation instructions do not expose the fixed NodePorts to the host, while URL resolution assumes either localhost exposure or direct routing to the Kind node container.

## Issue Context
Provide and document a Kind config with `extraPortMappings` for ports 30081 and 30422, or make the helper establish portable access such as `kubectl port-forward`.

## Fix Focus Areas
- deploy/scripts/k8s-host-urls.sh[17-32]
- deploy/k8s/environment-agent.yaml[62-70]
- deploy/k8s/nats.yaml[39-47]
- deploy/docs/in-cluster.md[26-31]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


6. In-cluster manifest uses unreachable host.docker.internal ✓ Resolved 🐞 Bug ≡ Correctness
Description
deploy/k8s/environment-agent.yaml hardcodes
DCM_REGISTRATION_URL=http://host.docker.internal:8080, but pods in a Kind cluster do not get this
hostname resolved (it is only wired via compose's extra_hosts, absent here), and no
dnsPolicy/hostAliases/CoreDNS customization is added by this PR. This causes make k8s-deploy to
ship a Deployment whose registration will fail to resolve DNS out of the box, contradicting the
documented in-cluster.md guidance to use a Kubernetes service name like dcm-control-plane.
Code

deploy/k8s/environment-agent.yaml[R35-37]

+            # Required; agent retries until reachable. Point at host control-plane when running one.
+            - name: DCM_REGISTRATION_URL
+              value: http://host.docker.internal:8080
Evidence
host.docker.internal is a Docker/compose-specific alias (only configured via extra_hosts in
deploy/compose.yaml); Kind pods use ClusterFirst DNS via CoreDNS and have no equivalent mapping. No
k8s-deploy.sh, kind-env.sh, or kustomization patch adds this DNS entry for the in-cluster path,
while docs (in-cluster.md) recommend a real service name instead.

deploy/k8s/environment-agent.yaml[35-37]
deploy/docs/in-cluster.md[88-91]
deploy/compose.yaml[70-73]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`deploy/k8s/environment-agent.yaml` sets DCM_REGISTRATION_URL to http://host.docker.internal:8080, which does not resolve inside Kind pods by default.

## Issue Context
This is the sample in-cluster manifest applied by `make k8s-deploy`; it's meant to work out of the box per `in-cluster.md`.

## Fix Focus Areas
- deploy/k8s/environment-agent.yaml[35-37]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View medium (2)
7. pull_policy always defeats local build ✓ Resolved 🐞 Bug ☼ Reliability
Description
The environment-agent service in deploy/compose.yaml sets pull_policy: always while also
declaring a build: section, so Compose attempts to pull
quay.io/dcm-project/environment-agent:main from the registry even when make compose-up passes
--build. This can break first-time/offline contributors if the tag doesn't exist or isn't
reachable, or silently use the pulled image instead of the freshly built local one.
Code

deploy/compose.yaml[R56-57]

+    image: quay.io/dcm-project/environment-agent:${ENVIRONMENT_AGENT_VERSION:-main}
+    pull_policy: always
Evidence
Compose applies pull_policy for services with both image and build; always pulls regardless of
--build. The file's own comment says local dev builds from the Containerfile, contradicting the
always-pull policy.

deploy/compose.yaml[52-57]
Makefile[40-41]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
deploy/compose.yaml sets pull_policy: always on a service that also has a build: section used for local development via `make compose-up --build`, which can cause compose to pull a possibly nonexistent/inaccessible remote image instead of building locally.

## Fix Focus Areas
- deploy/compose.yaml[52-57]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


8. Registration store is ephemeral ✓ Resolved 🐞 Bug ☼ Reliability
Description
Compose places the provider registration file in /app/data but mounts no volume there, so any
container recreation—including the advertised compose-up --build workflow—discards registered
provider state. This defeats the application's file-backed persistence for the standalone
deployment.
Code

deploy/compose.yaml[20]

+  AGENT_SP_PERSISTENCE_PATH: /app/data/registrations
Evidence
The application treats the configured path as its durable JSON FileStore, but the Compose service
mounts only /kubeconfig; the only declared named volume belongs to NATS. The image creates
/app/data, which makes writes possible but does not preserve them across container replacement.

internal/config/config.go[54-58]
internal/provider/store/file.go[14-32]
deploy/compose.yaml[20-20]
deploy/compose.yaml[62-75]
Containerfile[21-23]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The configured registration store lives only in the environment-agent container writable layer and is lost when Compose recreates the container.

## Issue Context
Mount a named volume at `/app/data` (or its parent) while retaining a file path for `AGENT_SP_PERSISTENCE_PATH`.

## Fix Focus Areas
- deploy/compose.yaml[20-20]
- deploy/compose.yaml[62-64]
- deploy/compose.yaml[74-75]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

9. k8s-deploy applies stale hardcoded image before update ✓ Resolved 🐞 Bug ☼ Reliability
Description
k8s-deploy.sh runs kubectl apply -k (which creates/updates the Deployment using the manifest's
hardcoded quay.io/dcm-project/environment-agent:dev image with imagePullPolicy: IfNotPresent)
before calling kubectl set image with the actual built ${IMAGE}. When CONTAINER_IMAGE_NAME is
customized, this can cause Kubernetes to initially schedule a pod against the wrong/inaccessible
hardcoded image before the follow-up set image corrects it, risking a transient ImagePullBackOff.
Code

deploy/scripts/k8s-deploy.sh[R32-34]

+echo "==> Applying manifests"
+kubectl apply -k "${K8S_DIR}"
+kubectl -n dcm set image deployment/environment-agent environment-agent="${IMAGE}"
Evidence
deploy/k8s/environment-agent.yaml hardcodes image quay.io/dcm-project/environment-agent:dev with
IfNotPresent; k8s-deploy.sh applies the kustomization then only afterward runs kubectl set image
with the customized IMAGE, creating a window where the wrong image is referenced.

deploy/scripts/k8s-deploy.sh[32-34]
deploy/k8s/environment-agent.yaml[20-22]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
k8s-deploy.sh applies the kustomization (hardcoded :dev image) before setting the actual built image via kubectl set image, causing a transient window where the wrong image reference is used.

## Fix Focus Areas
- deploy/scripts/k8s-deploy.sh[32-34]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 17 rules
Review mode: 🧠 Deep: This is a broad deployment change spanning Compose, Kubernetes manifests/RBAC, Kind networking, shell automation, Make targets, and message publishing, creating many independent, easy-to-miss failure paths.

Grey Divider

Tip of the day
💡 Did you know, you can route each action level your way: inline, summary, both, or drop

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread deploy/k8s/nats-init-job.yaml Outdated
Comment thread deploy/compose.yaml Outdated
Comment thread deploy/scripts/k8s-host-urls.sh
Comment thread Makefile Outdated
Comment thread deploy/k8s/environment-agent.yaml Outdated
Comment thread deploy/compose.yaml Outdated
Comment thread deploy/compose.yaml Outdated
Comment thread deploy/scripts/kind-disconnect.sh Outdated
Comment thread deploy/scripts/k8s-deploy.sh Outdated

@machacekondra machacekondra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good to me. The only thing is that some files are redudant or similar with what we have in the control-plane repo.

Maybe it's time to think of separating this into e2e repo? Maybe we can have discussion about this.

Assisted-By: Cursor AI

Signed-off-by: Jennifer Ubah <cju.cipher@gmail.com>
@jenniferubah

Copy link
Copy Markdown
Contributor Author

All looks good to me. The only thing is that some files are redudant or similar with what we have in the control-plane repo.

Maybe it's time to think of separating this into e2e repo? Maybe we can have discussion about this.

Sure, we can do the refactoring into separate repo in a follow up PR

Comment thread deploy/scripts/k8s-deploy.sh Outdated
fi

echo "==> Applying manifests (image: ${IMAGE})"
kubectl apply -k "${K8S_DIR}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k8s-deploy.sh builds and loads ${IMAGE} from ENVIRONMENT_AGENT_VERSION, but kubectl apply -k still uses the hardcoded newTag: dev in kustomization. Anything other than dev loads the right image into Kind then schedules :dev.
Prefer kustomize edit set image or kubectl set image

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, d749d80

Comment thread deploy/DEPLOY.md Outdated
See [in-cluster.md](docs/in-cluster.md) for full detail.

```bash
kind create cluster --name dcm-local

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In-cluster quick start uses plain kind create cluster.
On some hosts (Docker Desktop, rootless Podman) neither 127.0.0.1:30081 nor the node InternalIP is reachable, so make k8s-verify fails even when the pod is healthy. Maybe a small Kind config with extraPortMappings for 30081 and 30422? or documentation for port-forward as the fallback?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added the extraPortMappings config and port-forwards as a fallback in the troubleshooting section - d749d80

Comment thread deploy/DEPLOY.md Outdated

| Variable | Default | Notes |
|----------|---------|--------|
| `AGENT_EMBEDDED_SPS` | _(in example)_ | `container`, `vm`, `cluster` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default column says (in example) but .env.example keeps AGENT_EMBEDDED_SPS commented, which is fine as opt-in. Prefer empty (set in .env) or similar so the table matches the example file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated, 5566f3e

Comment thread deploy/compose.yaml
# DCM Environment Agent stack (standalone): NATS JetStream + environment-agent.
#
# Image version: defaults to :main. Pin with ENVIRONMENT_AGENT_VERSION in deploy/.env.
# Local dev: `make compose-up` builds the agent from the repo Containerfile.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compose defaults the agent image to :main, k8s-deploy to :dev. Each path builds and tags its own image, so switching from compose to in-cluster does not pick up the image you just built on the other path. Can we use one default tag for both workflows?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated it to default to dev for both workflows. d749d80

Comment thread Makefile Outdated
Comment on lines +44 to +45
kubeconfig-for-compose:
./deploy/scripts/kubeconfig-for-compose.sh

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make kubeconfig-for-compose runs ./deploy/scripts/kubeconfig-for-compose.sh.

That needs the file to be executable so the target fails with permission denied on a fresh clone.
k8s-host-urls.sh avoids this by calling bash in the Makefile.

Prefer chmod +x here, or invoke via bash like the other scripts.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated it to invoke bash - d749d80

Comment thread deploy/k8s/kustomization.yaml Outdated
- rbac.yaml
- environment-agent.yaml

# Image tag should match ENVIRONMENT_AGENT_VERSION in deploy/scripts/k8s-deploy.sh (default: dev).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment says the tag should match k8s-deploy.sh, but the script never updates this file. Either wire the tag in the script or soften the comment so it does not imply automatic sync.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the comment

Comment thread Makefile Outdated
Assisted-By: Cursor AI

Signed-off-by: Jennifer Ubah <cju.cipher@gmail.com>
Assisted-By: Cursor AI

Signed-off-by: Jennifer Ubah <cju.cipher@gmail.com>
Assisted-By: Cursor AI

Signed-off-by: Jennifer Ubah <cju.cipher@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants