FLPATH-4794 and FLPATH-2761 test plan updates/additions

[testetson22]

• Add test plan for FLPATH-4794: CLI-to-API contract testing with 6 test cases across static serialization, subsystem integration, and CI alerting tiers. I couldn't find an existing ticket around this type of thing, but this aims to prevent issues like FLPATH-4770
• Update FLPATH-2761 tech demo test plan to v1.3 with a "Preventive CI Recommendations" section documenting three proposed CI gates to prevent FLPATH-4770-class regressions from the other end of things.

Details

test-plans/FLPATH-4794-cli-api-contract-testing.md (new)

Defines a contract testing strategy between the CLI and control-plane API:

• TC-01/02: Static unit tests validating YAML→JSON round-trip preserves spec.resources and user_values[].resource
• TC-03/04: Subsystem tests running the CLI binary against a real control-plane (reusing existing catalog-subsystem compose stack)
• TC-05: Negative test confirming the CLI surfaces API errors for malformed payloads
• TC-06: Dependency freshness alert workflow (weekly go.mod staleness check)

test-plans/FLPATH-2761-tech-demo-poc.md (v1.2 → v1.3)

Added "Preventive CI Recommendations" section with:

1. Doc example schema validation (website repo PR workflow)
2. CLI-to-API contract test (references FLPATH-4794)
3. Dependency freshness alert (CLI repo scheduled workflow)

[qodo-code-review]

PR Summary by Qodo

Add CLI/API contract plan and preventive CI recommendations

🧪 Tests 📝 Documentation 🕐 40+ Minutes

AI Description

• Adds six CLI/API contract cases covering serialization, integration, errors, and dependency drift.
• Proposes three CI gates preventing schema, CLI, and documentation compatibility regressions.

Diagram

graph TD
  DOCS["Website YAML"] --> STATIC["Static Tests"] --> CLI["CLI Binary"] --> API["Control Plane"] --> DB[("Postgres")]
  API --> MOCK["WireMock"]
  CI["CI Workflows"] --> STATIC
  CI --> FRESH["Freshness Check"] --> CLI

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Use website examples as canonical CI fixtures

• ➕ Eliminates manually duplicated YAML fixtures and associated drift
• ➕ Tests the exact examples presented to users
• ➖ Introduces cross-repository checkout and availability dependencies
• ➖ Requires pinning or coordinating website revisions for reproducible runs

2. Validate generated clients against OpenAPI only

• ➕ Runs quickly without containers or subsystem infrastructure
• ➕ Directly detects generated-type divergence from the published schema
• ➖ Does not exercise the CLI YAML parsing and serialization path
• ➖ Cannot verify runtime API behavior or user-facing errors

Recommendation: Keep the proposed layered static and subsystem strategy because it covers both silent serialization loss and real API acceptance. Treat website examples as the source of truth where practical; otherwise, make the proposed byte-for-byte fixture synchronization check mandatory rather than relying on manually maintained copies.

Files changed (2) +1568 / -0

Other (2) +1568 / -0

FLPATH-2761-tech-demo-poc.mdDocument preventive CI gates for tech demo regressions +1042/-0

Document preventive CI gates for tech demo regressions

• Adds the FLPATH-2761 tech demo test plan at version 1.3, including nine validation scenarios, execution findings, and identified defects. It introduces preventive recommendations for documentation schema validation, CLI/API contract testing, and dependency freshness monitoring.

test-plans/FLPATH-2761-tech-demo-poc.md

FLPATH-4794-cli-api-contract-testing.mdDefine layered CLI-to-API contract testing strategy +526/-0

Define layered CLI-to-API contract testing strategy

• Adds six planned contract tests spanning static YAML serialization, real control-plane subsystem integration, malformed-payload handling, and dependency staleness alerts. The plan identifies reusable infrastructure, proposed artifact locations, coverage boundaries, and cross-repository risks.

test-plans/FLPATH-4794-cli-api-contract-testing.md

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Prerequisite deleted before TC-04 ✓ Resolved 🐞 Bug ≡ Correctness

Description

TC-03 deletes contract-test-item during cleanup even though TC-04 declares that item, and TC-03
itself, as prerequisites. Running the automated cases in their documented order or running TC-04
independently therefore leaves TC-04 without its required catalog item.

Code

test-plans/FLPATH-4794-cli-api-contract-testing.md[R277-280]

+```bash
+/tmp/dcm catalog item delete contract-test-item \
+  --control-plane-url http://localhost:28080
+```

Evidence

TC-03's cleanup deletes the exact fixed ID that TC-04 says must already exist. Existing repository
instance coverage instead creates the catalog item and instance in one lifecycle and removes them
together afterward.

test-plans/FLPATH-4794-cli-api-contract-testing.md[275-280]
test-plans/FLPATH-4794-cli-api-contract-testing.md[284-301]
tests/e2e/api_instances_test.go[80-107]
tests/e2e/api_instances_test.go[148-168]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
TC-03 cleanup destroys the catalog item required by TC-04, making the automated cases order-dependent and unusable independently.

## Issue Context
Prefer creating and cleaning up the catalog item within TC-04, or define TC-03 and TC-04 as one lifecycle with cleanup only after both assertions complete.

## Fix Focus Areas
- test-plans/FLPATH-4794-cli-api-contract-testing.md[275-280]
- test-plans/FLPATH-4794-cli-api-contract-testing.md[284-332]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Obsolete CLI commands tested ✓ Resolved 🐞 Bug ≡ Correctness

Description

TC-09 invokes dcm providers list and other flat/plural commands, while the current CLI uses nested
or singular forms such as dcm sp provider list and dcm policy list. These steps fail with
unknown-command errors instead of validating the user guide.

Code

test-plans/FLPATH-2761-tech-demo-poc.md[R609-612]

+```bash
+dcm providers list
+dcm providers --help
+```

Evidence

Current executable E2E tests invoke providers through sp provider, policies through singular
policy, and catalog instances through catalog instance; the plan itself records the same nested
structure while its test steps use incompatible commands.

tests/e2e/cli_providers_test.go[53-63]
tests/e2e/cli_policy_test.go[72-85]
tests/e2e/rehydration_cli_test.go[18-28]
test-plans/FLPATH-2761-tech-demo-poc.md[734-736]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
TC-09 uses obsolete flat and plural CLI subcommands, so its executable checks fail rather than validating the documentation.

## Issue Context
The repository's E2E tests use `dcm sp provider list`, `dcm policy list`, and nested `dcm catalog ...` commands. The plan's own execution results also identify this nested hierarchy.

## Fix Focus Areas
- test-plans/FLPATH-2761-tech-demo-poc.md[599-649]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Freshness threshold never evaluated ✓ Resolved 🐞 Bug ≡ Correctness

Description

TC-06 only extracts timestamp strings and contains no date conversion, subtraction, 14-day threshold
condition, or alert command. Consequently, stale and fresh dependencies produce identical behavior,
so acceptance criterion 3 is not tested.

Code

test-plans/FLPATH-4794-cli-api-contract-testing.md[R417-420]

+current_date=$(echo "$CURRENT" | grep -oP '\d{14}' | head -1)
+latest_date=$(echo "$LATEST" | grep -oP '\d{14}' | head -1)
+# Alert if difference > 14 days
+```

Evidence

The test promises an alert when the dependency is more than two weeks behind, but its implementation
ends immediately after extracting two strings and merely comments that an alert should occur.

test-plans/FLPATH-4794-cli-api-contract-testing.md[389-400]
test-plans/FLPATH-4794-cli-api-contract-testing.md[404-423]
test-plans/FLPATH-4794-cli-api-contract-testing.md[476-482]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The dependency freshness procedure extracts pseudo-version timestamps but never compares them or emits the promised alert.

## Issue Context
Add executable timestamp parsing, age calculation, a greater-than-14-days condition, and a warning or issue action. Handle extraction failures and tagged module versions explicitly.

## Fix Focus Areas
- test-plans/FLPATH-4794-cli-api-contract-testing.md[389-423]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View high (1)

4. Instance references wrong item ✓ Resolved 🐞 Bug ≡ Correctness

Description

TC-04 submits the byte-for-byte website fixture whose catalog_item_id is small-vm, but TC-03
creates the prerequisite item as contract-test-item. Instance creation therefore references an
item the setup did not create and fails before testing user_values[].resource.

Code

test-plans/FLPATH-4794-cli-api-contract-testing.md[R309-312]

+/tmp/dcm catalog instance create \
+  --from-file testdata/docs/my-vm.yaml \
+  --id contract-test-instance \
+  --control-plane-url http://localhost:28080

Evidence

TC-03 explicitly creates contract-test-item, while the documented source YAML copied into
my-vm.yaml hardcodes catalog_item_id: small-vm; the fixture-maintenance section requires those
files to be byte-for-byte copies.

test-plans/FLPATH-4794-cli-api-contract-testing.md[245-251]
test-plans/FLPATH-4794-cli-api-contract-testing.md[463-471]
test-plans/FLPATH-2761-tech-demo-poc.md[899-905]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
TC-04's instance fixture references `small-vm`, while its setup creates `contract-test-item`.

## Issue Context
The plan requires fixtures to remain byte-for-byte copies of website examples, so either TC-03 must create `small-vm` or the subsystem test must generate a test-specific instance payload with the created ID.

## Fix Focus Areas
- test-plans/FLPATH-4794-cli-api-contract-testing.md[245-273]
- test-plans/FLPATH-4794-cli-api-contract-testing.md[298-324]
- test-plans/FLPATH-4794-cli-api-contract-testing.md[463-472]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Cleanup resolves wrong directory ✓ Resolved 🐞 Bug ☼ Reliability

Description

After TC-02 changes into the newly cloned cli directory, its cleanup tries to enter
control-plane/deploy relative to that location. The cd fails and prevents `podman-compose down
-v` from removing the test stack and volumes.

Code

test-plans/FLPATH-2761-tech-demo-poc.md[R238-240]

+```bash
+cd control-plane/deploy && podman-compose down -v
+```

Evidence

TC-02 first enters control-plane/deploy, later enters a nested cli directory, and finally
attempts the original path as another relative child. The repository's supported teardown is
executed from its root through make e2e-down.

test-plans/FLPATH-2761-tech-demo-poc.md[191-197]
test-plans/FLPATH-2761-tech-demo-poc.md[217-224]
test-plans/FLPATH-2761-tech-demo-poc.md[236-240]
README.md[94-105]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
TC-02's cleanup path is invalid after the preceding directory changes, leaving the deployed stack running.

## Issue Context
The procedure first enters `control-plane/deploy`, then clones and enters `cli`. Cleanup must save the deployment directory, use an absolute path, or explicitly return to it.

## Fix Focus Areas
- test-plans/FLPATH-2761-tech-demo-poc.md[189-240]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

Tip of the day

💡 Did you know, you can add REVIEW.md to your repo root and Qodo follows it on every PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[chadcrum]

lgtm

[vkolodny]

The three-tier approach is the right fix for FLPATH-4770. I'd hold merge until the executable steps match the real CLI and catalog-subsystem setup — as written, TC-03/04 would fail before they test the contract.

FLPATH-4794

• TC-03 cleanup deletes contract-test-item, which TC-04 lists as a prerequisite. The website fixture also hardcodes catalog_item_id: small-vm while the setup creates contract-test-item. --id only names the instance — the item id comes from the YAML. These can't be both independent cases and byte-for-byte website copies. One lifecycle, or generate the instance YAML with the created item id.
• WireMock is not "already configured by compose." test/subsystem/catalog/docker-compose.yaml starts it with no mappings; the Ginkgo tests call stubPMCreateResource() in BeforeEach. A bash create against a bare stack will 404 on placement before user_values[].resource is tested.
• The sample tests use parseInputFileAs[CatalogItemFile] / InstanceFile. Those types don't exist. The path that dropped fields in 4770 is CreateCatalogItemJSONRequestBody / CreateCatalogItemInstanceJSONRequestBody.
• Plan AC doesn't match the Jira AC. Ticket AC1 is a static CLI-repo test; the plan's AC1 is subsystem create. Plan AC2 (website YAML vs OpenAPI on doc PRs) has no numbered TC.
• TC-06 extracts two timestamps and stops at a comment. Tagged versions have no 14-digit stamp, so the check is a no-op.

FLPATH-2761

This is a new 1042-line file, not a v1.2→v1.3 diff. Fine to archive the run, but the current steps are stale:

• FLPATH-2761 404s for me. The live testing ticket looks like FLPATH-4786. Branch is also named FLPATH-2671 (unrelated closed story).
• TC-09 still runs dcm providers list / dcm catalog-items list. Current CLI is nested (dcm sp provider, dcm catalog item). The execution results already recorded that as PASS.
• TC-03 URLs don't match TC-01 (create-catalog-item vs create-small-vm-catalog-item).
• TC-02 cleanup cd control-plane/deploy after cd cli won't find that path.
• The "Proposed Jira" block is now a duplicate of FLPATH-4770 — a link is enough.

The Preventive CI section is the useful v1.3 delta and matches 4794. The rest reads as a report now, not a plan to re-run as written.

Resubmitting with inline comments only.

[vkolodny]

See inline comments on the FLPATH-4794 test plan.

[jordigilh]

LGTM